asioso
Modules

Nutch Indexer for Elasticsearch 6.

The free indexer connects the Apache Nutch crawler with Elasticsearch 6. For operators of search systems who want to keep using Nutch with up-to-date text search.

Crawler
Apache Nutch 1.15
Benefit
Works with Elasticsearch 6
Language
Java
Available on
GitHub, free of charge

A gold standard among crawlers.

The Nutch project has been around for several years and is rightly considered a gold standard in crawling. At the time, SOLR was the state of the art in text search. Because Elasticsearch has become increasingly popular in recent years, Nutch also includes an integrated indexer for it.

The problem with Nutch 1.15.

However, version 1.15 of Nutch only supports Elasticsearch 5. This is likely to have caused massive problems, and not only on our systems. With our indexer, Nutch can now also be used with Elasticsearch 6. This keeps this tool available in combination with the current best solution for text search.

Elasticsearch and Nutch.

Elasticsearch is a search engine based on Lucene. It stores documents in a NoSQL format and is written in Java. Clients communicate with it via a RESTful web interface. Alongside Solr, Elasticsearch is the most widely used search server. Nutch is a crawler written in Java: a tool that grabs web page content and processes it further, for example to index it in a search engine.

Your result

What you get.

Revised dependencies

The dependencies used have been revised and aligned with Elasticsearch 6.

Adapted indexer code

The original Nutch indexer code has been adapted for Elasticsearch 6.

Free and open

You can download the indexer from GitHub free of charge and contribute to it yourself.

How it works.

Download the indexer

You download the indexer from GitHub free of charge.

Crawl web pages

Nutch grabs the content of web pages and processes it further.

Index in Elasticsearch 6

The indexer writes the crawled content to Elasticsearch 6.

Search via REST

Your clients query the index through the Elasticsearch RESTful web interface.

Contact

Hello.

[email protected]
PHONE
+49 89 9545706 10
ADDRESS
Wilhelmine-Reichard-Str. 26, München
Contact
Hello.

Tell us about your project. We will get back to you within one working day.