New Luke on JavaFX

Hello and Happy New Year to my readers!

I'm happy to announce release of completely reimplemented Luke -- using JavaFX technology.  Luke is the toolbox for analyzing and maintaining your Lucene / Solr / Elasticsearch index on low level. 

The implementation was contributed by Tomoko Uchida, who also did the honors of releasing it.

The excitement of this release is supported by the fact, that in this version Luke becomes fully compliant with ALv2 license! And it gets very close to be contributed to Lucene project. At this point we need lots of testing to make sure JavaFX version is on par with the original thinlet based one.

Here is how load index screen looks like in new JavaFX luke:

After navigating to the Solr 7.1 index and pressing OK, here is what luke shows:

I have loaded an index of Finnish wikipedia with 1,069,778 documents, and luke tells me that the index does not have deletions and was not optimized. Let's go ahead and optimize it:

Notice, that on this dialogue you can request only expunging of deleted docs, without merging (the costly part for large indices). After optimization's complete, you'll have a full log of actions in front of you to confirm the operation was successful:

You could also opt for checking the health of your index via Tools -> Check index menu item:

Let's move to the Search tab. It has changed slightly in that search box has moved to the right, while search settings and other knobs were moved to the left.

Thinlet version:

JavaFX version:

It is more intuitive UI now in terms of access to various tools like Analyzer, Similarity (now with access to parameters of new BM25 ranking model, that became default in Lucene and default in luke) and More Like This. There is a new Sort sub-tab that lets you choose a primary and secondary field to sort on. Collectors tab however is gone: please let us know, if you used it for some task -- would love to learn.

Moving on to the Analysis tab, I'd like to draw your attention towards really cool functionality of loading custom jars with your implementation of a character filter, tokenizer or token filter to form your custom analyzer. Test these right in the luke UI without the need to reload shards in your Solr / Elasticsearch installation:

Last, but not least is Logs tab. Essentially you should have been missing it for as long as luke exists: getting a handle of what's happening behind the scenes during an error case or a normal operation.

In addition, this version of Luke supports the recently released Lucene 7.2.0.

Luke 6.2.1 release and all things open source

Release

Indeed, luke 6.2.1 for lucene 6.2.1 is out of the oven. This is the proud moment for Tomoko Uchida, my co-committer to have been a release manager for the first time. Congrats, Tomoko!

Community

As luke gets more and more stargazers on github (520 at the time of this writing), I tend to glance over the list of them which sometimes makes my day. But beyond that and more importantly, this lays out the community of Lucene / Solr / Elasticsearch users and developers, that hopefully enjoy using luke too. 

Big names on user list

Having access to the stats of the luke repo gives insights on who and when might be talking about luke. This time, it is PayPal Engineering. And here is their nice technical writeup on indexing lots of data in Elasticsearch and field usage of luke for optimizing the lucene index data structures: https://www.paypal-engineering.com/2016/08/10/powering-transactions-search-with-elastic-learnings-from-the-field/

London Lucene/Solr hackday

Hackday is an amazing way to jump out of a routine and think big: what can be improved in the search land of Lucene / Solr technology and tooling? It was great to see that luke was picked up as one topic on the Lucene / Solr hackday in London: https://github.com/flaxsearch/london-hackday-2016. And there it is, Marple, browser-driven explorer for lucene indexes: https://github.com/flaxsearch/marple. Go check it out.

New contributors to luke

Tomoko and I have been active promoting luke on various occasions, Lucene / Solr Revolution 2015 and  ApacheCon 2015. And of course on twitter. Recently Florian Hopf has become active in sending pull requests to improve luke and fix various nagging issues. Welcome!

Luke 6.0 has been released

#luke 6.0 has been released. Major upgrade to #lucene 6.0 api: https://github.com/DmitryKey/luke/releases/tag/luke-6.0.0


There are other interesting features cooking, like access to DocValues: https://github.com/DmitryKey/luke/pull/53

If you feel like contributing, either by code or documentation, feel free to join the project:

https://github.com/DmitryKey/luke

[ANNOUNCE] Luke 5.3.0 released: naturally runs on Java 8

This release runs on Java8 and does not run on Java7.

This release includes a number of pull requests and github issues. Worth mentioning:

#38 upgrade to 5.3.0 itself

#28 Added LUKE_PATH env variable to luke.sh
#35 Added copy, cut, paste etc. shortcuts, using Mac command key
#34 Fixed lastAnalyzer retrieval (this feature remembers the last used analyzer on the Search tab)
#31 200 stargazers on github (by the time of this release the number crossed 260). Luke community is growing.

Everybody is welcome to contribute. If you feel like you care about search / indexing or would like to get deeper with Apache Lucene, go ahead and pick a ticket: https://github.com/DmitryKey/luke/issues

And, don't be afraid, we do not have any complaint departments:

All you need is your favourite beverage and a good debugger.

[ANNOUNCE] Luke 5.2.0 released

This is a major release supporting lucene / solr 5.2.0. Download the zip here:

https://github.com/DmitryKey/luke/releases/tag/luke-5.2.0

It supports elasticsearch 1.6.0 (lucene 4.10.4)

Issues fixed:
#20 Added support for reconstructing field values of indexed and not stored fields, that do not expose positions.

Pull requests:
#23 Elasticsearch support and Shade plugin for assembly
#26 added .gitignore to project
#27 Lucene 5x support
#28 Added LUKE_PATH env variable to luke.sh
#30 Luke 5.2

I'd like to highlight the contribution of Tomoko Uchida who has been recently very active in sending pull requests, including upgrade to lucene 5.x and first version of Apache Pivot based luke ui.

Luke gets support for Elasticsearch indices

That is that, really. The so long awaited proper support for elasticsearch indices.

#luke gets support for #elasticsearch #lucene indices! Just tested with es 1.5.0 https://t.co/3hsilTEaAR #apachecon cc .@elastic
— Dmitry Kan (@DmitryKan) 14 апреля 2015

@yevhen @elastic :) thanks!
— Dmitry Kan (@DmitryKan) 14 апреля 2015

Luke supported Apache Solr indices already. Why not Elasticsearch? The reason was, that ES uses its own SPI for postings format. If you tried to open an Elasticsearch index with luke before, you'd get something like:

A SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'es090' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene40, Lucene41]

The biggest issue of supporting custom SPI is that you'd need to hack the luke jar binary and add the ES SPI. I bet it is not what you would want to spend your time on.

With the excellent pull request by apakulov https://github.com/DmitryKey/luke/pull/23 luke uses shade maven plugin, that does all the magic. It magically updates the in-binary META-INF/services file with the following entry:

org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat
org.elasticsearch.search.suggest.completion.Completion090PostingsFormat
org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat

Currently this is available on luke master: https://github.com/DmitryKey/luke and a pre-release: https://github.com/DmitryKey/luke/releases/tag/luke-4.10.4-field-reconstruction

Luke keeps getting updates and now on Apache Pivot

Originally developed for fun and profit by Andrzej Bialecki, the lucene toolbox luke continues to be developed. Its releases are published at: https://github.com/DmitryKey/luke/releases


Most recently Tomoko Uchida has contributed into effort of porting Luke to an Apache License 2.0 friendly GUI framework Apache Pivot. New branch has been created to host this work:

https://github.com/DmitryKey/luke/tree/pivot-luke

Currently supported Lucene: 4.10.4.

It is far from completion, but already now you can:

• open your Lucene index and check its metadata

• page through the documents and analyze fields

• search the index

We will appreciate if you could test the pivot luke and give your feedback.

Exporting Lucene index to xml with Luke

Luke is the open source Lucene toolbox originally written by Andrzej Bialecki and currently maintained by yours truly. The tool allows you to introspect into your solr / lucene index, check it for health, fix problems, verify field tokens and even experiment with scoring or read the index from HDFS.

In this post I would like to illustrate one particular luke's feature, that allows you to dump index into an xml for external processing.

Task

Extract indexed tokens from a field to a file for further analysis outside luke.

 

Indexing data

In order to extract tokens you need to index your field with term vectors configured. Usually, this also means, that you need to configure positions and offsets.

If you are indexing using Apache Solr, you would configure the following on your field:

<field indexed="true" name="Contents" omitnorms="false" stored="true" termoffsets="true" termpositions="true" termvectors="true" type="text">

With this line you make sure you field is going to store its contents, not only index; it will also store the term vectors, i.e. a term, its positions and offsets in the token stream.

 

Extracting index terms

One way to view the indexed tokens with luke is to search / list documents, select the field with term vectors enabled and click TV button (or right-click and choose "Field's Term Vector").




If you would like to extract this data into an external file, there is a way currently to accomplish this via menu Tools->Export index to XML:



In this case I have selected the docid 94724 (note, that this is lucene's internal doc id, not solr application level document id!), that is visible when viewing a particular document in luke. This dumps a document into the xml file, including the fields in the schema and each field's contents. In particular, this will dump the term vectors (if present) of a field, in my case:

<field flags="Idfp--SV-Nnum--------" name="Contents">
<val>CENTURY TEXT.</val>
<tv>
<t freq="1" offsets="0-7" positions="0" text="centuri" />
<t freq="1" offsets="0-7" positions="0" text="centuryä" />
<t freq="1" offsets="8-12" positions="1" text="text" />
<t freq="1" offsets="8-12" positions="1" text="textä" />
</tv>
</field>

Lucene Toolbox Luke

As I was mentioning in some of the previous posts, I have been working on luke, the Lucene Toolbox. Recently, the releases have been more or less steady, providing the binaries right on the github site.

Here is the most recent release of luke for the most recent release of Lucene 4.6.0:

port to Lucene 4.6.
Tested against the solr-4.6 index of exampledocs xml files.
Changelist:
1. bumped up the version of maven-assembly plugin to the latest 2.4
2. added maven-jar plugin which copies all binary dependencies into lib/ subdirectory
3. Fixed AnalyzerToolPlugin to use token stream reset outside while loop over tokens
4. added lock factory handling in FsDirectory (untested)
5. updated trivia in about.xml

At the moment the generated all-inclusive jar can be launched by double-clicking it only. There is some work to do to enable command line launches.

If you feel like contributing (patches and new features alike) and enjoy using the tool, get in touch or just send a pull request.

Enjoy the season's holidays,

@dmitrykan