Deep learning: анализ текста и изображений при помощи Рекурсивных Нейронных сетей

Продолжая тему Deep Learning, получившую в последнее время большое внимание научного сообщества и индустрии, хочется рассказать про довольно необычную сферу применения нейронных сетей. Это сфера разбора (parsing) изображений рекурсивными нейронными сетями. В этом посте я кратко опишу суть и приведу ссылки для более детального ознакомления с материалом. Помимо этого будет ссылка на задачу из Stanford'а для желающих попробовать свои силы и получить практический результат в области deep learning для NLP.

Этот пост базируется на докладе Richard Socher, видео на английском доступно здесь.

Как оказывается, рекурсивной (или некоторой регулярной) структурой может обладать не только язык, но и изображения. Но в начале о рекурсивных свойствах предложений. Принцип композиционности (principle of compositionality) делает предположение, что любое предложение на естественном языке можно представить в виде иерархической структуры из связных составляющих. Смысл предложения можно представить в виде смыслов слов, в него входящих, и списка правил соединения слов в группы. Например, в предложении:

Это страна моего рождения.

Cлова "моего" и "рождения" образуют единую группу, "страна" и "моего рождения" над-группу, а всё предложение замыкает его смысл в виде иерархического представления указанных групп. Таким образом, получаются не только смыслы отдельных слов предложения, но и древесные структуры, в которые данные слова увязываются.

Как можно разобрать изображения по аналогии с текстовыми рекурсивными представлениями? Можно утверждать, что существует схожий принцип композиционности для изображений. Рассмотрим изображение:

Если рассмотреть некоторые сегменты изображения: здание, конусообразная крыша, оконный ряд, окна по отдельности. В указанном порядке они описывают "вложенную" рекурсивную структуру, которой можно описать здание целиком. Есть ещё параллельные зданию объекты -- люди, деревья и трава. Таким образом, изображение, например, дома можно представить виде древесной структуры, где узлы на нижних уровнях являются составляющими узлов-предков. В точности, как и в древесных структурах предложений на естественном языке.

Алгоритм на основе рекурсивных нейронных сетей авторов Socher и др. достигает 78,1% качества. Область применения таких алгоритмов -- распознавание сцен (область анализа изображений или image analysis). Исходные коды и датасеты можно посмотреть здесь.

Задачка. Для более детального ознакомления с deep learning в применении к задачам NLP на сайте Stanford'a предлагается к решению задачка: реализовать простой оконный распознаватель именных сущностей (Named Entity Recognition или NER). Описание задачки здесь. Стартовый исходный код на Java и тренировочный сет здесь. Тренировочный сет аннотирован персонами, например:

Franz PERSON
Fischler PERSON

У исходного кода есть два режима: собственно, исполнение и отправка решения на сервер. Понятно, что из этих двух режимов нас интересует первый.

Довольно много материалов по теме Deep Learning можно найти на сайте Richard Socher, включая применение данного направления к распознаванию тональности (sentiment analysis).

Why I started learning (j)ruby

I have been pretty comfortable with Perl for a while in my programming life, but at some point realized, that what Dijkstra says about an impact a programming language has on a programmer's mind, seems to hold. I.e. if you code long the same language eventually you will look at every problem at hand through the prism of what your programming language has to offer to solve it. By this I mean, the data structures, debugging, dumping the data contents into stdout, references, copying, working with file encoding, web and so on. I don't want to go too far stretched in suggesting that even how for or for each loop affect one's mind, just to want to say, that practicing other languages for the same tasks can be quite useful. This still assumes, that each language has its power, and in the case of Perl that are certainly regular expressions.

This post won't compare Perl and Ruby or Ruby to any other language. I just want to note along the way the Ruby features, that I find the most interesting to me.

  probabilities = 10.times.map{Float(rand(0))}
  probabilities.each {|p| print p.to_s + " "}

This prints:

0.972042584650313 0.148158594901043 0.109142777878581 0.825619772397228 0.177120402897994 0.411135204463207 0.0448148166075958 0.996025937730191 0.143679780727901 0.311015907725463

That is, with just two lines of in practice functional code you are able to create an array of 10 random real numbers between 0 and 1 and output them to stdout.

Another compelling feature of ruby is that it can be turned into jruby and then all the java mass of libraries becomes available at your scripting finger tips. Supposing, that you have a text file with some categories separated with semi-colons, you can load them into guava's ArrayListMultimap:

# using guava 13.0.1 in jruby
require 'java'
require '/home/user/.m2/repository/com/google/guava/guava/13.0.1/guava-13.0.1.jar'

def loadCategories
  myCategoryMultimap = com.google.common.collect.ArrayListMultimap.create
  File.open(fName, "r").each_line do |line|
    category = line[/Category=[^;]+/]
    myCategoryMultimap.put category, line
  end
  return myCategoryMultimap

end

To summarize so far, two features: functional style of writing code and java-friendliness make (j)ruby a compelling next language to learn if you come from the scripting / Java world.

P.S. If you are in Finland around Helsinki you might be interested in Helsinki Ruby Brigade where the sessions have been pretty technical and interesting.

Fixing X issues with jconsole and jvisualvm under ubuntu

This is merely a technical post describing how to solve the issues with running the aforementioned jdk tools on ubuntu without X servers installed.

It is sometimes possible that you need to run X based apps on ubuntu servers that do not have graphical libraries no GUI installed.

The easy way to check whether your ubuntu server is missing any libraries is to run the command recommended on stackoverflow.com:

jvisualvm -J-Dnetbeans.logger.console=true

This command will output names of shared libraries that are required to run the command but are missing. For example these libraries could be: libxrender1, libxtst6, libxi6. Their names can be printed also with .so. suffixes, like so: libXrender.so.1.

In order to install them you can run:

sudo apt-get install libxrender1
sudo apt-get install libxtst6
sudo apt-get install libxi6


After installing a library keep running the jvisualvm command above to see if there are any libraries still missing.

If all libraries are in order the jvisualvm should start given that you have connected to your ubuntu server with ssh -X command line parameter which will stream the graphical command's GUI output to your client machine.

Have fun monitoring!

Solr usability contest: make Apache Solr even cooler!

In august I took part in the Solr Usability Contest ran by Alexandre Rafalovitch, author of the Apache Solr for Indexing Data How-to book from Packt.

As I have already told Alexandre (@arafalov), it was a great idea to launch a contest like this. While Solr / Lucene mail-lists serve as a direct way of solving particular problems and Apache jira is a way of doing some feature requests and bug submissions, it is great to sometimes take a step back and have a look at a larger perspective of features / limitations / possible improvements and so on.

The following three are the winning suggestions of truly yours:

Make automic updates really atomic

Make dashboard more interactive and configurable

Add scripting capability

 

On atomic updates

It coincided that we have been evaluating new sexy sounding atomic updates feature and found out, that it wasn't easy to enable it. To actually make use of the feature, we essentially would've needed to make *all* fields stored. The past several years I have been "fighting" against storing fields unless really necessary. It is amongst one of the performance suggestions to avoid storing fields if possible that in turn helps avoiding extra disk seeks. Not all have SSD disks installed on their Solr servers. Having written some Lucene level code some time ago I could wildly guess, why would all fields be necessary stored for the atomic updates. Essentially upon atomic update Solr will have to retrieve an existing document with all its source values, update a value (or values) and push the document back into persistent storage (index). To my taste this describes a bulk update. There is one major advantage of atomic updates (given that all fields were made stored): saving on the network traffic. Indeed, instead of submitting an entire document with a couple updated fields, you can send only the fields with new values and provide a document id. There are other cool features, like setting a previously non-existent field on the document or deleting an existing field. These all will surely make an atomic update feature appealing to some folks. You will find real examples of how to use atomic updates feature in the Alexandre's book. So go and get your copy now.

In the course of reindexnig our data in solr4 we have found out a lot of improvements, one of them is index compression. The lossless compression algorithm used in Lucene / Solr 4 is lz4, which has made the index super compact. Our use case shows 20G vs 100G index size compression in solr4 vs solr3 battle, which is simply amazing. By the way the algorithm has a property of fast decompression (fast decoder), which makes it an ideal fit for an online algorithm.

In light of compactness of the index we are still considering to evaluate the atomic updates feature, merely from three perspectives:

• traffic savings
• speed of processing
• index size increase vs storing only necessary fields

On interactivity of Solr dashboard

As we have started evaluating goodies of Solr4 I was positively surprised about how usable and eye-catchy looking the Solr dashboard (admin) has become. Above all is usability (it is after all usability contest) and, oh yes, it has become usable for the first time. In Solr 1.4.1 and 3.4 times we have been merely consulting the cache statistics page and analysis page occasionally. In Solr4 one can now administer the cores directly from the dashboard, optimize indices, study the frequency characteristics of text data and so on. This is of course on top of mentioned features, like field analysis and monitoring the cache stats.

But.. something is still missing. We are running several shards with frontend solrs and for us it has always been a bit of a pain to monitor our cluster. We intentionally do not use SolrCloud, because of requirements for logical sharding. Sometime ago I have blogged about Solr on FitNesse, which helped to see the situation with the cluster with just one click. We have also set up RAM monitoring with graphite, but wait, all of these are external to Solr tools. It would be really great to be able to integrate some of them directly into Solr dashboard. So we hope this will change into the direction of "plug-n-play" type of interfaces that would allow implementing plugins to Solr dashboard. In the mean time good ol' jvisualvm is a tool helping to monitor a heavy shard during soft-commit runs:

On scripting capability

I also dared to fantasize about what could open Solr up for wider audience. Especially people that are not dreaming of reading and changing Solr source code. This can be enabled with a scripting capability. By this I mean a way of hacking into Solr via external interfaces in the language that fits your task and skillset best (ruby or scala or some other JVM friendly language or perhaps something outside JVM family altogether). The best thing this would offer is an opportunity to experiment fast with the Solr search: changing runtime order of analyzers or search components, affecting on scoring, introducing advertisement entries, calculating some analytics, refining facets etc etc. While some of these may sound too far stretched, the feature in general may open up for changing the Solr core behaviour without hacking into the heavy-duty source code recompilation (although personally I would recommend diving into that anyway).

 

Concluding remarks

I would like to conclude that Solr4 has brought lots of compelling features and improvements (an extremely great soft-commit feature, for example) and we are happy to see this blazingly fast search platform to evolve that fast. In these three usability suggestions I have tried to summarize what is great to do to make the platform even more compelling and cool.

yours truly,

@dmitrykan

Monitoring Solr with graphite and carbon

This blog post requires graphite, carbon and python to be installed on your *ux. I'm running this on ubuntu.

http://graphite.wikidot.com/
https://launchpad.net/graphite/+download


To setup monitoring RAM usage of Solr instances (shards) with graphite you will need two things:

1. backend: carbon
2. frontend: graphite

The data can be pushed to carbon using the following simple python script.

In my local cron I have:

1,6,11,16,21,26,31,36,41,46,51,56 * * * * \
   /home/dmitry/Downloads/graphite-web-0.9.10\
          /examples/update_ram_usage.sh

The shell script is a wrapper for getting data from the remote server + pushing it to carbon with a python script:

scp -i /home/dmitry/keys/somekey.pem \
    user@remote_server:/path/memory.csv \ 
    /home/dmitry/Downloads/MemoryStats.csv

python \
  /home/dmitry/Downloads/graphite-web-0.9.10\
    /examples/solr_ram_usage.py

An example entry in the MemoryStats.csv:

2013-09-06T07:56:02.000Z,SHARD_NAME,\
  20756,33554432,10893512,32%,15.49%,SOLR/shard_name/tomcat

The command to produce a memory stat on ubuntu:

COMMAND="ssh user@remote_server pidstat -r -l -C java" | grep /path/to/shard 

The python script is parsing the csv file (you may want to define your own format of the input file, I'm giving this as an example):

import sys
import time
import os
import platform
import subprocess
from socket import socket
import datetime, time

CARBON_SERVER = '127.0.0.1'
CARBON_PORT = 2003

delay = 60
if len(sys.argv) > 1:
  delay = int( sys.argv[1] )

sock = socket()
try:
  sock.connect( (CARBON_SERVER,CARBON_PORT) )
except:
  print "Couldn't connect to %(server)s on port %(port)d, is carbon-agent.py running?" % { 'server':CARBON_SERVER, 'port':CARBON_PORT }
  sys.exit(1)

filename = '/home/dmitry/Downloads/MemoryStats.csv'

lines = []

with open(filename, 'r') as f:
  for line in f:
    lines.append(line.strip())

print lines
 
lines_to_send = []

for line in lines:
  if line.startswith("Time stamp"):
    continue
  shard = line.split(',')
  lines_to_send.append("system."+shard[1]+" %s %d" %(shard[5].replace("%", ""),int(time.mktime(datetime.datetime.strptime(shard[0], "%Y-%m-%dT%H:%M:%S.%fZ").timetuple()))))

#all lines must end in a newline
message = '\n'.join(lines_to_send) + '\n'
print "sending message\n"
print '-' * 80
print message
print
sock.sendall(message)
time.sleep(delay)

After the data has been pushed you can view it in graphite GWT based UI. The good thing about graphite vs jconsole or jvisualvm is that it persists data points so you can view and analyze them later.

For Amazon users, an alternative way of viewing the RAM usage graphs is with CloudWatch, although at the moment of this writing it allows storing 2 weeks worth of data only.

ReVerb: Open Information Extraction

Кросспост моего второго поста об извлечении информации.

Предыдущий пост о семантических связях между словами представил открытый инструмент word2vec, позволяющий строить или выявлять в некотором смысле семантические сети слов и словосочетаний.

В этом посте мы рассмотрим систему, выявляющую связи между запросом и документами по тройке: Объект1-Связь-Объект2, где объекты {Объект1, Объект2} представлены в виде существительного либо семантического класса существительного, Связь -- в виде глагола или падежного типа.

Система называется ReVerb (от Relataion=связь, Verb=глагол). Её исходный код доступен на github. Система поддерживает только английский язык.

С попыткой представить знание в виде приведённых троек можно встретиться довольно часто где (например, этот подход упоминался в докладе Gerhard Weikum на RuSSIR'2011). Первое впечатление от такого подхода: слишком узкий взгляд на семантику и что ничего путного с этим не сделать. Однако это не совсем так. Часто перед решением задачи компьютерной лингвистики (будь то машинный перевод, анализ тональности или информационный поиск) нужно сделать первые шаги в изучении имеющихся данных. Эти шаги могут включать построение частотных таблиц слов или словосочетаний (N-грамм), выявление ключевых слов, представляющих документ и т.д. Кстати, многие начальные шаги можно оптимально сделать при помощи инструментов Linux, таких как cat, cut, grep, awk, sed, wc (от word count, а не то, что можно подумать) и других. Таким образом, воспользовавшись существующими инструментами обработки текста, можно решить начальные задачи, даже не написав строчки кода!

Демонстрация системы извлечения знаний из 500 млн веб-страниц находится здесь. Что в ней примечательного?

Например, можно получить список стран Африки, задав запрос:

Argument1: type:Country

Relation: is located in

Argument2: Africa

Система выводит список из 45 государств, видимо тех, о которых что-то публикуется в Сети (вообще, официально признанных суверенных государств в Африке 54, согласно Википедии).

Можно задавать общие вопросы: например, какие актёры играли в каких фильмах:

Argument1: what/who

Relation: starred in

Argument2: what/who

Например, Barbra Streisand снималась в фильме "Yentl", Jessica Alba в "Sin City", а Johny Depp в "Pirates of the Caribbean".

Воспользовавшись падежной связкой "symbol of", мы получаем список символов разных стран.

Argument1: what/who

Relation: symbol of

Argument2: type:Country

У Шотландии -- это единорог.

Индексировать и искать документы с мета-информацией можно, например, при помощи Apache Solr. Но это уже отдельная история.

Машинное обучение без учителя для определения смысла слов: open source инструмент от Google word2vec

Кросспост моего поста с http://mathlingvo.ru/

В блоге Google Open Source Blog появилось сообщение о новом open source инструменте word2vec. Исследователи Google утверждают, что при его помощи можно получить смысл слов, лишь прочитав огромные массивы данных. Инструмент применяет "распределённые представления" текстовых данных для обнаружения связей между концептами -- и всё это при помощи машинного обучения без учителя (unsupervised machine learning) на основе нейронных сетей (neural networks).

Интересно, что модель помещает близкие страны рядом, как и близкие столицы. Похожие связи возникают автоматически во время тренировки алгоритма.

У исходного кода хорошая лицензия: Apache License 2.0, которая позволяет менять его без опубликования изменений и встраивать его в том числе в коммерческие приложения.

В статье также упоминается ставший популярным в последнее время метод Deep Learning, дающий результаты, лучшие на порядок предыдущих методов. Кстати, большинство победителей конкурсов по машинному обучению на kaggle (ваш покорный слуга также имел честь участвовать) применяет либо ансамбли методов на Decision Trees, либо методы Deep Learning.

// ./demo_word.sh
Enter word or sentence (EXIT to break): machine translation

Word: machine  Position in vocabulary: 799

Word: translation  Position in vocabulary: 1206

          Word       Cosine distance
------------------------------------------------------------------------
          mmix              0.485542
    translator              0.484659
          msil              0.483476
        manual              0.479708
        turing              0.462978
  introduction              0.458771
      readable              0.449272
    unabridged              0.448343
      machines              0.447570
       rosetta              0.443270
      compiler              0.438949
    dictionary              0.437040
  translations              0.436334
    translated              0.429008
 specification              0.422286
    typewriter              0.422246
           awk              0.420415
       version              0.417623
   interpreter              0.415583
        itrans              0.414944
         tools              0.413505
     annotated              0.413150
        lincos              0.411448
      abridged              0.411152
          text              0.407197
      language              0.404664
        freedb              0.403896
       vulgate              0.402863
         xpath              0.401687
    calculator              0.397689
        enigma              0.394239
       klingon              0.394041
       opencyc              0.393687
       systran              0.391636
       multics              0.391623
           kli              0.389196
           apl              0.386948
      editions              0.383799
        skybox              0.383791
         algol              0.383730
Enter word or sentence (EXIT to break): weather

Word: weather  Position in vocabulary: 2693

          Word       Cosine distance
------------------------------------------------------------------------
          warm              0.634004
      humidity              0.611526
         humid              0.605240
       summers              0.594220
 thunderstorms              0.591256
      snowfall              0.590065
 precipitation              0.582246
       climate              0.580110
       winters              0.577238
      rainfall              0.570583
         rainy              0.566492
below_freezing              0.566140
  rainy_season              0.561857
        cooler              0.558795
         winds              0.558283
        colder              0.557494
  cold_winters              0.545980
           wet              0.545650
        frosts              0.539969
         drier              0.539645
      climatic              0.537766
        warmer              0.535417
        winter              0.532653
  warm_summers              0.530857
         el_ni              0.530692
  temperatures              0.528191
relative_humidity           0.527605
        summer              0.527042
  mild_winters              0.526249
       monsoon              0.524260
   trade_winds              0.523211
       daytime              0.523093
      seasonal              0.520377
           dry              0.519703
    hurricanes              0.517527
     subarctic              0.514771
    visibility              0.514740
     snowfalls              0.513660
     monsoonal              0.513538
   hot_summers              0.513050

What is it like to study mathematics at Saint Petersburg State University? (my answer on quora.com)

As it turns out, not all of my readers are on quora. So because of this and in the spirit of posting non-technical blogs too, I'm reposting an answer I gave to the question there: "What is it like to study mathematics at Saint Petersburg State University?"

I have studied math and other subjects (like physics, computer science and others) during 2002-2005 in Saint Petersburg State University (SPbU) for a Specialist program (comparable to that of Master's degree).

My experience was constantly comparative in the beginning: as I was advancing further into teaching style of SPbU professors and docents I was viewing it side by side with the style of another State University of my home city (10x smaller in population than Saint Petersburg that time).

So perhaps I can approach answering your question from the perspective of comparison.

1. (a) In my home university we were taught to learn long theorem proofs in the fashion that would enable a student to easily reproduce it on an (pre-)exam. I remember only one occasion, when a theorem was so long that learning all the low-level details was impossible (despite how many days I tried), therefore really deriving the proof was the only option. Of course you would learn the fundamental constructs and apparatus for deriving the proof, that is you wouldn't be doing it completely from scratch and finding your ways into it.

  (b) In SPbU, in contrast, you wouldn't be expected to learn the entire theorem proof at all, but instead be ready to derive it. Some of the practical tasks given along the theoretical proofs would require the same: derive a solution as you go. This was the first thing that struck me as largely different.

2. (a) In my home university I was expected to learn about 80% of definitions, theorem formulations, their proofs.

    (b) It was my first exam on Control Theory in SPbU where its professor told me, a student should learn about 35% (or even less): the _most_ important theorem formulations and their proofs plus the _most_ important definitions. The rest is derivable as explained in (1) (b)

3. (a) The highlight of fun part of studying in my home university that comes to mind was that once a professor of mathematical analysis came to the class and asked: "Do you want theory and tasks today or talk about life?" "Life" was the answer, and the first question from the audience was: "Girls of which country were the most beautiful?".

   (b) In SPbU there have been all sorts of surprises that opened student's mind or made studying more fun. One example: during one of the exams on electrodynamics (complex theory with integral calculus, Lie algebra and so on), a professor said 10 minutes past the start: "The ones who would like to get C mark (3 or "satisfactory" in Russia)" can get it right now without answering their questions. Few people rushed towards him and exited the exam room. About 10 mins later he continued: "The ones who would like to get B mark (4 or "good" in Russia)" can get it now, but you have to show me, what you have written. Some more people rushed towards him. 15 min later (and a few drops of sweat on our brave necks) he said: "The rest just get A's, because you have survived and didn't know in advance what to expect. " (5 or "excellent", the best mark). What I have learnt was that it is not always necessary to be an egg head and learn everything to be always ready to stand up. Sometimes it is important to be a good person, brave and keep courage in your heart. That may lead to more adventures and opportunities in the future!

With a few exceptions I would say, that studying math was both fun and rather instructive in that, it developed some fundamental skills of reasoning and attacking a problem at hand without having trained yourself specifically to solve that class of problems before -- what you need in real life, be it further PhD studies or solving other complex problems, including those occurring in life.

Controlled reflection and template methods in java

This was in drafts for a long time and since then I have lost the original context. But I thought I'll make it compilable and let you, the reader, decide, whether you find any use for this.

Suppose you have a base class A. Suppose also that you need to instantiate two classes B and C, sub-classes of A, with the same configuration data. One straightforward way to achieve this in java is to use constructors:

ConfigData configData = setConfigData();
B b = new B(configData);
C c = new C(configData);

The question is: is there is a way to keep everything just in one method of the base class, governing setting the config data that would return an instance of a subclass of A (B or C)?

Yes, there is! One way to set this is to implement a ctor in the base class A.

Another method is to use "templated" reflection. By "templated" I here refer to Java generics. In order to make sure we get the proper class instances, we should limit the accepted classes with <T extends A>:

public class A {
// declaration updated thanks to Pitko's comment below
public static <t extends A> t configureA(Class<a> ATemplateClass) {
t a;
 ConfigData configData = setConfigData();
 a = ATemplateClass.getConstructor(ConfigData.class).newInstance(configData);
 return a;
    }

private static ConfigData setConfigData() {
        ConfigData configData = new ConfigData("configParam1Value");
 return configData;
    }
}

public class ConfigData {
public String configParam1;

public ConfigData(String _configParam1) {
configParam1 = _configParam1;
}

/* (non-Javadoc)
 * @see java.lang.Object#toString()
 */
@Override
public String toString() {
   return "ConfigData [configParam1=" + configParam1 + "]";
}
}

The child classes will look alike (in practise they will have differrent implementation logic), illustrating with just B subclass:

public class B extends A {
ConfigData configData;

public B(ConfigData _configData) {
this.configData = _configData; 
}

/**
* @return the configData
*/
public ConfigData getConfigData() {
   return configData;
}
}

Now we can say:

B b = A.configureA(B.class);
C c = A.configureA(C.class);
  
System.out.println("Class B:" + b.getConfigData());
System.out.println("Class C:" + c.getConfigData()); 

// which outputs:
Class B:ConfigData [configParam1=configParam1Value]
Class C:ConfigData [configParam1=configParam1Value]

This post illustrates the usage of reflection and generics in java. We were able to access child classes in the base class using "controlled" reflection, that is we allowed only subclasses of the base class to be passed in the reflecitve method. We use generics to return proper subclass instances from the base class.

Solr on FitNesse

This year's Berlin Buzzwords conference was as intense as last year's. For me, in particular, it was heavier on the discussion side (hooked up with Robert Muir to discuss the "deduplication of postings lists" in Lucene and with Ted Dunning to speak some Russian), but some of talks have been interesting enough for me to try something practical immediately.

Dominik Benz of Inovex has presented on FitNesse tool.

In its own words: FitNesse is "the fully integrated standalone wiki and acceptance testing framework". Dominik was describing their experience with integrating it and told that the upfront investment is almost nil and suits to non-technical people. At this point I can confirm the former point, while the second needs more investigation really.

As the presentation concentrated quite heavily on how one would go about integrating FitNesse into the cycle of a Big Data project, I got curious whether this tool would be suitable for some of the tasks on Solr side. I have also compiled a presentation of my own, that summarizes what follows (some of the slides were borrowed from Dominik's slides).

Solr onfitnesse learningfromberlinbuzzwords from Dmitry Kan

A bit of thinking, and decided: implement a FitNesse fixture, that will check the health of solr cluster. Sometimes, when the cluster is too big (say, tens of nodes) someone could be overloading it with posting data or querying data. Some of the nodes (with solr shards) can go down or become unresponsive. It would be nice in a wiki setting to be able to say with a glance: is the cluster up and running or suffers for more CPU / RAM etc?

I'll present quite simple fixture for checking the solr health, which roughly took me 15 minutes to implement. I hope it can be useful for you too.

Here is how FitNesse UI looks like after executing the fixture:

The Java code:

package example;

import fit.ColumnFixture;
import org.apache.solr.client.solrj.SolrServer;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer;
import org.apache.solr.client.solrj.response.SolrPingResponse;

import java.io.IOException;
import java.net.MalformedURLException;

/**
 * Created with IntelliJ IDEA.
 * User: dmitry
 * Date: 6/14/13
 * Time: 3:48 PM
 * To change this template use File | Settings | File Templates.
 */
public class SolrShardsFixture extends ColumnFixture {

    private String shardURL;
    private String shardName;

    public boolean isShardUp() {
        if (shardURL == null || shardURL.isEmpty())
            throw new RuntimeException("shardURL url is empty");
        try {
            SolrServer serverTopic = new CommonsHttpSolrServer(shardURL);
            SolrPingResponse solrPingResponse = serverTopic.ping();

            if (solrPingResponse.getStatus() == 0)
                return true;

        } catch (MalformedURLException e) {
            throw new RuntimeException("Failed to create SolrServer instance: " + e.getMessage());
        } catch (IOException e) {
            throw new RuntimeException("Failed to ping the SolrServer instance: " + e.getMessage());
        } catch (SolrServerException e) {
            throw new RuntimeException(e.getMessage());
        }
        return false;
    }

    public void setShardURL(String shardURL) {
        this.shardURL = shardURL;
    }

    public void setShardName(String shardName) {
        this.shardName = shardName;
    }
}

MTEngine: switching UI languages

This is latest update from our test environment:

We have implemented a feature of respecting your browser language. That is, if you browser tells us en-us, we'll show the English version and if it is ru-ru, it is going to load the Russian version.

MTEngine: latest developments

Here are the latest developments going on on our test environment: MTEngine_test:

1. We took a snapshot of sentences from opencorpora.org and are working on pushing these into the new UI feature, called "tasks". Each task is one Russian sentence to be translated and rated by the user.

2. The feature with a free-form translation remains in the UI and is pushed into its own tab (screenshot in the Russian version of this message below).

For the production version of MTEngine we have done one improvement: when registering and using the system for the first time, the dictionary entries will be looked from the common dictionary, contributed by all our users.

Happy translations!


Same in Russian:

Свежие разработки в тест версии проекта MTEngine:

1. Мы взяли дамп предложений проекта opencorpora.org и работаем над новой фичей под названием "задания". Каждое задание -- это одно предложение на русском языке для перевода и оценки пользователем.

2. Фича с произвольным переводом пользовательских предложений на русском языке будет находится в отдельном табе:

Мы сделали улучшение и в продакшн версии: теперь, когда пользователь регистрируется и делает первые переводы, словарные единицы берутся из общего переводного словаря, который создали все пользователи проекта.

Успешных переводов и хороших выходных!

What grammatical challenges prevent Google Translate from being more effective?

Cross-posting my answer to the question in the topic on quora.com [1].

Google is pretty good at modeling close enough language pairs. By close enough I mean languages that share multiple vocabulary units, have similar word order, morphological richness level and other grammatical features.

Let's pick an example of a pair, where Google Translate (GT) is good. Round-trip method is one way to verify whether the languages are close enough, at least statistically, for GT:

(these examples are using GT only, no human interpretation involved)

English: I am in a shop.
Dutch: Ik ben in een winkel.
back to English I'm in a store. (quite ok)

English: I danced into the room.
Dutch: Ik danste in de kamer.
back to English: I danced in the room. (preposition issues)


Let's pick a pair of more unrelated languages (by the way, when we claim the languages are unrelated grammatically, they may also be unrelated semantically or even pragmatically: different languages were created by people to suit their needs at particular moments of history). One such pair is English and Finnish:

Finnish: Hän on kaupassa.
English: He is in the shop.
Finnish: Hän on myymälä. (roughly the original Finnish sentence)

This example has pronoun hän, which in Finnish is not gender specific. It should be resolved based on larger context, than just a sentence. Somewhere before this sentence in a text, there should have been a mention of who hän is referring to.

To conclude this particular example: Google Translate translates on a sentence level and that is a limitation in itself, that makes correct pronoun resolution impossible. Pronouns are useful, if we wanted to understand, what was the interaction between the objects in a text.


Let's pick another example of unrelated languages: English and Russian.

Russian: Маска бывает правдивее и выразительнее лица.
English: The mask is truthful and expressive face. (should have been: The mask can be more truthful and expressive than face)
back to Russian: Маска правдивым и выразительным лицом. (hard to translate, but the meaning roughly: The mask being a truthful and expressive face).

To conclude this example: languges with rich morphology that, in the case of the Russian language, convey grammatical case in just a word inflection and thus require deeper grammatical analysis, which pure statistical machine translation methods lack no matter how much data has been acquired. There exist methods of combining rules and statistics together.


Another pair and different example:
English: Reporters said that IBM has bought Lotus.
Japanese: 記者は、IBMがロータスを買っていると述べた。
back to English: The reporter said that IBM Lotus are buying.

Japanese has a "recursive syntax", that represents this English sentence, like:

Reporters (IBM Lotus has bought) said that.

i.e. the verb is syntacically placed after the subject-object pair of a sentence or a sub-sentence (direct / indirect object).

To conclude this example: there should exist a method of mapping syntax structures as larger units of the language and that should be done in a more controlled fashion (i.e. is hard to derive from pure statistics).


References
[1] http://www.quora.com/Linguistics/What-grammatical-challenges-prevent-Google-Translate-from-being-more-effective

MTEngine: new test UI features

MTEngine project is going forward and we are now testing new features in the test UI, that have been released last week:

1. Text boxes for editing the translation dictionary are wider now, comfortable for editing even on a mobile.
2. The translation progress indicator is now in the user focus area, right under the text area with the sentence in Russian.
3. The text area with the sentence in English has light-gray background now, similar to the one's of Google Translate service.

In the pipeline:
1. New translation history feature.
2. Linking VK profile for userpic in your user profile.
3. Content for pages "Download" and "About project" (in Russian only).

Feel free to join the testing! You need to know Russian and English.

Test UI URL:   http://semanticanalyzer.info/mtengine_test/

То же сообщение на русском:

Нововведения в тестовом UI:

1. Текстовые поля для правки словаря теперь шире, удобно даже на мобильном телефоне.
2. Индикатор прогресса перевода теперь находится в поле зрения пользователя, прямо под текстовым полем с предложением на русском языке.
3. Поле с переводом на английский теперь светло-серого цвета, как у Google Translate :)

На очереди:
1. Новая фича, где можно посмотреть историю своих переводов.
2. Подключение профиля вконтакте для загрузки юзерпика.
3. Контент для страниц "Скачать" и "О проекте".

Тестовый UI: http://semanticanalyzer.info/mtengine_test/

Java Garbage Collector magic in action (or how to improve your java code using jconsole and jmap)

In my Java experience it has been somewhat unobvious how to jump from monitoring GC and fancy memory graphs with tools like jconsole to actually improving your code.

The ingredient I was missing apparently before was jmap that is part of JDK.

What the tool does is that it allows you to attach to a live java process by process id (pid) and output the histogram of its objects. Here is how it works:

jmap -histo 22170 > histo_22170.log

In the example, 22170 is the java process pid and command line option -histo makes jmap to output a histogram of objects. One nice thing about jmap is that it allows you build an object histogram on JVM OutOfMemory crash (see some details here).

The first lines of the histo_22170.log look like this, before some bug fixing has been done to the code (more on this in a moment):

 num     #instances         #bytes  class name

----------------------------------------------

   1:      88210219     2943436216  [B

   2:       5198407      455421864  [[B

   3:       5162015      123888360  com.mysql.jdbc.ByteArrayRow

   4:       2005702       95264296  [C

   5:       2006883       64220256  java.lang.String

   6:         37819       53525280  [I

   7:        309332       44543808  com.mysql.jdbc.Field

   8:        923280       36931200  java.util.TreeMap$Entry

   9:         18744       25864528  [Ljava.lang.Object;

  10:        471843       15098976  java.util.HashMap$Entry

  11:         36577        5195248  [Ljava.util.HashMap$Entry;

  12:         18196        4512608  com.mysql.jdbc.JDBC4PreparedStatement

  13:         18196        3202496  com.mysql.jdbc.JDBC4ResultSet

  14:         54308        2606784  java.util.TreeMap

  15:         12809        1903312  

  16:         36571        1755408  java.util.HashMap

  17:         12809        1750136  

  19:         18196        1164544  com.mysql.jdbc.PreparedStatement$ParseInfo

I have marked the relevant parts of the histogram with the bold font. The java process was doing some heavy-duty task for thousands of files and talking to the MySQL DB in a loop to load some meta-information for each of the file. The process was given 4GB max heap size and was not properly finishing, producing OutOfMemory error that in turn crashed the JVM.

The code snippet that was producing this looked like this:

PreparedStatement sqlStatement = sqlConnection.prepareStatement(
                          "SELECT * FROM SOME_TBL WHERE SOME_ID=?");

for(int i = 0; i < some_number_less_than_100; i++) {
    sqlStatement.setString(1, companyIds.get(i));
    ResultSet sqlResult = sqlStatement.executeQuery();
    if (sqlResult != null) {
     while (sqlResult.next()) {
        // do some processing of the query results here
     }
    }
}

Intuitively by now you should feel that something is wrong with the code around the JDBC object management.

Let's have a look on the bolded parts from the top of the object histogram. Apparently, the trending JDBC related objects are com.mysql.jdbc.Field with 309332 instances, com.mysql.jdbc.JDBC4PreparedStatement with 18196 instances and com.mysql.jdbc.JDBC4ResultSet with 18196 instances. Two latter objects have exactly same number of instances and that is reflected in our code, where both objects are re-created in a loop. The visual monitoring tool jconsole was showing constant RAM usage growth and Eden Heap Space being saturated with lots of young objects, while the Survivor Heap Space was not trending at all.

What's missing is releasing the JDBC resources, by calling close() methods on both PreparedStatement and ResultSet.

So let's correct the code:

PreparedStatement sqlStatement = sqlConnection.prepareStatement(
                           "SELECT * FROM SOME_TBL WHERE SOME_ID=?");

for(int i = 0; i < some_number_less_than_100; i++) {

    sqlStatement.setString(1, companyIds.get(i));
    ResultSet sqlResult = sqlStatement.executeQuery();
    if (sqlResult != null) {
     while (sqlResult.next()) {
        // do some processing of query results here
     }
     // missing lines added
     sqlResult.close();
}

sqlStatement.close();

After the two missing statements have been added (sqlResult.close() and sqlStatement.close()), the DB resources started to release properly and the original process began to work properly, without big spikes in RAM usage. The JDBC related objects have also disappeared from the top of the histogram:

 num     #instances         #bytes  class name
----------------------------------------------
   1:       1301417       66169256  [C
   2:       1333656       42676992  java.lang.String
   3:        158410       29981072  [I
   4:        382702       12246464  java.util.HashMap$Entry
   5:        116080        5668216  [B
   6:        230584        5534016  java.lang.StringBuffer
   7:         56760        3632640  java.util.regex.Matcher
   8:         18320        2620552 
   9:         18320        2501360 
  10:           588        2187960  [Ljava.util.HashMap$Entry;
  11:          1460        1733744 
  12:         33570        1523672 
  13:         19495        1247680  java.util.regex.Pattern
  14:         19729        1201136  [Ljava.lang.Object;
  15:          1460        1130688 
  16:         19477        1090712  [Ljava.util.regex.Pattern$GroupHead;
  17:          1312        1080992 
  18:         19096         614976  [Ljava.lang.String;
  19:         18642         596544  java.util.RandomAccessSubList
  20:         18642         596544  java.util.AbstractList$ListItr

Now the process is happily completing with reasonable RAM usage:

Click the image to make it bigger

The diagram shows that Eden Heap Space became much more free of young objects and the Survivor Heap Space gets utilized more. See here, if you want more details on various pools of Heap and Non-Heap memory.

Interestingly enough, this bug was hiding for months in the code base and only manifested itself once more data had to be processed. This made the process to run longer and thus reach and overflow the allocated RAM bounds.

This trivial example shows the importance of monitoring your heavy (and not so) java processes.

Happy monitoring!

Berlin buzz words 2012: impressions

This year I have had a unique chance to participate in the Berlin buzz words conference for the first time. In brief, it is the event where search, store and scale people come together to exchange on the recent ideas / developments in the area. I must say that the conference level simply amazed me: the quality of the presentations and the audience maturity have clearly aligned together.

Urania building, the venue

To me, as a Solr / Lucene user and developer it was especially fun to meet in person people I have previously only seen on the mail-lists or in video talks on the Internet. These, in particular, include (in my case): Otis Gostpodnetić, Uwe Schindler, Simon Willnauer, Robert Muir, Grant Ingersoll, Ted Dunning, Rafał Kuć. There've been new folks I haven't heard of previously and got inspired by their presentations, like Alex Lloyd from Google and Markus Weimer from Microsoft (opps, GOOG and MSFT in the same sentence). Got to see sematext guys in action at their SPM booth.

Opening session kicks in

The wi-fi worked everywhere, which is unnatural usually to other conferences. Yet, I kept my laptop at a hotel in order to force myself do three things: 1) actually listen to the presenter and ask questions via mike or in person; 2) occasionally take pictures; 3) network during the coffee-breaks.

First day's keynote session by Leslie Hawthorn

As a result: I took some amount of pictures; felt less distracted and tired at the end of each day; asked questions from the audience and got (probably) recorded on the video and many more questions in person; networked with leaders in their areas to actually perceive how things are going in their communities. SO this is to say, that in the end, what mattered to me was people and not only the technologies they have talked about.

Eric Evan's presentation

Some observations (probably interesting more to the conference orgs), pros and cons mixed:
1) The personal badge could have name on each side because of two reasons: it tends to always flip so that the name isn't visible and second - the map on the other side of it was useless, because it was easy to learn where each auditorium was.
2) Food was great and free beer / ice-cream / snacks by sponsors -- awesome addition.
3) Small auditoriums tended to have been super-packed and the only big one have been super sparse (excluding opening and closing sessions). Could be addressed somehow next year?
4) 20 minutes talks have been a surprise for the presenters that expected to have 40 min. The result is usually running out of time to ask any questions from the audience and presenters getting slowly to the core of their presentation.
5) Party on Monday evening and cute small surprises on the bus seats from wooga were cool!

There was also sometime left to explore the beautiful city of Berlin and of course eat Schnitzel!

Thanks to all the #bbuzz team for excellent experience and hoping to come next year! yours truly,

(first?) virtual presentation on Dialogue conference

Just participated in one of the biggest Russian conferences which fuses together theoretic and applied linguists, Dialogue'12. This time I couldn't come there in person, so instead we decided with @vporoshin to try out some modern technology. The selection was pretty easy: skype Finland->Russia, directed through speakers onto microphone connected to an amplifier. Also injected a photo of myself to add to "physical" presence. The conference organizers have appreciated utilizing new advanced technologies in presenting scientific papers. Here is the presentation (no author's photo there, you had to be present on the conference to see it):

Rule based approach to sentiment analysis at romip’11 slides

View more PowerPoint from Dmitry Kan

Paper on rule-based sentiment accepted!

My paper on rule-based sentiment was accepted to Dialog'2012, special section on ROMIP'2011. The ROMIP had a track on 2-way and 3-way sentiment classification of texts in Russian last year. In our team with @vporoshin we had three major systems:

1. Rule-based described in the paper.
2. Modified multinomial Naive Bayes trained on unigrams and bigrams.
3. Classifier ensemble of the two above.

Rule-based approach largely relies on the pre-crafted polarity dictionary. It means, that it knows only those polarity word sequences, that it has in the dictionary. The MNB classifier in contrast learns such sequences from training set. They also have other differences. MNB is in a way a bag-of-words approach, but may work surprisingly well. In 2-way classification it has shown accuracy of 90+% for one of the domains. The rule-based algorithm has interesting linguistic features, like object oriented sentiment detection. Although this first time, the ROMIP's sentiment tracks did not require an object oriented detection, the test data had an object name (e.g. movie title or product name) attributed to each text to classify. Both object oriented and general sentiment detection has performed equally well and above 50% (i.e. above the accuracy of a coin tossing method). Overall accuracy of the general rule-based classification is 63% with 92% precision for the positive class. This generally means that more polarity words should be mined for the negative class and the existing negative polarity dictionary revised (some words could be of positive or ambiguous polarity).

Some more numbers in the paper:

Rule based approach to sentiment analysis at ROMIP 2011

View more documents from Dmitry Kan

Scientifc agenda of this year

This year stays promising in terms of the scientific happenings, first of all, I participated in the ROMIP contest on sentiment analysis. It was intense and interesting to dive into annotated and test data. More on this later, once information ready.

On the other note, this year's step up was to have been accepted on the committees list of the Second International Symposium on Business Modeling and Software Design (http://www.is-bmsd.org/). The research topics include and are not limited to the following:

BUSINESS MODELS AND REQUIREMENTS
- Business Analysis - Value Models and Process Models
- Essential Business Models
- Re-usable Business Models
- Relating Business Goals to Requirements
- Business Process Coordination
- Business Entities and Business Roles
- Business Data and Semantics
- Business Rules
- Behavior Modeling and Pragmatics
- Identification and Elicitation of Requirements
- Domain-imposed and User-defined Requirements
- Requirements Analysis

BUSINESS MODELS AND SERVICES
- Business Modeling and Service Science
- Relating Business Goals to the Identification of Services
- Service Modeling - Technology-independent and Platform-specific
- Business Rules and Service Composition
- Autonomic Service Behavior
- Context-aware Service Behavior
- Re-usable Service Models

BUSINESS MODELS AND SOFTWARE
- Business Modeling -driven Derivation of Software
- Business Innovation and Software Evolution
- Business-IT Alignment and Traceability
- Re-usable Business Models and Software Components
- Business Rules and Software Specification
- Business Goals and Software Integration
- Autonomic and Context-aware Business/Software Systems

INFORMATION SYSTEMS ARCHITECTURES
- Enterprise Architectures
- Service-Oriented Architectures
- Architectural Styles
- Architectural Viewpoints
- Crosscutting Concerns

My experience with airBaltic

UPD: Please read the entire post. I will not remove the story line written originally, because this is exactly what has happened. However airBaltic contacted me on the phone themselves and told about positive resolution of the case. Please read on.

Original story:
-----
First of all, I would like to assure you that I'm not the best at blaming, meaning I simply don't like doing it publicly. It's probably unfair to only publicly blame an air operator and never praise them. But that's how it works. A happy customer doesn't compile an entire blog post about how cool it was to fly with a certain operator. "If I'm happy, I stay silent" principle. But believe me, if the case I'll tell you here about would resolve positively, I wouldn't hesitate to blog about it.

Here is the case. We planned a 3 days trip to Moscow from Helsinki and back together with my wife. Using skyscanner we've found the cheapest option: fly with airBaltic via Riga. Quick friend survey, all's good, settled. I went online and started my ticket search last Saturday. The cheapest option was to depart on 16:25. Chosen that, prepared to pay. After double-checking dates and times it struck me: nope, no good, departure set to 8:25. Cancelled search, started all over. First question here: is it bad luck or bad system? You choose.
Re-ran my search, all is good, paid 531 euros. But when I printed the travel receipt, this time it REALLY struck me: the return flight date was set to one month later! Another bad luck or system fault? This time I'm inclined to choose the second option. No problems, calling to the Finnish office. "On the weekends office is closed". I decided to call first thing next Monday morning. This is my first mistake and I admit it: should have called to Latvia and pay some euro and a half a minute to change the date.

Calling first thing Monday morning: young lady's voice, I described her the problem. She refused to change the date without an additional fee. I have asked her to connect me to her manager. After a couple of minutes (yeah-yeah, customer is on the first place), manager's voice: teaching and preaching me how I should have used their system. "On every page of multi-page ticket booking process, you can look on your right and check the departure and return flight dates and times." All right, thanks! But look, attempted I to explain the system fault: "Unless you really travel back in one month,there is no way to choose that different month without extra movements. No does the system suggest you the best return flights from in a month period!" This was simply noise for her and she continued teaching me how to use the system. I asked her to give me her manager / director. Guess what was the answer: "This is not possible". "Why?". "I'm sorry, but this is not possible." Being a customer, I'm pretty sure, I can talk to almost any worker of the company, who stays in the customer relations line. This time it is your fault, airBaltic.

"I would like to change the return flight date back to what it should be". But the lady tought me another time: "This is only possible if you pay 150 euros. You should have called us on Saturday and explained the problem." Which I did! And the Finnish office was closed. Is this really my problem now? I doubt it. Because, it is YOUR REPRESENTATIVE, airBaltic! What if I wouldn't have an opportunity to call abroad (yes, Riga is abroad to me) and pay for an international call? And if it didn't work, make sure to take my call on Monday morning seriously, attend to it and make an exception or a good men deal. What on Earth does this rule "if you called after two days, it cannot be changed without a fee" policy mean? Do you want to keep a customer or loose it? What do you loose by changing the month standing away date? Afraid not to find any cusomer during an entire month?

"And if I cancel the entire trip, what sum can be refunded?" "You get 76 euros back". Excellent.

Without further ramblings, I would like to publicly thank airBaltic for 531 euros worth "stay at home and don't travel with us" service. It has really taught me not to use your services. Ever.
Everything seems to be mortal in this wolrd, and airBaltic's serivce will die as well. But by making this type of "friendly" customer service and policies you only bring the end faster.

Good luck and enjoy 531-76 euros for not taking us where we wanted.
-----

UPDATE to the story: airBaltic continues working on the case, here is what they posted on twitter: @DmitryKan Dmitry, your case is not closed. Please give us a bit more time and colleagues will come back to you.

UPDATE 2: The case has been resolved. I have received a call from airBaltic, where they said that the return flight date was changed without an extra fee. I don't know was it a result of my social media activity since yesterday evening, but airBaltic service was extremely fast and accurate this time. Since all the posts I have done on the Internet about airBaltic link here, the landed people will read these updates as well. Thank you, airBaltic.