My experience with airBaltic

UPD: Please read the entire post. I will not remove the story line written originally, because this is exactly what has happened. However airBaltic contacted me on the phone themselves and told about positive resolution of the case. Please read on.

Original story:
-----
First of all, I would like to assure you that I'm not the best at blaming, meaning I simply don't like doing it publicly. It's probably unfair to only publicly blame an air operator and never praise them. But that's how it works. A happy customer doesn't compile an entire blog post about how cool it was to fly with a certain operator. "If I'm happy, I stay silent" principle. But believe me, if the case I'll tell you here about would resolve positively, I wouldn't hesitate to blog about it.

Here is the case. We planned a 3 days trip to Moscow from Helsinki and back together with my wife. Using skyscanner we've found the cheapest option: fly with airBaltic via Riga. Quick friend survey, all's good, settled. I went online and started my ticket search last Saturday. The cheapest option was to depart on 16:25. Chosen that, prepared to pay. After double-checking dates and times it struck me: nope, no good, departure set to 8:25. Cancelled search, started all over. First question here: is it bad luck or bad system? You choose.
Re-ran my search, all is good, paid 531 euros. But when I printed the travel receipt, this time it REALLY struck me: the return flight date was set to one month later! Another bad luck or system fault? This time I'm inclined to choose the second option. No problems, calling to the Finnish office. "On the weekends office is closed". I decided to call first thing next Monday morning. This is my first mistake and I admit it: should have called to Latvia and pay some euro and a half a minute to change the date.

Calling first thing Monday morning: young lady's voice, I described her the problem. She refused to change the date without an additional fee. I have asked her to connect me to her manager. After a couple of minutes (yeah-yeah, customer is on the first place), manager's voice: teaching and preaching me how I should have used their system. "On every page of multi-page ticket booking process, you can look on your right and check the departure and return flight dates and times." All right, thanks! But look, attempted I to explain the system fault: "Unless you really travel back in one month,there is no way to choose that different month without extra movements. No does the system suggest you the best return flights from in a month period!" This was simply noise for her and she continued teaching me how to use the system. I asked her to give me her manager / director. Guess what was the answer: "This is not possible". "Why?". "I'm sorry, but this is not possible." Being a customer, I'm pretty sure, I can talk to almost any worker of the company, who stays in the customer relations line. This time it is your fault, airBaltic.

"I would like to change the return flight date back to what it should be". But the lady tought me another time: "This is only possible if you pay 150 euros. You should have called us on Saturday and explained the problem." Which I did! And the Finnish office was closed. Is this really my problem now? I doubt it. Because, it is YOUR REPRESENTATIVE, airBaltic! What if I wouldn't have an opportunity to call abroad (yes, Riga is abroad to me) and pay for an international call? And if it didn't work, make sure to take my call on Monday morning seriously, attend to it and make an exception or a good men deal. What on Earth does this rule "if you called after two days, it cannot be changed without a fee" policy mean? Do you want to keep a customer or loose it? What do you loose by changing the month standing away date? Afraid not to find any cusomer during an entire month?

"And if I cancel the entire trip, what sum can be refunded?" "You get 76 euros back". Excellent.

Without further ramblings, I would like to publicly thank airBaltic for 531 euros worth "stay at home and don't travel with us" service. It has really taught me not to use your services. Ever.
Everything seems to be mortal in this wolrd, and airBaltic's serivce will die as well. But by making this type of "friendly" customer service and policies you only bring the end faster.

Good luck and enjoy 531-76 euros for not taking us where we wanted.
-----

UPDATE to the story: airBaltic continues working on the case, here is what they posted on twitter: @DmitryKan Dmitry, your case is not closed. Please give us a bit more time and colleagues will come back to you.

UPDATE 2: The case has been resolved. I have received a call from airBaltic, where they said that the return flight date was changed without an extra fee. I don't know was it a result of my social media activity since yesterday evening, but airBaltic service was extremely fast and accurate this time. Since all the posts I have done on the Internet about airBaltic link here, the landed people will read these updates as well. Thank you, airBaltic.

axis2: serialization and deserialization of wsdl2java generated objects

Using axis2's wsdl2java tool and a third-party wsdl I have generated service stub and supporting classes (data holders). Since there was a need to do post-processing of loaded data from a service, there was a need to serialize one of the data holder objects.

Questions that I had and posted on stackoverflow.com were:

1) is there a standard axis2 tool / approach that can be used for the purpose?

2) since the data holder class does not implement Serializable interface what would be the easiest way of serializing the object into xml format with the ability to restore the original object?

Data binding option was used (-d jaxbri) and each field of the class in question is annotated with @XmlElement tag, e.g.:

@XmlElement(name = "ID", required = true)
protected String id;



Here is how I solved it:

1. axis2 generated java classes set (client side) had an object called ObjectFactory. Majority of its methods create JAXBElement objects with values of fields of the class holder
2. I had to implement a serializable wrapper class ASerializable for the class holder, such that it uses the ObjectFactory to create the JAXBElement objects for all the fields.
3. some external code uses the wrapper class to create an serializable object and writes it to the output stream.
4. on the receiving end:

ASerializable aSerializable;
        A a;
        aSerializable= (ASerializable)in.readObject();
        a.setID((String)aSerializable.getID().getValue());

It still looks like extra work for the pre-annotated class serialization, but better than serializing into some text format and manual type checking during deserialization.
Some good intro into serialization with java can be found here.

First international publication

Celebration!

Had my first international publication shown on the DBLP. Somehow it was something I wanted to achieve as an intermediate goal in the academic career. In a way this gives some visibility to what I have been doing for around 4 years. I mean NLP (Natural Language Processing) and Machine Translation more precisely. Before going international, I've had 5 publications in the Russian scientific journals and conferences.

On the same ICSOFT'11 conference where this publication has been presented in a form of poster, I had an honor to serve as knowledge-based systems track chair. Both presenting my work and leading the session were exciting. I wanted also to say thanks to the ICSOFT's organizing committee for giving me the participant's grant, that made my participation possible. Special thanks to Sergio Brissos.

ICSOFT 2011 Conference

ICSOFT is strictly not an NLP conference. However, it has a knowledge-based track, where rather relevant NLP related topics are listed:

Ontology Engineering
Decision Support Systems
Intelligent Problem Solving
Expert Systems
Reasoning Techniques
Knowledge Acquisition
Knowledge Mining
Machine Learning
Natural Language Processing
Human-Machine Cooperation

Two publications I remembered

From these, ontology engineering articles were strong. One of them (Barbara Furletti, Franco Turini: Mining Influence Rules out of Ontologies, see here) was about mining ontology and reasoning rules from the oldest Italian bank's data. This sounds exceptional to me, when some (even old) commercial data is given away to researchers.

Another, non-directly related to NLP, article I remembered was by Manolya Kavakli et al (Manolya Kavakli, Tarashankar Rudra, Manning Li: An Embodied Conversational Agent for Counselling Aborigines - Mr. Warnanggal.), where one of the challenges is providing health assistance to the Australlian aborigines via a computer based system, not very motivated people, poor, stealing food and other things. Another challenge is dealing with about 500 languages, that these aborigines speak. Here is a potential for interesting NLP problems.

Why would I recommend going to a conference not directly related to your research topic?

As a pre-word, I should mention, that in a way whatever we do in the NLP is materialized in the form of programming code. Therefore our work qualifies to a software engineering conference as well as to an NLP one.

Going to a strictly SW conference can give you the following benefits:
* concrete questions of you work in the light of software development practices. Some NLP researchers may think it is not very important to make their SW configurable, re-usable or performant. In the end of the day, this matters a lot, especially if you plan to implement you work into industrial level solution

* if you do a poster presentation, people can give you good insights into the quality of your poster and what can be improved. There were two extreme cases on the conference: one with the entire article text being pasted into the poster and another one with a couple of boxes and an arrow between them. The audience has reacted in an expected way: the first poster did not draw almost any attention, while the second had gathered the majority of the audience.

* you can pause and reflect a little bit: are you doing something valuable? Do you like what you do?


A couple of words about Spain, where ICSOFT'11 happened. +45 is something I have experienced for the first time; visiting royal palace Alcázar of Seville was extremely interesting and of course partying with conference peers over Spanish wine and tapas made the event memorable.

Enjoy you research life and publish your work as soon as possible.

Оценка системы машинного перевода

Есть система машинного перевода с русского на английский.
Нужно сделать ручную оценку работы системы.

Целевая аудитория: все, кто хочет сделать машинные переводчики лучше (примеры: translate.google.com, translate.ru) и люди, интересующиеся прикладной лингвистикой (Natural Language Processing). Умение программировать НЕ требуется.

Связь: dmitry.kan[+AT+]gmail.com, twitter: DmitryKan

Задача: получить у меня пакет предложений (объём: сколько возьмётесь).

В пакете: предложения на русском языке и их переводы экспертом на английский язык.

Прогнать предложения на русском через систему. Просмотреть вручную их переводы на английский язык.

Составить список слов, которые не были найдены (это просто: не найденные слова будут выведены на русском). Послать мне список слов, я добавлю их в систему машинного перевода.

На выходе от вас три группы предложений из пакета:
1. хорошо перевелись
2. приемлемо перевелись (понятно по английской фразе, что было в русской)
3. плохо перевелись (непонятно по английской фразе, что было в русской)

Работа волонтёрская. Начальный бонус: статья со мной в соавторстве на конференции либо в журнале, если вам это интересно. Если нет -- всяческий пиар вам.

Дальше: если сработаемся, предложу Вам работу в лингвистических проектах (умение программировать обязательно).

Ссылки для интересующихся
[1] http://www.slideshare.net/dmitrykan/icsoft-2011-51cr
[2] http://www.slideshare.net/dmitrykan/automatic-build-of-semantic-translational-dictionary
[3] http://ufal.mff.cuni.cz/umc/

interested in rule based machine translation (rbmt)? / Интересуетесь машинным переводом на правилах?

I'm looking for students and activists of rule-based machine translation to help me in the evaluation of my machine translation system from Russian into English. Details in the e-mail: dmitry.kan[+AT+]gmail.com (substitute characters from [ fro ] with @).

Я ищу студентов и активистов машинного перевода на правилах для оценки моей системы машинного перевода с русского на английский. Детали по почте: dmitry.kan[+AT+]gmail.com (замените символы с [ по ] знаком @).

Interested in machine translation between Russian and English?

Then mark August 15-19 2011 in your calendars. Web of Data'11 has accepted my poster on machine translation with semantic features, the full paper title is:

Semantic Feature Machine Translation System for Information Retrieval

Some details on the work from another poster, accepted to ICSOFT'11 can be checked here:

Poster: Method for an automatic generation of a semantic-level contextual translational dictionary

View more documents from Dmitry Kan

Пример кода, интегрирующего AOT морфологический лемматайзер в C#

АОТ предлагает свой лемматайзер для русского и английского языка на сайте www.aot.ru. Если Вам нужно интегрировать их COM внутри проекта на C#, читайте ниже.

После установки библиотеки при помощи Setup.exe, загрузите Lemmatizer.dll в C# проект. Скопируйте следующий метод или его тело, например, в main-class:

private static void initAOTMorphoanalyzer()
{
 LEMMATIZERLib.ILemmatizer lemmatizerRu = new LEMMATIZERLib.LemmatizerRussian();
 lemmatizerRu.LoadDictionariesRegistry();
 LEMMATIZERLib.IParadigmCollection piParadigmCollection = lemmatizerRu.CreateParadigmCollectionFromForm("мыла", 0, 0);

 Console.Out.WriteLine(piParadigmCollection.Count);       

    for (int j=0; j < piParadigmCollection.Count; j++)
    {
     object[] args = { j };     

     Type paradigmCollectionType = piParadigmCollection.GetType();

     if (paradigmCollectionType != null)
     {
      object Item = paradigmCollectionType.InvokeMember("Item", BindingFlags.GetProperty, null, piParadigmCollection, args);
      Type itemType = Item.GetType();
      if (itemType != null)
      {
       object Norm = itemType.InvokeMember("Norm", BindingFlags.GetProperty, null, Item, null);
       Console.Out.WriteLine(Norm);
      }
      else
       Console.Out.WriteLine("itemType is null");
     }
     else
      Console.Out.WriteLine("paradigmCollectionType is null");
    }
}

Результат:
2
МЫЛО
МЫТЬ

COM test example for C#: AOT lemmatizer

I will post this both in English and Russian for more people's benefit.

There is a Russian / English lemmatizer from AOT (www.aot.ru). If you need to use the COM that AOT provides inside C#, read on. Load the lemmatizer.dll inside your C# project. Insert the following method or its body inside your code, for example main class:

private static void initAOTMorphoanalyzer()
{
 LEMMATIZERLib.ILemmatizer lemmatizerRu = new LEMMATIZERLib.LemmatizerRussian();
 lemmatizerRu.LoadDictionariesRegistry();
 LEMMATIZERLib.IParadigmCollection piParadigmCollection = lemmatizerRu.CreateParadigmCollectionFromForm("мыла", 0, 0);

 Console.Out.WriteLine(piParadigmCollection.Count);       

    for (int j=0; j < piParadigmCollection.Count; j++)
    {
     object[] args = { j };     

     Type paradigmCollectionType = piParadigmCollection.GetType();

     if (paradigmCollectionType != null)
     {
      object Item = paradigmCollectionType.InvokeMember("Item", BindingFlags.GetProperty, null, piParadigmCollection, args);
      Type itemType = Item.GetType();
      if (itemType != null)
      {
       object Norm = itemType.InvokeMember("Norm", BindingFlags.GetProperty, null, Item, null);
       Console.Out.WriteLine(Norm);
      }
      else
       Console.Out.WriteLine("itemType is null");
     }
     else
      Console.Out.WriteLine("paradigmCollectionType is null");
    }
}

Output:
2
МЫЛО
МЫТЬ

Amazed by Scala #1: objects and compilation

7 minutes and here is an object, that can be compiled into java classes:

import scala.actors._
import Actor._

object TopStock {
 val symbols = List( "AAPL", "GOOG", "IBM", "MSFT")
 val receiver = self
 val year = 2008
 
  def main(args: Array[String]) = {
 symbols.foreach { symbol =>
   actor { receiver ! getYearEndClosing(symbol, year) }
 }
  
 val (topStock, highestPrice) = getTopStock(symbols.length)
 printf("Top stock of %d is %s closing at price %f\n", year, topStock, highestPrice)
  }

 def getYearEndClosing(symbol : String, year : Int) = {
   val url = "http://ichart.finance.yahoo.com/table.csv?s="+ 
   symbol + "&a=11&b=01&c=" + year + "&d=11&e=31&f=" + year+
             "&g=m"
   
   val data = io.Source.fromURL(url).mkString
   val price = data.split("\n")(1).split(",")(4).toDouble
   (symbol, price)
 }

 def getTopStock(count : Int) : (String, Double) = {
   (1 to count).foldLeft("", 0.0) { (previousHigh, index) =>
  receiveWithin(10000) {
    case (symbol : String, price : Double) =>
   if (price > previousHigh._2) (symbol, price) else previousHigh
  }
   }
 }  
}

Saved in TopStock.scala. Compiled with

> scalac TopStock.scala

ran with

> scala TopStock

Top stock of 2008 is GOOG closing at price 307,650000

Amazed by Scala

Top stock of 2008 is GOOG closing at price 307,650000 among (AAPL, GOOG, IBM, MSFT). Amazed by simplicity, clarity and beauty of the following code in Scala from this book.

import scala.actors._
import Actor._

val symbols = List( "AAPL", "GOOG", "IBM", "MSFT")
val receiver = self
val year = 2008

symbols.foreach { symbol =>
  actor { receiver ! getYearEndClosing(symbol, year) }
}
  
val (topStock, highestPrice) = getTopStock(symbols.length)

printf("Top stock of %d is %s closing at price %f\n", year,
           topStock, highestPrice)

def getYearEndClosing(symbol : String, year : Int) = {
 val url = "http://ichart.finance.yahoo.com/table.csv?s="+ 
 symbol + "&a=11&b=01&c=" + year + "&d=11&e=31&f=" + year+
       "&g=m"
  
 val data = io.Source.fromURL(url).mkString
 val price = data.split("\n")(1).split(",")(4).toDouble
 (symbol, price)
}

def getTopStock(count : Int) : (String, Double) = {
 (1 to count).foldLeft("", 0.0) { (previousHigh, index) =>
    receiveWithin(10000) {
  case (symbol : String, price : Double) =>
    if (price > previousHigh._2) (symbol, price) else
             previousHigh
  }
 }
}

StackOverflow

Do you know the famous resource http://stackoverflow.com by Joel Spolsky and team? Here is my own definition of stackoverflow in Java:

private static void log(String logStatement) {
   log(logStatement);
}

XML in Chrome

Tired of (manual) XML rendering in Google Chrome?
Try this:
XML Tree

NoSQL, Apache SOLR and Apache Hadoop

Check out this SlideShare Presentation -- talk for NёrdCamp April 23, 2011:

NoSQL, Apache SOLR and Apache Hadoop

View more presentations from Dmitry Kan.

NerdCamp

NerdCamp is coming and I'm going to present on Apache technologies, specifically on Apache Solr and a bit on Apache Hadoop. If you are into all this and like creative thinking, want to create a start-up and will come around Saint-Petersburg this weekend, you should definitely come to NerdCamp!

Confirmed Key Participants

Yury Lifshits, Sergey Poduzov, Alexander Shtuchkin,
Vladimir Gorovoy, Nikolay Vyahhi, Vladimir Aluferov, Yakov Sirotkin and myself

The preliminary program has juicy topics among which are:
How the Web will Transform Education by Yury Lifshits,
Geo Information Systems Around Us by Aleksander Klechikov,
Software Development for little ones by Yakov Sirotkin,
Introducing NoSQL: Apache SOLR and Hadoop by Dmitry Kan,
Introducing Cloud Services by Dmitry Petrov,
Nature of Enterpreneurship Craft, Lifestyle or Science by Alexey Baranov,
How to grow places for smart people -- workshop

and more!

Angry Birds -pelaaja: DK

Angry Birds -pelaaja: DK: "DK on kertakaikkisen kyllästynyt sikailuun! Seuraa hänen edesottamuksiaan Angry Birdsin SM-kisoissa ja tule itsekin mukaan."

concert of components?

"ZooKeeper: Because coordinating distributed systems is a Zoo"

This post is short and not useful in that it doesn't give you any code snippets or technical recommendations. I would like just to cite the SolrCloud's wiki. The SolrCloud is "the set of Solr features that take Solr's distributed search to the next level, enabling and simplifying the creation and use of Solr clusters." They use Apache ZooKeeper project (a subproject of Hadoop) as a distributed system of keeping cluster of SOLRs state updates. In a distributed system every component can potentially crash, yet the system is expected to provide it's service to its users. If a single SOLR crashes, its replica will take over, but if ZooKeeper crashes, the system will still continue serving the user requests, but no updates of the system are visible to the system (sounds interesting, I know). To improve that, this is what is possible:

"Running multiple zookeeper servers in concert (a zookeeper ensemble) allows for high availability of the zookeeper service. Every zookeeper server needs to know about every other zookeeper server in the ensemble, and a majority of servers are needed to provide service. For example, a zookeeper ensemble of 3 servers allows any one to fail with the remaining 2 constituting a majority to continue providing service. 5 zookeeper servers are needed to allow for the failure of up to 2 servers at a time."

So, if you have a big zoo with variety of animals in it, make sure you have 5 zoo keepers for at least 3 of them take care of your pigs and elephants, when 2 others got stuck somewhere else.

SOLR: speed up batch posting

If you are familiar with the Apache SOLR and deal with index worth of millions documents, it becomes quite important to be able to (re-)index fast. There are various techniques as to what to tweak for the indexing go faster. Another simple way to speed up your (batch) indexing without changing your SOLR schema is to modify logging.

Apparently, when deployed under Tomcat, SOLR logs each and every update request during POSTing process. Experience shows, that heavy http operation is done faster, when logging is minimal.

SOLR (as of 1.4 at least) has admin GUI which serves as a central information hub for the given SOLR core. Among other useful features, it has a page where one can set up logging levels of different SOLR components. In default SOLR installation you can access the page via http://localhost:8983/solr/admin/logging. By default, the logging levels amount mainly to INFO, which permits logging of all the select/update requests (imagine 1 million of such log entries for batch reindexing).

It would be handy to be able to automatically change the logging levels to, say, WARNING before batch POSTing and back to INFO after that. solr/admin/logging is declared as servlet in web.xml of the corresponding SOLR core:

  
    Logging
    org.apache.solr.servlet.LogLevelSelection
  

All the components which allow changing their logging levels are listed on the page http://localhost:8983/solr/admin/logging. Using curl we can send a post request to the servlet and set the desired levels. It is reasonable to implement a function, which takes logging level and url of SOLR core as parameters (choose your own favourite language, this is done in Perl):

sub setSolrLogLevel
{
 my ($url, $level) = @_;

    print "setting logging level to $level\n";
    my $res = system("curl --user user:pass -d \"submit=set&root=$level&fi=$level" .
"&fi.alphasense=$level&fi.alphasense.solr=$level&fi.alphasense.solr.query=$level" .
"&fi.alphasense.solr.query.AlphaSenseQParserPlugin=$level&httpclient=$level&httpclient.wire=$level&httpclient.wire.content=$level&httpclient.wire.header=$level&javax=$level&javax.management=$level&javax.management.mbeanserver=$level&org=$level&org.apache=$level&org.apache.catalina=$level&org.apache.catalina.core=$level&org.apache.catalina.core.ContainerBase=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5BLogging%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5BSolrServer%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5BSolrUpdate%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5Bdefault%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5Bjsp%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5Bping%5D=$level&org.apache.catalina.session=$level&org.apache.catalina.session.ManagerBase=$level&org.apache.commons=$level&org.apache.commons.digester=$level&org.apache.commons.digester.Digester=$level&org.apache.commons.digester.Digester.sax=$level&org.apache.commons.httpclient=$level&org.apache.commons.httpclient.ChunkedInputStream=$level&org.apache.commons.httpclient.HeaderElement=$level&org.apache.commons.httpclient.HttpClient=$level&org.apache.commons.httpclient.HttpConnection=$level&org.apache.commons.httpclient.HttpMethodBase=$level&org.apache.commons.httpclient.HttpMethodDirector=$level&org.apache.commons.httpclient.HttpParser=$level&org.apache.commons.httpclient.HttpState=$level&org.apache.commons.httpclient.MultiThreadedHttpConnectionManager=$level&org.apache.commons.httpclient.SimpleHttpConnectionManager=$level&org.apache.commons.httpclient.auth=$level&org.apache.commons.httpclient.auth.AuthChallengeProcessor=$level&org.apache.commons.httpclient.cookie=$level&org.apache.commons.httpclient.cookie.CookiePolicy=$level&org.apache.commons.httpclient.cookie.CookieSpec=$level&org.apache.commons.httpclient.methods=$level&org.apache.commons.httpclient.methods.EntityEnclosingMethod=$level&org.apache.commons.httpclient.methods.ExpectContinueMethod=$level&org.apache.commons.httpclient.methods.PostMethod=$level&org.apache.commons.httpclient.params=$level&org.apache.commons.httpclient.params.DefaultHttpParams=$level&org.apache.commons.httpclient.params.HttpMethodParams=$level&org.apache.commons.httpclient.util=$level&org.apache.commons.httpclient.util.EncodingUtil=$level&org.apache.commons.httpclient.util.ExceptionUtil=$level&org.apache.commons.httpclient.util.IdleConnectionHandler=$level&org.apache.jasper=$level&org.apache.jasper.EmbeddedServletOptions=$level&org.apache.jasper.JspCompilationContext=$level&org.apache.jasper.compiler=$level&org.apache.jasper.compiler.Compiler=$level&org.apache.jasper.compiler.JspConfig=$level&org.apache.jasper.compiler.JspRuntimeContext=$level&org.apache.jasper.compiler.TldLocationsCache=$level&org.apache.jasper.servlet=$level&org.apache.jasper.servlet.JspServlet=$level&org.apache.jasper.servlet.JspServletWrapper=$level&org.apache.solr=$level&org.apache.solr.analysis=$level&org.apache.solr.analysis.BaseTokenFilterFactory=$level&org.apache.solr.analysis.BaseTokenizerFactory=$level&org.apache.solr.client=$level&org.apache.solr.client.solrj=$level&org.apache.solr.client.solrj.impl=$level&org.apache.solr.client.solrj.impl.CommonsHttpSolrServer=$level&org.apache.solr.common=$level&org.apache.solr.common.util=$level&org.apache.solr.common.util.ConcurrentLRUCache=$level&org.apache.solr.core=$level&org.apache.solr.core.Config=$level&org.apache.solr.core.CoreContainer=$level&org.apache.solr.core.JmxMonitoredMap=$level&org.apache.solr.core.RequestHandlers=$level&org.apache.solr.core.SolrConfig=$level&org.apache.solr.core.SolrCore=$level&org.apache.solr.core.SolrResourceLoader=$level&org.apache.solr.handler=$level&org.apache.solr.handler.AnalysisRequestHandler=$level&org.apache.solr.handler.XmlUpdateRequestHandler=$level&org.apache.solr.handler.admin=$level&org.apache.solr.handler.admin.LukeRequestHandler=$level&org.apache.solr.handler.admin.SystemInfoHandler=$level&org.apache.solr.handler.component=$level&org.apache.solr.handler.component.QueryElevationComponent=$level&org.apache.solr.handler.component.SearchHandler=$level&org.apache.solr.handler.component.SpellCheckComponent=$level&org.apache.solr.highlight=$level&org.apache.solr.highlight.SolrHighlighter=$level&org.apache.solr.request=$level&org.apache.solr.request.BinaryResponseWriter=$level&org.apache.solr.request.XSLTResponseWriter=$level&org.apache.solr.schema=$level&org.apache.solr.schema.FieldType=$level&org.apache.solr.schema.IndexSchema=$level&org.apache.solr.search=$level&org.apache.solr.search.SolrIndexSearcher=$level&org.apache.solr.servlet=$level&org.apache.solr.servlet.LogLevelSelection=$level&org.apache.solr.servlet.SolrDispatchFilter=$level&org.apache.solr.servlet.SolrRequestParsers=$level&org.apache.solr.servlet.SolrServlet=$level&org.apache.solr.servlet.SolrUpdateServlet=$level&org.apache.solr.spelling=$level&org.apache.solr.spelling.AbstractLuceneSpellChecker=$level&org.apache.solr.spelling.FileBasedSpellChecker=$level&org.apache.solr.spelling.IndexBasedSpellChecker=$level&org.apache.solr.update=$level&org.apache.solr.update.SolrIndexConfig=$level&org.apache.solr.update.UpdateHandler=$level&org.apache.solr.util=$level&org.apache.solr.util.SolrPluginUtils=$level&org.apache.solr.util.plugin=$level&org.apache.solr.util.plugin.AbstractPluginLoader=$level\" $url");
    print "Result code:$res\n";
}

There you go. Call setSolrLogLevel("http://localhost:8983/solr/admin/logging", "WARNING"); before the batch POSTing and setSolrLogLevel("http://localhost:8983/solr/admin/logging", "INFO"); after the batch POSTing has finished.

"OOPness" in Java

Optimisation in my code has led me to a code refactoring, where several methods with return values became one. Since I could not use "pass by reference" feature for all my parameters (primitives and objects as collections) I had to come up with an inner class for holding all return parameters.

So by not supporting "pass by reference" feature Java makes your code even more OOP-ish. Even though the topic can be well known, this article helps to put things together and make a memory refresh.

Successive replacement in regular expressions (java)

Actually not sure, how often people out there do the successive replacement in a target text having a regular expression pattern, but Java has rather neat solution for it. I'm publishing it here, because I know, that esp. younger developers can re-invent a wheel here and have longer debugging sessions.

So the task is like this: you have a text T, like "cat-1 dog-1 cat-1 elephant-1 cat-2 dog-2 cat-3".
Suppose we want to change numerals attached to the words "cat" to their word representations: "1" to "one", "2" to "two".

One straightforward way would be to match all "cat-([0-9])+" subsequences and then run replace operation on T.

So the code would look something like this:

String T = "cat-1 dog-1 cat-1 elephant-1 cat-2 dog-2 cat-3";
Pattern catPattern = Pattern.compile("cat-([0-9]+)");
Matcher catMatcher = catPattern.matcher(T);
Map numToWord = new HashMap();
numToWord.add("1", "one");
numToWord.add("2", "two");
numToWord.add("3", "three"); // ...
while (catMatcher.find())
{
    T = T.replaceFirst(catMatcher.group(1), numToWord.get(catMatcher.group(1)));
}

This code produces:

cat-one dog-one cat-1 elephant-1 cat-two dog-2 cat-three

Which is missing one substitution. Ok, let's use replaceAll instead and make sure we touch only cats:

{
    T = T.replaceAll("cat-" + catMatcher.group(1), "cat-" + numToWord.get(catMatcher.group(1)));
}

which produces what we want:

cat-one dog-1 cat-one elephant-1 cat-two dog-2 cat-three

But now what happens inside the loop is logically out of sync with the loop condition: we iterate over matches, but call replaceAll (probably not efficient either, as replaceAll will be attempted even when not needed anymore, for duplicate matches).

Any more elegant and correct solution?

Yes! It is called Matcher.appendReplacement

  Pattern catPattern = Pattern.compile("cat-([0-9]+)");
  Matcher catMatcher = catPattern.matcher(T);
  Map numToWord = new HashMap();
  numToWord.put("1", "one");
  numToWord.put("2", "two");
  numToWord.put("3", "three"); // ...
  
  StringBuffer sb = new StringBuffer();
  
  while (catMatcher.find())
  {
   System.out.println("Match:" + catMatcher.group(1));
   catMatcher.appendReplacement(sb, "cat-" + numToWord.get(catMatcher.group(1)));
  }
  catMatcher.appendTail(sb);

now sb.toString() contains:

cat-one dog-1 cat-one elephant-1 cat-two dog-2 cat-three

If you append System.out.println(sb.toString()); inside the while loop, you will also see, that replacements happen in sync with the while loop's state, so that what is inside the loop and what while loops over are in sync.

B2B: what is WSDL (simple explanation)

Just for those wondering around what is WSDL file or technology in the area of Business 2 Business (when companies speak to each other automatically over the network): you can think of it as a declaration of the client-server conversation. In this case one B (server) provides some functionality (like taking square roots of big numbers) and another B (client) has a lot of these numbers and needs square roots of them.

So the server declares a method (which is in fact a remote method):

Double squareRoot(Double number)

in its WSDL file. The client side then takes this file and generates the client side code, which handles the protocol of communication and remote method invocation automatically. What client needs is simply to implement a business logic around this invocation, like a web page with a text field for the number or the entire hardware with touch screen display.

In general the input and return types of remote methods can be simple and complex -- in fact as complex and "proprietary" as you need, because the entire WSDL file is nothing but xml as well as the messages being sent over the network.

One of the libraries used in the industry is Axis and Axis2. Using the library you can generate the java code from wsdl for you client (and even generate wsdl from your java code for you server).

Zero-witdth negative lookahead group: example

Suppose you have to split a comma separated string in java where meaningful parts are separated by continious spaces, say from 2 to infinity. But the problem is that sometimes a meaningfull part (like a person name) contains 2 spaces. You want to exclude such cases and retain the name's parts together. Example:

name1[one_space]surname1[two_spaces]age
surname2[two_spaces]surname2[two_spaces]age

In this example it is easy to solve the problem with zero-width negative lookahead group (?!\\w), because age is always supposed to be numeric. So the final code will be something like this:

String s = current_string.split("\\s{2}(?!\\w)");

Парижские впечатления

Вернулся из второй поездки в Париж. Вот сейчас, уже по возвращению, я подумал: в Париже все отдыхают. Или это только видимость? Как бы там ни было, отдыхать там легко. Понравилось кататься на public велосипедах!

Dumper and sorting of keys

When you use Dumper for debugging via comfortable logging data structures by reference, you can additionally apply sorting to the output. Say, you have a hash addressed by $hash_ref reference. If you need to output the hash contents, but have its keys sorted, you can do:

use strict;
use Data::Dumper;

$hash_ref = generate your hash here

$Data::Dumper::Sortkeys = \&my_filter;
print Dumper($hash_ref), "\n";

sub my_filter {
 my ($hash) = @_;
 # return an array ref containing the hash keys to dump
 # in the order that you want them to be dumped
        return [
            sort {$a <=> $b} keys %$hash
        ];
}

Сходимость

Код программы аппроксимирует решение задачи. Сходимость кода к решению осуществляется посредством связки "изменение кода"-"компиляция/интерпретация"-"запуск". Получается что-то вроде временного ряда: код может не сходиться неделю и сойтись за один день. Если код не сходится, стоит посмотреть на саму концепцию в решении / задачу / компилятор.

Всё это к тому, что полезно преподавать программирование, рассматривающее программу как целостную математическую схему, а не только как набор алгоритмов, приёмов и дизайна.

Finnish is like C++

, because it is highly typed.

An idea for GUI designers and developers -- a standalone app / OS level change.

When I'm under VPN & putty I sometimes need to run an sftp client. Currently I have two options to use: WinSCP (I find it easier to use in some cases, like quickly watching / editing the contents of different text files) and Secure Shell Client from SSH. For some reason the first client loads in about 2-3 minutes after I log in. So: it would be great to have an option to configure launch of a program depending on a certain event, like putty execution, connected to a certain host.

This may give a lot of opportunities to quite flexibly configure your GUI world and save a lot of time spent otherwise on the monkey repetitions.

giza++ under windows: episode 2

It turned out, that in order to comfortably run GIZA++ under win32 with cygwin I had to recompile it without the flag -DBINARY_SEARCH_FOR_TTABLE (credits: http://code.google.com/p/giza-pp/issues/detail?id=9).

The full list of steps goes here (I suppose that source corpus is stored in corpus.ru and target corpus is stored in corpus.en):

1. produce vcb and dictionary files with "plain2snt.out corpus.ru corpus.en" (credits: http://vee-r.blogspot.com/2006/12/giza-guide.html)
2. produce cooc file with "snt2cooc.out corpus.ru.vcb corpus.en.vcb corpus.ru_corpus.en.snt > ru_en.cooc" (credits: myself, after analyzing the train-factored-phrase-model.perl from Moses package)
3. run GIZA++ with config file:

outputfileprefix play_giza
sourcevocabularyfile corpus.ru.vcb
targetvocabularyfile corpus.en.vcb
c corpus.ru_corpus.en.snt
CoocurrenceFile ru_en.cooc
model1dumpfrequency 1
model4smoothfactor 0.4
nsmooth 4
onlyaldumps 1
nodumps 1
p0 .999
m1 5
m2 0
m3 3
m4 3
o giza

if these options are stored in giza.config then run "GIZA++ giza.config". This produces giza.A3.final file, typical entry of which is:

# Sentence pair (1) source length 4 target length 7 alignment score : 2.25315e-10
there is a book on the table
NULL ({ }) на ({ }) столе ({ 5 6 7 }) лежит ({ 1 2 }) книга ({ 3 4 })

which means the following mapping:

"столе" --> "on the table"
"лежит" --> "there is"
"книга" --> "a book"

Running GIZA++ under win32 and under linux gives same results in terms of word mappings, except that the alignment scores may slightly differ due to possibly different float point precision models.

giza++ under windows

After an 'outrageous' attempt to compile giza++ under Visual Studio C++ Express 2008 (with more than 2000 compile time errors) I switched over to cygwin and installed:

gcc version 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)
make GNU Make 3.81 (built for i686-pc-cygwin)

With

$ make

under giza-pp-v1.0.3\giza-pp I have obtained two executables: GIZA++-v2/GIZA++.exe and mkcls-v2/mkcls.exe which perfectly run under Windows XP Professional Version 2002 SP 2.

RuSSIR'2010

What is RuSSIR? It is the summer school on Information Retrieval. Where is it held? Usually in Russia, this time in Voronezh. Want to apply? Read on.

4th Russian Summer School in Information Retrieval (RuSSIR 2010)
Monday September 13 - Saturday September 18, 2010
Voronezh, Russia
http://romip.ru/russir2010/eng/

FIRST CALL FOR COURSE PROPOSALS

The 4th Russian Summer School in Information Retrieval (RuSSIR 2010) will be held on September 13-18, 2010 in Voronezh, Russia, one of the major cities in south-western Russia. The mission of the school is to teach students about modern problems and methods in Information Retrieval; to stimulate scientific research in the field of Information Retrieval; and to create an opportunity for informal contacts among scientists, students and industry professionals. The Russian Conference for Young Scientists in Information Retrieval will be co-located with the school. RuSSIR 2010 will offer 4 or 5 courses and host approximately 100 participants. The working languages of the school are English (preferable) and Russian. The target audience of RuSSIR is advanced graduate and PhD students, post-doctoral researchers, academic and industrial researchers, and developers.

The RuSSIR 2010 Organizing Committee invites proposals for courses on a wide
range of IR-related topics, including but not limited to:
- IR theory and models
- IR architectures
- Algorithms and data structures for IR
- Text IR
- Multimedia (including music, speech, image, video) IR
- Natural language techniques for IR tasks
- User interfaces for IR
- Web IR (including duplicate detection, hyperlink analysis, query logs)
- Text mining, information and fact extraction
- Mobile applications for IR
- Dynamic media IR (blogs, news, WIKIs)
- Social IR (collaborative filtering, tagging, recommender systems)
- IR evaluation.

Each course should consist of five 90-minute-long sessions (normally in five consecutive days). The course may include both lectures and practical exercises in computer labs.

RuSSIR 2010 organizers will cover travel expenses and accommodations at the school for one lecturer per course, but there is no additional honorarium. The RuSSIR organizers would highly appreciate if, whenever possible, lecturers could find alternative funding to cover travel and accommodation expenses and indicate this possibility in the proposal.

Course proposals for RuSSIR 2010 must be submitted by email to Pavel Braslavski (pb@yandex-team.ru), by February 14, 2010. A course proposal should contain a brief description of the course (up to 200 words), preferred schedule, prerequisites, equipment needs, a short description of teaching/research experience and contact information of the lecturer(s). All proposals will be evaluated by the RuSSIR 2010 program committee according to the school goals, presentation clarity, lecturer’s qualifications and experience. Topics not featured at previous RuSSIRs are preferred. All submitters will be notified by March 1, 2010. Early informal inquiries about the school or the proposal evaluation process are encouraged.

About RuSSIR: The Russian Summer School in Information Retrieval is co-organized by the Russian Information Retrieval Evaluation Seminar (ROMIP) and Voronezh State University. Previous schools took place in Ekaterinburg, Taganrog, and
Petrozavodsk. Previous RuSSIR courses included IR Models (by Djoerd Hiemstra), Modeling Web Searcher Behavior and Interactions (by Eugene Agichtein), Computational Advertising (by James Shanahan), Text Mining, Information and Fact Extraction (by Marie-Francine Moens), Natural Language Processing for Information Access (by Horacio Saggion), Music IR (by Andreas Rauber), and other. Ricardo Baeza-Yates, VP of Research for Europe and Latin America at Yahoo, has confirmed as an invited lecturer for RuSSIR 2010 with the course 'Web data mining'.

About the RuSSIR 2010 location: Voronezh is a major city in southwestern Russia, spanning both sides of the Voronezh River, with population of 850,000. Express trains from Moscow to Voronezh take about 10 hours. There are also regular flights from Moscow, Munich, Prague, Tel-Aviv, and Istanbul. The town was founded in 1586. In the 17th century, Voronezh gradually evolved into a sizeable town, especially after Tsar Peter the Great built a dockyard in Voronezh. Currently, Voronezh is an administrative, economic and cultural center of the Voronezh region. Voronezh surrounding area has many attractions including archeological museum, nature and historical reserve Divnogorie, Kostomarovo cave monastery, Orlov trotter stud farm at Khrenovoe. Voronezh has a large student population: 37 institutions of higher education and 53 colleges educating over 127,000 students today. Voronezh State University was founded in 1918 and is one of the largest universities in Russia, with a total enrollment of 22,000.

Contacts

Use the e-mail address and substitute [at] with @ and [dot] with "." school[at]romip[dot]ru.

Semantic Analysis: theory, applications and use cases

Gave a talk at 6th FRUCT seminar by Nokia Research Center and Helsinki University of Technology and Nokia Siemens Networks:

Semantic Analysis: theory, applications and use cases

View more documents from dmitrykan.

Augmented Reality with Adobe Flash

Having carefully followed the Adobe MAX conference, I have found a ground shaking presentation on the Augmented Reality with Flash.

As the presenters define it, AR is superimposing "graphics over real-world environments in realtime". So I checked out the exciting (and long) tutorial on how to set up the development environment to hit the road. For running a quick demo from the tutorial I printed a black and white pattern (it makes sense to make it smaller, than I made, as it is much easier to handle). Before diving into details I decided to have a clue on how it is going to look like and made a small video which I share with you (please be patient about video quality and mute your player).



It is early to draw any conclusions on the future of AR, but it sounds like a very exciting field of software development in the future. It is where creative people get together and come up with an exciting business card and all sorts of interesting things.

Sting's programmer's mind

First impression of Sting's web-site is his quote of the day:

"I quite like using songs as a modular system where you can mix and match lines from different songs. It's a tradition now and people expect it. Basically, it's all one big song. You could say it was an aspect of postmodernism if you liked but you'd be called pretentious if you said that."

Perl: concise way to map one array onto antother in perl hash

Suppose I have two arrays. Suppose further elements in one of this array logically map onto elements of the other.

my @ar1 = (...);
my @ar2 = (...);

Easy way to map ar1 (keys) onto ar2 in perl is:

my %hash;
@hash{@ar1} = (@ar2) x @ar1;

Important assumption: the order in this two arrays matters. In other words first element of ar1 maps to first element of ar2, ..., n-th element of first array ar1 maps onto n-th element of ar2 and there exactly n elements in both arrays.


Examples

It is OK to have unique keys, obviously for the hash to preserve correct mapping (include use Data::Dumper in your code):

sub unique_mapping
{
 my @ar1 = ('a', 'b', 'c', 'd', 'e');
 my @ar2 = ('1', '2', '3', '4', '5');
 
 print Dumper(\@ar1);
 print Dumper(\@ar2);
 
 my %hash;
 @hash{@ar1} = (@ar2) x @ar1;
 
 
 print Dumper(\%hash);
 
}

Result:

$VAR1 = [
          'a',
          'b',
          'c',
          'd',
          'e'
        ];
$VAR1 = [
          '1',
          '2',
          '3',
          '4',
          '5'
        ];
$VAR1 = {
          'e' => '5',
          'c' => '3',
          'a' => '1',
          'b' => '2',
          'd' => '4'
        };

The mapping is not what you might want to have in the case when keys are not unique:

sub keys_non_unique_mapping
{
 my @ar1 = ('a', 'b', 'b', 'd', 'e');
 my @ar2 = ('1', '2', '3', '4', '5');
 
 print Dumper(\@ar1);
 print Dumper(\@ar2);
 
 my %hash;
 @hash{@ar1} = (@ar2) x @ar1;
 
 
 print Dumper(\%hash);
 
}

Result:

$VAR1 = [
          'a',
          'b',
          'b',
          'd',
          'e'
        ];
$VAR1 = [
          '1',
          '2',
          '3',
          '4',
          '5'
        ];
$VAR1 = {
          'e' => '5',
          'a' => '1',
          'b' => '3',
          'd' => '4'
        };

Logging: helpful perl snippet to start with

I needed to establish simple logging in my small perl app which serves as a plugin for a company "big" product. Here is what I have come up with:

use strict;

my $log_file=file_string_here;

my $LOG_HANDLE = open_log_file_for_writing($log_file);

log_entry($LOG_HANDLE, "Logging started");

log_entry($LOG_HANDLE, "Logging finished");

close_log_file($LOG_HANDLE);


sub open_log_file_for_writing
{
   my $log_file = shift;
   my $LOGGING_HANDLE;

   print "INFO Opening log file...\n";

   unless(open $LOGGING_HANDLE, ">> ", $log_file) {
      return undef;
   }

   my $current_time = localtime;
   print $LOGGING_HANDLE "\n".$current_time."\n";

   return $LOGGING_HANDLE;
}

sub log_entry
{
   my $LOGGING_HANDLE = shift;
   my $log_entry      = shift;

   print $LOGGING_HANDLE $log_entry."\n";
}

sub close_log_file
{
   my $LOGGING_HANDLE = shift;
   print "INFO Closing log file...\n";
   close($LOGGING_HANDLE);
}

upd: the logging handle can be externalized for easing the use of the logging. It comes at cost of global variable, but might still suit moderately sized perl-scripts. Code will change a bit:

my $g_LOGGING_HANDLE = open_log_file_for_writing($log_file);
log_entry("Logging started");

sub log_entry
{
   my $log_entry      = shift;

   print $g_LOGGING_HANDLE $log_entry."\n";
}

Now you can call log("log entry goes here") from where you want quickly, without the need to pass down as well the logging handle, for example deep inside some procedure or function.

upd1: If you want to control whether to log or not, another small modification will do it for you:

my $LOG_ENABLED = 1; # put 1 to enable logging, 0 to disable logging

my $log_file = "plugin_request.log";
...

my $g_LOGGING_HANDLE;
undef $g_LOGGING_HANDLE;

if ($LOG_ENABLED)
{
   $g_LOGGING_HANDLE = open_log_file_for_writing($log_file);
}


...

sub log_entry
{
   my $log_entry      = shift;

   return if (!defined($g_LOGGING_HANDLE));

   print $g_LOGGING_HANDLE $log_entry."\n";
}

sub close_log_file
{
   my $LOGGING_HANDLE = shift;

   return if (!defined($g_LOGGING_HANDLE));

   print "INFO Closing log file...\n";
   close($LOGGING_HANDLE);
}

C++: if file line length exceeds array (buffer) length

Suppose we have implemented the following scenario:

// read input file line by line
// allocate 256 characters for each line

ifstream input_file("some_file.txt");
const int BUF_SIZE=256;
char buf[BUF_SIZE];
string s, strCurString;

if (!input_file.is_open())
{
   cerr << "File some_file.txt coudl not be open!" << endl;
   getch();
   exit(EXIT_FAILURE);
}

while(!input_file.eof()) {
   input_file.getline(buf, BUF_SIZE);
   strCurString = buf;
   s += strCurString;
}

cout << "File contents: " << endl << s << endl;

But what if the current file length exceeds BUF_SIZE? Well, in this case the while loop will never end, becoming an infinite loop. Why? Simply, because in the input file stream object a special bit (failbit) will be set, saying that the last getline() operation has failed (in this case not due to the end of a file, but due to the buffer length exceeding). In this case all subsequent calls to getline() will fail to read anything (can be seen by calling input_file.gcount(), which constantly returns 0 (zero) after the last getline() call that led to setting a failbit).


To overcome this, we can use a trick found here:

// read input file line by line
// allocate 256 characters for each line

ifstream input_file("some_file.txt");
const int BUF_SIZE=256;
char buf[BUF_SIZE];
string s, strCurString;

if (!input_file.is_open())
{
   cerr << "File some_file.txt coudl not be open!" << endl;
   getch();
   exit(EXIT_FAILURE);
}

while(!input_file.eof()) {
   input_file.getline(buf, BUF_SIZE);

   // remember about failbit when amount of
   // characters in the current line is
   // more than BUF_SIZE
   if (input_file.fail() && !input_file.eof())
       // clear up the failbit and 
       // continue reading the input file
      input_file.clear();
   strCurString = buf;
   s += strCurString;
}

cout << "File contents: " << endl << s << endl;

HackDay'09

The HackDay'09 is about to happen in the glorious city on Neva river -- Saint-Petersburg! Together with our friends we have decided to participate with our ideas based on semantic analysis. Let's see where the HackDay brings us. On Friday we are heading there!

Porvoo is a unique old city, but old town is disgustful

Introduction
Surprised with the topic formulation? Read on for the details.

Visiting Porvoo
The unique old city is the old city of Porvoo [1] we have visited today. It's been a pleasure, though it was raining almost all the time. Some photos here [2]. Comments in Russian though (well, as it was once said in a humourous russian show KVN, "learn Russian in order to understand humour" and vice versa). Why Porvoo might have become widely known is thanks to a fact Finland had been declared autonomous Grand Duche of Russia in the local cathedral.

Related work and criticism
I usually try to avoid posting any anti-ads in the blog, but this time it's unavoidable. It is about the second part - old town, which is the restaurant with reported refined taste and special service. Well, it was about 16:00 already, but should it make any difference for the service quality? First we have waited way too much time for placing an order, basically from being *very* hungry to the point of thoughts to leave getting nothing for our pains. When the waitress came down, she missed the soup story completely leading to two of us having a soup (delicious, nothing to say!) instead of one. But the topic adjective starts with pasta. Have you ever eaten the fast food spaghetti? Somewhere in a student dormitory may be or in a camp. So take this spaghetti, add some sea food (which is nice however) and put it into the micro. The taste was something like this. We experienced culmination for an ordered and never arrived dessert. As some random passers-by might have fetched all the cakes we have ordered coupled with black tea, the wairtress wasn't very confused to inform us that only one cake left. Why that happened? Because the restaurant had another side feature: a cafe. Sounds like a multitier technology, right? Which in practice averages to a (restaurant + cafe) / 2 quality level. The place has managed to blur the impression about Porvoo as right after that we have left the city, but thinking about it now, when about 5h passed, I generally feel the journey was pleasant. We decided not to slap our wrathful feelings into their face, but to publish this post.

Discussion
We have thought about why the restaurant quality was *that* not good and came to the conclusion that the target audience of it was tourists. Which means mostly occasional visitors, who will never remember it (as they usually don't come back) even if it wasn't worthwhile. Also, if you do care about your clients, think twice of coupling a restaurant with a cafe in the same cramped premise.

Conclusion
Instead of blurring, use sharpening: go for a home pizza or may be visit some really expensive place to eat and polish the collected impressions of your day in Porvoo.

Bibliography
[1] http://en.wikipedia.org/wiki/Porvoo_Cathedral
[2] http://picasaweb.google.ru/dmitry.kan/Porvoo#

Perl: trust programmatic access to VB Project

It has been already a tradition to cross-post useful information bits, so let me continue.

If you came to the following problem when accessing the the VB Project programmatically from your Perl script:

"Programmatic access to Visual Basic Project is not trusted"

, you will need to allow the access and trust in your target MS Office app:

Office 2003 and Office XP

1. Open the Office 2003 or Office XP application in question. On the Tools menu, click Macro, and then click Security to open the Macro Security dialog box.
2. On the Trusted Sources tab, click to select the Trust access to Visual Basic Project check box to turn on access.
3. Click OK to apply the setting. You may need to restart the application for the code to run properly if you automate from a Component Object Model (COM) add-in or template.

Office 2007

1. Open the 2007 Microsoft Office system application in question. Click the Microsoft Office button, and then click Application Options.
2. Click the Trust Center tab, and then click Trust Center Settings.
3. Click the Macro Settings tab, click to select the Trust access to the VBA project object model check box, and then click OK.
4. Click OK.

Java: encoding and listing files in a directory

Two code snippets tested to be working under win32 at least;

Encoding. When your input text files are in encoding different from your default platform's there is no way to use FileReader in this case. Instead you should go deeper in the class hierarchy and specify a file enconding, which you know in advance. Code (inspired by answers here):

  
 private static String loadFileContents(String filename) 
throws FileNotFoundException, IOException {
  StringBuilder contents = new StringBuilder();
  
  InputStream is = 
new BufferedInputStream(new FileInputStream(filename));
  Reader reader = new InputStreamReader(is, "UTF8");
  
  BufferedReader bufferedReader = 
new BufferedReader(reader);
  
  String line;
  while ( (line = bufferedReader.readLine()) != null ) {
   contents.append(line + "\n");
  }
  
  return contents.toString();
 }

Directory listing. Suppose you want to list files in a directory based on some filename criteria. Easy way to do this is to implement a custom filename filter. Entire code (watched here):

  
/**
 * 
 */
package org.dk.mt;

import java.io.File;
import java.io.FilenameFilter;

/**
 * @author Dmitry_Kan
 *
 * Lists given directory based on filename pattern
 */
public class DirectoryLister {

 private String filenamePattern;
 private String dirname;

 public DirectoryLister(String filepat, String directory) {
  filenamePattern = filepat;
  dirname = directory;
 }

 public String[] getListOfFiles() {
  class CustomFilter implements FilenameFilter {
   public boolean accept(File dir, String s) {
    if (s.contains(filenamePattern) && s.contains("txt") )
     return true;
    return false;
   }
  }
  return new java.io.File(dirname).list(new CustomFilter());
 }

}

tabs in Firefox 3.5.1

Just got amazed: it is so easy to drag-n-drop tabs in Firefox 3.5.1. If you have two browser instances opened and you want to drag-n-drop tab from one of them to another, drag the tab and drop it to the tab area of another Firefox instance.

Post builder script for CGI development in Eclipse

Hi,

In CGI development it may be a lot of help to have some sort of a post builder script. The primary aim of this script should be updating the cgi scenario(s) and modules when you build your perl project under Eclipse.

So I have come up with the following post_builder_script.pl:

#!/usr/bin/perl -w

use strict;

# append "/" to the target path if required
my $target_path = $ARGV[0];
if (not( $target_path =~ m/\/$/)) {
 $target_path .= "/";
}

# copy cgi script: what, from where, to where
my $file = "uni_search.cgi"; 
my $copy_base_path = "../";
my $copy_from = $copy_base_path.$file;
my $copy_to   = $target_path.$file;

copy($copy_from, $copy_to);

# copy Perl modules
my @modules = ("Module1.pm", "Module2.pm", "Module3.pm");
my $modules_path = "some_path/";
$copy_base_path = "./";
my $target_copy_base_path = $target_path.$modules_path;

foreach my $module(@modules) {
 copy($copy_base_path.$module, 
$target_copy_base_path.$module);
}

sub copy
{
 my $copy_from = shift;
 my $copy_to   = shift;
 
 my $result;
 
 # chmod'ing
 $result = system("echo ".[your_sudo_password_here].
" | sudo -S chmod +w ".$copy_to);
 print "Changing write rights: ".$result."\n";
 
 # updates / copies the cgi script to server
 $result = system("sudo cp ".$copy_from." ".$copy_to);
 if ($result == 0) {
  print "Result of copying the ".$copy_from.
" to server's cgi-bin: ".$result."\n\n";
 }
}

It is easy to add a builder which will call the script. In Perl perspective go to Project->Properties->Builders. Select "New..." on the right. Choose "Program" in the popup window. Configure the program by choosing what to launch, where and command line arguments:




There you go. After the builder has been successfully configured, edit your code and do full rebuild (select the root node of your project and Project->Build Project), when you want to deploy your cgi script set to your web server.

Information retrieval: conceptual problem

From my experience with grep for example, I come to a conclusion that the main problem of current search engine
interfaces (not grep ones as grep serves other aims) is that they lack one major feature. It is the feature of visualizing the search space or from other perspective -- a feature of search hints.

Suppose I want to search *something* about HTTP protocol (some very specific detail) and I know very little about HTTP (an artificial example). So for example, I even don't know which category does that particular detail belongs too.
Needless to say, that I don't know the term which I'm looking for. Now I'm stuck. I should have some starting hints, which would lead me to first query terms apart from HTTP. Because HTTP is very wide term in a tree of terms (like the tree root), I will spend hours and hours reading through millions of returned hits.

Instead, I would use a concept graph, visualized, which directly corresponds to the search index of a searching engine.
Now I can easily jump in a concise way over those categories getting deeper and deeper into HTTP topic and discovering
some previously unknown terms, which I can further search for. Thus narrowing down the search space with the parallel learning I would come to a result faster.

Perl is truly weird

Otherwise how else would you treat the following: $#$ar_ref?

It is actually $#($ar_ref) which is the last index of an array referenced via $ar_ref.

Michael's last curtain call

It suddenly came to my mind right after I got to know about Michael's passing away: the world has had or is about to change when *SUCH* people leave it..

But the strangest thing is - when now I see him performing on my screen, I do not *feel* him passed away. It is half sad and half light feeling about what happened.


"This is it. This is the final curtain call." Michael Jackson

Gmail: spam handling as a user experience

It might be a better idea to post below as a feature request to Gmail team directly, but I post it here because of two reasons:

1) I'm rather busy at work to find their blog
2) let's check Google's track_everything_happening_on_the_web property


About spam again. Recently I have been receiving quite a bulk of these pleasant messages (giving the unique opportunity to enlarge something on a recipient's body etc) - which results in about 100+ messages / day filtered out to the spam folder.

Not sure about the majority of e-mail user habits, but I have an attitude to check all the spam messages (don't worry, only the titles; well the body rarely as well) and get rid of them manually, rather than leaving the engine to automatically delete them after N days. I should notice the precision of the spam filtering algorithm Google has in Gmail: it works perfect in my case. But still even the tiniest 0,(0*)1% probability to overlook the valid message in spam folder may give hard times and irreversible processes in my brain, which I do not want to happen.


So my idea follows next. Suppose I have 10 pages of junk piled up in the spam folder. When I open one page and start looking through the e-mail topics, I obviously spend some time (like 2-3 minutes for 100 messages). Once I have done this, I may or may not proceed to the next spam page, but with no regard to it Gmail marks internally the page as "read". When I sign out or after a while (another configurable parameter), Gmail silently deletes all the "read" messages.

For the sake of flexibility there can be the third feature: a button (or a fixed option) , explicitly prohibiting to silently remove messages even though they have been read.

Словарный запас: результаты теста

Итак, ваш запас: Тоже очень хороший результат. Ваш словарный запас значительно выше среднего

Пройти тест

Perl's flexibility

I was suspecting Perl to be rather flexible and implementing the principle "type as you think and get it working" (which does not apply always as Perl is 'weird' by definition), but could not imagine it goes that far:

sub getHeaderName
{
 return (split " - ", shift)[1]; # let's see if it works. Upd: it works!
}

, where input string is of type: "process_type - process_name" and I need process_name value from it.

spam: get phd diploma!

"GET YOUR DIPLOMA TODAY!If you are looking for a fast and cheap way to get a diploma, this is the best way out for you. Choose the desired field and degree and call us right now: For US: some_num Outside US: some_another_num
"Just leave your NAME & PHONE NO. (with CountryCode)" in the voicemail.

Our staff will get back to you in next few days!"


I wonder, if this fake diploma will help me, e.g. when applying to Google jobs and during work.

Google's search suggestions

Is it "null" which Google predicted to be next in the sequence, or is it that Google predicted turned out to be null (system issue)?

Syntax and semantics

It may be the implications of the specifics of my phd topic, which made my eye keener, but I tend to come across the above two terms every now and then. People refer to syntax as to rules and to semantics as to in a way an invariant of those rules and something that one basically wishes to express using the rules.

Today I've bumped into an article with the following title: "Syntax is from Mars while Semantics from Venus! Insights from Spectral Analysis of Distributional Similarity Networks.":

"We observe that while the syntactic network has a hierarchical structure with strong communities and their mixtures, the semantic network has several tightly knit communities along with a large core without any such well-defined community structure."

Another example brought by one functional specification which has found me at work: "The [some company's] command language provides syntax and semantics to perform
service-specific and service-independent management operations."

It is important to notice, that in general one can bind any semantics to a particular syntactic structure, i.e. what exactly happens after applying this particular structure. And at this point we come to another level: pragmatics. What will happen refers to pragmatics behind an action rather than semantics. The semantics is just a formal intermediate representation of used syntactic structures. I will try to gather illustrations in English later.

Приготовление еды и программирование

Приготовление еды - это как программирование или компилирование проекта с исходников.
При компилировании с исходников иногда тоже как и при написании кода надо поисхитряться, может, даже поизменять код. Поэтому тоже процесс творческий. И ещё: важное свойство компилирования проект.. приготовления любимого блюда - это то, что это можно проделать в любой стране, даже при условии, что в этой стране такое не готовят.

Переведу, пожалуй, для Гугл:

Cooking is like programming or compiling a project from its source code. When compiling from a source code at times one should be skilful, sometimes change the code. Therefore it is also a creative process. One important property of compiling a proj.. cooking you favourite dish is that one can do it in any country, even given a fact that this dish is not cooked in this particular country.

Кредо исследователя или как прожить в науке

Сегодня был на очень интересной лекции по машинному переводу, где
в неофициальной части докладчик (Matti Kääriäinen) рассмотрел проблемы академической
атмосферы. Главной проблемой он считает то, что основная масса
исследователей ориентируется на публикацию статей с целью публикации
статей. Он также привёл этапы аккуратного на его взгляд распорядка:

1) поставь научную цель и представь, что она решена - что тогда? Как
изменится мир? Если дальше, чем решение задачи, продумать не можешь,
выбери другую задачу;
2) keep the pace: делай даже маленькое сегодня (если нет веских причин
отложить на завтра);
3) публикуй свои достижения в любом виде: статьи, исходный код и т. д.

Также нужно убрать ненужные риски с плеч исследователя. Если,
например, одна группа исследует геном и полагает, что ген интеллекта
находится в хромосоме 7, а вторая - в хромосоме 9, то полугодичная
работа обеих групп с результатом "ген в хромосоме 9" не должен
означать конец карьеры участников первой группы. Эта ответственность
должна ложиться на профессора как главу исследовательской группы.

А что на практике: один исследователь получил контракт на 5 лет с
университетом. Полгода прошли активно, а после этого он стал постоянно
думать: а что через 4,5 года? И так очень многие: нужно жить, и люди
думают об этом очень много, стараясь выполнить все формальные
требования для продления контракта, забывая о главной цели.

Есть над чем подумать.

IR workshop @ AM-CP

has hosted another talk of mine. This time I have brought in more details on natural language functional theory and computer semantics. The audience has been rather wide as well: 10-15 people. After diving into computer semantics we've turned to Machine Translation topic, briefly covering statistical and classical approaches. It has been also important to see how the students follow the talk as well as keep them laughing from time to time to break the ice wall.

IR workshop @ AM-CP

Computer Semantics And Machine Translation

View more presentations from dmitrykan.

CPS'09

For those of you, who silently keep track of what's going on, my following message: I have made a conference talk last week at CPS'09 (Control Processes and Stability). The talk is in Russian. In brief: the article proposes an automatic method for creating a semantic translational dictionary ru->en. The method is based on statistical approaches and implemented with the use of GIZA++. Comparison of an experimental machine translation system based on the dictionary with the SMT system based on Moses is provided.

Automatic Build Of Semantic Translational Dictionary

View more presentations from dmitrykan.

autoflush feature in ActivePerl

I tend to post here only those receipts that have practically worked for me.

There have been another feature I couldn't make working under win32 for ActivePerl. It is the autoflush, i.e. auto flushing the buffer content to a physical file.

Spied here. The solution is to make the filehandle "hot":

select((select(FILEHANDLE), $| = 1)[0]); # make the handle hot

Analogy of a face search in a human brain

While I was preparing my breakfast a thought of a face expression flashed accross my mind. The small picture was so bright,
that I immediately started to search the owner in my memory. This is the exact verb, that I should use
here: to search. The search took just about two seconds (lucky me) and I started to analyze
how did it happen: what algorithm did I potentially use, what was the complexity. As I met the owner of the
face about a week ago I believe the search took place in cache and looked a linear search on one hand.
On the other hand I distinctly realized while this was happenning that I clearly was skipping some
faces without further detailed analysis (i.e. calculating some metrics for determining relevance level).
One important observation is that I didn't draw a clear line in a search space based on gender: rather it was a quick search (with no reference
to the existing algorithm, but an allegory) with very fast process of prunning the search space.


It might turn out, that existing search algorithms are not that far from what happens inside a human brain.


mood: thoughtful :)

podcasting on Google

I've decided to make the first podcast with a certain purpose: there is a podcast contest by budam on rpod.ru. The general idea is to create a podcast, speaking entirely in English which must not be your native language.

The podcast is on how I have had an interview with Google in Trondheim (pity, pity, recently closed office) and - Google be safe - does not disclose a single task from the NDA signed tasks list I got that long day. It actually was a long day - with 5 interviews and one lunch. I bet I have drunk a 5 litres bottle of water - when you are speaking up for 45 minutes per interview your throat gets easily parched.

Microsoft fights itself

No, no, no. It is not any kind of political post related to the company's current state. It is not a holy war type of post either. I have just got the below interesting message from which I concluded the topic.



I'll translate the key phrases as well:

Prevention of data execution - Microsoft Windows

For defending the computer this program was shut down by the system

Name: Windows®installer
Publisher: Microsoft Corporation

Локальная гравитация

Представьте, что вы отправились покупать вашу любимую пиццу в местной пиццерии. Предположим, вы не просто пошли туда, а поехали на велосипеде (да, захватите ещё рюкзак, ниже я объясню, зачем). Когда пицца, горячая и ароматная, запакованная в картонную коробку, у вас в руках, вы раздумываете, а как бы добраться теперь до дома с комфортом.

Если пиццу держать на руле, сохраняя её горизонтальное положение, будет сложно маневрировать. Особенно, если дорога бугристая ото льда. Можно пиццу привязать к багажнику, но это будет вызывать неудобство: нужно всякий раз оборачиваться и смотреть, на мести ли пицца.

Тогда можно положить пиццу в рюкзак. Стоп! Как же можно класть пиццу в рюкзак, она ведь соберёт все свои компоненты внизу рюкзака. Можно! Можно положить пиццу в рюкзак с с локальной гравитацией, вектор действия которой вы можете задать самостоятельно. Таким образом, гравитация для пиццы в рюкзаке будет направлена соосно вашему движению на велосипеде, но в противоположную сторону.

Осталось подумать о том, как достичь следующего:

1) локальная гравитация должна быть именно локальной, то есть не действовать, к примеру, на вашу шапку на голове;

2) локальная гравитация не должна конкурировать с естественной гравитацией Земли, в идеальном случае они должны существовать параллельно друг к другу, без каких бы то ни было интерференций;

3) можно произвольно менять вектор локальной гравитации.

Приятного аппетита! :)

Mr. Lecturer

Last week I have given lectures on Machine Translation (in Russian) in my home Saint-Petersburg State University.

Introduction To Machine Translation

View more presentations from guestae31e.

Introduction To Machine Translation (continuation)

View more presentations from guestae31e.

I let my students know beforehand that the course is experimental and they happen to be the pioneers whom the course is going to be tested on.

After the 3,5 h lecture I have asked them, how do they feel about the experiment.

The answer was: The course was interesting. In case it would have been uninteresting, they would sleep. I believe it is the best compliment, especially taking into account the fact, that a human concentrates only first 40 (15?) minutes.

Unicode in Perl

Sometimes it feels that perl's power in string manipulations comes at a cost of its synthax being awkward.

When you open a file for reading without caring in what encoding is its contents, you do:

open FILE, "<".$filename or die $!;

But if you do care of an encoding you should open the file using the following instruction:

open ENC_FILE, "<:encoding(cp1251)", $enc_filename or die $!;

Now the key point is in comma following the encoding instruction. If you put there "." instead (which I believe does the concatenation of stream direction sign "<" and the filename), the file fails to open.

Another important addition is: if you know in advance in which encoding the file contents is represented, specify it using the above encoding instruction. Doing this you get all the string data to be in internal perl's representation which is by default utf8.

I feel like on top of the world (c)

.. when I manage to make an unhadled exception in compiler / interpreter. This time it has happened with Perl Command Line Interpreter:

Perl: file or directory

To check this, the prescription says:

if (-d $file)
{
   print $file." is a directory\n";
} else {
   print $file." is a file\n";
}

When this is used in pair with IO:Dir, which helps you to enumerate contents of a given directory, one non-obvious step should not be forgotten:

tie %dir, 'IO::Dir', $dir;
foreach my $entry(keys %dir) {
   next if ($entry eq '.' or $entry eq '..');
   # important part is here: concatenation with the full path
   if (-d $dir."/".$entry)
   {
      print $entry." is a directory\n";
   }
}

Simple Perl modules

Making it already a rool to post technical details for which I have spent more than 20 minutes, I decided to post as well this.

keywords: How to write perl modules
PageRank: http://www.netalive.org/tinkering/serious-perl/

Answer: it's simple!

Create file StringManip.pm in Lib/ directory where you like with the following contents:

package Lib::StringManip;

use strict;

use base 'Exporter';
our @EXPORT = ('trim');

# Perl trim function to remove whitespace from the start and end of the string
sub trim
{
 my $string = shift;
 $string =~ s/^\s+//;
 $string =~ s/\s+$//;
 return $string;
}

1;

Add the full path to Lib/StringManip.pm to PERL5LIB environment variable. In my case (win32) it is: PERL5LIB=%PERL5LIB%;D:\Programming\Perl

i.e. inside D:\Programming\Perl I have Lib/StringManip.pm. In Linux/Unix: export PERL5LIB=some_path/Lib/StringManip.pm.

Usage snippet:

#!perl -w

use strict;
use Lib::StringManip;

print trim("  trim me!  ");

Natural Langauge Processing and preparation of a human brain

Having read a number of articles dealing with natural language processing (NLP), cognition and linguistics, like Beyond Zipf's law: Modeling the structure of human language, I come to a conclusion, that NLP in essence is one of the most accurate and non-intruding ways to understand how human brain works.

Compare NLP, for example to neuropsychology.

Moscow journey video via Animoto

That is very easy to make a video with Animoto and brings some freshness into your world.

How do I load an org.w3c.dom.Document from XML in a string?

The entire header of this message and it's contents were copied from http://stackoverflow.com/questions/33262/how-do-i-load-an-orgw3cdomdocument-from-xml-in-a-string with one aim: to give higher rank for the page in Google PageRank.

Good reason for this is that good things should be replicated.




I have a complete XML document in a string and would like a Document object. Google turns up all sorts of garbage. What is the simplest solution? (In Java 1.5)

Solution Thanks to Matt McMinn, I have settled on this implementation. It has the right level of input flexibility and exception granularity for me. (It's good to know if the error came from malformed XML - SAXException - or just bad IO - IOException.)

public static org.w3c.dom.Document loadXMLFrom(String xml)
    throws org.xml.sax.SAXException, java.io.IOException {
    return loadXMLFrom(new java.io.ByteArrayInputStream(xml.getBytes()));
}

public static org.w3c.dom.Document loadXMLFrom(java.io.InputStream is) 
    throws org.xml.sax.SAXException, java.io.IOException {
    javax.xml.parsers.DocumentBuilderFactory factory =
        javax.xml.parsers.DocumentBuilderFactory.newInstance();
    factory.setNamespaceAware(true);
    javax.xml.parsers.DocumentBuilder builder = null;
    try {
        builder = factory.newDocumentBuilder();
    }
    catch (javax.xml.parsers.ParserConfigurationException ex) {
    }  
    org.w3c.dom.Document doc = builder.parse(is);
    is.close();
    return doc;
}

GUI vs Command Line

For those in software development and aiming at more, my following observation:

when one launches a task in GUI and it gets by some means frozen, there is no way to painlessly stop that particular task but kill the entire GUI.

when one launches a task in the Command Line and you suspect it got frozen, you simple stop the task by CTRL+X or CTRL+Z or CTRL+C (any key combination that does it).

The point is: I would aim at multithreading (or even creating separate processes) of consuming tasks so that I could easily undo any task-related actions. I would be really happy to have a functionality where I can press CTRL+Z to undo any action be it new thread/process for a task or typing a letter in a sentence.

Unusual Billie Jean video

It has been a fun and honour to be a fan of Michael in my youth. This video looks very unusual and at the same time unique vs what you can find for the video search query "Billie Jean":

More Videos

credits: http://www.michaeljackson.com