Conferences in 2019

As Christmas and New Year holidays are coming up, I wanted to reflect on two conferences in two different countries -- Estonia and Ukraine, that I had a pleasure to participate and / or organize this year.

AINL

AINL Conference (https://ainlconf.ru/2019/program), held in Tartu in November, focused a lot on applying deep learning to NLProc, with two tutorials by Dmitry Ustalov (Yandex) on Crowdsourcing on Language Resources and Evaluation and by Andrey Kutuzov (University of Oslo) on Diachronic Word Embeddings for Semantic Shifts Modelling. Andrey Kutuzov's tutorial was practical and involved some Python coding, resulting in a pull request: https://github.com/wadimiusz/diachrony_for_russian/pull/5 that I submitted for the task of comparing semantic shifts in meaning between Soviet and Post Soviet eras. This code uses Jaccard similarity as a local method for detecting shifts in meaning. There are also global methods, like Procrustes alignment, the only downside of which is it is slower, than Jaccard. You can read more detail on the task in Andrey's AINL slides.

Credit: Dmitry Kan

In terms of submitted papers -- the review process was double-blind and involved at least 3 reviewers per paper. The result was 30% acceptance rate and 12 out of 40 papers that did make it, focused on data acquisition and annotation, human-computer interaction, statistical NLProc (including paper by Ansis Bērziņš on usage of speech recognition for determining language similarity -- video) and neural language models (one of the works for morpheme segmentation using Bi-LSTM model cited the work of Mathias Creutz with whom we worked at AlphaSense 2010-2016).

Last day of the conference focused on the industrial applications of AI in NLProc. By invitation of Lidia Pivovarova (University of Helsinki) I presented on the search engine and NLProc work we've done at AlphaSense, including smart synonyms, sentiment analysis, named entity recognition and salience resolution, theme modelling and high-precision search.

One of the challenges for the industrial presentation was that it had to last for 1,5 hours. If you consider your audience ability to focus only for 40 minutes, you have got to do something else than 65 slides. I decided to make about 30 slides and then handle the rest of my talk with Q&A. The outcome has been very surprising to myself, because the audience did want to learn details of AlphaSense product, making the Q&A last for 50 minutes. Quite a few questions I managed to answer with the product itself -- this sparks genuine interest in understanding the UI of an AI product powering the financial industry. I hope this was beneficial for the audience to dive into the workflows of financial knowledge workers and how NLProc can help solve their daily routine tasks better.

Customer Development Marathon

Customer development is the topic that interests me from the point of the product development. Just recently I've learnt about jobs-to-be-done approach to mining for real jobs that your customers hire your product for. One example with which Clayton Christensen of Harvard Business School motivates this approach is the job that male consumers of milkshakes had on the their way to work every day: stay engaged in life during monotonous driving and stay full until 10 a.m.

The conference (or marathon as we called it) on customer development attracted 70 participants at iHUB co-working center in Kyiv, Ukraine. Speakers from various established companies -- YouScan, MacPaw, PromoRepublic, Competera, AlphaSense, Kyivstar, Terrasoft, PMLab, Portmone.com, Weblium, VARUS, SendPulse, EVO.company -- presented 5 min talks about specific cases on engaging with their customers to grow conversion, retention and happiness with their products. Following the presentations, the discussion panels dug deeper into how to implement a customer-centric business. 

Credit: Maria Kudinova

We've organized the marathon in 3 panels: 

1. Idea. Analysis. Validation 
2. Creation. Delivery. Launch and
3. Sales. Feedback. Innovation. 

Each of these panels focused on a particular stage of product development from idea to post-sale feedback and innovation loop. The audience learnt about how to conduct an efficient user interview, what tools help reach out to new or existing clients, how not to push your product into consulting or outsource, how to establish an internal company-wide communication to stay on the same page when shaping the product, marketing and sales around customer needs.

Both events were full of networking, meeting new and familiar faces in the industry and academia and learning a lot. For anything you aspire to build next year, focusing on real value and ease of use of your NLP / AI / search products, and thinking what job your users hire your products for will help you serve them better.

Automatic writing with Deep Learning: Preface

This article was also reblogged at: https://dzone.com/articles/automatic-writing-with-deep-learning-preface


Quite many machine and deep learning problems are directed at building a mapping function of roughly the following form:

Input X ---> Output Y,

where:

X is some sort of an object: an email text, an image, a document; 

Y is either a single class label from a finite set of labels, like spam / no spam, detected object or a cluster name for this document or some number, like salary in the next month or stock price.

While such tasks can be daunting to solve (like sentiment analysis or predicting stock prices in realtime) they require rather clear steps to achieve good levels of mapping accuracy. Again, I'm not discussing situations with lack of training data to cover the modelled phenomenon or poor feature selection.

In contrast, somewhat less straightforward areas of AI are the tasks that present you with a challenge of predicting as fuzzy structures as words, sentences or complete texts. What are the examples? Machine translation for one, natural language generation for another. One may argue, that transcribing audio to text is also such type of mapping, but I'd argue it is not. Audio is a "wave" and the speech detection is an okay solved task (with state of the art above 90% of accuracy), however such an algorithm does not capture the meaning of the produced text,  except for where it is necessary to do the disambiguation of what was said. Again, I have to make it clear, that audio->text problem is not at all easy with its own intricacies, like handling speaker self corrections, noise and so on.

Lately, the task of writing texts with a machine (e.g. here) caught my eye on twitter. Previously, papers from Google on writing poetry or other text producing software were giving me creepy feelings. I somehow undermined the role of such algorithms in the space of natural language processing and language understanding and saw only diminishing value of such systems to users. Again, any challenging tasks might be solved and even bring value to solving other challenging tasks. But who would use an automatic poetry writing system? Why would somebody, I thought, use these systems -- just for fun? My practical mind battled against such "fun" algorithms. Again, making an AI/NLProc system capable of producing anything sensible is hard. Take the task of sentiment analysis, where it is quite unclear what the agreement between experts is, not to mention non-experts.

I think this post has poured enough of text onto the heads of my readers. I will use this post as a self-motivating mechanism to continue the research with systems producing text. My target is to complete the neural network training on the text from my Master thesis and show you some examples for your judgement of the usefulness of such systems.

Weka template project for sentiment classification of an English text

Internet is buzzing about machine learning. Many folks use it for fun and profit. 

credits: Customers Who Bought This Item Also Bought PaulsHealthBlog.com, 11.04.2014

But! When a non-expert gets around started with these topics in practice, it becomes increasingly difficult to just get going. There are of course quick solutions, like scikit-learn library for Python. If you are a Java developer, there are a few options as well: Weka, Apache Mahout. Both of these are generic enough to be applied to different machine learning problems, including text classification. More tailored libraries and packages for text oriented machine learning in Java are Mallet and Stanford's set of libraries. There are as well some less known machine learning toolkits, like Java-ML.

This post will focus on Weka and will give you a very simple and working template project for classifying sentiment in the English text. Specifically, we will create three way sentiment classifier using Multinomial Naive Bayes algorithm.

In my view, there are three main practical problems, that a programmer seeks to find solutions for using a machine learning library: setting up a classifier algorithm, adding training instances (effectively, training the classifier) and I/O (storing and retrieving a model). Beyond this and of high importance is measuring the quality of the trained model that we will take a look at as well.

Setting up a classifier

As mentioned above, we will use the Multinomial Naive Bayes algorithm. To get going, let's set it up for the three way sentiment classification:

public class ThreeWayMNBTrainer {
    private NaiveBayesMultinomialText classifier;
    private String modelFile;
    private Instances dataRaw;

    public ThreeWayMNBTrainer(String outputModel) {
        // create the classifier
        classifier = new NaiveBayesMultinomialText();
        // filename for outputting the trained model
        modelFile = outputModel;

        // listing class labels
        ArrayList<attribute> atts = new ArrayList<attribute>(2);
        ArrayList<string> classVal = new ArrayList<string>();
        classVal.add(SentimentClass.ThreeWayClazz.NEGATIVE.name());
        classVal.add(SentimentClass.ThreeWayClazz.POSITIVE.name());
        atts.add(new Attribute("content",(ArrayList<string>)null));
        atts.add(new Attribute("@@class@@",classVal));
        // create the instances data structure
        dataRaw = new Instances("TrainingInstances",atts,10);
    }

}

What goes in the above code is:

• Create the classifier
• List the target labels: NEGATIVE and POSITIVE
• Create the instances data structure

In a similar fashion, but with more classes (target labels) we'd set up a five way classifier, using the same algorithm under the hood:

public class FiveWayMNBTrainer {
    private NaiveBayesMultinomialText classifier;
    private String modelFile;
    private Instances dataRaw;

    public FiveWayMNBTrainer(String outputModel) {
        classifier = new NaiveBayesMultinomialText();
        classifier.setLowercaseTokens(true);
        classifier.setUseWordFrequencies(true);

        modelFile = outputModel;

        ArrayList<Attribute> atts = new ArrayList<Attribute>(2);
        ArrayList<String> classVal = new ArrayList<String>();
        classVal.add(SentimentClass.FiveWayClazz.NEGATIVE.name());
        classVal.add(SentimentClass.FiveWayClazz.SOMEWHAT_NEGATIVE.name());
        classVal.add(SentimentClass.FiveWayClazz.NEUTRAL.name());
        classVal.add(SentimentClass.FiveWayClazz.SOMEWHAT_POSITIVE.name());
        classVal.add(SentimentClass.FiveWayClazz.POSITIVE.name());
        atts.add(new Attribute("content",(ArrayList<String>)null));
        atts.add(new Attribute("@@class@@",classVal));

        dataRaw = new Instances("TrainingInstances",atts,10);
    }
}

Adding training instances (training a classifier)

Training the classifier is the process of showing examples to the algorithm. An example usually consists of a set of pairs (x,y), where x is a feature vector and y is a label for this vector. In the context of sentiment analysis specifically, a feature vector can be words (unigrams) in a sentence and a label is sentiment: NEGATIVE, NEUTRAL or POSITIVE in the case of three way sentiment classification. The algorithm is expected to learn from the example set and generalize to predict labels y' for the previously unseen vectors x'.

Engineering the features is both the mix of art and mechanical work, as I've once mentioned. And also finding good classifier options can be a task for statistical analysis with visualization.

Let's implement the method for adding the training instances for three way classification:

public void addTrainingInstance(SentimentClass.ThreeWayClazz threeWayClazz, String[] words) {
        double[] instanceValue = new double[dataRaw.numAttributes()];
        instanceValue[0] = dataRaw.attribute(0).addStringValue(Join.join(" ", words));
        instanceValue[1] = threeWayClazz.ordinal();
        dataRaw.add(new DenseInstance(1.0, instanceValue));
        dataRaw.setClassIndex(1);
    }

So basically we put input unigrams (words) as a String x value and integer of label as y value, thus forming a training instance for the algorithm. Next the algorithm will internally tokenize the input string sequence and update the necessary probabilities.

For five way classification the above method looks almost the same, except the first parameter is of type SentimentClass.FiveWayClazz.

Training the model after we have finished adding the training examples is quite simple:

public void trainModel() throws Exception {
        classifier.buildClassifier(dataRaw);
    }

That's it!

I/O (storing and retrieving the trained model)

It is ok to train a model and classify right a way. But, that does not work, if you want to develop your model and ship that to production. In production mode your trained classifier will do its main work: classify new instances. So your model must be pre-trained and exist on disk. Storing and loading a trained model with Weka is extremely easy. This is thanks to the fact the classifiers extend abstract class AbstractClassifier, which in turn implements Serializable interface among others.

Storing the trained model is as easy as:

public void saveModel() throws Exception {
        weka.core.SerializationHelper.write(modelFile, classifier);
    }

And loading the model is easy too:

public void loadModel(String _modelFile) throws Exception {
        NaiveBayesMultinomialText classifier = (NaiveBayesMultinomialText) weka.core.SerializationHelper.read(_modelFile);
        this.classifier = classifier;
    }

After we have loaded the model, let's classify some texts. The method for the three way classification is:

public SentimentClass.ThreeWayClazz classify(String sentence) throws Exception {
        double[] instanceValue = new double[dataRaw.numAttributes()];
        instanceValue[0] = dataRaw.attribute(0).addStringValue(sentence);

        Instance toClassify = new DenseInstance(1.0, instanceValue);
        dataRaw.setClassIndex(1);
        toClassify.setDataset(dataRaw);

        double prediction = this.classifier.classifyInstance(toClassify);

        double distribution[] = this.classifier.distributionForInstance(toClassify);
        if (distribution[0] != distribution[1])
            return SentimentClass.ThreeWayClazz.values()[(int)prediction];
        else
            return SentimentClass.ThreeWayClazz.NEUTRAL;
    }

Notice the line #12. Remember, that we have defined the target classes for the three way classifier as {NEGATIVE, POSITIVE}. So in principle our classifier should be capable to do the binary classification. But! In the event when the probability distribution between the classes is exactly equal, we can safely assume it is NEUTRAL class. So we get the three way classifier. The following test case should ideally pass:

@org.junit.Test
    public void testArbitraryTextPositive() throws Exception {
        threeWayMnbTrainer.loadModel(modelFile);
        Assert.assertEquals(SentimentClass.ThreeWayClazz.POSITIVE, threeWayMnbTrainer.classify("I like this weather"));
    }

Neat!

To wrap things up, here is the "runner" class that builds the three-way classifier, saves the model and tests it for quality over the training data:

public class ThreeWayMNBTrainerRunner {
    public static void main(String[] args) throws Exception {
        KaggleCSVReader kaggleCSVReader = new KaggleCSVReader();
        kaggleCSVReader.readKaggleCSV("kaggle/train.tsv");
        KaggleCSVReader.CSVInstanceThreeWay csvInstanceThreeWay;

        String outputModel = "models/three-way-sentiment-mnb.model";

        ThreeWayMNBTrainer threeWayMNBTrainer = new ThreeWayMNBTrainer(outputModel);

        System.out.println("Adding training instances");
        int addedNum = 0;
        while ((csvInstanceThreeWay = kaggleCSVReader.next()) != null) {
            if (csvInstanceThreeWay.isValidInstance) {
                threeWayMNBTrainer.addTrainingInstance(csvInstanceThreeWay.sentiment, csvInstanceThreeWay.phrase.split("\\s+"));
                addedNum++;
            }
        }

        kaggleCSVReader.close();

        System.out.println("Added " + addedNum + " instances");

        System.out.println("Training and saving Model");
        threeWayMNBTrainer.trainModel();
        threeWayMNBTrainer.saveModel();

        System.out.println("Testing model");
        threeWayMNBTrainer.testModel();
    }
}

The quality of the model

Testing the trained model is fairly easy with Weka as well. Knowing the quality of your model is important because you want to make sure that there is no under- or overfitting happening. Underfitting means you haven't found an optimum of features describing your fenomena to fully utilize your training data, thus the model is long-sighted or too simple. Overfitting means you deal with over-learning your training data and over-adjusting for it, i.e. the model does not generalize for real world instances and becomes too short-sighted or too complex.

There are different ways to test the model, one is use part of you training data as test data (for example one third) and perform N fold cross-validation. I.e. on each iteration take a new piece of training data for test data and compute sensible metrics, like precision / recall / accuracy etc. In the end of the cross-validation take average over computed values. This will be your "amortized" quality.

We can also take a peek look at the quality by just counting the number of correctly classified instances from the training data:

    public void testModel() throws Exception {
        Evaluation eTest = new Evaluation(dataRaw);
        eTest.evaluateModel(classifier, dataRaw);
        String strSummary = eTest.toSummaryString();
        System.out.println(strSummary);
    }

The method outputs the following statistics:

Correctly Classified Instances       28625               83.3455 %
Incorrectly Classified Instances      5720               16.6545 %
Kappa statistic                          0.4643
Mean absolute error                      0.2354
Root mean squared error                  0.3555
Relative absolute error                 71.991  %
Root relative squared error             87.9228 %
Coverage of cases (0.95 level)          97.7697 %
Mean rel. region size (0.95 level)      83.3426 %
Total Number of Instances            34345     

The code can be found on my github. It utilizes the data posted on kaggle. So if you want to use the code as is (and perhaps even make a submission) you need to accept the terms of the kaggle competition and download the training set. I leave the exercise of implementing the full code for five-way classification and code for classifying kaggle's test set to the reader.

Training on NLProc and Machine Learning

Just did a training on NLProc (imho, better abbreviation for natural language processing than NLP) and Machine Learning for OK.ru (Russian Facebook owned by Mail.Ru Group) in person in Saint-Petersburg, Russia.

OK.ru has a nice office not far from Petrogradskaya subway station in Saint-Petersburg (the central office is in Moscow).

If you feel like your project is somewhat stuck and needs a fresh look or you need to widen you knowledge in NLProc and / or machine learning, feel free to contact me on g+. At the moment folks at SemanticAnalyzer can do this in Europe / western part of Russia. At SemanticAnalyzer we also offer a full package of services for natural language processing development in case there isn't expertise in your house. This includes project scoping, breaking down by technical tasks, time estimation, development, testing / evaluation and delivery.

SentiScan: блог-пост о технологии распознавания сентимента (тональной окрашенности сообщений)

На днях наш партнёр YouScan опубликовал интервью с моим участием о нашей совместной технологии распознавания сентимента или, говоря иначе, эмоциональной окрашенности в текстах. Эта задача известна теоретической и практической компьютерной лингвистике довольно давно и создано множество подходов. Традиционно выделяют две группы методов: основанные на машинном обучении и статистике и подходы, основанные на правилах. Есть также и методы соединения обоих подходов, а также новомодный алгоритм на нейронных сетях.

В технологии SentiScan мы также сочетаем оба подхода и добавляем нашу собственную изюминку: объектную ориентированность. Это не ООП (объектно-ориентированное программирование), а поиск именнованных сущностей и определение сентимента по отношению к ним. Список сущностей мы получаем из поисковых запросов пользователей, описывающих некий бренд, название продукта, имя человека или других явлений. Задача системы найти данные объекты в тексте, выделить сентиментный контекст и распознать сам сентимент.

Мы использовали методы машинного обучения для поиска полярных единиц, т.е. таких, которые имеют однозначное тональное значение -- позитивное либо негативное. Примеры таких однозначно окрашенных единиц:

позитив

благородный

доход

изысканный

лояльный

необыкновенный

оперативный

передовой

негатив

абсурд

винить

вымогательство

грабеж

идиотский

нытье

отвратительный

Как можно заметить, в словарях присутствуют представители любых частей речи: не только имён прилагательных, но имён существительных, глаголов. Есть и наречия (отвратительно). 

После того, как входной текст был разделён на отдельные предложения, алгоритм производит синтаксический анализ с целью определения объектов в тексте, а также их взаимного влияния. Правила синтаксического анализа подобраны специально для задачи распознавания сентимента и не подойдут, например, для некоторой общей задачи синтаксического анализа либо его применения (машинный перевод или spell-cheker).

В процессе синтаксического анализа производится наращивание информации и статистики о сентиментном потоке (его силе и полярном окрасе -- позитив либо негатив) и его направленности на целевой объект. Накопив информацию об отдельных предложениях, алгоритм переходит на уровень текста, на котором вычисляется финальная информация. В итоге алгоритм выносит вердикт по всему тексту (который может также состоять и из одного предложения или даже слова): позитив либо негатив. Текст может быть также помечен и нейтральным флагом, в двух случаях:

1. в тексте не было ни одной тонально окрашенной единицы либо синтаксического противопоставления (объект А, но объект Б)

2. в тексте был смешанный сентимент и неясно, что хотел сказать своим высказыванием автор. В этом случае алгоритм может опционально поставить метку "смешанный сентимент", то есть позитив+негатив.

У описанного здесь вкратце алгоритма есть также и отдельная функциональность определения объективности ("беспристрастноси") и субъективности текста либо сообщения. Если автор текста не использует эмоционально окрашенных выражений, то его текст можно в целом считать объективным или беспристрастным. И субъективным, если использует. Распознавание субъективности автора может быть полезна тем брендам, которые ищут "подлинные" обзоры их продукции, т.е. опирающиеся на факты.

Попробовать эту систему в действии можно, написав нам письмо на info@semanticanalyzer.info. Техническая документация и характеристики быстродействия описаны в документации.