Conferences in 2019

As Christmas and New Year holidays are coming up, I wanted to reflect on two conferences in two different countries -- Estonia and Ukraine, that I had a pleasure to participate and / or organize this year.

AINL

AINL Conference (https://ainlconf.ru/2019/program), held in Tartu in November, focused a lot on applying deep learning to NLProc, with two tutorials by Dmitry Ustalov (Yandex) on Crowdsourcing on Language Resources and Evaluation and by Andrey Kutuzov (University of Oslo) on Diachronic Word Embeddings for Semantic Shifts Modelling. Andrey Kutuzov's tutorial was practical and involved some Python coding, resulting in a pull request: https://github.com/wadimiusz/diachrony_for_russian/pull/5 that I submitted for the task of comparing semantic shifts in meaning between Soviet and Post Soviet eras. This code uses Jaccard similarity as a local method for detecting shifts in meaning. There are also global methods, like Procrustes alignment, the only downside of which is it is slower, than Jaccard. You can read more detail on the task in Andrey's AINL slides.

Credit: Dmitry Kan

In terms of submitted papers -- the review process was double-blind and involved at least 3 reviewers per paper. The result was 30% acceptance rate and 12 out of 40 papers that did make it, focused on data acquisition and annotation, human-computer interaction, statistical NLProc (including paper by Ansis Bērziņš on usage of speech recognition for determining language similarity -- video) and neural language models (one of the works for morpheme segmentation using Bi-LSTM model cited the work of Mathias Creutz with whom we worked at AlphaSense 2010-2016).

Last day of the conference focused on the industrial applications of AI in NLProc. By invitation of Lidia Pivovarova (University of Helsinki) I presented on the search engine and NLProc work we've done at AlphaSense, including smart synonyms, sentiment analysis, named entity recognition and salience resolution, theme modelling and high-precision search.

One of the challenges for the industrial presentation was that it had to last for 1,5 hours. If you consider your audience ability to focus only for 40 minutes, you have got to do something else than 65 slides. I decided to make about 30 slides and then handle the rest of my talk with Q&A. The outcome has been very surprising to myself, because the audience did want to learn details of AlphaSense product, making the Q&A last for 50 minutes. Quite a few questions I managed to answer with the product itself -- this sparks genuine interest in understanding the UI of an AI product powering the financial industry. I hope this was beneficial for the audience to dive into the workflows of financial knowledge workers and how NLProc can help solve their daily routine tasks better.

Customer Development Marathon

Customer development is the topic that interests me from the point of the product development. Just recently I've learnt about jobs-to-be-done approach to mining for real jobs that your customers hire your product for. One example with which Clayton Christensen of Harvard Business School motivates this approach is the job that male consumers of milkshakes had on the their way to work every day: stay engaged in life during monotonous driving and stay full until 10 a.m.

The conference (or marathon as we called it) on customer development attracted 70 participants at iHUB co-working center in Kyiv, Ukraine. Speakers from various established companies -- YouScan, MacPaw, PromoRepublic, Competera, AlphaSense, Kyivstar, Terrasoft, PMLab, Portmone.com, Weblium, VARUS, SendPulse, EVO.company -- presented 5 min talks about specific cases on engaging with their customers to grow conversion, retention and happiness with their products. Following the presentations, the discussion panels dug deeper into how to implement a customer-centric business. 

Credit: Maria Kudinova

We've organized the marathon in 3 panels: 

1. Idea. Analysis. Validation 
2. Creation. Delivery. Launch and
3. Sales. Feedback. Innovation. 

Each of these panels focused on a particular stage of product development from idea to post-sale feedback and innovation loop. The audience learnt about how to conduct an efficient user interview, what tools help reach out to new or existing clients, how not to push your product into consulting or outsource, how to establish an internal company-wide communication to stay on the same page when shaping the product, marketing and sales around customer needs.

Both events were full of networking, meeting new and familiar faces in the industry and academia and learning a lot. For anything you aspire to build next year, focusing on real value and ease of use of your NLP / AI / search products, and thinking what job your users hire your products for will help you serve them better.

Automatic writing with Deep Learning: Progress

This is a continuation of the post https://dmitrykan.blogspot.com/2018/05/automatic-writing-with-deep-learning.html. This item was reblogged at Writer's DZone: https://dzone.com/articles/automatic-writing-with-deep-learning-progress

Fast forward few months (apologies for the delay) I can share some findings.

Again, I think, we should take AI co-writer exercises with a grain of salt. However, during this time I have come across practical usage example areas for such systems.

One of them is augmentation of a news article writer. More specifically, when writing a news item, one of the most challenging tasks is to coin a catchy title. Does the title have some trendy phrases in it? Or perhaps it mentions an emerging topic, that captures attention at this given moment? Or reuses a pattern that worked well for this given author? Or just spurs an idea in the author's head?

Copyright: https://www.rogerwilco.co.za/blog/robot-writers-how-ai-will-affect-copywriting

In the following exercise I have set a very modest goal: train a co-writer on previously written texts with an attempt to suggest something useful from them. I could imagine, that this could be extended to texts that are trending or a collection of particularly interesting titles. What have you.

To train such a model I have used Robin Sloan's RNN writer: https://github.com/robinsloan/rnn-writer. The goodies of the project are:

• Trained on Torch. Nowadays, Torch is leveraged via PyTorch, a deep learning Python library that is nearing its production readiness time.
• The trained model gets exposed into an Atom -- pluginable editor (I'd imagine, real writers would want to have the model integrated into their favourite editor, like Word).
• API is available too to integrate into custom apps (and this is exactly how it is integrated with Atom).

I will skip the installation of Torch and training the network and proceed to examples. The rnn-writer github repository has a good set of instructions to proceed with. I have installed Torch and trained the model on a Mac.

First things first: RNN trained on my Master's Thesis "Design and Implementation of Peer-to-Peer Network" (University of Kuopio, 2007).

The text of the Master's Thesis is about 50 pages in English with diagrams and formulas. On one hand, having more data makes NNs learn more word representations and should have larger probability space to predict next word given the condition of the current word or phrase. On the other hand, limiting the input corpus to phrases that have certain domain goal, like writing an email, could leverage a clean set of phrases that a user employs in many typical email passages.

As I got an access to Fox articles, I thought, this could warrant another RNN model and a test. Something to share next time.

Automatic writing with Deep Learning: Preface

This article was also reblogged at: https://dzone.com/articles/automatic-writing-with-deep-learning-preface


Quite many machine and deep learning problems are directed at building a mapping function of roughly the following form:

Input X ---> Output Y,

where:

X is some sort of an object: an email text, an image, a document; 

Y is either a single class label from a finite set of labels, like spam / no spam, detected object or a cluster name for this document or some number, like salary in the next month or stock price.

While such tasks can be daunting to solve (like sentiment analysis or predicting stock prices in realtime) they require rather clear steps to achieve good levels of mapping accuracy. Again, I'm not discussing situations with lack of training data to cover the modelled phenomenon or poor feature selection.

In contrast, somewhat less straightforward areas of AI are the tasks that present you with a challenge of predicting as fuzzy structures as words, sentences or complete texts. What are the examples? Machine translation for one, natural language generation for another. One may argue, that transcribing audio to text is also such type of mapping, but I'd argue it is not. Audio is a "wave" and the speech detection is an okay solved task (with state of the art above 90% of accuracy), however such an algorithm does not capture the meaning of the produced text,  except for where it is necessary to do the disambiguation of what was said. Again, I have to make it clear, that audio->text problem is not at all easy with its own intricacies, like handling speaker self corrections, noise and so on.

Lately, the task of writing texts with a machine (e.g. here) caught my eye on twitter. Previously, papers from Google on writing poetry or other text producing software were giving me creepy feelings. I somehow undermined the role of such algorithms in the space of natural language processing and language understanding and saw only diminishing value of such systems to users. Again, any challenging tasks might be solved and even bring value to solving other challenging tasks. But who would use an automatic poetry writing system? Why would somebody, I thought, use these systems -- just for fun? My practical mind battled against such "fun" algorithms. Again, making an AI/NLProc system capable of producing anything sensible is hard. Take the task of sentiment analysis, where it is quite unclear what the agreement between experts is, not to mention non-experts.

I think this post has poured enough of text onto the heads of my readers. I will use this post as a self-motivating mechanism to continue the research with systems producing text. My target is to complete the neural network training on the text from my Master thesis and show you some examples for your judgement of the usefulness of such systems.

AI for lip reading

It is exciting to push your imagination for where else can you apply AI, machine learning and most certainly -- deep learning, that is so popular these days. I came across this question on quora that provoked me to think a bit how would one go about training a neural network to lip read. I don't actually know what made me answer this question more: that found myself in an unusual context sitting on an Angularjs meetup at Google offices in New York City (after work, usual level tired) or the question itself. Whatever the reason, here is my answer:

Source: http://theconversation.com/our-lip-reading-technology-promises-to-make-hearing-aids-more-human-45166

I would probably first start with formalizing what is lip reading process from a human understandable algorithm point of view. May be it is worth to talk to a professional, like a spy or something. Obviously you need training data. Understanding, what is lip reading from the algorithm perspective will affect on what data you need.

1. To read a word of several syllables you’d need a sequence of anchor lip positions, that represent syllables. Or probably vowels / consonants. See, I don’t know, which one is best. But you’d need to start with the lowest level possible out of which you can compose larger sequences, like letters -> syllables -> words. Let’s call these states.
2. A particular lip posture (is that the right word?) will most probably map to ambiguous states.
3. Now the interesting part is how to resolve the ambiguities. Number 2 produces several options. Out of these you can produce a multitude of words that we can call candidates.
4. Then you need to score candidates based on some local context information. Here it turns into a natural language understanding.
5. I'd start with seq2seq.

Will deep learning make other machine learning algorithms obsolete?

The fourth (fifth?) quoranswer is here! This time we'll talk a bit about deep learning and its role in making other state of the art machine learning methods obsolete.

Will deep learning make other machine learning algorithms obsolete?

I will try to take a look at the question from the natural language processing perspective.

There is a class of problems in NLProc, that might not be benefited from deep learning (DL), at least directly. For the same reasons, machine learning  (ML) cannot help so easily. I will give three examples, which share more or less the same property so hard to model with ML or DL:

1. Identifying and analyzing a sentiment polarity oriented towards a particular object: person, brand etc. Example: I like phoneX, but dislike phoneY. If you monitor the sentiment situation for the phoneX you'll expect this message to be positive, while negative polarity for the phoneY. One can argue, it is easy / doable with ML / DL, but I doubt you can stay solely within that framework. Most probably you'll need a hybrid with rule-based system, syntactic parsing etc, which somewhat defeats the purpose of DL: be able to train neural network on a large amount of data without domain (linguist) knowledge.

2. Anaphora resolution. There are systems that use ML (and hence DL can be tried?), like BART coreference system , but most of the research I have seen so far is based around some sort of rules / syntactic parsing (this presentation is quite useful: Anaphora resolution). There is a vast application area for AR, including sentiment analysis and machine translation (also fact extraction, question-answering etc).

3. Machine translation. Disambiguation, anaphora, object relations, syntax, semantics and more in a single soup. Surely, you can try to model all of these with ML, but commercial systems in MT are more or less done with rules (+ml recently). I'm expecting DL can produce advancements in MT. I'll cite one paper here that uses DL and improves on phrase-based SMT: [1409.3215] Sequence to Sequence Learning with Neural Networks Update: some recent fun experiment with DL based machine translation.

The list can be extended to knowledge bases etc, but I hope I made my point.

More fun with Google machine translation

Having posted in quoranswer tag specifically on machine translation tricks and challenges + looking at some fun with Mongolian->Russian translation with Google, I decided to experiment with Mongolian->English pair. To make this work, you'd need a Cyrillic keyboard and type only Russian letters 'а' as input on Mongolian language side. Throughout the text I'll refer to Google Translate as "neural network" or "network", as it has been known that Google has switched its translation system over to a Neural Network implementation.

So let's get going. It all starts rather sane:

а   -> a
аа -> ah

And as we stack up more letters on the left, we start getting more interesting translations:

ааа -> Well
аааа -> ahaha
ааааа -> sya
аааааа -> Well
ааааааа -> uh

and skipping a bit:

ааааааааа -> that's all

(at this point you'd imagine that deep neural network had some fun you teasing it and wants you to stop. But no).

аааааааааа -> that's ok
аааааааааааааа -> that's fine

ааааааааааааааааа -> everything is fine

ааааааааааааааааааа -> it's a good thing


And a bit more letters stacked up, the network begs to stop again, threatening:

ааааааааааааааааааааааааааааааааааааа -> it's all over

Then, after having enough of statements, the network starts asking questions.

ааааааааааааааааааааааааааааааааааааааааа -> is it a good thing?

and answers own question:

аааааааааааааааааааааааааааааааааааааааааа -> it's a good thing

few comments here and there:

ааааааааааааааааааааааааааааааааааааааааааааааааааа -> a good time

аааааааааааааааааааааааааааааааааааааааааааааааааааа-> to have a good time

Eventually, more dictionary entries crop in:

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> to a whirlwind

And, unexpectedly:

ааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> to make a date
аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> to make a living

Then, the network starts to output:

ааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> to make a dicision

And begs me to put some sane words in instead of the letter non-sense:

ааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> put your own word

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> a whistle-blower

The latter one is probably meant as an offence to add colour to network's ask.

ааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> have a private time in the world

Notice how general words are, like "private", "time", "world". Still they are grammatical and make sense, except unlikely as translations.

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> a mortal year

And to begging again:

ааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> have a kindness in the world

Again, all my commentary is meant as fun, I'm not intending to (mis)lead you to something here.

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> a dead dog

ааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> put ā € |

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> have a deadline

And more threats, again:

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> a hash of you

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> a mortal beefed up

ааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> have a heartbroker

A heartbroker? Really? Something new.

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> a hash of a tree

ааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> to put a lot of light on it

And finally, the network gets hungry:

ааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> to have a meal

And positively concludes:

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> a date auspicious

аааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааааа -> a friend of a thousand years

Hope you had fun reading these, and please try some for yourselves.

Deep learning: анализ текста и изображений при помощи Рекурсивных Нейронных сетей

Продолжая тему Deep Learning, получившую в последнее время большое внимание научного сообщества и индустрии, хочется рассказать про довольно необычную сферу применения нейронных сетей. Это сфера разбора (parsing) изображений рекурсивными нейронными сетями. В этом посте я кратко опишу суть и приведу ссылки для более детального ознакомления с материалом. Помимо этого будет ссылка на задачу из Stanford'а для желающих попробовать свои силы и получить практический результат в области deep learning для NLP.

Этот пост базируется на докладе Richard Socher, видео на английском доступно здесь.

Как оказывается, рекурсивной (или некоторой регулярной) структурой может обладать не только язык, но и изображения. Но в начале о рекурсивных свойствах предложений. Принцип композиционности (principle of compositionality) делает предположение, что любое предложение на естественном языке можно представить в виде иерархической структуры из связных составляющих. Смысл предложения можно представить в виде смыслов слов, в него входящих, и списка правил соединения слов в группы. Например, в предложении:

Это страна моего рождения.

Cлова "моего" и "рождения" образуют единую группу, "страна" и "моего рождения" над-группу, а всё предложение замыкает его смысл в виде иерархического представления указанных групп. Таким образом, получаются не только смыслы отдельных слов предложения, но и древесные структуры, в которые данные слова увязываются.

Как можно разобрать изображения по аналогии с текстовыми рекурсивными представлениями? Можно утверждать, что существует схожий принцип композиционности для изображений. Рассмотрим изображение:

Если рассмотреть некоторые сегменты изображения: здание, конусообразная крыша, оконный ряд, окна по отдельности. В указанном порядке они описывают "вложенную" рекурсивную структуру, которой можно описать здание целиком. Есть ещё параллельные зданию объекты -- люди, деревья и трава. Таким образом, изображение, например, дома можно представить виде древесной структуры, где узлы на нижних уровнях являются составляющими узлов-предков. В точности, как и в древесных структурах предложений на естественном языке.

Алгоритм на основе рекурсивных нейронных сетей авторов Socher и др. достигает 78,1% качества. Область применения таких алгоритмов -- распознавание сцен (область анализа изображений или image analysis). Исходные коды и датасеты можно посмотреть здесь.

Задачка. Для более детального ознакомления с deep learning в применении к задачам NLP на сайте Stanford'a предлагается к решению задачка: реализовать простой оконный распознаватель именных сущностей (Named Entity Recognition или NER). Описание задачки здесь. Стартовый исходный код на Java и тренировочный сет здесь. Тренировочный сет аннотирован персонами, например:

Franz PERSON
Fischler PERSON

У исходного кода есть два режима: собственно, исполнение и отправка решения на сервер. Понятно, что из этих двух режимов нас интересует первый.

Довольно много материалов по теме Deep Learning можно найти на сайте Richard Socher, включая применение данного направления к распознаванию тональности (sentiment analysis).