AI for lip reading

It is exciting to push your imagination for where else can you apply AI, machine learning and most certainly -- deep learning, that is so popular these days. I came across this question on quora that provoked me to think a bit how would one go about training a neural network to lip read. I don't actually know what made me answer this question more: that found myself in an unusual context sitting on an Angularjs meetup at Google offices in New York City (after work, usual level tired) or the question itself. Whatever the reason, here is my answer:

Source: http://theconversation.com/our-lip-reading-technology-promises-to-make-hearing-aids-more-human-45166

I would probably first start with formalizing what is lip reading process from a human understandable algorithm point of view. May be it is worth to talk to a professional, like a spy or something. Obviously you need training data. Understanding, what is lip reading from the algorithm perspective will affect on what data you need.

1. To read a word of several syllables you’d need a sequence of anchor lip positions, that represent syllables. Or probably vowels / consonants. See, I don’t know, which one is best. But you’d need to start with the lowest level possible out of which you can compose larger sequences, like letters -> syllables -> words. Let’s call these states.
2. A particular lip posture (is that the right word?) will most probably map to ambiguous states.
3. Now the interesting part is how to resolve the ambiguities. Number 2 produces several options. Out of these you can produce a multitude of words that we can call candidates.
4. Then you need to score candidates based on some local context information. Here it turns into a natural language understanding.
5. I'd start with seq2seq.

Will deep learning make other machine learning algorithms obsolete?

The fourth (fifth?) quoranswer is here! This time we'll talk a bit about deep learning and its role in making other state of the art machine learning methods obsolete.

Will deep learning make other machine learning algorithms obsolete?

I will try to take a look at the question from the natural language processing perspective.

There is a class of problems in NLProc, that might not be benefited from deep learning (DL), at least directly. For the same reasons, machine learning  (ML) cannot help so easily. I will give three examples, which share more or less the same property so hard to model with ML or DL:

1. Identifying and analyzing a sentiment polarity oriented towards a particular object: person, brand etc. Example: I like phoneX, but dislike phoneY. If you monitor the sentiment situation for the phoneX you'll expect this message to be positive, while negative polarity for the phoneY. One can argue, it is easy / doable with ML / DL, but I doubt you can stay solely within that framework. Most probably you'll need a hybrid with rule-based system, syntactic parsing etc, which somewhat defeats the purpose of DL: be able to train neural network on a large amount of data without domain (linguist) knowledge.

2. Anaphora resolution. There are systems that use ML (and hence DL can be tried?), like BART coreference system , but most of the research I have seen so far is based around some sort of rules / syntactic parsing (this presentation is quite useful: Anaphora resolution). There is a vast application area for AR, including sentiment analysis and machine translation (also fact extraction, question-answering etc).

3. Machine translation. Disambiguation, anaphora, object relations, syntax, semantics and more in a single soup. Surely, you can try to model all of these with ML, but commercial systems in MT are more or less done with rules (+ml recently). I'm expecting DL can produce advancements in MT. I'll cite one paper here that uses DL and improves on phrase-based SMT: [1409.3215] Sequence to Sequence Learning with Neural Networks Update: some recent fun experiment with DL based machine translation.

The list can be extended to knowledge bases etc, but I hope I made my point.

What are some funny Google Translate tricks?

This is the third quoranswer blog post, answering the question What are some funny Google Translate tricks? I have decided to update the Google translations based on the current situation. I think they are still a lot of fun. Let me know in comments, if you came across some funny translations!

There used to be a funny politically coloured trick for Russian->English, where sense was inverted on translation depending on what President names were used in positive vs negative context. I can’t reproduce it right now, but GT produces this at the moment:

Обама не при чём, виноват Путин.

human: Obama is innocent, Putin is to blame.

GT: Obama has nothing to do with Putin. (Previously in Aug 4, 2016: "Obama is not to blame, blame Putin.")

Путин не при чём, виноват Обама

human: Putin is innocent, Obama is to blame.

GT: Putin has nothing to do with Obama's fault. (Previously in Aug 4, 2016: "Putin is not being Obama's fault.")

What grammatical challenges prevent Google Translate from being more effective?

Here is one more Quora question on the exciting topic of machine translation and my answer to it.

The question had some sub-questions:

• Is there a set of broad grammatical rules which decreases its efficacy?
• How can these challenges be overcome? Is it possible to fully automate good quality translation?

Below is my answer, hoping it will be interesting to learn about machine translation and different language pairs. Note, that translations given currently by Google Translate might differ from below as they were obtained in 2013. UPD: and they do! See comments to this post.

Google is pretty good at modeling close enough language pairs. By close enough I mean languages that share multiple vocabulary units, have similar word order, morphological richness level and other grammatical features.

Let's pick an example of a pair, where Google Translate (GT) is good. Round-trip method is one way to verify whether the languages are close enough, at least statistically, for GT:

(these examples are using GT only, no human interpretation involved)

English: I am in a shop.
Dutch: Ik ben in een winkel.
back to English I'm in a store. (quite ok)

English: I danced into the room.
Dutch: Ik danste in de kamer.
back to English: I danced in the room. (preposition issues)


Let's pick a pair of more unrelated languages (by the way, when we claim the languages are unrelated grammatically, they may also be unrelated semantically or even pragmatically: different languages were created by people to suit their needs at particular moments of history). One such pair is English and Finnish:

Finnish: Hän on kaupassa.
English: He is in the shop.
Finnish: Hän on myymälä. (roughly the original Finnish sentence)

This example has pronoun hän, which in Finnish is not gender specific. It should be resolved based on larger context, than just a sentence. Somewhere before this sentence in a text, there should have been a mention of who hän is referring to.

To conclude this particular example: Google Translate translates on a sentence level and that is a limitation in itself, that makes correct pronoun resolution impossible. Pronouns are useful, if we wanted to understand, what was the interaction between the objects in a text.


Let's pick another example of unrelated languages: English and Russian.

Russian: Маска бывает правдивее и выразительнее лица.
English: The mask is truthful and expressive face. (should have been: The mask can be more truthful and expressive than face)
back to Russian: Маска правдивым и выразительным лицом. (hard to translate, but the meaning roughly: The mask being a truthful and expressive face).

To conclude this example: languges with rich morphology that, in the case of the Russian language, convey grammatical case in just a word inflection and thus require deeper grammatical analysis, which pure statistical machine translation methods lack no matter how much data has been acquired. There exist methods of combining rules and statistics together.


Another pair and different example:
English: Reporters said that IBM has bought Lotus.
Japanese: 記者は、IBMがロータスを買っていると述べた。
back to English: The reporter said that IBM Lotus are buying.

Japanese has a "recursive syntax", that represents this English sentence, like:

Reporters (IBM Lotus has bought) said that.

i.e. the verb is syntacically placed after the subject-object pair of a sentence or a sub-sentence (direct / indirect object).

To conclude this example: there should exist a method of mapping syntax structures as larger units of the language and that should be done in a more controlled fashion (i.e. is hard to derive from pure statistics).

What's a good topic for a bachelor's thesis in Sentiment Analysis?

Preamble

Over the past few months (soon close to a year) you, my readers, might have noticed decline in frequency of my blogging. There are few reasons, including practical (absence of time), but still the most two important are:

1. Blogger has not developed too much as a tool over time. It probably continues to be relatively popular and bringing some ad money, so Google did not shut it down. Moving over to medium.com might be a better idea in order to produce visually "shinier" posts and actually enjoy writing.

2. There are other interesting and more interactive ways to share one's knowledge. One of such, that I personally like, is quora.com. The site offers a reverse model compared to blogging: you answer questions. This way you ensure, that at least the questioner will read your answer, but so might do other respondents. Rating of your answers is another component, that contributes to statistics and getting analogy of payment - credits, that you can later use for instance for boosting your answers to a larger audience. But I would say the latter is of lesser importance to me.

Since I have never actually figured out, whether Quora allows you to read posts without being registered, re-posting my answers here from time to time could be a good way to also maintain this blog alive.

So here we go (slightly edited version):

What's a good topic for a bachelor's thesis in Sentiment Analysis?

Apart from applying deep neural networks to sentiment analysis being exciting, another topic that is exciting both from research and practice perspective is sarcasm detection. It goes somewhat outside of the topic of sentiment analysis per se out to the opinion mining. Sentiment analysis precision and recall are affected by the sarcastic posts. This is because sarcastic posts tend to be positive on the surface (in fact to the conventional algorithms — ML based or rule-based ones), but suggest negative context.

There are interesting situations that arise as a result of failing to recognize sarcasm. Borrowing from [1]:

User 1 tweet:

You are doing great! Who could predict heavy travel between #Thanksgiving and #NewYearsEve. And bad cold weather in Dec! Crazy!

Response from a major U.S. Airline:

We #love the kind words! Thanks so much.

User 1:

wow, just wow, I guess I should have #sarcasm

User 2:

Ahhh..**** reps. Just had a stellar experience w them at Westchester, NY last week. #CustomerSvcFail

Response from a major U.S. Airline:

Thanks for the shout-out Bonnie. We’re happy to hear you had a #stellar experience flying with us. Have a great day.

User 2:

You misinterpreted my dripping sarcasm. My experience at Westchester was 1 of the worst I’ve had with ****. And there are many.
[1]
Rajadesingan A. et al. Sarcasm Detection on Twitter: A Behavioral Modeling Approach Sarcasm Detection on Twitter

What is it like to study mathematics at Saint Petersburg State University? (my answer on quora.com)

As it turns out, not all of my readers are on quora. So because of this and in the spirit of posting non-technical blogs too, I'm reposting an answer I gave to the question there: "What is it like to study mathematics at Saint Petersburg State University?"

I have studied math and other subjects (like physics, computer science and others) during 2002-2005 in Saint Petersburg State University (SPbU) for a Specialist program (comparable to that of Master's degree).

My experience was constantly comparative in the beginning: as I was advancing further into teaching style of SPbU professors and docents I was viewing it side by side with the style of another State University of my home city (10x smaller in population than Saint Petersburg that time).

So perhaps I can approach answering your question from the perspective of comparison.

1. (a) In my home university we were taught to learn long theorem proofs in the fashion that would enable a student to easily reproduce it on an (pre-)exam. I remember only one occasion, when a theorem was so long that learning all the low-level details was impossible (despite how many days I tried), therefore really deriving the proof was the only option. Of course you would learn the fundamental constructs and apparatus for deriving the proof, that is you wouldn't be doing it completely from scratch and finding your ways into it.

  (b) In SPbU, in contrast, you wouldn't be expected to learn the entire theorem proof at all, but instead be ready to derive it. Some of the practical tasks given along the theoretical proofs would require the same: derive a solution as you go. This was the first thing that struck me as largely different.

2. (a) In my home university I was expected to learn about 80% of definitions, theorem formulations, their proofs.

    (b) It was my first exam on Control Theory in SPbU where its professor told me, a student should learn about 35% (or even less): the _most_ important theorem formulations and their proofs plus the _most_ important definitions. The rest is derivable as explained in (1) (b)

3. (a) The highlight of fun part of studying in my home university that comes to mind was that once a professor of mathematical analysis came to the class and asked: "Do you want theory and tasks today or talk about life?" "Life" was the answer, and the first question from the audience was: "Girls of which country were the most beautiful?".

   (b) In SPbU there have been all sorts of surprises that opened student's mind or made studying more fun. One example: during one of the exams on electrodynamics (complex theory with integral calculus, Lie algebra and so on), a professor said 10 minutes past the start: "The ones who would like to get C mark (3 or "satisfactory" in Russia)" can get it right now without answering their questions. Few people rushed towards him and exited the exam room. About 10 mins later he continued: "The ones who would like to get B mark (4 or "good" in Russia)" can get it now, but you have to show me, what you have written. Some more people rushed towards him. 15 min later (and a few drops of sweat on our brave necks) he said: "The rest just get A's, because you have survived and didn't know in advance what to expect. " (5 or "excellent", the best mark). What I have learnt was that it is not always necessary to be an egg head and learn everything to be always ready to stand up. Sometimes it is important to be a good person, brave and keep courage in your heart. That may lead to more adventures and opportunities in the future!

With a few exceptions I would say, that studying math was both fun and rather instructive in that, it developed some fundamental skills of reasoning and attacking a problem at hand without having trained yourself specifically to solve that class of problems before -- what you need in real life, be it further PhD studies or solving other complex problems, including those occurring in life.