private static void log(String logStatement) {
log(logStatement);
}
Wednesday, May 25, 2011
StackOverflow
Monday, May 2, 2011
Monday, April 25, 2011
NoSQL, Apache SOLR and Apache Hadoop
Tuesday, April 19, 2011
NerdCamp
Confirmed Key Participants
Yury Lifshits, Sergey Poduzov, Alexander Shtuchkin,
Vladimir Gorovoy, Nikolay Vyahhi, Vladimir Aluferov, Yakov Sirotkin and myself
The preliminary program has juicy topics among which are:
How the Web will Transform Education by Yury Lifshits,
Geo Information Systems Around Us by Aleksander Klechikov,
Software Development for little ones by Yakov Sirotkin,
Introducing NoSQL: Apache SOLR and Hadoop by Dmitry Kan,
Introducing Cloud Services by Dmitry Petrov,
Nature of Enterpreneurship Craft, Lifestyle or Science by Alexey Baranov,
How to grow places for smart people -- workshop
and more!
Friday, March 25, 2011
Angry Birds -pelaaja: DK
Monday, March 21, 2011
concert of components?
This post is short and not useful in that it doesn't give you any code snippets or technical recommendations. I would like just to cite the SolrCloud's wiki. The SolrCloud is "the set of Solr features that take Solr's distributed search to the next level, enabling and simplifying the creation and use of Solr clusters." They use Apache ZooKeeper project (a subproject of Hadoop) as a distributed system of keeping cluster of SOLRs state updates. In a distributed system every component can potentially crash, yet the system is expected to provide it's service to its users. If a single SOLR crashes, its replica will take over, but if ZooKeeper crashes, the system will still continue serving the user requests, but no updates of the system are visible to the system (sounds interesting, I know). To improve that, this is what is possible:
"Running multiple zookeeper servers in concert (a zookeeper ensemble) allows for high availability of the zookeeper service. Every zookeeper server needs to know about every other zookeeper server in the ensemble, and a majority of servers are needed to provide service. For example, a zookeeper ensemble of 3 servers allows any one to fail with the remaining 2 constituting a majority to continue providing service. 5 zookeeper servers are needed to allow for the failure of up to 2 servers at a time."
So, if you have a big zoo with variety of animals in it, make sure you have 5 zoo keepers for at least 3 of them take care of your pigs and elephants, when 2 others got stuck somewhere else.
Saturday, January 29, 2011
SOLR: speed up batch posting
Apparently, when deployed under Tomcat, SOLR logs each and every update request during POSTing process. Experience shows, that heavy http operation is done faster, when logging is minimal.
SOLR (as of 1.4 at least) has admin GUI which serves as a central information hub for the given SOLR core. Among other useful features, it has a page where one can set up logging levels of different SOLR components. In default SOLR installation you can access the page via http://localhost:8983/solr/admin/logging. By default, the logging levels amount mainly to INFO, which permits logging of all the select/update requests (imagine 1 million of such log entries for batch reindexing).
It would be handy to be able to automatically change the logging levels to, say, WARNING before batch POSTing and back to INFO after that. solr/admin/logging is declared as servlet in web.xml of the corresponding SOLR core:
Logging
org.apache.solr.servlet.LogLevelSelection
All the components which allow changing their logging levels are listed on the page http://localhost:8983/solr/admin/logging. Using curl we can send a post request to the servlet and set the desired levels. It is reasonable to implement a function, which takes logging level and url of SOLR core as parameters (choose your own favourite language, this is done in Perl):
sub setSolrLogLevel
{
my ($url, $level) = @_;
print "setting logging level to $level\n";
my $res = system("curl --user user:pass -d \"submit=set&root=$level&fi=$level" .
"&fi.alphasense=$level&fi.alphasense.solr=$level&fi.alphasense.solr.query=$level" .
"&fi.alphasense.solr.query.AlphaSenseQParserPlugin=$level&httpclient=$level&httpclient.wire=$level&httpclient.wire.content=$level&httpclient.wire.header=$level&javax=$level&javax.management=$level&javax.management.mbeanserver=$level&org=$level&org.apache=$level&org.apache.catalina=$level&org.apache.catalina.core=$level&org.apache.catalina.core.ContainerBase=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5BLogging%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5BSolrServer%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5BSolrUpdate%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5Bdefault%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5Bjsp%5D=$level&org.apache.catalina.core.ContainerBase.%5BCatalina%5D.%5Blocalhost%5D.%5B%2Fsolrtopic%5D.%5Bping%5D=$level&org.apache.catalina.session=$level&org.apache.catalina.session.ManagerBase=$level&org.apache.commons=$level&org.apache.commons.digester=$level&org.apache.commons.digester.Digester=$level&org.apache.commons.digester.Digester.sax=$level&org.apache.commons.httpclient=$level&org.apache.commons.httpclient.ChunkedInputStream=$level&org.apache.commons.httpclient.HeaderElement=$level&org.apache.commons.httpclient.HttpClient=$level&org.apache.commons.httpclient.HttpConnection=$level&org.apache.commons.httpclient.HttpMethodBase=$level&org.apache.commons.httpclient.HttpMethodDirector=$level&org.apache.commons.httpclient.HttpParser=$level&org.apache.commons.httpclient.HttpState=$level&org.apache.commons.httpclient.MultiThreadedHttpConnectionManager=$level&org.apache.commons.httpclient.SimpleHttpConnectionManager=$level&org.apache.commons.httpclient.auth=$level&org.apache.commons.httpclient.auth.AuthChallengeProcessor=$level&org.apache.commons.httpclient.cookie=$level&org.apache.commons.httpclient.cookie.CookiePolicy=$level&org.apache.commons.httpclient.cookie.CookieSpec=$level&org.apache.commons.httpclient.methods=$level&org.apache.commons.httpclient.methods.EntityEnclosingMethod=$level&org.apache.commons.httpclient.methods.ExpectContinueMethod=$level&org.apache.commons.httpclient.methods.PostMethod=$level&org.apache.commons.httpclient.params=$level&org.apache.commons.httpclient.params.DefaultHttpParams=$level&org.apache.commons.httpclient.params.HttpMethodParams=$level&org.apache.commons.httpclient.util=$level&org.apache.commons.httpclient.util.EncodingUtil=$level&org.apache.commons.httpclient.util.ExceptionUtil=$level&org.apache.commons.httpclient.util.IdleConnectionHandler=$level&org.apache.jasper=$level&org.apache.jasper.EmbeddedServletOptions=$level&org.apache.jasper.JspCompilationContext=$level&org.apache.jasper.compiler=$level&org.apache.jasper.compiler.Compiler=$level&org.apache.jasper.compiler.JspConfig=$level&org.apache.jasper.compiler.JspRuntimeContext=$level&org.apache.jasper.compiler.TldLocationsCache=$level&org.apache.jasper.servlet=$level&org.apache.jasper.servlet.JspServlet=$level&org.apache.jasper.servlet.JspServletWrapper=$level&org.apache.solr=$level&org.apache.solr.analysis=$level&org.apache.solr.analysis.BaseTokenFilterFactory=$level&org.apache.solr.analysis.BaseTokenizerFactory=$level&org.apache.solr.client=$level&org.apache.solr.client.solrj=$level&org.apache.solr.client.solrj.impl=$level&org.apache.solr.client.solrj.impl.CommonsHttpSolrServer=$level&org.apache.solr.common=$level&org.apache.solr.common.util=$level&org.apache.solr.common.util.ConcurrentLRUCache=$level&org.apache.solr.core=$level&org.apache.solr.core.Config=$level&org.apache.solr.core.CoreContainer=$level&org.apache.solr.core.JmxMonitoredMap=$level&org.apache.solr.core.RequestHandlers=$level&org.apache.solr.core.SolrConfig=$level&org.apache.solr.core.SolrCore=$level&org.apache.solr.core.SolrResourceLoader=$level&org.apache.solr.handler=$level&org.apache.solr.handler.AnalysisRequestHandler=$level&org.apache.solr.handler.XmlUpdateRequestHandler=$level&org.apache.solr.handler.admin=$level&org.apache.solr.handler.admin.LukeRequestHandler=$level&org.apache.solr.handler.admin.SystemInfoHandler=$level&org.apache.solr.handler.component=$level&org.apache.solr.handler.component.QueryElevationComponent=$level&org.apache.solr.handler.component.SearchHandler=$level&org.apache.solr.handler.component.SpellCheckComponent=$level&org.apache.solr.highlight=$level&org.apache.solr.highlight.SolrHighlighter=$level&org.apache.solr.request=$level&org.apache.solr.request.BinaryResponseWriter=$level&org.apache.solr.request.XSLTResponseWriter=$level&org.apache.solr.schema=$level&org.apache.solr.schema.FieldType=$level&org.apache.solr.schema.IndexSchema=$level&org.apache.solr.search=$level&org.apache.solr.search.SolrIndexSearcher=$level&org.apache.solr.servlet=$level&org.apache.solr.servlet.LogLevelSelection=$level&org.apache.solr.servlet.SolrDispatchFilter=$level&org.apache.solr.servlet.SolrRequestParsers=$level&org.apache.solr.servlet.SolrServlet=$level&org.apache.solr.servlet.SolrUpdateServlet=$level&org.apache.solr.spelling=$level&org.apache.solr.spelling.AbstractLuceneSpellChecker=$level&org.apache.solr.spelling.FileBasedSpellChecker=$level&org.apache.solr.spelling.IndexBasedSpellChecker=$level&org.apache.solr.update=$level&org.apache.solr.update.SolrIndexConfig=$level&org.apache.solr.update.UpdateHandler=$level&org.apache.solr.util=$level&org.apache.solr.util.SolrPluginUtils=$level&org.apache.solr.util.plugin=$level&org.apache.solr.util.plugin.AbstractPluginLoader=$level\" $url");
print "Result code:$res\n";
}
There you go. Call setSolrLogLevel("http://localhost:8983/solr/admin/logging", "WARNING"); before the batch POSTing and setSolrLogLevel("http://localhost:8983/solr/admin/logging", "INFO"); after the batch POSTing has finished.
Friday, November 19, 2010
"OOPness" in Java
So by not supporting "pass by reference" feature Java makes your code even more OOP-ish. Even though the topic can be well known, this article helps to put things together and make a memory refresh.
Wednesday, November 3, 2010
Successive replacement in regular expressions (java)
So the task is like this: you have a text T, like "cat-1 dog-1 cat-1 elephant-1 cat-2 dog-2 cat-3".
Suppose we want to change numerals attached to the words "cat" to their word representations: "1" to "one", "2" to "two".
One straightforward way would be to match all "cat-([0-9])+" subsequences and then run replace operation on T.
So the code would look something like this:
String T = "cat-1 dog-1 cat-1 elephant-1 cat-2 dog-2 cat-3";
Pattern catPattern = Pattern.compile("cat-([0-9]+)");
Matcher catMatcher = catPattern.matcher(T);
Map numToWord = new HashMap();
numToWord.add("1", "one");
numToWord.add("2", "two");
numToWord.add("3", "three"); // ...
while (catMatcher.find())
{
T = T.replaceFirst(catMatcher.group(1), numToWord.get(catMatcher.group(1)));
}
This code produces:
cat-one dog-one cat-1 elephant-1 cat-two dog-2 cat-three
Which is missing one substitution. Ok, let's use replaceAll instead and make sure we touch only cats:
{
T = T.replaceAll("cat-" + catMatcher.group(1), "cat-" + numToWord.get(catMatcher.group(1)));
}
which produces what we want:
cat-one dog-1 cat-one elephant-1 cat-two dog-2 cat-three
But now what happens inside the loop is logically out of sync with the loop condition: we iterate over matches, but call replaceAll (probably not efficient either, as replaceAll will be attempted even when not needed anymore, for duplicate matches).
Any more elegant and correct solution?
Yes! It is called Matcher.appendReplacement
Pattern catPattern = Pattern.compile("cat-([0-9]+)");
Matcher catMatcher = catPattern.matcher(T);
MapnumToWord = new HashMap ();
numToWord.put("1", "one");
numToWord.put("2", "two");
numToWord.put("3", "three"); // ...
StringBuffer sb = new StringBuffer();
while (catMatcher.find())
{
System.out.println("Match:" + catMatcher.group(1));
catMatcher.appendReplacement(sb, "cat-" + numToWord.get(catMatcher.group(1)));
}
catMatcher.appendTail(sb);
now sb.toString() contains:
cat-one dog-1 cat-one elephant-1 cat-two dog-2 cat-three
If you append System.out.println(sb.toString()); inside the while loop, you will also see, that replacements happen in sync with the while loop's state, so that what is inside the loop and what while loops over are in sync.
Saturday, August 21, 2010
B2B: what is WSDL (simple explanation)
So the server declares a method (which is in fact a remote method):
Double squareRoot(Double number)
in its WSDL file. The client side then takes this file and generates the client side code, which handles the protocol of communication and remote method invocation automatically. What client needs is simply to implement a business logic around this invocation, like a web page with a text field for the number or the entire hardware with touch screen display.
In general the input and return types of remote methods can be simple and complex -- in fact as complex and "proprietary" as you need, because the entire WSDL file is nothing but xml as well as the messages being sent over the network.
One of the libraries used in the industry is Axis and Axis2. Using the library you can generate the java code from wsdl for you client (and even generate wsdl from your java code for you server).
Tuesday, August 17, 2010
Zero-witdth negative lookahead group: example
name1[one_space]surname1[two_spaces]age
surname2[two_spaces]surname2[two_spaces]age
In this example it is easy to solve the problem with zero-width negative lookahead group (?!\\w), because age is always supposed to be numeric. So the final code will be something like this:
String s = current_string.split("\\s{2}(?!\\w)");
Thursday, July 1, 2010
Парижские впечатления
Tuesday, April 27, 2010
Dumper and sorting of keys
use strict;
use Data::Dumper;
$hash_ref = generate your hash here
$Data::Dumper::Sortkeys = \&my_filter;
print Dumper($hash_ref), "\n";
sub my_filter {
my ($hash) = @_;
# return an array ref containing the hash keys to dump
# in the order that you want them to be dumped
return [
sort {$a <=> $b} keys %$hash
];
}
Wednesday, April 21, 2010
Сходимость
Всё это к тому, что полезно преподавать программирование, рассматривающее программу как целостную математическую схему, а не только как набор алгоритмов, приёмов и дизайна.
Tuesday, April 20, 2010
Thursday, April 8, 2010
An idea for GUI designers and developers -- a standalone app / OS level change.
This may give a lot of opportunities to quite flexibly configure your GUI world and save a lot of time spent otherwise on the monkey repetitions.
Saturday, March 13, 2010
giza++ under windows: episode 2
The full list of steps goes here (I suppose that source corpus is stored in corpus.ru and target corpus is stored in corpus.en):
1. produce vcb and dictionary files with "plain2snt.out corpus.ru corpus.en" (credits: http://vee-r.blogspot.com/2006/12/giza-guide.html)
2. produce cooc file with "snt2cooc.out corpus.ru.vcb corpus.en.vcb corpus.ru_corpus.en.snt > ru_en.cooc" (credits: myself, after analyzing the train-factored-phrase-model.perl from Moses package)
3. run GIZA++ with config file:
outputfileprefix play_giza
sourcevocabularyfile corpus.ru.vcb
targetvocabularyfile corpus.en.vcb
c corpus.ru_corpus.en.snt
CoocurrenceFile ru_en.cooc
model1dumpfrequency 1
model4smoothfactor 0.4
nsmooth 4
onlyaldumps 1
nodumps 1
p0 .999
m1 5
m2 0
m3 3
m4 3
o giza
if these options are stored in giza.config then run "GIZA++ giza.config". This produces giza.A3.final file, typical entry of which is:
# Sentence pair (1) source length 4 target length 7 alignment score : 2.25315e-10
there is a book on the table
NULL ({ }) на ({ }) столе ({ 5 6 7 }) лежит ({ 1 2 }) книга ({ 3 4 })
which means the following mapping:
"столе" --> "on the table"
"лежит" --> "there is"
"книга" --> "a book"
Running GIZA++ under win32 and under linux gives same results in terms of word mappings, except that the alignment scores may slightly differ due to possibly different float point precision models.
Saturday, February 27, 2010
giza++ under windows
gcc version 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)
make GNU Make 3.81 (built for i686-pc-cygwin)
With
$ make
under giza-pp-v1.0.3\giza-pp I have obtained two executables: GIZA++-v2/GIZA++.exe and mkcls-v2/mkcls.exe which perfectly run under Windows XP Professional Version 2002 SP 2.
Saturday, December 19, 2009
RuSSIR'2010
4th Russian Summer School in Information Retrieval (RuSSIR 2010)
Monday September 13 - Saturday September 18, 2010
Voronezh, Russia
http://romip.ru/russir2010/eng/
FIRST CALL FOR COURSE PROPOSALS
The 4th Russian Summer School in Information Retrieval (RuSSIR 2010) will be held on September 13-18, 2010 in Voronezh, Russia, one of the major cities in south-western Russia. The mission of the school is to teach students about modern problems and methods in Information Retrieval; to stimulate scientific research in the field of Information Retrieval; and to create an opportunity for informal contacts among scientists, students and industry professionals. The Russian Conference for Young Scientists in Information Retrieval will be co-located with the school. RuSSIR 2010 will offer 4 or 5 courses and host approximately 100 participants. The working languages of the school are English (preferable) and Russian. The target audience of RuSSIR is advanced graduate and PhD students, post-doctoral researchers, academic and industrial researchers, and developers.
The RuSSIR 2010 Organizing Committee invites proposals for courses on a wide
range of IR-related topics, including but not limited to:
- IR theory and models
- IR architectures
- Algorithms and data structures for IR
- Text IR
- Multimedia (including music, speech, image, video) IR
- Natural language techniques for IR tasks
- User interfaces for IR
- Web IR (including duplicate detection, hyperlink analysis, query logs)
- Text mining, information and fact extraction
- Mobile applications for IR
- Dynamic media IR (blogs, news, WIKIs)
- Social IR (collaborative filtering, tagging, recommender systems)
- IR evaluation.
Each course should consist of five 90-minute-long sessions (normally in five consecutive days). The course may include both lectures and practical exercises in computer labs.
RuSSIR 2010 organizers will cover travel expenses and accommodations at the school for one lecturer per course, but there is no additional honorarium. The RuSSIR organizers would highly appreciate if, whenever possible, lecturers could find alternative funding to cover travel and accommodation expenses and indicate this possibility in the proposal.
Course proposals for RuSSIR 2010 must be submitted by email to Pavel Braslavski (pb@yandex-team.ru), by February 14, 2010. A course proposal should contain a brief description of the course (up to 200 words), preferred schedule, prerequisites, equipment needs, a short description of teaching/research experience and contact information of the lecturer(s). All proposals will be evaluated by the RuSSIR 2010 program committee according to the school goals, presentation clarity, lecturer’s qualifications and experience. Topics not featured at previous RuSSIRs are preferred. All submitters will be notified by March 1, 2010. Early informal inquiries about the school or the proposal evaluation process are encouraged.
About RuSSIR: The Russian Summer School in Information Retrieval is co-organized by the Russian Information Retrieval Evaluation Seminar (ROMIP) and Voronezh State University. Previous schools took place in Ekaterinburg, Taganrog, and
Petrozavodsk. Previous RuSSIR courses included IR Models (by Djoerd Hiemstra), Modeling Web Searcher Behavior and Interactions (by Eugene Agichtein), Computational Advertising (by James Shanahan), Text Mining, Information and Fact Extraction (by Marie-Francine Moens), Natural Language Processing for Information Access (by Horacio Saggion), Music IR (by Andreas Rauber), and other. Ricardo Baeza-Yates, VP of Research for Europe and Latin America at Yahoo, has confirmed as an invited lecturer for RuSSIR 2010 with the course 'Web data mining'.
About the RuSSIR 2010 location: Voronezh is a major city in southwestern Russia, spanning both sides of the Voronezh River, with population of 850,000. Express trains from Moscow to Voronezh take about 10 hours. There are also regular flights from Moscow, Munich, Prague, Tel-Aviv, and Istanbul. The town was founded in 1586. In the 17th century, Voronezh gradually evolved into a sizeable town, especially after Tsar Peter the Great built a dockyard in Voronezh. Currently, Voronezh is an administrative, economic and cultural center of the Voronezh region. Voronezh surrounding area has many attractions including archeological museum, nature and historical reserve Divnogorie, Kostomarovo cave monastery, Orlov trotter stud farm at Khrenovoe. Voronezh has a large student population: 37 institutions of higher education and 53 colleges educating over 127,000 students today. Voronezh State University was founded in 1918 and is one of the largest universities in Russia, with a total enrollment of 22,000.
Contacts
Use the e-mail address and substitute [at] with @ and [dot] with "." school[at]romip[dot]ru.