Google News + Google Translate could be much better

Google News is one of many Google Technologies that I love. Everyday I check both the US version and the HK version. Google Translate is another fantastic service, even if the lack of a free API is a bit short sighted in my mind.

Imagine my disappointment, when Google News US clustered a Chinese article poorly. Click through the image below to see the grey "translate" button next to the text "From China".

That article is about a Chinese Badminton player, Chen Jin, getting the last spot at the London Olympics. The news cluster is about activist Chen Guangcheng, a blind, self-taught human rights lawyer, escaping prison and getting to the U.S. embassy. (He has since left the U.S. embassy).

Mis-classified news article does appear fairly frequently in Google News. Had the mis-classified article been in English, I would probably have just ignored it.

Instead, I took a relevant news article and the wikipedia entry about Guang Cheng, and compared it against two translated articles. One translated article is the one listed by Google News US, about Jin, the badminton player. The other is an article about Chen Guang Cheng, from Google News HK. Basic word frequency shows clearly that Google News US made a mistake. I leave it as an exercise for the reader to sum up the total percentage of word hits and misses in both cases. I imagine using techniques to weigh information content of the words (e.g. IDF) and phrasing would make it even more obvious. Behavior-based learning from bilingual users should help also.

[In the tables below, the columns are, from left to right: word frequency_in_article frequency_in_cluster percentage_in_article, percentage_in_cluster]

Frequently used words in translated article on Chen Jin that appear in cluster


chinese 7  66 1.79 0.59
beijing 3  17 0.77 0.15
   chen 3 133 0.77 1.20
  china 2  46 0.51 0.41
     's 1  64 0.26 0.58
   also 1   9 0.26 0.08
    may 1  14 0.26 0.13
   news 1  17 0.26 0.15
 people 1  14 0.26 0.13
   said 1  12 0.26 0.11


Frequently used words in translated article on Chen Jin that do not appear in cluster


         london 8 0 2.05 0.00
          seats 8 0 2.05 0.00
        olympic 6 0 1.53 0.00
      badminton 5 0 1.28 0.00
       olympics 5 0 1.28 0.00
            six 5 0 1.28 0.00
          games 4 0 1.02 0.00
           gold 4 0 1.02 0.00
        project 4 0 1.02 0.00
     advantages 3 0 0.77 0.00
       capacity 3 0 0.77 0.00
         medals 3 0 0.77 0.00
       shooting 3 0 0.77 0.00
      strengths 3 0 0.77 0.00
            win 3 0 0.77 0.00
        ability 2 0 0.51 0.00
     delegation 2 0 0.51 0.00
         diving 2 0 0.51 0.00
    eligibility 2 0 0.51 0.00
            jin 2 0 0.51 0.00
          medal 2 0 0.51 0.00
          order 2 0 0.51 0.00
       projects 2 0 0.51 0.00
        qualify 2 0 0.51 0.00
          rifle 2 0 0.51 0.00
          skeet 2 0 0.51 0.00
          table 2 0 0.51 0.00
         tennis 2 0 0.51 0.00
  weightlifting 2 0 0.51 0.00
        achieve 1 0 0.26 0.00
       addition 1 0 0.26 0.00
           army 1 0 0.26 0.00
        aroused 1 0 0.26 0.00
         become 1 0 0.26 0.00
            big 1 0 0.26 0.00
            bus 1 0 0.26 0.00
         caught 1 0 0.26 0.00
            cnr 1 0 0.26 0.00
...
In contrast, taking a translated article on Guang Cheng


Frequently used words in translated article on Guang Cheng that appear in cluster


      chen 14 133 2.57 1.20
guangcheng 11  29 2.02 0.26
        hu  9   8 1.65 0.07
   embassy  8  15 1.47 0.13
   beijing  6  17 1.10 0.15
      u.s.  6  20 1.10 0.18
        's  5  64 0.92 0.58
  activist  3  34 0.55 0.31
     china  3  46 0.55 0.41
    escape  3  31 0.55 0.28
      name  3   7 0.55 0.06
    rights  3  29 0.55 0.26
  security  3   7 0.55 0.06
government  2   7 0.37 0.06
     human  2  17 0.37 0.15
      said  2  12 0.37 0.11
      also  1   9 0.18 0.08
    arrest  1  19 0.18 0.17
     blind  1  30 0.18 0.27
      case  1   8 0.18 0.07
    family  1  16 0.18 0.14
   foreign  1   7 0.18 0.06
      free  1   7 0.18 0.06
     house  1  24 0.18 0.22
    lawyer  1  10 0.18 0.09
     legal  1   9 0.18 0.08
     linyi  1   7 0.18 0.06
      news  1  17 0.18 0.15
    people  1  14 0.18 0.13
    return  1  25 0.18 0.22
     state  1   9 0.18 0.08
     times  1  15 0.18 0.13
      wife  1   9 0.18 0.08


Frequently used words in translated article on Guang Cheng that do not appear in cluster


        mainland 7 0 1.28 0.00
         affairs 3 0 0.55 0.00
       political 3 0 0.55 0.00
          states 3 0 0.55 0.00
       yesterday 3 0 0.55 0.00
           enter 2 0 0.37 0.00
         evening 2 0 0.37 0.00
            hide 2 0 0.37 0.00
        informed 2 0 0.37 0.00
     questioning 2 0 0.37 0.00
          screen 2 0 0.37 0.00
         station 2 0 0.37 0.00
      2012-04-30 1 0 0.18 0.00
        addition 1 0 0.18 0.00
           armed 1 0 0.18 0.00
             ask 1 0 0.18 0.00
          assist 1 0 0.18 0.00
           audio 1 0 0.18 0.00
         butcher 1 0 0.18 0.00
         capture 1 0 0.18 0.00
           chase 1 0 0.18 0.00
         chasing 1 0 0.18 0.00
           clock 1 0 0.18 0.00

Uncle Wah and Ping Pong Rhymes

=== 1 ===

Earlier this year, Uncle Wah died. Uncle Wah was not my real uncle, but a prominent politician in Hong Kong. He founded a teachers' union in the 70s, helped draft the Basic Law in the 80s, and pushed for democracy in China and Hong Kong, most notably after June 4, 1989. Browsing memorial sites, I came across some of his short articles [0], published every third day in a respected Hong Kong newspaper. These short pieces cover a large number of topics - from poetry to politics, from meetings with his students to meetings with powerful world leaders - I find them to be a great read. To help me read, I pulled the articles to my Kindle.

If you read Chinese, feel free to ask me for a copy. If you don't, I hope to translate some snippets as time allows. Feedback/comments/alternate translations are most welcome. Below are two snippets from an article published 2003/12/31, titled "Couplets by Mr. Sun Yat-Sen" [1].

=== 2 ====

On a visit to Cheung Chi-Tung, governor of a southern province, the young Sun sent in a name card, with the following note:

"Dear Brother Tung, Student Sun-Wen requests an audience." [2]

Cheung was unhappy with that, and sent back these words on the card:

"Three lines of scribble for
One powerful duke
How dare a school boy claim to be my equal" [3]

This was obviously a Ping Pong Rhyme challenge [4]. Sun read it with a slight smile, and quickly wrote back:

"Ten thousand books read and
One thousand miles walked
More proud could be a peasant than a noble" [5]

Chung realized the visitor was no ordinary Joe, and hurriedly welcomed him in as if he were an important official.

---

In 1915, Yuan Shikai abolished the new Republic of China, and made himself King. Meanwhile, Sun married Soong Ching-ling in Japan, at the same time actively making plans against Yuan. One day, as the Suns took a stroll in a garden, their conversation turned to Yuan. Soong came up with an opening of a Ping Pong Rhyme for Sun to complete:

"To the Garden we go
To drive out the King
And revive our Home" [6]

This opening rhyme plays a neat trick with the way the words are written: word 3 (園 - garden) becomes word 11 (國 - country) by replacing word 8 (袁 - yuan) with word 1 (或 - perhaps/by chance).

Sun thought for a while, then said:

"On this path I walk
No turning back
No idle talk" [7]

Not only is this response a good match in sound and meaning, it also plays the same trick with word composition: word 3 (道 - road) becomes word 11 (途 - path) by replacing word 8 (首 - head) with word 1 (余 - first person reference: "I")

=== 3 ====

To check my translation, I sent English versions of the Ping Pong Rhymes to a few friends. My brother wrote back with another famous rhyme that I might have read a long time ago:

Among the rocks in the mountains is an old tree - this tree is firewood;
Besides the white splashes of the river stream is a good lady - young ladies are wonderful. [8]

There are neat word tricks for both lines: putting word 1 above word 2 becomes word 3, word 5 next to word 6 gets word 7, words 8 and 9 form word 11.

Isn't this amazing? Are there examples of anagrams in poetry?

In any case, I thought writing this up would be a good way to remember Uncle Wah, as well as the Centenary of the Xinhai Revolution of 1911.

[0] http://www.hkptu.org/szetowah/
[1] http://www.hkptu.org/szetowah/?p=1402
[2] Chinese people back in the day have many names - see http://en.wikipedia.org/wiki/Sun_Yat-sen
[3] 持三字帖，見一品官，儒生妄敢稱兄弟
[4] ping pong rhyme is my translation for duilian, also known as couplets. words (in all positions) have to match in sound and meaning.
[5] 讀萬卷書，行千里路，布衣亦可傲王侯
[6] 或入園中，逐出老袁還我國
[7] 余行道上，義無回首瞻前途
[8] 山石岩下古木枯，此木是柴; 白水泉邊女子好，少女最妙

Tool of the Month

The best applications not only solves real problems, they put a smile on your face. Here is a fantastic resignation letter generator for yahoo.

Honorable mention: PicLens

Bill Gate's last day

Today is Bill Gate's last day at Microsoft. New York Times has a nice article. Hate him or love him, the man has delivered solutions for people all around the world. And to dedicate his earnings and his time and energy to philanthropy, I for one admire him.

In a loosely related article, Wall Street Journal pokes fun at the relationship between Microsoft, Yahoo, and Google. A delightful read.

Google speller needs a grammar lesson?

Google is amazing. Among other things, their speller does a great job. Which is why it was shocking to me that the other day, when I was comparing Google and Powerset, I found an obvious error.

My query "who did leonard marry in yeomen of the guard" got the speller to suggest "who did leonard married in yeomen of the guard". Even a third-grader knows that that is wrong. While plain old n-gram may not fix that (bi-grams almost certainly cannot), the resulting document count may help: the original query had 92100 results, and the suggestion only 9640.

My tiny test with Powerset

As a small, anecdotal evaluation of Powerset, I tried two queries. On both queries, Powerset did not do as well as Google, yet there are promising signs of what PowerSet can do.

The first query was a variation of Ron Kaplan's example (see previous post) - to see who John McCain praised I used the query "McCain praises". The top two snippets showed that McCain praised Bush in 2004. The other snippets are somewhat problematic - the lexical parser seems to have ignored word ordering: "Tom Coburn [...] praising McCain" was returned as a result, so are "McCain was promoted" and "McCain's bill". The highlighting suggests that the words "promoted" and "bill" were matched with "praises". If done right, this could be a tremendous boost in recall.

In comparison, Google returned two recent instances of McCain praising someone. In June 2008, McCain praised Hillary and Jindal, the Governor of Louisiana. It appears that Google beat out Powerset, 2-1.

If you look closely at the top section of Powerset's result page, you will see a section under Factz: McCain praised Tatopoulos. Now who is this Tatopoulous? Unfortunately for Powerset, this factoid was taking from an article on the film Outlander. Interesting? Yes; Relevant? Not really.

Names can be very ambiguous. "Michael Jordan" usually refers to arguably the best basketball player who ever lived; yet in the information retrieval world, that name sometimes refer to a professor at Berkeley. In the same way, "McCain" refers to two persons. Can a system like Factz do a better job? Perhaps enumerate various ambiguities by tabs? That could be quite useful.

The second query was, I thought, a bit more tricky. In Gilbert and Sullivan's play, the Yeomen of the Guard, Colonel Fairfax married Elsie, but Elsie thought Fairfax was another guy, Leonard. With such a mess, perhaps the brains in Powerset will do a better job. My query, "who did fairfax marry in yeomen of the guard" is phrased in a question, which Powerset is built to handle. But yet again, Google came up with better results. The forth and fifth results, are relevant: "Elsie agrees to be Fairfax's bride", "The disguised Fairfax discovers that it is Elsie that he has married". Powerset, on the other hand, did not even give any female character's name from the play.

That may seem a bit pathetic, but notice that Google's results do not come from Wikipedia. Since Powerset only indexed Wikipedia, it can only do as good as Wikipedia. Google did not return anything from Wikipedia for that query on the first page, so perhaps that Wikipedia article was not very helpful. Clicking through from Powerset's result, a fancy outline of the Wikipedia article showed up on the right. In the bottom of that outline tool, there is a search bar - I typed in the word "marry", and sections on the page became highlighted. That by itself is nothing new, but if you look closer, the words highlighted are not all "marry" - instead, words such as "wedding", "marriage", and "wed" are matched. That is neat.

PowerSet - a recent talk

Recently I attended a talk by PowerSet's chief scientist, natural language processing guru, Ron Kaplan. He explained how usage patterns in natural languages (e.g. English) make bag-of-word search engines suffer in precision and recall. For example, to search for "who Obama critized" one may use the query words "obama critized". In that case, a bag-of-word search engine (e.g. Google) would return documents with the phrases "Hillary critized Obama" and "McCain critized Obama" - these are cases when a search engine returned results that were irrelevant,  a precision problem. In other cases, synonyms for "critized" are not matched, so documents with "Obama rejected" are not returned - this means relevant results are not returned, a recall problem.

He then went on to discuss what technologies PowerSet is built on: Lexical Functional Grammar, Transfer/Glue Semantics, Finite state morphology - technologies to learn/detect structure from free-form documents.  Crowd-sourced, structured datastreams are also used - freebase and wikipedia come to mind. By acquiring content (indexing terms with lexical/semantic information), packing ambiguity into compact states, a new generation of software is ready to improve for a wide range of tasks: summarization, question answering, translation, entity extraction... As for search, with less than 1 second in query response time, and less than 1 second per sentence in indexing time, he thinks natural language search is ready for prime time. Or at least, ready to convince investors to put in more money.

A handful of my co-workers also went to the talk. Most of them were underwhelmed. "Sure it was a good talk, but I heard it two years ago," said one. "I tried something similar to his example and it didn't work," said another. To be fair, PowerSet has been over-hyped - some thought it would be a Google Killer. And by only indexing wikipedia, it would not be far-fetched to say they under-delivered. However, in terms of bringing innovation to the table, and trying to improve search experience, I like their efforts. I won't put money on them yet, but I'll be watching.

Harry Potter Competition

ye poets in heart
write to win a thousand bucks
overcome black art

http://www.amazon.com/gp/feature.html?ie=UTF8&docId=1000207461

Guns and Democracy

As a democracy matures, she must let go of the diaper of violence. It may be true that the right to bear arms is still essential in the early days of a free State in the 21st century. Yet as the people of the State become more accustomed to the machinery of democracy, and as revolution by election becomes routine, tolerance for violence must be adjusted. In a mature democracy, where dissent is respected and encouraged in the chambers of debate, and efficient checks and balances are in place to prevent lunatic tyrants, proponents that fail to win the popular vote must not be allowed to infringe on anyone's right to life. As such, the right to bear arms by individuals would have served its purpose, and must be allowed to retire.

Random newspaper stories

Patriots head injury

Super Coaches with Character

Court martial

Global Warming - Scientists silenced?

Tool of the Year

If anyone had any hope that this award was unbiased, wake up! Check out this year's best website: http://www.michiganupdate.com. In a few days you will see a much nicer layout, but the content itself should give you hours of joy, if you are a Michigan fan.

About this website

In the twelfth year of Lloyd Carr's reign of Michigan Football, a few fans saw a flaw in the system. No, the flaw was not with Michigan football - the State of the Program was strong. Rather, the flaw was with the internet. With blogs, tubes, web 1.0 sites, and bulletin boards, there are now so much stuff about Michigan Football that trying to read up on the team effectively is more difficult than understanding Jim Herrmann's playbook.

To tackle this problem, MichganUpdate was born. Now, available at your fingertips, is an up-to-date selection of stories about Michigan sports around the internet. With a proprietary algorithm somewhere between the complexity of the BCS and PageRank, MichiganUpdate clips and sorts the information you want, so you can just kickback and relax.

REIT 1: AFR

Looking at AFR, an REIT, the first thing that caught my eyes was the 9.4% dividend. Then I looked at the 5-year graph, which looks rather predictable.



If I bought it in Jan 05 at $16, and sold it at $12 in Jan 06, the dividend had better be greater than 25%. I wonder what the deal is with this REIT.

My portfolio

Lately I've been thinking more about my investments. When I was an intern, I put some money away in money market accounts. After I got out of school, I started putting money into indexed funds. Since my investment and retirement accounts are spread out among a few institutions, I never quite knew how my portfolio was doing. A few days ago I collected data from my accounts, and plotted these two charts:





The legend is made up of [fund name, percent] pairs. Fund names are either sticker symbols, cash accounts (sbmm, emdir, fdrxx, boa, compa, nati), or funds restricted/chosen by 401K companies. I think it is pretty obvious that I should move some cash into better investment vehicles. Perhaps reit or house?

Bowling 2.0

Went bowling last week for the first time in many years. The game itself is still quite boring, although it was great meeting entrepreneur-type folks. I even made it to the video!

Free PC to anything calls

The sound quality on Skype seems a little bit better, but Gizmo is free*

Mac tricks

Here are some mac tricks i recently learned:

1. command-tab switches between applications, command-` switches between windows of the same application. ctrl-f4 does the same, but it's much quicker to hit command-`.

2. scrolling on firefox is better done with space, shift-space than page up (fn-up), page down (fn-down). it actually works on other major browser as well.

3. active four corners is fun. now that i just point my mouse around to use expose, i actually use it a lot.

4. setting the wallpaper to change every 5 seconds is nice, especially when an active corner is set to show desktop.

5. instead of ctrl-click, try "tap trackpad using two fingers for seconary click"

6. move applications folder to the dock, combined with trick 5 this makes for fast application starting. there is a similiar free keyboard tool as well.

7. when using garage band, window -> music typing lets you type to write music...

Tool of the Month

The award goes to...Stumble Upon. You could surf the web, search for goodies, or stumble upon sites. This is not new, but it's still good.

Runner up: Mashable. This is a good blog about new web tools. I like it.

2nd runner up: Free anagram server. I stumbled upon this old site.

Memorial Day Special - My violin teacher, Mr. L

[Yesterday was Memorial Day. To a lot of people, myself included, that might just as well be called Barbeque Day. Chatting at a barbeque yesterday, I somehow brought up my old violin teacher, Mr. L, whom I have not seen for over 10 years. It didn't cross my mind at the time, but it struck me later on, that Mr. L fought in the Vietnam war. In fact, he is the only veteran that I know personally. It is with fond memories that I write this.]

=====

Mr. L was an esteemed violinist. He was the First Assistant Concert Master for a cosmopolitan philhamonic orchestra. What that means is he plays violin for a living in a big band. He doesn't just play, he is the third in command. If the conductor and the concert master dies in a concert, he's the one resposible to lead them through the rest of the music. He would be the commander-in-chief. I don't think that has ever happened, just like there has never been a Speaker of the House that sat in as President. What did happen was when the concert master was out, Mr. L would lead the orchestra through rehearsals and concerts, and play solo.

Some people are scared to play solo. I have never been scared of that. If you know what it means to be in crossfire, to face bullets and dodge land mines, you won't be afraid to play solo. What's the worst thing that can happen to you? You make a mistake, you miss a note, you miss a beat, you are still alive. When someone is trying to kill you, you make a mistake, you die.

Mr. L is a decorated veteran. He's got a bronze star and a purple heart. He always says he's not worthy of the medals. He says the people who are worthy are dead. One day in Vietnam, he left his unit to relieve himself. On his way back, he saw four communist soldiers close by. It was either them or him, someone had to die. Well, he didn't.

To this day I get nightmares about those four people. I know I had no choice. If I didn't kill them, they would have killed me. It's not like we could have sat down for a cup of coffee and then go our separate ways. But I still feel - I'm not sure what I feel - it just feels wrong to take someone else's life. I mean, these guys are people too. They have families just like us. They are husbands, fathers, sons, boyfriends. I never wanted to kill any one.

=====

I first started taking lessons from Mr. L in 3rd grade. I couldn't understand half of what he said, so for the first couple years, my dad would come with me and translate. Most of Mr. L's students called him by his firstname; my dad insisted on calling him Mr. L. So I also called him Mr. L. When I first started, my dad half-joked that we were getting a good deal - on top of a violin lesson, I was getting an English lesson for free.

When I was in 7th or 8th grade, Mr. L's wife divorced him. Not long afterwards he got into an accident - some people said he got drunk and fell down the stairs. In any case, he couldn't play the violin any more. His students left one after the other, and in the end, I was his only student. My violin-english lessons then became mostly English lessons. I would get about an hour's worth of violin lesson, mingled with a few hours of lecture on whatever happened to be on his mind. These ranged from music theory, economy, religion, to politics. I never could guess what he was going to talk about, but he could always relate these random thoughts back to the Brahms concerto that I was learning. And he would always mention something about Vietnam.

=====

Most music teachers don't like their students playing sports, especially high-risk sports like basketball and volleyball. In these games, fingers are routinely jammed, and months of musical training could all go down the drain. Mr. L was different. He couldn't care less.

Don't worry! Go play! Your fingers will be fine. Besides, at your age, the important thing is to find something that you love so much, that you are willing to die for it. Martin Luther King said that you have not truly lived if you do not have anything that you are willing to give up your life for. What kind of things are worth dying for? Probably not violin or volleyball; some people say Freedom, Justice, Faith. But that is something you have to find out for yourself.

=====

Mr. L is an American. Unlike other teachers, he was always positive and encouraging. I had a report card from the music school one year, and the examiner said my intonation was poor in one of the pieces. I showed Mr. L the report card and he just laughed.

I know this guy. Trust me, he doesn't know what he's talking about. Your intonation is fine. If anything, this guy is jealous of your vibrato. I told you you are gifted with the virbrato. Next time you play this piece, just remember to show off your vibrato.

I don't think I thought twice about that - but I am quite sure the examiner was not jealous. Even now, my intonation is not the best, and I could only imagine how bad it could be back then. Vibrato, by the way, is the best cover up on the violin if you missed a note.

Not all teachers are like that. After my wining performance in a solo competition, another teacher with Soviet Union and Chinese backgrounds came up to me and said, "Your slow movement was too slow, too dragging."

The Chinese has a saying that "the deeper you love, the harder you criticize". If Mr. L were Chinese, he didn't like me at all.

=====

In my 8 years with Mr. L, his life changed a lot. His wife left, he lost his livelihood, and all his students left him. Yet despite all these horrible things, I have never seen him angry at anyone. The only time I thought he was angry was when Clinton became President.

I don't understand how anyone could vote for Clinton. The man is a coward. I don't think he should even be called a man. You have respnsibilities as a citizen; if you don't get them done, you are not fit to become President. When you country needs you to go to war, you don't run away, and come back to become commander-in-chief. If you are too chicken to be shot at, you can't decide if anything is worth sending someone else's kid to get shot at. If you don't understand the consequences of war, you can't make the decision to go to war.

I don't know what Mr. L has to say about the war on terror; he never said whether he thought Vietnam was justified as something worth dying for. I also wonder, if he were to meet with the current commander-in-chief today, what he would have to say.

Buy groceries online!

Do you know that Amazon started selling groceries online? Well, now you know. And not just any groceries, organic groceries.

Search for "Gatorade", and you can find some pretty sweet deals. Don't say I didn't tell you.

Firefox keyboard shortcut

I'm a big fan of keyboard shortcuts, vi, gmail, alias...Now thanks to a friend at work, I am more efficient at browsing as well. Especially when combined with search strings, these firefox shortcuts are great.

This is all you have to do:
1. copy/change the html below into a file ending in .html
2. select Bookmarks > Manage Bookmarks in Firefox
3. select File > Import... in the bookmark manager
4. select "From File"
5. find your file and continue

After you are done, you can type "s love" on your browser address bar to find love.

<dt><a href="http://a9.com/%s" shortcuturl="s">A9.com >Search
</a></dt><dt><a href="http://www.google.com/search?q=%s" shortcuturl="g">www.google.com > Search</a>
</dt><dt><a href="http://dictionary.reference.com/search?q=%s" shortcuturl="d">www.dictionary.com</a>
</dt><dt><a href="https://mail.google.com/mail" shortcuturl="m">mail.google.com/mail</a>
</dt><dt><a href="http://bloglines.com/myblogs" shortcuturl="b">bloglines.com/myblogs</a></dt>