Remember this Tweet about Flight 1549 crashing into the Hudson River?
http://twitpic.com/135xa – There’s a plane in the Hudson. I’m on the ferry going to pick up the people. Crazy.
— Janis Krums (@jkrums) January 15, 2009
How about this one about Osama bin Laden getting killed:
Helicopter hovering above Abbottabad at 1AM (is a rare event).
— Sohaib Athar (@ReallyVirtual) May 1, 2011
Or this one when President Obama was re-elected?
Four more years. twitter.com/BarackObama/st…
— Barack Obama (@BarackObama) November 7, 2012
What do they all have in common? (besides the conspiracy theories!)
They all broke on Twitter.
Twitter’s engineering department recently explained how they use humans to help improve their search results.
In a nutshell:
- They monitor which search queries are currently popular using Storm topology that tracks statistics on search queries.
Real World Example: the query [Big Bird] may suddenly see a spike in searches from the US.
- As soon as a new popular search query is discovered, it’s sent by “Storm” via “Thrift” to Amazon’s Mechanical Turk for human evaluators, to answer a variety of questions about it.
Real world example: As soon as Twitter notices “Big Bird” spiking, they may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps Twitter serve relevant Tweets and ads.
- After responses from evaluators are received, the information is pushed to Twitter’s backend systems, so that the next time a user searches for a similar query, the machine learning models will make use of the additional information to provide better search results and ads.
Real World Example: suppose evaluators tell Twitter that [Big Bird] is related to politics; the next time someone performs this search, Twitter knows to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.
Watch this catchy crowdsourced singing telegram created to celebrate the launch of the project.
My $0.02
Very impressive and scalable.
For years, Google et al. have tried to crack the real time search nut but to little/no avail. Twitter just may have provided the answer and taken the lead on shaping the future of search.
Well done guys.
