Future of Search: How Twitter Serves Real Time Results

twitter search

Remember this Tweet about Flight 1549 crashing into the Hudson River?


How about this one about Osama bin Laden getting killed:


Or this one when President Obama was re-elected?

What do they all have in common? (besides the conspiracy theories!)

They all broke on Twitter.

Twitter’s engineering department recently explained how they use humans to help improve their search results.

In a nutshell:

  1. They monitor which search queries are currently popular using Storm topology that tracks statistics on search queries.

    Real World Example: the query [Big Bird] may suddenly see a spike in searches from the US.

  2. As soon as a new popular search query is discovered, it’s sent by “Storm” via “Thrift” to Amazon’s Mechanical Turk for human evaluators, to answer a variety of questions about it.

    Real world example:  As soon as Twitter notices “Big Bird” spiking, they may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps Twitter serve relevant Tweets and ads.

  3. After responses from evaluators are received, the information is pushed to Twitter’s backend systems, so that the next time a user searches for a similar query, the machine learning models will make use of the additional information to provide better search results and ads.

    Real World Example: suppose evaluators tell Twitter that [Big Bird] is related to politics; the next time someone performs this search, Twitter knows to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.

Watch this catchy crowdsourced singing telegram created to celebrate the launch of the project.

My $0.02

Very impressive and scalable.

For years, Google et al. have tried to crack the real time search nut but to little/no avail.  Twitter just may have provided the answer and taken the lead on shaping the future of search.

Well done guys.

If you enjoyed this post, make sure you subscribe to my RSS feed!

New Age AdMan & proud father of 2 incredible kids. Omar is Chief Strategy Officer at Sandstorm Digital; a new age brand communications agency. Omar has over 20 years experience in marketing and advertising, the last 10 being in digital within large media agency environments. During that time, Omar developed and implemented cutting edge digital campaigns for some of the world's best known brands. For the latest in digital marketing and industry news updates, follow Omar's Twitter stream (@OmarKattan) or add him to one of your Google+ Circles: Omar Kattan's Google+ Profile.

Share This Post

Related Articles

Powered by WordPress · Designed by Theme Junkie