Powerset for Wikipedia search

When I first started browsing the web, I used AltaVista. Later, I switched to Yahoo! and when Google came along, I changed to it and have since found no need to use any other search engine.

Many search start-ups like Wikia Search, Cuill and Hakia have emerged to try to become the next Google but so far, to no avail. But it's not just the small players trying to get a piece of the search action. Both Microsoft and Yahoo! have tried, unsuccessfully, to challenge Google by refining their search engine.

To be fair, many of these search engines do a pretty good job of searching the web. The efforts of Microsoft and Yahoo!, in particular, produce results that are arguably as good as Google's. But if their results are only as good as Google's, why should I switch from something I'm used to?

Last week, a new kind of search engine was launched. It's called Powerset and its mission is to provide a better way to search, using "natural language". That means instead of typing in keywords like "Oon" and "Yeoh", you type in questions like "Who is Oon Yeoh?"

The service is still in its infancy and it doesn't even attempt to index the World Wide Web. For now, its search is confined to content on Wikipedia. I know this seems incredibly underwhelming but think about it. There is a real need for a good search engine for Wikipedia.

First, Wikipedia's built-in search engine gives new meaning to the word "inadequate". Have you tried using it? It's barely usable. Second, Wikipedia is such an important and useful source of knowledge and information that it has become pretty indispensable for people working in the knowledge industry.

To give accurate results to your questions, Powerset has created an index of Wikipedia by studying the meaning of whole sentences rather than keywords.

I decided to give it a spin and typed in the navel-gazing question "Who is Oon Yeoh?"

Here's what I found. It gives a pretty good summary of my career as a journalist but it wrongly lists my country as "China".
What's really interesting are the other articles in Wikipedia where I'm mentioned for one reason or another.

For example, under "Khairy Jamaluddin" I'm mentioned, together with Jeff Ooi, as labelling him "the new Malay-Ultra".
Under the entry "Anwar Ibrahim", there is a link to an article entitled "The Return of Anwar Ibrahim", which I wrote for a now-defunct magazine called Chrome.

An article I wrote for theSun, entitled "Meritocracy: The Truth Must Be Well Told", is also referenced in the Wikipedia entries on "Social Contract (Malaysia)" and "Article 153 of the Constitution of Malaysia".

All this is new — and amusing — to me!

I will probably use Powerset whenever I have questions to which I think Wikipedia can provide an answer. For other purposes, I'll stick to Google. But Powerset has shown that it has great potential. In a sense, what it has done with Wikipedia is a type of "proof of concept".

Wikipedia is a highly organised, highly structured website. This makes it relatively easy for Powerset to analyse and index the meaning of the content it finds there. To do this for the entire web is a different kettle of fish. It's an undertaking that seems almost too challenging.

To manage expectations, Powerset's CEO Barney Pell has said Powerset is not meant to be a Google killer. Instead, it will focus on "high value" websites, such as Wikipedia. There's speculation that it will next index blogs or news articles.

If it can do that well, it would be a major achievement in itself but there is still the question of whether people would actually bother to use it, given that Google is so well entrenched.

Natural language search has the potential to produce more accurate and more interesting results but for most people, keyword search is more than adequate. Thus, Powerset is no replacement for Google. If anything, it complements it.

Perhaps, Google should just buy it up. Perhaps, that's just what Powerset wants.

Comments

[John Lim]

I'll bet the staff at Powerset wait everyday at their phones for a phone call from Google. If you can't beat 'em, join 'em. It'll probably be chump change for them. The Guardian did an interesting piece on something similar about Google's precarious position in the internet search business, and adding Powerset would certainly increase its core competency.
http://www.guardian.co.uk/technology/2008/may/22/internet.google