Compiling the history of search engines, how they evolved into the giants we see today. The milestones in the search history are penned down in this article showing the exponential growth of web search market.
How Search Engines Evolved
!!Waiting For Infographics!!
Even before the World Wide Web, there were search engines that attempted to organize the Internet. The first of these was the Archie search engine from McGill University in 1990, by Alan Emtage. He built a program that downloaded the file directory structure on local databases creating search-able index. It didn’t index the content of the files.
The rise of Gopher (created in 1991 by Mark McCahill at the University of Minnesota) led to two new search programs, Veronica and Jughead. Like Archie, they searched the file names and titles stored in Gopher index systems. Veronica (Very Easy Rodent-Oriented Net-wide Index to Computerized Archives) provided a keyword search of most Gopher menu titles in the entire Gopher listings. Jughead (Jonzy’s Universal Gopher Hierarchy Excavation And Display) was a tool for obtaining menu information from specific Gopher servers1991 by WAIS and Gopher. All three of those systems predated the invention of the World Wide Web but all continued to index the Web and the rest of the Internet for several years after the Web appeared. There are still Gopher servers as of 2006, although there are a great many more web servers.
1993, Matthew Gray, at MIT, produced what was probably the first “web robot”, the Perl-based “World Wide Web Wanderer”, and used it to generate an index called ‘Wandex‘. The purpose of the Wanderer was to measure the size of the World Wide Web.
The web’s second search engine “Aliweb” appeared in November 1993. Aliweb did not use a web robot, but instead depended on being notified by website administrators of the existence at each site of an index file in a particular format.
As the Web grew, search engines and Web directories were created to track pages on the Web and allow people to find things. The first full-text Web search engine was WebCrawler in 1994. Before WebCrawler, only Web page titles were searched. Acquired many times with last in 2001 by InfoSpace.
Another early search engine, Lycos, was created in 1993 as a university project, and was the first to achieve commercial success. Soon after, many search engines appeared and vied for popularity. These included Magellan, Excite, Infoseek, Inktomi, Northern Light, and AltaVista. Yahoo! was among the most popular ways for people to find web pages of interest, but its search function operated on its web directory, rather than full-text copies of web pages.
During the late 1990s, both Web directories and Web search engines were popular—Yahoo! (founded 1994) and Altavista (founded 1995) were the respective industry leaders.
By August 2001, the directory model had begun to give way to search engines, tracking the rise of Google (founded 1998), which had developed new approaches to relevancy ranking. Directory features, while still commonly available, became after-thoughts to search engines. Database size, which had been a significant marketing feature through the early 2000s, was similarly displaced by emphasis on relevancy ranking, the methods by which search engines attempt to sort the best results first. Algorithms for relevancy ranking have continuously improved. Google’s PageRank method for ordering the results has received the most press, but all major search engines continually refine their ranking methodologies with a view toward improving the ordering of results.
Yahoo! acquired Inktomi in 2002, and Overture (which owned AlltheWeb and AltaVista) in 2003. Yahoo! switched to Google’s search engine until 2004, when it launched its own search engine based on the combined technologies of its acquisitions.
Microsoft first launched MSN Search in the fall of 1998 using search results from Inktomi. In early 1999 the site began to display listings from Looksmart blended with results from Inktomi except for a short time in 1999 when results from AltaVista were used instead. In 2004, Microsoft began a transition to its own search technology, powered by its own web crawler (called msnbot).
As of 2006, search engine rankings are more important than ever, so much so that an industry has developed (“search engine optimizers”, or “SEO”) to help web-developers improve their search ranking, and an entire body of case law has developed around matters that affect search engine rankings, such as use of trademarks in metatags.
Microsoft’s rebranded search engine, Bing, was launched on June 1, 2009. On July 29, 2009, Yahoo! and Microsoft finalized a deal in which Yahoo! Search would be powered by Microsoft Bing technology.
The algorithms are continuously changing since the beginning of web search. With the advent of newer technologies and RIAs, search engines are adapting more rapidly and more competitively. The search market is huge, and has been mostly monopolistic with 80% of search share being with Google at its peak in Dec 2008. With universal search into the market search engines are now making a shift to unify all the media present on the net while searching for relevant results.
