"Search engines are the key to finding specific information on the vast expanse of the World Wide Web," explains Webopedia. There are robot-like engines called "web crawlers" and human-powered search engines that submit data to be indexed.
Each search engine visits a website, reads the information on it, follows any links that connect that site to others, and uses an "algorithm" to measure the relevance of the material it finds to what the online searcher appears to be looking for. Periodically, the crawler returns to the site to see if any of the information has changed.
When you type in a phrase on Google (or Yahoo, MSN, About.com, etc.), you’re actually searching the index for that phrase.
Remember me saying in a former blog post that business blogging uses both science and art? Proper use of keywords to enhance Search Engine Optimization is the "science" part the equation; writing interesting and relevant content for the blogs is where the "art" comes in. In fact, the reason ghost blogging is so fascinating a profession for me is that very combination of art and science.
One of my favorite "reads" is a magazine called MentalFloss, and a recent article dealt with modern technologies whose discovery was based on studying the animal kingdom.
A featured creature was the lobster. A new x-ray technology is based on the way lobsters see things. X-rays don’t like to bend, and are therefore difficult to manipulate. The only way we have been able to scan bags at airports or view bones and organs is by using very large machines that emit a torrent of radiation.
Lobsters, crawling 300 feet below the surface of the ocean, have more efficient x-ray vision. They can take different reflections and focus them together to a single point, forming an image by "bending" those reflections. (The Department of Homeland Security has already invested millions in the LEXID – Lobster Eye Xray Imaging Device – they hope LEXID will find contraband).
In a way, web crawlers use a LEXID-like process to gather many bits of information from many different locations on the World Wide Web and index that information into one source. Web crawlers are the ultimate lobsters of the Internet!