Tricking Web's `spiders'

Competition: Web sites owners try to manipulate search engines meant to find information based on relevance.

September 05, 2002|By Bethany Halford | Bethany Halford,SPECIAL TO THE SUN

To anyone who's struggled with the Dewey Decimal System in a library, an Internet search engine looks like a gift from heaven. Just type in whatever you're looking for, and seconds later, a list of relevant Web pages appears.

But how relevant are those pages? And why do the sites at the top of the list show up where they are?

Thanks to competition for top search engine rankings and the economics of the Web, you might not always see links to sites with the best information, but sites with the biggest bankroll or craftiest programmers. Increasingly, merchants are paying search engines for top placement or tinkering with the code of their Web pages to fool search engines into putting them atop the lists.

Type "George W. Bush" into the search prompt at AltaVista, HotBot, or iWon and you'll get a million hits.

But high on each list, long before the White House Web site or the Republican National Committee home page, you'll find a link to Political-, which sells "Lub Ya Dubya" bobblehead dolls bearing a likeness of the commander in chief.

In small print above the top listings are the words "Sponsored Links," a euphemism meaning that the Web site operator or broker has paid the search engine to put the site at the top of the list.

The trend has consumer groups concerned, and the Federal Trade Commission has warned search engine operators to label advertising-supported links clearly.

The stakes are big because search engines are so popular, and many users rely on them to find information. Given a specific term or just a keyword, a search engine will sift through an index of hundreds of millions of Web pages to produce a list of links.

Web page designers say they have to break into the top 30 sites on a search engine list if they want to draw visitors. With surfers ignoring banner ads and newer pop-up windows, search engines are making the most of their position by charging Web sites for top placement when a user enters specific terms.

Tom Steinhagen, who operates Political Pizazz with his wife, Valerie, contracted for the "George W. Bush" placement with Overture Services Inc., which acts as a middleman between advertisers and search engine operators.

Steinhagen pays 7 cents every time someone clicks on one of his sponsored links. He originally listed his company under keywords such as "bobbing head doll." But when stiff competition in the trendy bobble-head market drove the price up to 25 cents per click, Steinhagen shifted his keyword listing to the president's name, which was cheaper.

"We thought it was more cost-effective," he said.

When deciding on the order in which it displays sponsored links, a search engine might take into account the actual relevance of the site. But driving force is still money.

"The more that you've paid, the more likely you are to rank higher," said Danny Sullivan, editor of, an online industry publication. During peak shopping seasons, the cost for some gift-oriented listings can soar to over a dollar per click.

Steinhagen said shifting to the "George W. Bush" keyword has increased the number of visitors, but not his overall sales. So he's pondering yet another change. The key, he said, is picking the right keywords and search engine optimization service.

Otherwise, he said, "It's kind of like advertising when everybody's television is off."

Until recently, "sponsored listings" appeared under more ambiguous names such as "featured listings" or "recommended sites." Last year, the advertising watchdog group Commercial Alert filed a complaint with the Federal Trade Commission.

"This is a bread-and-butter issue for us," said Gary Ruskin, Commercial Alert's executive director. "We're trying to defend the advertising-editorial line."

Commercial Alert said offending search engines that had gained credibility by ranking sites on content were now posting paid links in exactly the same format with no indication they were ads.

The FTC took no formal action but did send letters to the offenders, warning them to clarify which hits were ads or face possible sanctions.

While a few search engines, including popular, still list paid results brokered through Overture with no special markings, most sites heeded the FTC's warning and indicate which are "sponsored."

Ruskin considers the change a victory: "`Sponsored' means advertising. Everybody understands that."

Outright advertising isn't the only way to climb in the rankings. Many Web site programmers expend considerable effort to attract or even fool search engine "spiders." These are programs deployed by Google and other "crawler" search engines that prowl the Web, looking for new sites and hunting for keywords that form the basis of the search engine's index.

Baltimore Sun Articles
Please note the green-lined linked article text has been applied commercially without any involvement from our newsroom editors, reporters or any other editorial staff.