April 28, 1999

Changes in Search Industry Create Strange Bedfellows

By LISA NAPOLI

he announcement last week that Netscape Communications Corp. had agreed to license, at no charge, its Open Directory Web guide to a competitor, Lycos, left a lot of people in the industry scratching their heads.

Related Article Alta Vista Invites Advertisers to Pay for Top Ranking (April 15, 1999)

Last November, Netscape paid an unspecified sum for the Open Directory, a guide to the Web created by two computer programmers in California that is now maintained by 8,900 volunteers. At the time, Netscape announced it would make the Open Directory available to other Web sites that wanted to use it, free -- perhaps to avoid alienating the unpaid staff contributing to it, industry watchers said.

But by allowing Lycos or any other search service to use -- free -- a service it paid for, Netscape is essentially financing the competition.

"It's like Wal-Mart selling to CVS," said Danny Sullivan, the editor of Search Engine Watch, an online newsletter. "This isn't the concept of co-opetition, where they're competing with each other but also helping each other. This is just weird and strange."

Whether or not it makes sense for either party, the deal points to a key challenge facing search services in the current competitive landscape: how to keep up with the exponential growth of the Web, while at the same time building a profitable business.

As search services have refashioned themselves as "portals," focusing on revenue-generating deals like partnerships with e-commerce companies and advertisers, their strategy for developing search features has changed. Rather than trying to offer access to the broadest number of sites on the Web, as they once did, the leading search services now focus on offering the most discerning guides to the Internet. To do so, most employ staff members who sift through site submissions and decide whether and where to include them in their directories.

But even the most profitable search service could never afford to pay a staff large enough to keep up with the 5.4 million sites that are now estimated to be online, not to mention address changes and dead links. As a result, they are looking for ways to economize. Many are partnering with other search companies to gain technology or content they have not developed in house -- in some cases, a search engine (technology that enables users to type in a query and search a database) in other cases, a directory (which allows users to browse through categorized listings).

For instance, AltaVista, a search engine, has partnerships with LookSmart, a human-powered directory with a staff of 160 editors, and Ask Jeeves, another human-powered search tool. Meanwhile, Excite licenses its search technology to both America Online and Netscape, which is now owned by AOL, and the Walt Disney Co.'s Go site uses Infoseek's search engine.

Top 10 Search Sites 1 Yahoo.com 31,019 2 Go.com 21,184 3 Excite.com 16,652 4 Lycos.com 16,102 5 Altavista.com 10,458 6 Snap.com 9,761 7 Hotbot.com 7,443 8 Looksmart.com 7,766 9 Goto.com 4,072 10 Webcrawler.com 3,252 All figures in (1000s) Source: Media Metrix Media Metrix Definitions:

By using the Open Directory, Lycos has taken the partnership frenzy one step further, relying not just on another company but on a community of volunteers.

Freeing up staffers to do other work was a major impetus in the decision to use Netscape's Open Directory said Andrew de Vries, a spokesman for Lycos.

"It didn't make sense to scale the directory to the growth of the Web with the in-house staff," de Vries said. "Now we don't have to add 300 people each year to keep up."

In the future, both Lycos and HotBot, another search service it owns, will use listings created by the volunteers who contribute to the Open Directory, de Vries said. The attentions of the company's 60 editorial employees will be redirected toward developing "premium content" in categories like finance, shopping and news -- areas where the services can really distinguish themselves, he said.

While a fluid staff that works for no pay may seem like an easy -- and economical -- solution to the complex problem of simultaneously maintaining a directory and the bottom line, not everyone in the search business supports the idea of relying on volunteers to create a useful Web guide.

Executives at several other major Web directories were quick to argue that volunteers, no matter how passionate and attentive they are, cannot offer the quality of work that their paid, trained editorial staff members do. But they did acknowledge that they could never hire as many people as the Open Directory has working for it.

"We have a finite set of resources to manually editorially filter, aggregate and present information," said Srinija Srinivasan, vice president of editorial for Yahoo. "We couldn't possibly scale head count to the growth of the Web. Is that a bug or a feature? For us, we've felt it's really been a benefit. What we can do with a tightly integrated team is put together a tightly integrated directory."

For competitive reasons, Srinivasan would not disclose how many people compile the site's directory. However, a recent move by Yahoo illustrates how difficult it is for the staff to keep up with the volume of Web site submissions Yahoo receives. In February, the company debuted an "express" service, allowing commercial sites to pay a $199 fee in order to expedite consideration of their site for inclusion in Yahoo's directory.

Shawn Hardin, a senior vice president at Snap, which employs 60 people to create and maintain its directory, also defended the "quality, not quantity" approach, saying users want fewer results when they search. "You want the best stuff that represents a given category," he said. "You want relevant results."

The landscape of the search industry could become even more intertwined if the commercial search services change their opinions about the value of an Open Directory and decide that they, too, will use it. Since trends on the Web shift almost as quickly as Web sites are created, the next step in the competition for information-searching surfers could come soon.

In the meantime, there are no easy or immediate answers to the problem of how to tame the unwieldy information beast that is the Web.

"In one way, it's a good problem to have, because it says, 'Gee, there is a lot of interest in the Web,'" said Harley Manning, an analyst with Forrester Research. "But it's a bad problem in that if I try to find something, it's impossible. If the information is on the Net and I could never find it in a million years, it might as well not be there."

---

Related Sites
These sites are not part of The New York Times on the Web, and The Times has no control over their content or availability.

• Netscape Communications Corp.

• Open Directory

• Lycos

• Search Engine Watch

• HotBot

• Yahoo

• Snap

• AltaVista

• LookSmart

• Ask Jeeves

• America Online

• Go.com

• Infoseek

---

Lisa Napoli at napoli@nytimes.com welcomes your comments and suggestions.

---