Open source browser development may have hit a snag lately, but the same concept applied to indexing the Web looks like it's catching on.
Lycos has adopted the Open Directory Project, a directory of Web sites compiled and managed by an army of 8,000 volunteer subject editors.
"Unlike other Internet directories that rely on a small staff of paid editors," proclaims Lycos, "the Open Directory Project leverages the unique insights of a wide variety of Internet users to build the best directory on the planet."
Lycos, Mozilla.org, and the HotBot search engine collaborated on the project. Lycos is in the process of acquiring Wired News parent company Wired Digital, which also owns HotBot.
Some call the idea an "open source" directory, since it depends on the same spirit of open software development used by Mozilla.org, developers of the Apache Web server, and others. In this case, a wannabe editor chooses a topic and joins at the open directory site.
Netscape's Netcenter incorporates the directory into its listings. But Netscape also licenses out the service, similar to the way the company licenses out the Mozilla browser software.
"We believe this validates and endorses the open directory model as a standard for directories on the Internet," said Chris Tolles, a Netscape senior product manager, in a statement.
"This is the largest endoresement of our model so far."
Lycos said that its Open Directory represents the largest human-edited Internet directory in the world, with over 54,000 topics and 8,000 editors.
HotBot will also add the Open Directory Project to its listing index.
Bomis.com, a smaller Web directory, made a similar move last month by licensing a portion of the Open Directory Project. The company said the move increased the listings in its index to an amount 10 times its current volume.
"By utilizing the Open Directory category structure, Bomis gets the benefits of a great category structure, while helping grow the directory by posting the way to become an editor on their site," said a company spokesman at the time.
The Open Web Directory originated as NewHoo -- a direct swing at Yahoo. NewHoo launched with a plan to build the largest, most active Web directory "on the planet."
Like Yahoo, the Open Directory Project is a directory-based index of Web sites, whose link listings are selected and categorized by actual humans. But in contrast to Yahoo's staff of in-house editors, the service is maintained by volunteer indexers from around the Web.
"If the goal is to get to pages on the Web and somehow catalog every document on the Web, then I agree that a contained finite approach will not scale," said Srinija Srinivasan, Yahoo's director of surfing. "For us, the goal has always been to deliver what users want. And we believe that what we're doing works."
Instead of trying to catalog every Web site out there, Yahoo says its group of between 50 to 80 "surfers" shares collective knowledge to be more discerning about the sites that are catalogued.
Indeed, even as the Web continues its massive growth, Yahoo has been criticized for falling behind in new listings, as well as not doing enough to add new sites or modify existing ones. Critics say the site needs to be more up-front to users about the content of its directory.
Tolles says Yahoo cannot keep up.
"Their model won't scale. We belieive this is the only way to scale. The only way to scale is through harnessing the energies of the Web itself.
And Tolles claims the Open Directory Project has less than one quarter of one percent of dead links in its directory of 430,000 sites in 65,000 categories.
