Fixing The Open Directory Project?

People who think they know how to fix something should make sure they have the experience to understand how it worked before they judge it. I was annoyed to read a post about how to fix the problems with dmoz, The Open Directory Project on DirPopulus today. Here is what I wrote:

I read your Problems & Solutions. Some of what you have written about dmoz is incorrect and based on the viewpoint of someone who did not see how the directory worked in reality. Although I understand your biased point of view, it is annoying to read someone making incorrect assumptions and judging the directory I spent over ten years working on.

Looks like you are using the dmoz software, or something based on it. So, that won’t help me really. I don’t want to deal with that. We had volunteer editors trying to fix that, not staff. AOL decided to dump dmoz because no one there was interested in supporting it. For the last few years dmoz was run entirely by volunteers while the AOL staff forgot the directory existed. In the end they did not find any value in keeping it on their servers but they did feel the domain and the dmoz/ Open Directory name were worth holding onto.

The main directory, with some active editors, is being set up on Curlie. Other projects were started and discussed but that is the one which has the best chance of becoming active again. Most of the volunteers who worked on the old dmoz software went to Curlie and have been working on the updating the software.

As volunteers we did not send out notifications every time we reviewed, edited, or added a site to the directory. We were already running with few active editors so trying to send out notices for every submission would have meant the end of getting any reviews done. Waiting for three editors to approve (while good in some ways) would also mean submitted sites would take ages to be listed.

We did have bots checking links and moving them into unreviewed for volunteers to check the links. Some bots were able to check for things like the new http:// versus https:// so an editor just needed to verify the change and re-list the site. We also had bots which checked for general link rot and expired domains. These doubled the amount of links to be reviewed leaving editors which huge amounts of links waiting for attention. Also, dmoz had a feature giving people a chance to leave a note about their link, letting us know if a correction was needed. This was a very seldom used feature and yet the first thing I would check when I began reviewing links in a category. Often this was abused and suggested changes were about spam, deleting another site’s listing, or some other junk.

Also, we were able to check links with the Wayback Machine and Google’s archived version of the domain/ link. This was a good help in tracking down an old submission/ broken link. I often found broken links, one way or another. It was one of my favourite things to do.

Reviewing submitted links took hours, especially in categories involving businesses and, of those, anything involving marketing became so flooded with junk submissions it was too much for a volunteer editor to want to deal with. When I tried to work on these categories my computer slowed down to a crawl just trying to load the page with all the sites to be reviewed. It was aggravating to work there when most of the submissions were junk – the link was already listed and descriptions were full of keywords, CAPITAL LETTERS and so on. Of course, these are the very people who complained about dmoz and dmoz editors the most. They did not understand we were running as a directory for the public to search, not for businesses to be listed. The priority was not listing every business or service but to have resources for people searching for a business, service or information (with the Regional listings to help people find local resources).

There are duplicate listings for some sites because they fit into more than one place. Also, sites could be double listed in Regional and the topic or business. Once you get into organizing and deciding where sites (you call them resources) should go you will see it is a much more complicated project than it seems from the outside looking in. We had a forum just for ontology issues. Due to many opinions from active and inactive volunteers, making category changes was time consuming and tended to get lost along the way.

I wish you best of luck with your directory. But, you have a lot of years to go before you should judge how another directory was run.

Leave a Reply