Exchange formats and common pools for free data

I feel pretty strongly about this, having organized a BOF on "Open Source/Open Data" at the last O'Reilly Open Source Conference helped found a company that's working to build an open file metadata catalog. Like the projects mentioned in the article, our catalog is currently centralized, probably for a similar reason: too few resources to work on domain problems and innovate in the area of distributed read/write datasets.

Libre datasets are increasingly important because much of the functionality provided by computers now resides in the net. Suppose you have a Debian box with only software from main installed, and your favorite services include Mapquest and Amazon book reviews. Are you still a free software saint? :-) What if the proprietary "data" you access starts to act more like code? What if your free software application can't compete in its niche without a dependency on proprietary data (e.g., imagine a world without FreeDB or MusicBrainz providing/working on free CDDB alternatives)?

Apart from the physical world and book metadata hinted at above, I'd really like to see distributed web annotations. Without them the promise of hypertext remains unfulfilled.

Since you brought it up, there's been a lot of thinking already in how to move content between wikis. The first step is generally believed to define a MeatBall:XmlRpcToWiki protocol and then to define a MeatBall:WikiInterchangeFormat. The goal originally was to make some sort of Wiki:InterWiki, but maybe in the interim we can build a MeatBall:DistributedWikiForum. One successful bridging technology has been Wiki:SisterSites.

Personally, I don't think it's really necessary to use too much cryptography. Wikis don't need them. If you think hard enough, you won't need to for this this either. For every bit of encryption you misuse, you half your userbase.

The major problem isn't security, it's copyright and control. The best example was when Wikipedia forked FOLDOC. I recommended using sister sites, but Larry didn't like that. He prefered absorbing as much content into Wikipedia as possible as then everything would be inside Wikipedia. Considering the FOLDOC wasn't under the GFDL, this is reasonable (is it?). In the end, Denis converted his entire site to the GFDL, which was nice of him.

Similarly, FOLDOC incorporates The Jargon File, which is in the public domain. This is perhaps nicer for an automated syndication system because it's less restrictive than the GFDL.

Insert that whole BSD vs. GPL debate.

Your description gets pretty much exactly both motivation behind and the basic structur of the Project Askemos.

To clarify: the project is about building a virtual public library. The software (current prototype version is 0.6.13) askemos is the infrastructure you basically described.

And a shameless plug: we really need helping hands!

We also kinda began such a thing. We publish our database in XML (yes, I know it's not that great, but we don't have a good XML guy). We did this to emulate the dmoz distribution model.

We also began talking to coopx about an implementable schema for sites that host projects so the info could be shared among everyone. Here's the mailing list archive.

I hope this is helpful.

If I didn't know better, I would have sworn that you were listening in on the discussions jlatour and I -- tnt -- have been having, about project Sand Dunes. Because virtually all of what you have said, has also been said in our conversations.

Now project Sand Dunes has not yet been announced, because (as you may have noticed if you visited the home page) we are not ready (to announce it). We are in the process of writing a white paper for it (which is the work in progress that you see on the home page).

However, having said that, the similarities between what you have proposed and project Sand Dunes are so close, that I thought that we should let you know about it... and see if you wanted to work with us.

It's too bad the white paper isn't done though.... Reading through what we have, again, I'm noticing that the similarities between what you have proposed and project Sand Dunes hasn't even been written up yet. But just to add to what you see on the home page, project Sand Dunes has things such as:

• Expert Groups... which are trust metric groups with different seeds,
• Forks... the ability to fork the project... without leaving the main system,
• A Distributed System of servers,
• Trust Metrics between expert groups... and not just people with in expert groups,
• multilingual stuff,
• and more.

Obviously that summary is very very inadequate. But combine that with what's correctly there on the home page... an... ummm... I guess that's still inadequate; but it gives you some idea. Anyways, e-mail me (or reply here) if you are interested in working with us.

Good ideas here. I'd like to emphasise the importance of something like dmoz in providing `high-level structure' for the web.

Here is an extract of an email I sent in November 2000 to a number of people who had left/been kicked out of dmoz around the time the new copyright guidelines came into force. I resigned my categories (I was editor cas) in protest at the removal of George Ruban (gruban) at this time. I've attached it here because I think it is important that there is a free alternative to dmoz that can `scale' at least as well as dmoz.

I have been thinking for a while about the possibility of constructing a decentralised alternative to DMoz that has many of the strengths of that project but without the defects that I found to be increasingly tiresome.

The strengths of DMoz (some of which it learnt from Yahoo), as I see them are:

1. It promotes cooperation in editing and figuring out organisational issues;
2. It brings together people with common interests;
3. It turns the product of the directory into a standard, world-readable document based on open W3C standards, making a natural separation between editing and publishing;
4. It supports editor privacy in the forums;
5. It managed to promote a generally quite high minimum level of editing, and a high level of directory-wide consistency;
6. It has good safeguards against editor abuse.

Its weaknesses are mostly due to its centralised form:

1. A courtly mode of operation, where influence is based on favours granted by staff's indulgence.
2. Editor permissions are governed by metas who generally are not sensitive to, for example, local academic standards in particular disciplines.
3. Directory-wide rules are imposed that are frequently an irrelevance or an inconvenience to certain parts of the directory. Eg. the debate over `cooled' sites.
4. A small number of metas govern an enormous number of editors, leading to a bureaucratic and remote feel. Many problems have to be solved that would not exist in smaller groups where everyone knows each other. And it makes volunteering feel like working for a company.
5. Disagreements are resolved by higher-level intervention, essentially consensus at all costs, as opposed to the open-source model is based upon the more free and more intelligent idea of allowing the parties to fork and allow the users decide who was right;
6. The centralised directory becomes a central point of failure, attractive to lawsuits, and vulnerable to malicious hackers, and changes in corporate strategy.

And some weaknesses due to the data representation:

1. It proves terribly hard to avoid duplicating work, and maximising opportunities for `lateral' linking without endlessly multiplying categories, which makes navigating the directory tiresome and error-prone.
2. Changing ontology is unnecessarily difficult. It's a matter of hard consensus, and so there is little experimentation to see if alternative ontologies work better. And it is fraught with issues of who-gets-to-edit-what which is irrelevant to the users of the directory.

I made a concrete proposal along these lines, that for various reasons never came to anything. The key idea was to split content from ontology, and use a modular representation for ontology that allows one ontology easily to be joined and divided. I think the design was sound and I'd be interested in reviving the project.