Scalable Open Storage

Research project

With the advent of Open Archives and the OAI-PMH protocol, scalable and robust Meta-data storage is becoming more and more important. Any Open Archive Data Provider (Repository) as well as Service Provider depends on it. The former requiring a transparent long-term stable archive while the latter requires it to be scalable with high speed/low latency access. This research dealt with the question of 1) how to store meta-data in a way that allows it to be correctly retrieved and restored in about 100 years, and 2) how to provide both random and sequentially ordered access (such as in SRU, OAI-PMH, RSS) in an efficient way, without complicating 1.

The first idea's about how to answer both questions came during our work on the e-Depot for the Rotterdam Municipal Archives. It accelerated when we needed to find a solution for supporting DAREnet search & retrieve and later on for an SRU/SRW implementation on top of FAST's FDS (FAST Data Search).

Current research consists of building an OAI-PMH on top of it (the final proof) and by adding support for our Weightless web-server to improve scalability and reduce memory usage.

Seek You Too