Introducing GSS

During my recent work-induced blog hiatus, I've been working on a new software system, called GSS. I've been more than enjoying the ride so far and since we have released the code as open-source, I thought discussing some of the experience I've gained might be interesting to others as well.

GSS is a network, er grid, er I mean cloud service, for providing access to a file system on a remote storage space. It is the name of both a service (currently in beta) for the Greek research and academic community and the open source software used for it, that can also be used by others for deploying such services. It is similar in some ways to services like iDrive, DropBox and drop.io, but it can also be regarded as a more high-level Amazon S3. Its purpose is to let the desktop computer's file system meet the cloud. The familiar file system metaphors of files and folders are used to store information in a remote storage space, that can be accessed from a variety of user and system interfaces, from any place in the world that has an Internet connection. All usual file manager operations are supported and users can share their files with selected other users or groups, or even make them public. Currently there are four user interfaces available, a web-based application, a desktop client, a WebDAV interface and an iPhone web application, in various stages of development. Underlying these user interfaces is a common REST-like API that can be used to extend the service in new, unanticipated ways.


The main focus of this service was to provide the users of the Greek research and academic community with a free, large storage space that can be used to store, access, backup and share their work, from as many computer systems as they want. Since the available user base is close to half a million (although the expected users of the service are projected to the low ten thousands), we needed a scalable system, that would be able to accommodate high network traffic and a high storage capacity at the same time. A Java Enterprise Edition server coupled with a GWT-based web client and a stateless architecture were our solution. In future posts I will describe the system architecture with all the gory details. The exposed virtual file system features file versioning, trash bin support, access control lists, tagging, full text search and more.

All of these features are presented through an API for third party developers to create scripts, applications or even full blown services that will fulfill their own particular needs or serve other niches. This API has a REST-like design and though it will probably fail a formal RESTful definition, it sports many of the advantages of such architectures:

• system resources such as users, groups, files and folders are represented by URIs

• GET, HEAD, POST, PUT and DELETE methods on resources have the expected semantics

• HTTP caching is explicitly supported via Last-Modified, ETag & If-* headers

• resource formats for everything besides files are simple JSON representations

• only authenticated requests are allowed, except for public resources

Users are authenticated through the GRNET Shibboleth infrastructure. User passwords are never transmitted to the GSS service. Instead GSS-issued authentication tokens are used by both client and server to sign the API requests after the initial user login. SSL transport can provide even stronger privacy guarantees, but it is not required, nor enabled by default.

The GSS code base is GPL-licensed and therefore anyone can use it as a starting point to implement his own file storage service. We have yet to provide binary downloads, due to the various dependencies, but the build instructions should be enough to get someone started. We are always interested in source or documentation patches, of course (did I mention it's open source?). Most importantly, the REST API will ensure that clients developed for one such service can be reused for every other one.

I will have much more to say about the API in a future post. In the meantime you can peruse the code and documentation, or even try it out yourself. I'd be very interested in any comments you might have.

How to create a WebDAV interface in 10 easy steps

Test Driven Development (TDD) is all the rage these days. And rightly so, if you ask me. Compare developing software against constantly changing client requirements and developing against a specification in the form of tests that your code must pass. I've done my share of the former for the better part of the last 9 years and I've got the gray hair to prove it. However, I only got a chance to practice the latter very recently and I must say I'm hooked.

My TDD endeavor concerns a small project to add a WebDAV interface to an existing server-side code base. The system already featured other interfaces for manipulating the resources stored in it, but a cross-platform, standards-based, nicely integrated, native client was deemed necessary. Enter WebDAV, a.k.a. HTTP on steroids. Most desktop systems nowadays have a WebDAV client installed by default (Windows calls it Web Folders, others prefer not to rename it), so testing a WebDAV server is not that complicated: if you can mount, browse, store and retrieve files and folders using your desktop client, the server implementation is fine. Nevertheless, we can do better than that. Using litmus, a "WebDAV server protocol compliance suite", we can make sure that our implementation respects the protocol, even in its more obscure corner cases and more importantly, perform the testing automatically, without painstaking clickety-click.

Implementing a WebDAV interface from scratch by reading the protocol specification is not the easiest nor the smartest path. You will have to read it in order to figure out what goes where, but you could also reuse some parts from an existing implementation. If you are extending an existing Java codebase, like me, your best bet is reusing the Tomcat implementation. You can run the litmus suite against a vanilla Tomcat installation, just to verify its WebDAV functionality. Not all tests pass, since the implementation is incomplete, but don't worry, most clients don't implement the spec completely either. However take a note of the results, since this is your target outcome.

For your own convenience, I have described the whole development process in 10 easy steps, ready to print, free of charge, batteries not included:

1. Download the litmus test suite.
2. Download the sources for Tomcat, particularly WebdavServlet.java and DefaultServlet.java.
3. Create your own version of WebdavServlet that will delegate the actual functionality to your own backend and fill it with empty stub methods that correspond to the ones in Tomcat.
4. Build & deploy your server.
5. Run the litmus test. All tests fail the first time, but when it eventually succeeds (i.e. matches the result of vanilla tomcat), go to step 9.
   
6. Fill one method at a time from Tomcat's sources, substituting the JNDI communications with your backend logic.
   
7. Add any other helper methods from other Tomcat classes as necessary.
8. Go to step 4.
9. Most tests succeeded, so you are done!
10. For extra credit: fix the code to pass some more tests and send your changes upstream.

You are bound to come across some hair-splitting problems when modifying the codebase to fit in you own backend, but it beats writing it all from scratch hands down, unless you are Greg Stein or Jim Whitehead. In my case, after two weeks and a submitted patch, I was done and had the tests to prove it. Furthermore, when the need for a code refactoring arises, I can rest assured that my changes won't break the protocol functionality if the litmus tests still pass. Not to mention that I have now contributed to Tomcat.

Man, I haven't felt that good in ages.