GSS/Pithos update: iPhone, Android, Mongo & getting to 2.0

It has been a while since my last post on GSS/Pithos, and there have been quite a few important announcements that I have not blogged about, only tweeted:

• a shiny native iPhone client
• a brand new Android client
• a back-office application for statistics and admin tasks
• public folders for hosting static web sites
• a slew of performance, usability and bug fixes for the web & desktop clients
• new user registration and quota upgrade workflows

And since the code base has been adopted by other European NRNs, we have also improved our release engineering process and documentation. This however was just the tip of the iceberg. What we are now close to completing, is nothing short of an almost complete rewrite of the core GSS server, aiming at even greater performance and scalability. We are about to release GSS 2.0.

The initial plan was far more modest. We had identified the PostgreSQL installation that stores all of the the system’s data besides the actual files, as a future bottleneck when user adoption would start to accelerate. In such an event there were a few low-to-medium-cost solutions that we could pursue, like replication and manual sharding, but since the problem seemed best suited for a NoSQL solution, we decided to bite the bullet and attempt a transition.

Our field search included lots of products, like Mongo, Couch, Cassandra, Riak, Redis, HBase, MemcacheDB and others. The basic requirements were high write throughput, sharding, replication and easy migration. Most of the products boasted high performance for writes and lots supported replication. In the end we discarded the ones that didn’t support sharding, which limited the options a lot, and we investigated the various APIs, trying to figure out what it would mean to rewrite GSS for one of these. In the end we chose Mongo for its rich query support, since it was the best fit for our problem at hand, making the transition of the existing code base easier. Although we’d love to have had the opportunity to try different ports for each of those datastores, it is most likely that we would have had to rewrite the entire server from scratch, which was not an option.

The initial plan was to try to isolate the changes to the data access layer, essentially building a new adapter for the middle tier that would appear almost identical to the existing code. This effort was fruitless however, since we could not find satisfactory solutions for various issues, like transactions, data mapping, entity lifecycle, etc. We came to the conclusion that even if we managed to build a not-too-leaky abstraction for Mongo, it would have probably killed the performance, making the switch from PostgreSQL pointless.

So we ditched JPA and container-managed transactions and decided to make a simpler POJO mapping to Mongo using Morphia, a Java data mapper for Mongo that takes care of the nitty-gritty details and repetitive coding, for only a minor performance hit. Since Mongo only supports atomicity for a single query, we reverted to manual transaction handling and decided to ditch the EJBs that formed the backbone of GSS altogether, since they didn’t buy us anything at that point. The new GSS server would be a set of servlets, service and domain objects, wired together via Google Guice. Since we could host the new server in a plain servlet container, we went from JBoss to Jetty, which vastly reduced the server startup time from 60 seconds down to 2. As a consequence, development round-trip times went down, and coding was fast again. Lots of object copying and conversions from entities to DTOs and back again were now unnecessary, since there was no more a transaction boundary at the session bean layer. This kept memory usage low, increased cache locality and boosted performance. There are lots of interesting details about our approach, but they would be better discussed in separate blog posts. If you are interested, you can already check out the code in the gss2 branch of the GSS repository.

The next generation GSS server is not production ready yet, but we'll be getting there soon, and so far it has been a great ride!

Lessons from Startup Weekend

I had an exhausting but fun weekend at the Athens Startup Weekend a few days ago. Along with Christos I joined Yannis, Panagiotis Christakos and Babis Makrinikolas on the Newspeek project. When Yannis pitched the idea on Friday night, the main concept was to create a mobile phone application that would provide a better way to view news on the go. I don't believe it was very clear in his mind then, what would constitute a "better" experience, but after some chatting about it we all defined a few key aspects, which we refined later with lots of useful feedback and help from George. Surprisingly, for me at least, in only two days we managed to design, build and present a working prototype in front of the judges and the other teams. And even though the demo wasn't exactly on par with our accomplishments, I'm still amazed at what can be created in such a short time frame.

Newspeek, our product, had a server-side component that periodically collected news items from various news feeds, stored them and provided them to clients through a simple REST API. It also had an iPhone client that fetched the news items and presented them to the user in a way that respected the UI requirements and established UX norms for that device.

So, in the interest of informing future participants about what works and what doesn't work in Startup Weekend, here are the lessons I learned:

1. If you plan to win, work on the business aspect, not on the technology. Personally, I didn't go to ASW with plans to create a startup, so I didn't care that much about winning. I mostly considered the event as a hackathon, and tried my best to end up with a working prototype. Other teams focused more on the business side of things, which is understandable, given the prize. Investors fund teams that have a good chance to return a profit, not the ones with cool technology and (mostly) working demos. Still, the small number of actual working prototypes was a disappointment for me. Even though the developers were the majority in the event, you obviously can't have too many of them in a Startup Weekend.
2. For quick server-side prototyping and hosting, Google App Engine is your friend. Since everyone in the team had Java experience, we could have gone with a JavaEE solution and set up a dedicated server to host the site. But, since I've always wanted to try App Engine for Java and the service architecture mapped nicely to it, we tried a short experiment to see if it could fly. We built a stub service in just a few minutes, so we decided it was well worth it. Building our RESTful service was really fast, scalability was never a concern and the deployment solution was a godsend, since the hosting service provided for free by the event sponsors was evidently overloaded. We're definitely going to use it again for other projects.
3. jQTouch rocks! Since our main deliverable would be an iPhone application, and there were only two of us who had ever built an iPhone application (of the Hello World variety), we knew we had a problem. Fortunately, I had followed the jQTouch development from a reasonable distance and had witnessed the good things people had to say, so I pitched the idea of a web application to the team and it was well received. iPhone applications built with web technologies and jQTouch can be almost indistinguishable from native ones. We all had some experience in building web applications, so the prospect of having a working prototype in only two days seemed within the realm of possibility again. The future option of packaging the application with PhoneGap and selling it in the App Store was also a bonus point for our modest business plan.
4. For ad-hoc collaboration, Mercurial wins. Without time to set up central repositories, a DVCS was the obvious choice, and Mercurial has both bundles and a standalone server that make collaborative coding a breeze. If we had zeroconf/bonjour set up in all of our laptops, we would have used the zeroconf extension for dead easy machine lookup, but even without it things worked flawlessly.
5. You can write code with a netbook. Since I haven't owned a laptop for the last three years, my only portable computer is an Asus EEE PC 901 running Linux. Its original purpose was to allow me to browse the web from the comfort of my couch. Lately however, I'm finding myself using it to write software more than anything else. During the Startup Weekend it had constantly open Eclipse (for server-side code), Firefox (for JavaScript debugging), Chrome (for webkit rendering), gedit (for client-side code) and a terminal, without breaking a sweat.
6. When demoing an iPhone application, whatever you do, don't sweat. Half-way through our presentation, tapping the buttons didn't work reliably all the time, so anxiety ensued. Since we couldn't make a proper presentation due to a missing cable, we opted for a live demo, wherein Yannis held the mic and made the presentation, and I posed as the bimbo that holds the product and clicks around. After a while we ended up both touching the screen, trying to make the bloody buttons click, which ensured the opposite effect. In retrospect, using a cloth occasionally would have made for a smoother demo, plus we could have slipped a joke in there, to keep the spirit up.

All in all it was an awesome experience, where we learned how far we can stretch ourselves, made new friends and caught up with old ones. Next year, I'll make sure I have a napkin, too.

GSS authentication

As I've described before, the GSS API is a REST-like interface for interacting with a GSS-based service, like Pithos. Using regular HTTP commands a client can upload, download, browse and modify files and folders stored in the GSS server. These interactions have two important properties: they store no client state to the server and they are not encrypted. I have already mentioned the most important benefits from the stateless architecture of the GSS protocol, namely scalability and loose coupling along the client-server boundary. In the same line of thought, SSL/TLS encryption of the transport was avoided for scalability reasons. Consequently, these two properties of the communication protocol, lead to another requirement: message authentication for each API call.

Since no session state is stored in the server, the client must authenticate each request as if it were the first one. Traditional web applications use an initial authentication interaction between client and server, that creates a unique session ID, which is transmitted with each subsequent request in that same session. While using this ID, the client does not need to authenticate again to the server. Stateless protocols, like WebDAV for instance, cannot rely on such an ID and have to transmit authentication data in each call. Ultimately the tradeoff is a minor increase in the message payload, in return for a big boost in scalability. The HTTP specification defines an Authorization header for use in such cases and WebDAV uses the HTTP Digest Access Authentication scheme. The GSS API uses a slight variation in that theme, blended with some ideas from OAuth request signing.

Essentially the standard HTTP Authorization header is populated with a concatenation of the username (for identifying the user making the request) and a HMAC-SHA1 signature of the request. The request signature is obtained by applying a secret authentication token to a concatenated string of the HTTP method name, the date and time of the request and the actual requested path. The GSS API page has all the details. The inclusion of the date and time helps thwart replay attacks using a previously sniffed signature. Furthermore, a separate header with timestamp information, X-GSS-Date, is used to thwart replay attacks with a full copy of the payload. The authentication token is issued securely by the server to the client and is the time-limited secret that is shared between the client and server. Since it is not the user password that is used as a shared secret, were the authentication token to be compromised, any ill effects would have been limited to the time period the token was valid. There are two ways for client applications to obtain the user's authentication token, and they are both described in detail in the API documentation.

In my experience, protocol descriptions are one thing and working code is a totally different one. In that spirit, I'm closing this post with a few tested implementations of the GSS API signature calculation, for client applications in different languages. Here is the method for signing a GSS API request in Java (full code here):

public static String sign(String httpMethod, String timestamp,
        String path, String token) {
  String input = httpMethod + timestamp + path;
  String signed = null;

  try {
    System.err.println("Token:" + token);
    // Get an HMAC-SHA1 key from the authentication token.
    System.err.println("Input: " + input);
    SecretKeySpec signingKey = new SecretKeySpec(
        Base64.decodeBase64(token.getBytes()), "HmacSHA1");

    // Get an HMAC-SHA1 Mac instance and initialize with the signing key.
    Mac hmac = Mac.getInstance("HmacSHA1");
    hmac.init(signingKey);

    // Compute the HMAC on the input data bytes.
    byte[] rawMac = hmac.doFinal(input.getBytes());

    // Do base 64 encoding.
    signed = new String(Base64.encodeBase64(rawMac), "US-ASCII");

  } catch (InvalidKeyException ikex) {
    System.err.println("Fatal key exception: " + ikex.getMessage());
    ikex.printStackTrace();
  } catch (UnsupportedEncodingException ueex) {
    System.err.println("Fatal encoding exception: "
                + ueex.getMessage());
  } catch (NoSuchAlgorithmException nsaex) {
    System.err.println("Fatal algorithm exception: "
                + nsaex.getMessage());
    nsaex.printStackTrace();
  }

  if (signed == null)
    System.exit(-1);
  System.err.println("Signed: " + signed);
  return signed;
}

Here is a method for signing GSS API requests in JavaScript, that uses the SHA-1 JavaScript implementation by Paul Johnston (full code here):

function sign(method, time, resource, token) {
  var q = resource.indexOf('?');
  var res = q == -1? resource: resource.substring(0, q);
  var data = method + time + res;
  // Use strict RFC compliance
  b64pad = "=";
  return b64_hmac_sha1(atob(token), data);
}

Here is a Tcl implementation by Alexios Zavras (full code here):

proc ::rest::_sign {id} {
  variable $id
  upvar 0 $id data
  set str2sign ""
  append str2sign $data(method)
  append str2sign $data(date)
  set idx [string first ? $data(path)]
  if {$idx < 0} {
    set str $data(path)
  } else {
    incr idx -1
    set str [string range $data(path) 0 $idx]
  }
  # append str2sign [::util::url::encode $str]
  append str2sign $str
  puts "SIGN $str2sign"
  set sig [::base64::encode [::sha1::hmac -bin -key $::GG(gsstoken) $str2sign]]
  set data(signature) $sig
}

I'd love to show implementations of the signature calculation for other languages as well, but since my time is scarce these days, I could use some help here. If you've written one yourself and you'd like to share, leave a comment and I'll update this post, with proper attribution of course.

Retrying transactions with exponential backoff

The method I described in my previous post about retrying transactions could use some improvement. A deficiency that will only become relevant in highly congested servers is the constant retry interval. When two or more transactions try to commit at the same time and fail, with the code from the last post they will retry probably simultaneously again. Random runtime events (like process/thread scheduler decisions, JIT compiler invocations, etc.) might help avoid collisions, but in general the collided transactions may well collide again. And again. And then again, until the specified maximum number of retries is reached.

The general method to alleviate such problems is to randomize the retry intervals. The most well-known algorithm in this category is called exponential backoff. This is one way to implement it for the utility method in my previous post (full code in gss):

public T tryExecute(final Callable<T> command) throws Exception {
  T returnValue = null;
  // Schedule a Future task to call the command after delay milliseconds.
  int delay = 0;
  ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
  for (int i = 0; i < TRANSACTION_RETRIES; i++) {
    final int retry = i;
    ScheduledFuture<T> future = executor.schedule(new Callable<T>() {

      @Override
      public T call() throws Exception {
        return command.call();
      }
    }, delay, TimeUnit.MILLISECONDS);

    try {
      returnValue = future.get();
      break;
    } catch (ExecutionException e) {
      Throwable cause = e.getCause();
      if (!(cause instanceof EJBTransactionRolledbackException) ||
          retry == TRANSACTION_RETRIES - 1) {
        executor.shutdownNow();
        throw new Exception(cause);
      }
      delay = MIN_TIMEOUT + (int) (MIN_TIMEOUT * Math.random() * (i + 1));
      String origCause = cause.getCause() == null ?
          cause.getClass().getName() :
          cause.getCause().getClass().getName();
      logger.info("Transaction retry #" + (i+1) + " scheduled in " + delay +
          " msec due to " + origCause);
    }

  }
  executor.shutdownNow();
  return returnValue;
}

The notable changes in this version are the delayed invocations of the Callable and the retry interval calculation. The former is accomplished using a ScheduledFuture from the excellent utilities in java.util.concurrent, which gets executed after a random time period. The interval calculation is something that can be implemented in a number of ways, either with monotonically increasing intervals or not. I opted for the formula above since it provided the fastest collision elimination in my tests, much faster than the monotonically increasing interval formulas I tried. The MIN_TIMEOUT constant is more of a black art. It should be tuned to the particular hardware and network setup in order to attain maximum efficiency: the minimum number of retries with the minimum interval between retries.

Another issue raised in a comment in my previous post was that the exception I am using as guard, EJBTransactionRolledbackException, may be too generic for this purpose. This is definitely true, EJBTransactionRolledbackException just wraps an OptimisticLockException as one would expect for this case, which in turn wraps the Hibernate-specific StaleObjectStateException that is thrown initially. However, contention in a database table, index or row might not necessarily result in optimistic locking violations, but in deadlocks as well, when one transaction holds exclusive locks on resources that are needed by the other and vice versa. Deadlock detection is performed by the DBMS, which forces a rollback of both transactions. This time however the initial exception (at least in my testing, might be DBMS or JDBC driver-specific) is GenericJDBCException which gets wrapped in a PersistenceException, which in turn is put inside a EJBTransactionRolledbackException before being received by our call site. Therefore, the generic nature of the guard is not a problem in this case. On the contrary it covers more relevant cases and one might argue even that there is no better service that can be offered to the caller, than retrying in all such occasions.

Retrying transactions in Java

When building a web-based system, the most often encountered persistence solution is the use of a relational DBMS. This provides many benefits, but probably the most important one is that its behavior and usage are widely understood after so many decades of use. The simplicity of the ACID transactional model, coupled with the nice fit of the HTTP request/response protocol, make for a killer combination.

The simplicity starts to break down however, when the load on the DBMS server increases, causing transactions to start failing. The DBMS can guarantee that no data corruption will occur, but handling such cases is not that simple. The scalable Optimistic Locking model requires transactions to be resubmitted when failure occurs. Since this is supposed to be rather rare (otherwise optimistic locking should probably not be used), developers often cheat by not dealing with such errors, letting them bubble up to the user, who is then responsible for resubmitting the transaction. This leads to simple, elegant codebases and unhappy users.

Adding ORM to the mix only makes things worse. In JavaEE, the JPA API hides much of the complexity of dealing with relational databases, but corner cases become harder. Add the container-managed transaction boundaries provided by session EJBs and one might be tempted to raise his hands up in despair.

However a solution is not as hard as it sounds. Provided that session EJB methods are sufficiently free from side effects, one can replay a transaction for a number of times from the call site (external to the session bean) with a small amount of code and a small added complexity to the call sites. Since I seem to come up against this requirement quite often, I thought I should describe one way to do it. The following tryExecute() helper method, retries any action supplied in a standard Callable interface for a number of times, in the face of optimistic locking violations:

public T tryExecute(Callable<T> command) throws Exception {
  T returnValue = null;
  for (int i = 0; i < TRANSACTION_RETRIES; i++) {
    try {
      returnValue = command.call();
      break;
    } catch(EJBTransactionRolledbackException trbe) {
      if (i == TRANSACTION_RETRIES - 1)
        throw trbe;
      } catch (Exception e) {
        throw e;
      }
      logger.debug("Transaction retry: " + (i+1));
    }
  return returnValue;
}

By abstracting away the session bean method code in a parameterized class we can reuse this method in every place that needs to retry transactions. Using a Callable as a parameter lets us return any values received from the session bean. For supplying bean methods that do not return values, we could create Runnable instances and wrap them to Callable using Executors.callable(). You can see the original code here.

If you use the above code in a JavaSE environment without EJB, you should probably need to catch OptimisticLockException for JPA, or even StaleObjectStateException, if you don't mind depending on a Hibernate API. I use the above helper method like this in my code:

file = new TransactionHelper<FileHeaderDTO>().tryExecute(new Callable<FileHeaderDTO>() {
  @Override
  public FileHeaderDTO call() throws Exception {
    return sessionBean.createFile(user.getId(), folder.getId(), name, mimeType, uploadedFile);
  }
});

Java's verbosity makes this not that easy to understand at first sight, but what happens is very simple indeed: I wrap a one-liner call to the session bean method createFile(), in a Callable and supply it to tryExecute(). The result from the bean method is received in a local variable as usual. One thing to keep in mind though is that since Java doesn't have closures, the above anonymous function can only receive final variables as parameters. In my example user, folder, name, mimeType and uploadedFile are all final. You might have to resort to using temporary local variables for storing final values on some occasions.

Reconciling Apache Commons Configuration with JBoss 5

There are many ways to configure a JavaEE application. Among the available solutions are DBMS tables, JNDI, JMX MBeans and even plain old files in a variety of formats. While we have used most of the above in various occasions, I find that plain files resonate with all types of sysadmins, when no other administrative interface is available. For such scenarios, Apache Commons Configuration is undoubtedly the best tool for the job. Recently, I came across an undocumented incompatibility when using Commons Configuration with JBoss 5 and I thought I should describe our solution for the benefit of others.

Usually we are storing our configuration files in the standard place for JBoss, which is JBOSS_HOME/server/default/conf, for the default server configuration. This has the disadvantage that is not as easy to remember as /etc in UNIX/Linux or \Program Files and \Windows in Windows systems, but it has the important advantage of being specified as a relative path in our code, making it more cross-platform without cluttering it with platform-specific if/else path resolution checks.

Commons Configuration can reload the configuration files automatically when changed in the file system, which helps prolong the server uptime. However JBoss 5 has introduced the concept of a virtual file system that caches all file system accesses performed through the context class loaders using relative paths. Unfortunately this generates resource URLs in the form vfsfile:foo.properties that Commons Configuration does not know how to deal with. Fixing this requires extending FileChangedReloadingStrategy, like we do in gss. Alternatively, one could patch Commons Configuration with the following change and use the standard FileChangedReloadingStrategy unchanged:

Index: FileChangedReloadingStrategy.java
===================================================================
--- FileChangedReloadingStrategy.java    (revision 764760)
+++ FileChangedReloadingStrategy.java    (working copy)
@@ -46,6 +46,9 @@
  /** Constant for the jar URL protocol.*/
  private static final String JAR_PROTOCOL = "jar";

+    /** Constant for the JBoss MC VFSFile URL protocol.*/
+    private static final String VFSFILE_PROTOCOL = "vfsfile";
+
  /** Constant for the default refresh delay.*/
  private static final int DEFAULT_REFRESH_DELAY = 5000;

@@ -161,7 +164,8 @@

  /**
   * Helper method for transforming a URL into a file object. This method
-     * handles file: and jar: URLs.
+     * handles file: and jar: URLs, as well as JBoss VFS-specific vfsfile:
+     * URLs.
   *
   * @param url the URL to be converted
   * @return the resulting file or null 
@@ -181,6 +185,18 @@
              return null;
          }
      }
+        else if (VFSFILE_PROTOCOL.equals(url.getProtocol()))
+        {
+            String path = url.getPath();
+            try
+            {
+                return ConfigurationUtils.fileFromURL(new URL("file:" + path));
+            }
+            catch (MalformedURLException mex)
+            {
+                return null;
+            }
+        }
      else
      {
          return ConfigurationUtils.fileFromURL(url);

Neither of these solutions covers the case of storing configuration files in zip or jar containers, but since it is something I haven't found a use for yet, I can't test a fix for it. If anyone is interested in such a use case, I'd advise extending FileChangedReloadingStrategy, combining the logic in jar: and vfsfile: URL handling.

GSS architecture

Handling a large number of concurrent connections requires many servers. Not only because scaling vertically (throwing bigger hardware at the problem) is very costly, but also because even if an application can be designed to scale vertically, the underlying stack probably can not. Java applications for instance, like GSS, run on the JVM and although the latter is an excellent piece of engineering, using huge amounts of heap is not something it's tuned for. Big Iron servers with many cores and 20+ GB of RAM are usually running more than one JVM, since garbage collection is not all that efficient with huge heaps. And since running application instances with a 4-8 GB heap size can be done with cheap off-the-shelf hardware, why spend big bucks on Big Iron?

So having a large number of servers is a sane choice, but brings it's own set of problems. Unless one partitions users to servers (having all requests a particular user makes be delivered to the same server), all servers must have a consistent view of the system data, in order to deliver meaningful results. Assigning user requests to particular servers, usually requires expensive application layer load-balancers or customized application code on each server, so it would rarely be your first option. Having all servers work on the same data is a more tractable problem, since it can be solved by having the application state being replicated among server nodes. Usually, only a small part of the application state needs to be replicated, for each user, that is the part which concerns his current session. But even though session clustering solutions have been a well studied field and implementations abound, having no session to replicate is an even better option.

For GSS we have implemented a stateless architecture for the core server, that should provide us with good scalability in a very cost-effective manner. The most important part in this architecture is the REST-like API that moves part of the responsibility for session state maintenance to the client applications, effectively distributing the system load to more systems than the available server pool. Furthermore, client requests can be authenticated without requiring an SSL/TLS transport layer (even though it can be used if extra privacy is required), which would entail higher load on the application servers or require expensive load balancers. In the server side, API requests are being handled by servlets that enlist the services of stateless session beans, for easier transaction management. Our persistence story so far is JPA with a DBMS backing, plus high-speed SAN storage for the file contents. If or when this becomes a bottleneck, we have various contingency plans, depending on the actual characteristics of the load that will be observed.


The above image depicts the path user requests will travel along, while various nodes interact in order to serve them. A key element in this diagram is the number of different servers that can be found, effectively specializing in their own particular domain. Although the system can be deployed on a single physical server (and regularly is, for development and testing), consisting of a number of standalone sub-services instead of a big monolithic service is a boon to scalability.

This high-level overview of the GSS architecture should help those interested to find their way around the open-source codebase and explain some of the design decisions. But the most interesting part from a user's point of view would be the REST-like API, that allows one to take advantage of the service for scratching his own itch.

So that will be the subject of my next post.

Introducing GSS

During my recent work-induced blog hiatus, I've been working on a new software system, called GSS. I've been more than enjoying the ride so far and since we have released the code as open-source, I thought discussing some of the experience I've gained might be interesting to others as well.

GSS is a network, er grid, er I mean cloud service, for providing access to a file system on a remote storage space. It is the name of both a service (currently in beta) for the Greek research and academic community and the open source software used for it, that can also be used by others for deploying such services. It is similar in some ways to services like iDrive, DropBox and drop.io, but it can also be regarded as a more high-level Amazon S3. Its purpose is to let the desktop computer's file system meet the cloud. The familiar file system metaphors of files and folders are used to store information in a remote storage space, that can be accessed from a variety of user and system interfaces, from any place in the world that has an Internet connection. All usual file manager operations are supported and users can share their files with selected other users or groups, or even make them public. Currently there are four user interfaces available, a web-based application, a desktop client, a WebDAV interface and an iPhone web application, in various stages of development. Underlying these user interfaces is a common REST-like API that can be used to extend the service in new, unanticipated ways.


The main focus of this service was to provide the users of the Greek research and academic community with a free, large storage space that can be used to store, access, backup and share their work, from as many computer systems as they want. Since the available user base is close to half a million (although the expected users of the service are projected to the low ten thousands), we needed a scalable system, that would be able to accommodate high network traffic and a high storage capacity at the same time. A Java Enterprise Edition server coupled with a GWT-based web client and a stateless architecture were our solution. In future posts I will describe the system architecture with all the gory details. The exposed virtual file system features file versioning, trash bin support, access control lists, tagging, full text search and more.

All of these features are presented through an API for third party developers to create scripts, applications or even full blown services that will fulfill their own particular needs or serve other niches. This API has a REST-like design and though it will probably fail a formal RESTful definition, it sports many of the advantages of such architectures:

• system resources such as users, groups, files and folders are represented by URIs

• GET, HEAD, POST, PUT and DELETE methods on resources have the expected semantics

• HTTP caching is explicitly supported via Last-Modified, ETag & If-* headers

• resource formats for everything besides files are simple JSON representations

• only authenticated requests are allowed, except for public resources

Users are authenticated through the GRNET Shibboleth infrastructure. User passwords are never transmitted to the GSS service. Instead GSS-issued authentication tokens are used by both client and server to sign the API requests after the initial user login. SSL transport can provide even stronger privacy guarantees, but it is not required, nor enabled by default.

The GSS code base is GPL-licensed and therefore anyone can use it as a starting point to implement his own file storage service. We have yet to provide binary downloads, due to the various dependencies, but the build instructions should be enough to get someone started. We are always interested in source or documentation patches, of course (did I mention it's open source?). Most importantly, the REST API will ensure that clients developed for one such service can be reused for every other one.

I will have much more to say about the API in a future post. In the meantime you can peruse the code and documentation, or even try it out yourself. I'd be very interested in any comments you might have.

Murdered by Numbers

"JavaScript has a single number type. Internally, it is represented as 64-bit floating point, the same as Java's double. Unlike most other programming languages, there is no separate integer type, so 1 and 1.0 are the same value. This is significant convenience because problems of overflow in short integers are completely avoided, and all you need to know about a number is that it is a number. A large class of numeric type errors is avoided."

Douglas Crockford, JavaScript: The Good Parts

Bitscuits © Barney Livingston / CC

When I was twenty-something, I liked watching The Young Indiana Jones Chronicles. Besides the regular stuff that your average Indy fan loves, I was particularly fond of Indy's apparently insatiable appetite for traveling and learning different languages. So while I was contemplating what it would take for me to follow in his footsteps, I figured I'd better start with learning foreign languages. By that time I had already got my English certificate and I had a few years of learning French under my belt that had at least provided me with some means to court women (with little success, regrettably). So I started learning Italian, which everyone said was easy to pick up, especially if you had already mastered some other foreign language. That turned out to be true, and I managed to get a degree in Italian after two years of intensive studies. What made that period frustrating (and occasionally funny) however, were the times that I would inadvertently mix all three languages in the same sentence, creating my own version of Esperanto. Due to the similarities among them, I might be trying to speak Italian, but use an English noun while constructing the past tense of a verb in French. Sometimes I would even be oblivious to my mistake until someone else pointed it out to me. It all seemed very natural as I was doing it.

Lately I'm having a déjà vu, when I find myself coding in Java, JavaScript and C++, often in the same day. More than once I tried to initialize a Java object using a JavaScript object literal. Sadly, the compiler was not very accommodating. While writing a GWT-based front-end, I often transmitted POJOs without being aware that a long value was silently converted to a JavaScript Number, which essentially amounts to a Java double. Reading David Flanagan's "Rhino" book had already left me with the impression that JavaScript was rather flawed for not having separate types for integers, bytes, chars and all the other goodies that languages like C/C++ and Java spoil us with. But after getting a copy of Douglas Crockford recent, highly opinionated, "Good parts" book, his argument resonated with me: "A large class of numeric type errors is avoided." It's Richard Gabriel's "Worse Is Better" principle, or alternatively "Less Is More", in new clothes. Recent events made sure that the message was permanently bolted in my brain.

I've been writing a desktop application in Java that communicates with various Bluetooth and USB devices. The communication protocol is some sort of terminal-like commands and responses that initiate at the Java application and travel through a thin JNI layer down to the C/C++ device driver, and then to the actual device. The protocol documentation describes in... er, broad terms, the sequences of bytes that constitute the various requests and responses. Suffice it to say that my system administrator's experience in sniffing and deciphering network packets proved invaluable.

Sending the command was easy (or so I thought): fill a byte[] array with the right numbers and flush it through the JNI layer. There we get the jbyteArray and put it inside an unsigned char array, which is later send through the device driver to the actual device. When receiving responses the sequence was reversed. It all seemed to work fine for quite some time, until suddenly I discovered that one particular command caused the device to misbehave. I couldn't be sure if the device was faulty or my code was buggy, but since I had zero chances of proving the former, I focused on investigating the latter. A couple of days of debugging later I was still on square one, since as far as I could tell the command reached the device unscathed. Logic says that if a fine command reaches a fine device, then one would be entitled to a fine response. Since I wasn't getting it, I began questioning my assumptions.

I resisted the urge to blame the device, since I couldn't prove it conclusively, and started blaming the command. There was definitely something fishy about it, and to be honest, I had a bad feeling all along. The other commands were simple sequences, like:

0x14, 0x18, 0x1a, 0x3e

or

0x13, 0x17, 0x19, 0x26

This particular one however, was icky:

0x11, 0x15, 0x17, 0xf0

If you can't see the ickiness (and I won't blame you), let me help you.

Java's byte type is a signed 8-bit integer. C++'s unsigned char type is an unsigned 8-bit integer (at least in 32-bit Windows). Therefore we can represent values from -128 to 127 in Java and values from 0 to 255 in C. So, if you have a value between 128 and 255, ickiness ensues. 0xf0 is, you guessed it, between 128 and 255. It is 240 to be precise, if Windows Calculator is to be trusted.

Now, of course I am not that dumb. I knew that you can't assign 0xf0 to a byte in Java, so I had already made the conversion. You see, what is actually transmitted is a sequence of 8 bits. If you get the sequence right, it will reach it's destination no matter what. When you convert 0xf0 to bits you get 11110000. The first bit is the sign bit, which is causing all the trouble. If it was zero instead, we would be dealing with 1110000, or 0x70, or 112 if you're into decimal numbers.

So that's what I had done. I'd constructed the negative version of 112 and used that to fill my command buffer:

0x11, 0x15, 0x17, -112

Looking at it made me feel a bit uneasy, without being able to explain why. I used to think it was the mixed hex and decimal numbers. Yeah, I'm weird like that. However, the zillionth time I reread the Java's byte type definition, a lightbulb lit up over my head. I actually paid attention to the words in front of me: "The byte data type is an 8-bit signed two's complement integer".

Sure, it's 8 bits wide and yes it's signed, using the most common representation for negative binary numbers, two's complement. What's new here? I know how to negate in two's complem...

Whoa! Wait a minute. What I just described above isn't how you negate a number in two's complement. It's actually how you do it in the sign-and-magnitude variant. In two's complement you invert the bits and add one to the result:

11110000 -> 00001111 -> 00010000 or 16, or 0x10

Yep, 16, not 112. So the proper command sequence becomes

0x11, 0x15, 0x17, -16

and, yes, the device seems quite happy with that. As happy as devices get, that is.

So, basically, I wasted many many hours and suffered innumerable hair-pulling episodes for falling prey to a numeric type error. The kind Doug Crockford alludes to in his book. Now, don't get me wrong, I like mucking with bits as much as the next guy. But had I been living solely in JavaScript-land, with the single number type, I'd feel less tired and probably not hate my work as much. Though, granted, I might be needing a haircut right now.

How to create a WebDAV interface in 10 easy steps

Test Driven Development (TDD) is all the rage these days. And rightly so, if you ask me. Compare developing software against constantly changing client requirements and developing against a specification in the form of tests that your code must pass. I've done my share of the former for the better part of the last 9 years and I've got the gray hair to prove it. However, I only got a chance to practice the latter very recently and I must say I'm hooked.

My TDD endeavor concerns a small project to add a WebDAV interface to an existing server-side code base. The system already featured other interfaces for manipulating the resources stored in it, but a cross-platform, standards-based, nicely integrated, native client was deemed necessary. Enter WebDAV, a.k.a. HTTP on steroids. Most desktop systems nowadays have a WebDAV client installed by default (Windows calls it Web Folders, others prefer not to rename it), so testing a WebDAV server is not that complicated: if you can mount, browse, store and retrieve files and folders using your desktop client, the server implementation is fine. Nevertheless, we can do better than that. Using litmus, a "WebDAV server protocol compliance suite", we can make sure that our implementation respects the protocol, even in its more obscure corner cases and more importantly, perform the testing automatically, without painstaking clickety-click.

Implementing a WebDAV interface from scratch by reading the protocol specification is not the easiest nor the smartest path. You will have to read it in order to figure out what goes where, but you could also reuse some parts from an existing implementation. If you are extending an existing Java codebase, like me, your best bet is reusing the Tomcat implementation. You can run the litmus suite against a vanilla Tomcat installation, just to verify its WebDAV functionality. Not all tests pass, since the implementation is incomplete, but don't worry, most clients don't implement the spec completely either. However take a note of the results, since this is your target outcome.

For your own convenience, I have described the whole development process in 10 easy steps, ready to print, free of charge, batteries not included:

1. Download the litmus test suite.
2. Download the sources for Tomcat, particularly WebdavServlet.java and DefaultServlet.java.
3. Create your own version of WebdavServlet that will delegate the actual functionality to your own backend and fill it with empty stub methods that correspond to the ones in Tomcat.
4. Build & deploy your server.
5. Run the litmus test. All tests fail the first time, but when it eventually succeeds (i.e. matches the result of vanilla tomcat), go to step 9.
   
6. Fill one method at a time from Tomcat's sources, substituting the JNDI communications with your backend logic.
   
7. Add any other helper methods from other Tomcat classes as necessary.
8. Go to step 4.
9. Most tests succeeded, so you are done!
10. For extra credit: fix the code to pass some more tests and send your changes upstream.

You are bound to come across some hair-splitting problems when modifying the codebase to fit in you own backend, but it beats writing it all from scratch hands down, unless you are Greg Stein or Jim Whitehead. In my case, after two weeks and a submitted patch, I was done and had the tests to prove it. Furthermore, when the need for a code refactoring arises, I can rest assured that my changes won't break the protocol functionality if the litmus tests still pass. Not to mention that I have now contributed to Tomcat.

Man, I haven't felt that good in ages.

How programming languages grow

Q: How do programming languages grow?
A: With pain.

Unless you were lost on a deserted island far away from civilization, you must have heard something about the ECMAScript v4 shenanigans. ECMAScript 4 is the future standard specification for what most people will undoubtedly refer to as JavaScript 2. This forthcoming specification brings a whole slew of features into the JavaScript language, about a decade after the current ECMAScript 3 standard came out. Many of the new features are about facilitating the creation of large programs in the language and improving their safety and robustness. As an intentional side-effect, they maintain and improve the syntactic similarity between JavaScript and Java, so that most young programmers will find it easy to learn and use (in case you didn't know, Java is one of the languages students learn in almost every CS curriculum nowadays).

The breadth of the changes as well as the increased similarity to Java has led some people to protest against them, while they portray the future of their beloved language with gloom and doom. Some members of the ECMAScript standardization committee decided to get off the standardization effort and pursue an ECMAScript v3.1, that is much less ambitious in scope, though still quite vague. These members are Microsoft and Douglas Crockford and while Redmond's move has set off the (usually correct) conspiracy reflexes of the community, Crockford's objections carry more weight to those who pay attention.

Doug Crockford has been known for many things in the JavaScript community, but what I respect most is his simplified JavaScript parser, which is itself written in the same simplified JavaScript. This is the second JavaScript parser in JavaScript that I'm aware of, the first being Brendan Eich's Narcissus. This subset of JavaScript that Crockford advocates for (and is still available in the ECMAScript 4 specification) prefers a more functional style of programming with heavy use of prototypal inheritance, compared to the traditional object-oriented style used in Java and proposed for ECMAScript 4. To cut a long story short, the creator of JavaScript convincingly makes the point that JavaScript must evolve if it is to keep up with the needs of its users, otherwise they will pick something else, like Silverlight or Flex.

In that presentation, Brendan Eich makes a reference to a talk given by Guy Steele in 1998, titled "Growing a Language". It was about the time that the Java community was having a debate about adding generics, operator overloading, complex numbers and other features. In the end some of the proposed changes made it to the standard, like generics, while others like operator overloading, did not. Today another debate is raging in the Java community, about adding closures to the language. Though perhaps less emotional than the JavaScript debate, it is still highly divisive and the reactions are occasionally overboard. It seems changing a language always involves fear, anxiety and pain. Guy Steele's talk provides some insights.

Its been about a decade since that talk, and it shows. No Powerpoint/Keynote slides, just hand-written ones manually placed on the projector. Even the haircut is out of fashion. However the actual presentation is sheer genius. Steele uses the form of his talk to illustrate the point of his argument. In order to demonstrate the difference between small languages and big languages in terms of the programs they can create and the difficulty of their use, he picks a subset of English as a given, all words with one syllable, and every time he needs to use another word he provides a definition. In the same way you provide a class definition of a Person in Java and then go on talking about that person in your business code as if it was a language primitive, Steele makes a talk sound like a computer program and as he confesses in the end, creating that talk was a lot like writing a program.

It may seem weird at first, but as you get the hang of it, it's an eye-opener. Not suitable for computer-language illiterate people of course. Your girlfriend definitely ain't gonna like it. Even the jokes are kinda geeky. Just watch it when you have an hour to spare and nobody is watching you. Then perhaps you might be able to understand Brendan Eich's passion. And do the right thing: use Firefox.

GWT and animation effects

I've already praised GWT and its highly productive environment for generating AJAX applications. I've been using it in my Very Cool Project with great pleasure, I must confess. Creating a highly interactive web application, brings back some of the entertaining aspects of programming again. Although, to be honest, it would have been nice to get rid of all those anonymous inner functions I find myself dealing with. Life (and code) would be a lot simpler with closures and higher order functions. Sometimes I find it ironic that GWT translates my anonymous-inner-classes-using code to JavaScript code with closures, higher order functions and all the other goodies. Apparently, a change of perspective is always an eye-opener.

My nagging aside, GWT is an enormous leap over other solutions for interactive web applications in Java. However, one area that has received little attention so far, is support for animation effects. The stuff that puts the candy in eye-candy. Certainly, there is always the option to use other JavaScript libraries that provide such effects through JSNI, like Scriptaculous, YUI, Ext JS and others. But that is not an efficient nor elegant way to do it. Memory leaks may creep up (particularly on Internet Explorer), separate download requests for the other scripts must be made and JSNI references are, er, how should I put it, um, chatty?

A better way involves using a GWT wrapper around the external effects JavaScript library. This way you get to reap the usual benefits of GWT, namely the minimum number of roundtrips to the server, compilation of just the code you need (the effects you actually use) and no detour to JavaScript-land (though lately I'm having second-thoughts about the goodness of this). The main solutions that have emerged so far are various Scriptaculous wrappers, an ext wrapper and gwt-fx.

I've tried both one Scriptaculous wrapper and the gwt-fx codebase and I would wholeheartedly recommend the latter. For starters, it feels like things are done "the GWT way": a single EffectPanel to decorate the widget that will be animated, separate effect classes for the most popular effects, sequential and parallel combinations of effects, deferred binding for browser-specific functionality, Listeners and Adapters and much more. Implementing a fade effect took me about 10 lines of Java code. And we all know Java is not the most succinct girl in town, right? Furthermore, while the Scriptaculous solution worked on IE, Firefox and Opera, it would barf on Safari (version 2 if you must know). gwt-fx worked on all of them without any special-casing in my code.

Also, when things don't work as expected, the author, Adam Tacy, is very cooperative and helpful. He has endured a flood of patches from me to solve some problems that I encountered, and when not satisfied with them, he implemented even better solutions after careful consideration and thoughtful reasoning. Just make sure to get the latest version (1.0.0) that has every bug I encountered so far, fixed.

Adding a supported effects library in the main GWT distribution has been discussed a few times on the contributors list, but apparently it is not a high priority task at the moment. For a good reason I might add, since the release of GWT 1.5 is just around the corner, with support for Java 5 and a whole slew of compiler optimizations. However, with the recent introduction of the GWT incubator project it might be appropriate to start trying out some design ideas on such a library and see how it goes. If you care for such an outcome, don't hesitate to let the GWT developers know about it, by joining the discussion in the recent thread on the subject.

An exceptional interview

My friend Alfred was in a job interview the other day, with a guy named Quincy. It was for a Java programming gig and they were apparently getting too many resumes, so they sat him down with a paper and a pen and asked various questions in order to determine his skills. I hear it is very trendy nowadays, all the cool kids are doing it. After a lot of rather boring, Java-for-dummies kind of stuff, he was given a paper with something more-or-like the following code printed on it:

public String getSomething(String one, Object two, Integer three) {
 if (one != null) {
   if (two != null) {
     if (three != null) {
       String result = reallyGetSomething(one, two, three);
       doSomethingMore(one, two);
       return result;
     } else throw new ObjectNotDefinedException("three is undefined");
   } else throw new IllegalArgumentException("two is not legal");
 } else throw new NullPointerException("one is null");
}

After taking a few moments to read the code, Quincy started firing away:

Q: What is wrong with this method?
A: Lot's of things.

Q: You mean more than one?
A: Of course. The most obvious mistake is that the method does not declare that it can throw ObjectNotDefinedException.

Q: Yep, that's right. Is there anything else?
A: Sure. That's the easy part that the compiler can tell you right away. Furthermore, the method is needlessly complex and has a questionable API.

Q: Really?
A: Sure, the order the null checks are in, force the indentation to increase, making the code difficult to follow. Moreover, if doSomethingMore() and reallyGetSomething() do not modify their parameters and have no side-effects that require this particular ordering, they could be swapped, saving one line:

public String getSomething(String one, Object two, Integer three)
   throws ObjectNotDefinedException {
 // Validate.
 if (one == null)
   throw new NullPointerException("one is null");
 if (two == null)
   throw new IllegalArgumentException("two is not legal");
 if (three == null)
   throw new ObjectNotDefinedException("three is undefined");

 // Do the actual work.
 doSomethingMore(one, two);
 return reallyGetSomething(one, two, three);
}

Q: OK, now what's wrong with the API?
A: Check out the exceptions the method throws back to the caller.

Q: What about them?
A: Only one of them, ObjectNotDefinedException, is a checked exception that would give the caller a chance to react to such an anomaly. The others are unchecked exceptions that will bubble up the call stack and probably blow up in the users face.

Q: Well, perhaps the API designer wanted to differentiate the handling of the three error cases.
A: Perhaps, that's why I said "questionable". But looking at the similarities in the way the method handles its parameters, that would be hard to believe, don't you think?

Q: I'll ask the questions.


Then, after a bit of fiddling with the internals of his briefcase, Quincy presented another sheet of paper with the following printed code listing:

public class ThrowNull {

 public static void main(String[] args) {
   System.out.println(getFluff(null));
   System.out.println(getStuff(null));
 }

 public static String getFluff(String name) {
   if (name == null)
     throw new NullPointerException("BOOM!");
   return "Fluff: "+name;
 }

 public static String getStuff(String name) {
   if (name == null)
     throw null;
   return "Stuff: "+name;
 }
}

Alfred was given a few moments to digest it and then Quincy went on:

Q: What does the program print?
A: That's easy. It prints nothing. At least, nothing useful. It throws a NullPointerException since the caller of getFluff() is null.

Q: Why does getFluff() throw an NPE if the name parameter is null?
A: I suppose the getFluff() API designer considered that it's the caller's job to make sure it is not called with a null argument. He should have put that in the javadoc, though. You do write javadoc comments don't you?

Q: I'll ask the questions. What will the program output if we change the first println() in main to

System.out.println(getFluff("bad"));

A: It's not much of a change. The line "Fluff: bad" will be printed, followed by a new NullPointerException.

Q: So the "throw null" syntax is legal?
A: Hell yeah! Even Neal Gafter is using it.

Q: Which way of throwing an NPE is better?
A: Better in what way?

Q: I'll ask the questions.
A: Well, each one has its merits. The first form is more explicit and traditional, but longer. The second is more succinct and closer to the natural language, but cryptic to newcomers.

Q: So what would you recommend that we use?
A: Does that mean I got the job?

Q: I'll ask the questions.
A: Well, I kinda like the short, concise form, but I don't feel particularly strong about it.

Q: Good answer, because we picked the other one after a three-hour long debate last week.
A: You mean you were debating about ways to throw a NullPointerException for three hours?

Q: Yeah, why?
A: Don't you have real work to do?

Q: I'll ask the questions.



I don't believe Alfred has heard back from them ever since.


I'm actually glad about it.

The case of the disappeared Exception message in GWT RPC

Regular readers of this blog (both of you) will remember my recent love confession for GWT. While I am glad to say that my feelings have not changed one bit, like all maturing relationships, we are getting more intimately familiar with one another. To the point where you know things about your significant other that you'd rather never found out. And since I know there are many others with feelings for GWT, I thought I'd give you a tip or two, in case you want to make a pass.

In my previous post I didn't elaborate on the marvelous feature of GWT RPC. The RPC feature gives your GWT client the ability to communicate with your Java-based backend in a very powerful, yet painless, way. I still plan to discuss it more thoroughly in a future post, but for now I'd like to mention a particular bug (or feature) of the current implementation.

The RPC mechanism allows a client to make a Remote Procedure Call to the server supplying the necessary call parameters and getting back the results. Quite often (as it turns out) the server will encounter an exceptional condition during the execution of this call and will throw an exception to the caller. Luckily an Exception in Java contains a detailed message about the problem encountered (or more accurately its ancestor, Throwable) that the client can rely upon to present the situation to the user. To give an example, you could have an Exception class EarthShatteringException like the following:

public EarthShatteringException extends Exception implements Serializable {

private static final long serialVersionUID = 1L;

public EarthShatteringException() {
}

public EarthShatteringException(String message) {
super(message);
}

public EarthShatteringException(Throwable cause) {
super(cause);
}

public EarthShatteringException(String message, Throwable cause) {
super(message, cause);
}

}


This is pretty much elementary stuff, you extend the base Exception class and also implement the Serializable interface, in order to give GWT a chance to convert your exception object to tiny little bits, transmit them over the wire and recreate the object on the client. So the way you actually throw such an exception on your server might be something like the following:

public void doGreatStuff() throws EarthShatteringException {
if (something.isWrong())
throw new EarthShatteringException("God help us!");
else
doTheGreatStuff();
}

On the client you deal with the exception in a callback function you provide, called onFailure:

public void onFailure(Throwable caught) {
displayError(caught.getMessage());
}

In this example we just display the message contained in the exception to the user.

At least that's what you would expect. In fact, what this code will do in GWT 1.4.60 (the latest release as of this writing) is display the string null.

How could that be you say? I'm glad you asked.

Serialization in GWT is a little different than in the rest of Javaland. This is because the client in runtime is actually JavaScript code, compiled from our original Java code by the miraculous GWT compiler. The compiler uses its own private implementation of the standard Java class libraries, that implements a quite large, but still limited subset of it. In our particular case the implementation of the Throwable class does not implement the Serializable interface and the side effects that entails.

So when our exception gets transmitted over to the client, what gets instantiated is an EarthShatteringException type, but its supertypes are not serialized (hence transmitted) themselves. Therefore our stored error message gets lost somewhere in Cyberspace, never to be seen again. If your server-side code attempted to throw a standard exception subclass from the standard class library, like IllegalArgumentException, things could be even worse, like getting a ClassCastException.

The good news is that this is a known issue among the GWT developers. The bad news is that it is not fixed yet. However, a simple workaround exists. If you store a detailed error message in your own exception object and override the getMessage method (or its cousins) to return that copy, instead of the supertype's, it will produce the excpected outcome. For the previous example we could do it like this (changes in bold):


public EarthShatteringException extends Exception implements Serializable {

private static final long serialVersionUID = 1L;

private String message;

public EarthShatteringException() {
}

public EarthShatteringException(String message) {
super(message);
this.message = message;
}

public EarthShatteringException(Throwable cause) {
super(cause);
}

public EarthShatteringException(String message, Throwable cause) {
super(message, cause);
this.message = message;
}

public String getMessage() {
return message;
}
}


If you are a grumpy kind of person and hasten to whine about the apparent intricacies of GWT, rest assured that most of the time GWT RPC is painless as advertised. In fact it seems so easy that people are already hooking it to other server-side frameworks, like Seam and Spring. It provides an elegant way to hook a modern server architecture with the most innovative web client technology on the planet.

I promise you, you'll love it.

From CVS to Subversion

My friend George, whose day job requires the artistic and scientific skills of a system administrator (which pretty much boils down to: keep the users happy and yourself invisible), was envious of a guy who quit his job to write a web application framework in Lisp.

Common Lisp.

Of all languages.

Anyway, George is a great guy, at least when he is not having these wacky ideas, and an excellent system administrator. So the fact that he expressed envy for a programming gig, struck a chord in me. George loves challenges, too.

As a teenager I was a huge fan of MacGyver, a TV series where the main character is constantly facing intractable problems and always finds a solution through improvisation. In the same spirit, I have always liked doing challenging things, even when they appeared useless to some, rather than take the easy tasks with the most fanfare. As a software developer I seek challenges mainly through diversity, constantly seeking to conquer another problem space, another language, another framework in the vast field of computer science. But I've learned by now that nothing can be more challenging and satisfying at the same time for me, than a system administration task.

So I got myself one. I've signed up for the conversion of our source code repository from CVS to Subversion, as a break from my coding assignments. After eight productive years of using CVS as our main version control system, we became Subversion converts a couple of weeks ago. It hasn't been a wholesale migration yet, mainly for risk management reasons, but I've been committing exclusively to the new repository all last week. We began hosting new codebases (like my VCP project) in our shiny new repository, but for the time being we are still keeping the old ones in CVS. The process was rather painless. Not that it was a surprise for me. I have been making migration simulations and testing for about a year now and they always worked fine.

We had three main reasons for migrating to Subversion:

1. Versioning renaming and copying files and directories. Since we are working mostly in Java where code refactoring often leads to renaming (for a class name change) and moving files around (for a package name change), it is imperative that we keep the change history of a file after renaming it. Till now the history was just lost after a rename, something that tended to make refactoring happen only when absolutely necessary.
   
2. Atomic commit sets. Having the ability to commit every changed file in a single batch with the same revision number, made rolling back changes easier. It also removed the need to keep a separate log of our CVS commits in order to find out what files were modified along with a specific one in the same commit.
   
3. Cheap branching and tagging. Experimental branches for fleshing out risky ideas should become more common now, since Subversion has better support for them. Also, no more tag-sliding to cope with unexpected last-minute fixes before a release, although, to be fair, Eclipse already helped with that.
   

However we are grateful for other benefits as well, perhaps less critical, but ones that provide an air of modernity to our deployment:

1. The repository is now served via HTTP, allowing far more versatile deployments, than cvs over ssh. SSL tunnels allow for remote access to the repository without a VPN connection, something that was out of the question in the past, since it would mean that shell access to our development server would be open to the world. Also, the ability to use LDAP for authentication and authorization, removed the need for shell accounts for every committer. That had been partly achieved in the past through PAM and the pam_ldap module, but setting it up involved screaming, crying, weeping, swearing and hair pulling. Not looking forward to it, ever again.
   
2. Binary diffs make storing binary files in the repository cheaper. If you store large binary files in the repo, like third-party libraries, CVS adds the whole new version, not just the changes. Not that we worry too much about the repository space, but it definitely feels like we are in the 21st century.
   
3. Versioning symbolic links. Not that I expect to use them a lot, but still a nice touch.
   
4. We got hook scripts from the vendor, for tasks like e-mail messages on commit, log message cleanup, etc. We had all that with CVS, but I had blatantly copied most of them from the FreeBSD project's excellent repository and while they were of the highest quality, they came without much documentation. Since I already had a working setup it did not seem like a big deal, but if I had to do it all over again copying scripts from, say Apache repositories, I think I would have posted this blog on September.
   
5. We also got repository mirroring and backup without file system specific tricks. Backing up the repository was a straight file copy for CVS, but since Subversion is more sophisticated (even for the fsfs backend that we use), having to resort to file system snapshots for a proper backup would have been a pain. Our development server is a FreeBSD system and UFS2 snapshots are relatively cheap, but not as simple as hot-backup.py. Also, repository mirroring only came up once and it would have required a nullfs mount from another jail in the server, but having the option of svnsync makes things simpler.
   
6. Fine-grained authorization for repository access with CVS over SSH, required enabling ACL support in UFS2 and manually handling the permissions in each folder or file. Subversion over HTTP however moves the authorization configuration on a separate place, as it should be.
   

Subversion definitely seems like CVS done right. There is a clear scent of modernity in every corner I look, compared to CVS. I was surprised to find various hook scripts and assorted infrastructure programs written in Python, Perl and Ruby. These are pretty much all the mainstream languages in a UNIX system these days, which says something about something.

The only thing I'm not terribly excited about, is the Subversion support in Eclipse. I've been using Subclipse so far and although I haven't had any real problems, the integration is not as good as with CVS. Which brings me to the main reason I picked Subversion for our next-gen version control system: Eclipse support. Had it not been for mature plugins like Subclipse and Subversive, all the advantages over CVS would have meant squat. Having to resort to command-line tools in order to perform diff, update and commit would have sent our team's productivity to the drain, before you can say Alt-Tab.

If I valued technical excellence over productivity, I would have moved us to Bazaar, Mercurial or git instead of Subversion. Every cool project these days seems to pick one of those for a VCS. They even have alpha-quality Eclipse plugins. Then I would need to schedule a weekend for the whole team in a Mediterranean resort and clue them in the joys of distributed version control systems.

Hmm, not a bad idea now that I think of it.

Then again, had I chosen technical excellence over getting work done, I would probably quit my job and build a web application framework in Lisp. Common Lisp.

Damn. George was right, again.

I hate it when he is right.