Showing posts with label open-source. Show all posts
Showing posts with label open-source. Show all posts

Tuesday, November 6, 2012

JavaScript debugging updates

There have been a lot of interesting developments in the JavaScript debugging front since my last update, so let me give you a brief overview of the most important changes.

Let's begin with the eagerly awaited Firefox OS. Starting the debugger in Firefox OS has gotten way easier. There is a toggle in the settings app that will set or reset the devtools.debugger.remote-enabled pref, so that you don't have to mess with files and adb at all. At least for production builds that have marionette disabled by default. Developer builds will have to keep disabling marionette for a bit longer. You can find the Remote Debugging toggle in Settings -> Device Information -> More Information -> Developer. Note that you will have to reboot the device (or restart the b2g process) for the change to have effect. Also, as you can see in the screenshot, it's a good idea to disable out-of-process support while debugging, at least until the debugger can support it.

Firefox OS settings
Firefox OS settings
While we are on the subject of debugging JavaScript on mobile, I shouldn't neglect to mention that Mihai Sucan has landed web console support for Firefox OS and Firefox for Android. There are still a few rough edges, particularly in the case of Firefox OS, but now you can connect your desktop Firefox to your mobile device and see the errors and logged messages, as well as evaluate JS code in the context of the mobile page.

Speaking of desktop Firefox, the amazing Victor Porof has landed an obscene number of patches in the last few weeks, with many fixes and tweaks for the debugger frontend. Of those I'd like to highlight the new functionality to search for variables in the current scope chain while the debugger is paused. Prepend your search with a star to search in the variables view. When you deal with large objects with long scope chains, finding the variable you are looking for can be like searching for a needle in a haystack. Now it's a piece of cake.
Searching for variables
Searching for variables
And last, but certainly not least, chrome debugging support has appeared in Nightly this past weekend! This will make debugging add-ons or Firefox itself way easier than ever before. You will need to enable remote debuging (devtools.debugger.remote-enabled) and browser chrome debugging as well (devtools.chrome.enabled, the same pref you switch to get Scratchpad to run against chrome).

Chrome debugging preferences
Chrome debugging preferences
This will get you a nice new Browser Debugger menu item in the Web Developer menu, which launches a separate debugger process to inspect your browser instance.

Browser Debugger menu item
Browser Debugger menu item

There has never been a better time to dive into Firefox internals and become a valued contributor. Take a look at this bug, to see how simple debugging has just become. Help us make the web better, by improving the lives of more than 400 million users!

Friday, October 5, 2012

Debugging Firefox OS

A few months ago, long before the Firefox debugger had shipped, I had tried to see what it would take to get it to debug B2G, or as we call it now, Firefox OS. The hack proved successful and with time it grew into an important debugging target for us. I demoed it at JSConf last April, but at that time a lot of the debugger frontend functionality had not been available in nightlies. As of yesterday, however, nightly builds of Firefox can debug nightly builds of Firefox OS, both in the device and desktop versions. Let me give you a quick rundown of the necessary steps.

First of all you need to go to about:config on your desktop Firefox and turn devtools.debugger.remote-enabled to true. After restarting Firefox you will have a new Remote Debugger menu item in your Web Developer menu.

You need to change the same setting in Firefox OS as well, and that requires finding the prefs.js file. For desktop builds, this will be in gaia-repository/profile/prefs.js. For devices, you should connect it to the desktop through a USB cable, use adb shell to connect to the device, and find the name of the default profile in /data/b2g/mozilla/. The prefs.js file wil be in that subdirectory, which in my case is /data/b2g/mozilla/ngsweij3.default/prefs.js. You can get that file off the device using:

adb pull /data/b2g/mozilla/ngsweij3.default/prefs.js prefs.js
 
That will store a copy of that file in your current working directory. You need to modify this file and then send it back to the device with:

adb push prefs.js /data/b2g/mozilla/ngsweij3.default/prefs.js

The necessary changes to that file are the following two lines:

user_pref("devtools.debugger.remote-enabled", true);
user_pref("marionette.defaultPrefs.enabled", false);

They enable the debugger server and at the same time disable the Marionette server, to avoid a conflict that we haven’t resolved yet.

That is all that is necessary for the desktop Firefox OS case, but you need to do one more thing for debugging Firefox OS on the device. If you are going to debug over a USB connection (which will also provide helpful console logs via adb logcat) you will need to forward a local port to the device using adb:

adb forward tcp:6000 tcp:6000

Port number 6000 is the default and if you need to change it for some reason, you can do so with the following pref:

user_pref("devtools.debugger.remote-port", 7000);

Debugging over WiFi is also possible, although disabled by default, so you will need to change the following pref to do so:

user_pref("devtools.debugger.force-local", false);

In that case, you don’t have to do the port forwarding step, but you may change the port number in the same way.

In order for the new settings to take effect in the device you have to restart it with ‘adb reboot’.

That’s all folks! Now you just start your nightly Firefox and select the Remote Debugger menu item. A dialog will pop up asking for the remote address and port number to connect to. The defaults will work for the desktop Firefox OS or the device debugging-over-USB cases (unless you changed the port number of course!). If you opted for debugging over WiFi, you will need to provide the IP address of the device. You can find it by tapping on the connection name in the settings.

After the connection is established, you will see a list of the sources loaded on the device, giving you the option to inspect the code, set breakpoints, step through the execution or even modify the values of variables. Here is a screencast that shows the debugger in action:



A better introductory guide to debugging Firefox OS will be available soon on MDN. This is just the first step in supporting Firefox OS development with Firefox developer tools. Just around the corner are web console support and profiling, with more to follow in the near future. If you have issues or suggestions for these tools, please get in touch! Post a comment, file a bug or pop in #devtools for a chat. Let's make the web the best platform to develop on!

Monday, October 1, 2012

Debugging Firefox

Last week we had a Firefox Developer Tools team get-together at the Mozilla London office. These meetups are an extremely valuable tool for the project to tackle hard problems, because when you put all of the experts on a particular hard problem in the same room, hard problems tend to run away screaming.

We managed to attack a lot of hard problems this week: a profiler, source maps, the developer toolbox, web console remoting and chrome debugging, were just a few of them. And not only that, but we also landed a few exciting new features as well, with fullscreen scratchpad and markup panel preview being my personal favorites. I'm sure others will cover the rest in appropriate detail, but I'd like to say a few things about chrome debugging.

Last March, during the previous team meetup we had managed to get a chrome debugging proof-of-concept working. We were thrilled by it, even though it was a big hack, because the implications were enormous. Something that not many people know is that the Firefox frontend is built in JavaScript, CSS and XUL, which is an HTML derivative with additional capabilities. That is, we use the exact same technologies that power your favorite web applications, to build the browser that displays said web apps to the user. This similarity has always been front and center to our thinking when considering tools for web developers: how can we use these tools to make hacking on Firefox easier?

Without easy to use tools, fixing bugs is time-consuming and adding features demands a very disciplined approach to programming. It hampers our ability to move the project faster and innovate rapidly. Furthermore, having a non-profit foundation to back the development of an open-source browser means that you don't have the option to simply throw more money at the problem, hiring more engineers than everybody else to increase the pace of innovation in the product. This is a browser created by thousands of contributors to take care of their own interests. We have to make their work easier, by giving them the tools they need to work faster.

When we did the chrome debugging prototype last March, the Firefox Debugger had been in trunk for a short while, disabled by default. No visual styling had been done on it and only the most basic of functionality was there. Six months later we have shipped the first version and we keep iterating on it.

The proof-of-concept couldn't debug chrome window globals, only modules, and was hijacking the remote debugging protocol in a hackish way that was guaranteed never to make it into a Firefox release. This time we have both kinds of globals supported and a sound protocol design that was agreed upon a few months ago.

This new tool, which we dubbed "Browser Debugger", will be useful to add-on developers, too. In my demo last week I used it to debug Firebug. We have patches under review or close enough that implement a first draft of the experience we aim to present users with. I want to land that as soon as we can, enable the feature by default (but conditioned on the devtools.chrome.enabled pref like other tools) and keep iterating.

It was without a doubt a very rewarding week. Some might expect us to also treat these rare events as opportunities to hang out, get drunk and take embarrassing pictures of each other. I am not going to either confirm or deny such allegations. At least not until I'm presented with hard, photographic evidence. Preferably on Flickr.

Sunday, February 19, 2012

Debugging mobile phones

Last week I posted an update on the progress we are making towards a functional JavaScript debugger in Firefox. We still have a ways to go before we get there, but the foundation this is based on, namely the Remote Debugging Protocol, provides a solid basis for debugging mobile browsers, and as it turns out, even mobile operating systems. Supporting these use cases is still farther down our list, but I decided to spend a few hours the other day trying to estimate how far we are from this goal. It turns out we are not that far, and I have a screencast to prove it.

I had done some prototyping work for Firefox for Android a while back, which was rather straightforward, so I was curious to know what would it take to tackle a more ambitious project: debugging Boot to Gecko. After a few hours of hacking and orienting myself around the B2G and Gaia code base, I got this far:



Not too bad I think for a few hours work. Don't expect this to be ready of course, before we have a working debugger in desktop Firefox. If, however, you'd like to follow along and maybe even lend a hand, you are always welcome!

Tuesday, October 12, 2010

GSS/Pithos update: iPhone, Android, Mongo & getting to 2.0


It has been a while since my last post on GSS/Pithos, and there have been quite a few important announcements that I have not blogged about, only tweeted:
  • a shiny native iPhone client
  • a brand new Android client
  • a back-office application for statistics and admin tasks
  • public folders for hosting static web sites
  • a slew of performance, usability and bug fixes for the web & desktop clients
  • new user registration and quota upgrade workflows
And since the code base has been adopted by other European NRNs, we have also improved our release engineering process and documentation. This however was just the tip of the iceberg. What we are now close to completing, is nothing short of an almost complete rewrite of the core GSS server, aiming at even greater performance and scalability. We are about to release GSS 2.0.

The initial plan was far more modest. We had identified the PostgreSQL installation that stores all of the the system’s data besides the actual files, as a future bottleneck when user adoption would start to accelerate. In such an event there were a few low-to-medium-cost solutions that we could pursue, like replication and manual sharding, but since the problem seemed best suited for a NoSQL solution, we decided to bite the bullet and attempt a transition.

Our field search included lots of products, like Mongo, Couch, Cassandra, Riak, Redis, HBase, MemcacheDB and others. The basic requirements were high write throughput, sharding, replication and easy migration. Most of the products boasted high performance for writes and lots supported replication. In the end we discarded the ones that didn’t support sharding, which limited the options a lot, and we investigated the various APIs, trying to figure out what it would mean to rewrite GSS for one of these. In the end we chose Mongo for its rich query support, since it was the best fit for our problem at hand, making the transition of the existing code base easier. Although we’d love to have had the opportunity to try different ports for each of those datastores, it is most likely that we would have had to rewrite the entire server from scratch, which was not an option.

The initial plan was to try to isolate the changes to the data access layer, essentially building a new adapter for the middle tier that would appear almost identical to the existing code. This effort was fruitless however, since we could not find satisfactory solutions for various issues, like transactions, data mapping, entity lifecycle, etc. We came to the conclusion that even if we managed to build a not-too-leaky abstraction for Mongo, it would have probably killed the performance, making the switch from PostgreSQL pointless.

So we ditched JPA and container-managed transactions and decided to make a simpler POJO mapping to Mongo using Morphia, a Java data mapper for Mongo that takes care of the nitty-gritty details and repetitive coding, for only a minor performance hit. Since Mongo only supports atomicity for a single query, we reverted to manual transaction handling and decided to ditch the EJBs that formed the backbone of GSS altogether, since they didn’t buy us anything at that point. The new GSS server would be a set of servlets, service and domain objects, wired together via Google Guice. Since we could host the new server in a plain servlet container, we went from JBoss to Jetty, which vastly reduced the server startup time from 60 seconds down to 2. As a consequence, development round-trip times went down, and coding was fast again. Lots of object copying and conversions from entities to DTOs and back again were now unnecessary, since there was no more a transaction boundary at the session bean layer. This kept memory usage low, increased cache locality and boosted performance. There are lots of interesting details about our approach, but they would be better discussed in separate blog posts. If you are interested, you can already check out the code in the gss2 branch of the GSS repository.

The next generation GSS server is not production ready yet, but we'll be getting there soon, and so far it has been a great ride!

Friday, February 19, 2010

The Cloud Desktop


This was originally posted in the mynetworkfolders.com blog.

We live in a browser these days. E-mail, word-processing, spreadsheets, chat, are all occupying a browser window in front of me. Maybe not for everyone yet, but the trend is clear, and has been clear for quite some time now. This is a good thing. The Web of today bears little similarity to the Web of the millennium. It's fast, it's powerful and it's ubiquitous. A consequence of that evolution is the simplicity with which I can continue my work when leaving my Linux workstation at work and sit in front of my Mac at home. Since my applications are web-based, they are OS- and browser-agnostic. More importantly, my data are always there when I need them. Readily accessible from my desktop, laptop or mobile phone. I may keep local backups for archival purposes, but my master copies are always online.

Assuming this trend is indeed inescapable, one can't help but wonder: what are the applications that have not migrated to the cloud, yet? What kind of things would I like to do online, seamlessly, from every device I own? What are the barriers that make our current online experience less powerful than the desktop one?

One thing that we noticed was missing from the online desktop, was the... desktop! Yes, my spreadsheets are online. Sure, my pictures are online. Yes, I can upload arbitrary files in various storage services. But, I couldn't find a way to use a familiar file manager interface to work with my data in the cloud, as I do on my computer. This was the need we decided to fulfill with MyNetworkFolders.

MyNetworkFolders attempts to provide an answer to the question: how would my desktop look in the cloud? And even though we are nowhere near done yet, the results so far are exciting and very promising.

MyNetworkFolders offers a familiar file manager interface for my cloud storage needs, with advanced functionality and ease of use. Documents can be viewed, modified, searched or sent to the trash. They can be shared with other users or among the members of a group using flexible ACLs, or even be publicly accessible from the whole internet. They can be versioned for tracking changes more effectively. The service can be accessed from a variety of ways, including a web browser, a desktop client, a WebDAV network share or (really soon) a mobile phone. But most importantly, there is no vendor lock-in. There is a REST-like API available, battle-tested from all the above client implementations, giving everyone the opportunity to access his data in his own particular way. Or even get them out of our service altogether.

Oh, and one other thing. It's all based on open-source software. The core infrastructure, the various clients we have built, even API helpers in various programming languages, are all available for anyone and everyone to see, use and modify. Or maybe used to setup and run YourNetworkFolders, if you feel really ambitious.

If however you just want a true desktop environment, with unparalleled flexibility, and maybe also see how far the cloud desktop rabbit-hole goes, then come along. It's going to be an exciting ride.

Thursday, August 20, 2009

Spell-checking in JavaScript

I just heard of Atwood's Law a few days ago. Apparently Jeff Atwood first published his discovery a couple of years ago, which is like a century in Internet time, but I can't keep up with everything. The Law states that "any application that can be written in JavaScript, will eventually be written in JavaScript". As Atwood explains, it is based on Tim Berners-Lee's Principle Of Least Power:

Computer Science spent the last forty years making languages which were as powerful as possible. Nowadays we have to appreciate the reasons for picking not the most powerful solution but the least powerful. The less powerful the language, the more you can do with the data stored in that language.


I think there is a common theme between Atwood's Law, Zawinski's Law ("Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can") and Greenspun's Tenth Rule ("Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp"). The inevitability of the specified outcome makes the world seem like a much simpler place. There is no need to argue. Believe and ye shall be redeemed.

I might have been in this state of mind, when I came across Peter Norvig's "How to Write a Spelling Corrector", because I knew instantly I had to port the algorithm to JavaScript. The algorithm is quite simple and has already been ported to many different languages, so I seized the opportunity to study the differences in expressiveness, performance and style, between these languages and my latest affection, JavaScript.

After a few night's work I have it working and free (as in beer and speech) for anyone to use. Check out the speller project on GitHub. If you want to understand how the algorithm works, you should go and read Norvig's article, although you might get some hints from the comments in my code. The modestly condensed version of the code is 53 lines:


var speller = {};
speller.train = function (text) {
var m;
while ((m = /[a-z]+/g.exec(text.toLowerCase()))) {
speller.nWords[m[0]] = speller.nWords.hasOwnProperty(m[0]) ? speller.nWords[m[0]] + 1 : 1;
}
};
speller.correct = function (word) {
if (speller.nWords.hasOwnProperty(word)) return word;
var candidates = {}, list = speller.edits(word);
list.forEach(function (edit) {
if (speller.nWords.hasOwnProperty(edit)) candidates[speller.nWords[edit]] = edit;
});
if (speller.countKeys(candidates) > 0) return candidates[speller.max(candidates)];
list.forEach(function (edit) {
speller.edits(edit).forEach(function (w) {
if (speller.nWords.hasOwnProperty(w)) candidates[speller.nWords[w]] = w;
});
});
return speller.countKeys(candidates) > 0 ? candidates[speller.max(candidates)] : word;
};
speller.nWords = {};
speller.countKeys = function (object) {
var attr, count = 0;
for (attr in object)
if (object.hasOwnProperty(attr))
count++;
return count;
};
speller.max = function (candidates) {
var candidate, arr = [];
for (candidate in candidates)
if (candidates.hasOwnProperty(candidate))
arr.push(candidate);
return Math.max.apply(null, arr);
};
speller.letters = "abcdefghijklmnopqrstuvwxyz".split("");
speller.edits = function (word) {
var i, results = [];
for (i=0; i < word.length; i++)
results.push(word.slice(0, i) + word.slice(i+1));
for (i=0; i < word.length-1; i++)
results.push(word.slice(0, i) + word.slice(i+1, i+2) + word.slice(i, i+1) + word.slice(i+2));
for (i=0; i < word.length; i++)
speller.letters.forEach(function (l) {
results.push(word.slice(0, i) + l + word.slice(i+1));
});
for (i=0; i <= word.length; i++)
speller.letters.forEach(function (l) {
results.push(word.slice(0, i) + l + word.slice(i));
});
return results;
};


It may not be as succinct as Norvig's Python version (21 lines), or the record-holding Awk and F# versions (15 lines), but is much better than C (184 lines), C++, Perl, PHP, Rebol and Erlang. There is even a Java version with 372 lines. It must contain some sort of spell-checking framework in there, or something. The condensed version above, although correct, has terrible performance in most JavaScript engines, however. For real-world use you should pick the regular version which may be slightly longer, but performs much better.

Since this was a toy project of mine, I wanted to play with ServerJS modules as well, in order to run it as a shell script. I turned the code into a securable module, so you can run it from the command line, using narwhal. I have a couple of scripts to that end. Of course since this is JavaScript, you can try it from your browser, by visiting the demo page, courtesy of GitHub Pages. Be sure to use a modern browser, like Firefox 3.5, Safari 4 or Chrome 3 (beta), otherwise you won't be able to run the test suite, since I used the brand-new Web Workers to make the long-running tasks run in the background.

Norvig's Python implementation took 16 seconds for test 1 and the best I got was 25 seconds with Safari 4 on my Mac. Narwhal is using Rhino by default, so it is definitely not competitive in such tests (139 seconds), but I'm planning to fix support for v8cgi and give that a try.

And it goes without saying that I'd love to hear about ways to improve the performance or the conciseness of the code. If you have any ideas, don't be shy, leave a comment or even better fork the code and send me a pull request on GitHub.

Tuesday, July 7, 2009

GSS authentication

As I've described before, the GSS API is a REST-like interface for interacting with a GSS-based service, like Pithos. Using regular HTTP commands a client can upload, download, browse and modify files and folders stored in the GSS server. These interactions have two important properties: they store no client state to the server and they are not encrypted. I have already mentioned the most important benefits from the stateless architecture of the GSS protocol, namely scalability and loose coupling along the client-server boundary. In the same line of thought, SSL/TLS encryption of the transport was avoided for scalability reasons. Consequently, these two properties of the communication protocol, lead to another requirement: message authentication for each API call.


Since no session state is stored in the server, the client must authenticate each request as if it were the first one. Traditional web applications use an initial authentication interaction between client and server, that creates a unique session ID, which is transmitted with each subsequent request in that same session. While using this ID, the client does not need to authenticate again to the server. Stateless protocols, like WebDAV for instance, cannot rely on such an ID and have to transmit authentication data in each call. Ultimately the tradeoff is a minor increase in the message payload, in return for a big boost in scalability. The HTTP specification defines an Authorization header for use in such cases and WebDAV uses the HTTP Digest Access Authentication scheme. The GSS API uses a slight variation in that theme, blended with some ideas from OAuth request signing.

Essentially the standard HTTP Authorization header is populated with a concatenation of the username (for identifying the user making the request) and a HMAC-SHA1 signature of the request. The request signature is obtained by applying a secret authentication token to a concatenated string of the HTTP method name, the date and time of the request and the actual requested path. The GSS API page has all the details. The inclusion of the date and time helps thwart replay attacks using a previously sniffed signature. Furthermore, a separate header with timestamp information, X-GSS-Date, is used to thwart replay attacks with a full copy of the payload. The authentication token is issued securely by the server to the client and is the time-limited secret that is shared between the client and server. Since it is not the user password that is used as a shared secret, were the authentication token to be compromised, any ill effects would have been limited to the time period the token was valid. There are two ways for client applications to obtain the user's authentication token, and they are both described in detail in the API documentation.

In my experience, protocol descriptions are one thing and working code is a totally different one. In that spirit, I'm closing this post with a few tested implementations of the GSS API signature calculation, for client applications in different languages. Here is the method for signing a GSS API request in Java (full code here):


public static String sign(String httpMethod, String timestamp,
String path, String token) {
String input = httpMethod + timestamp + path;
String signed = null;

try {
System.err.println("Token:" + token);
// Get an HMAC-SHA1 key from the authentication token.
System.err.println("Input: " + input);
SecretKeySpec signingKey = new SecretKeySpec(
Base64.decodeBase64(token.getBytes()), "HmacSHA1");

// Get an HMAC-SHA1 Mac instance and initialize with the signing key.
Mac hmac = Mac.getInstance("HmacSHA1");
hmac.init(signingKey);

// Compute the HMAC on the input data bytes.
byte[] rawMac = hmac.doFinal(input.getBytes());

// Do base 64 encoding.
signed = new String(Base64.encodeBase64(rawMac), "US-ASCII");

} catch (InvalidKeyException ikex) {
System.err.println("Fatal key exception: " + ikex.getMessage());
ikex.printStackTrace();
} catch (UnsupportedEncodingException ueex) {
System.err.println("Fatal encoding exception: "
+ ueex.getMessage());
} catch (NoSuchAlgorithmException nsaex) {
System.err.println("Fatal algorithm exception: "
+ nsaex.getMessage());
nsaex.printStackTrace();
}

if (signed == null)
System.exit(-1);
System.err.println("Signed: " + signed);
return signed;
}


Here is a method for signing GSS API requests in JavaScript, that uses the SHA-1 JavaScript implementation by Paul Johnston (full code here):


function sign(method, time, resource, token) {
var q = resource.indexOf('?');
var res = q == -1? resource: resource.substring(0, q);
var data = method + time + res;
// Use strict RFC compliance
b64pad = "=";
return b64_hmac_sha1(atob(token), data);
}

Here is a Tcl implementation by Alexios Zavras (full code here):


proc ::rest::_sign {id} {
variable $id
upvar 0 $id data
set str2sign ""
append str2sign $data(method)
append str2sign $data(date)
set idx [string first ? $data(path)]
if {$idx < 0} {
set str $data(path)
} else {
incr idx -1
set str [string range $data(path) 0 $idx]
}
# append str2sign [::util::url::encode $str]
append str2sign $str
puts "SIGN $str2sign"
set sig [::base64::encode [::sha1::hmac -bin -key $::GG(gsstoken) $str2sign]]
set data(signature) $sig
}

I'd love to show implementations of the signature calculation for other languages as well, but since my time is scarce these days, I could use some help here. If you've written one yourself and you'd like to share, leave a comment and I'll update this post, with proper attribution of course.

Thursday, June 4, 2009

Ubuntu on EEE PC

A few weeks ago I set out to upgrade Ubuntu on my EEE PC 901 from Intrepid to Jaunty. It was an, um, interesting experience, so I'm jotting down these notes for next time. Upgrading from Update Manager was not an option, because my root partition was not big enough to host the upgrade. The 901 has a 4GB SSD that I'm using as root and an 8GB (or 12GB) SSD that I've put /home on. It turns out that 4GB is OK for running a modern operating system, but a little short for updating it.

The usual procedure for upgrading is to use the official ISO images to boot from. As always you need a USB stick, or an external optical drive. I settled for the former, as you can see below.

The good news with Jaunty is that a special netbook edition is available for download. With previous editions you had to start with the desktop edition and do lots of tweaking, to arrive at a satisfactory result. With the netbook edition however things were unbelievably smooth. Everything worked out of the box with my EEE PC. Well, almost everything, but I will get to that in a while.

The first step after downloading the ISO (for the security paranoid at least) is verifying the download:


$ md5sum ubuntu-9.04-netbook-remix-i386.img
8f921e001aebc3e98e8e8e7d29ee1dd4 ubuntu-9.04-netbook-remix-i386.img

Then, we can write the image to the disk. I used Ubuntu ImageWriter (package usb-imagewriter) from another Ubuntu PC:


That went very smooth. Next step was booting the EEE PC with the newly formatted USB disk. Hitting ESC on boot, allows one to change the boot drive sequence. If this is the first time you are replacing the original OS, you probably need to disable Boot Booster from the BIOS first.

The installation process is quite simple. The only important thing was partitioning in my case, since I wanted to preserve my /home partition intact. Last time I installed Ubuntu I went with ext2 filesystems, trying to squeeze as much life from my solid state disks as I could. After doing some more research on the subject, I decided that I'll probably throw away my EEE long before the disks begin to wear out. Either that, or I'll throw an SD card in the expansion slot. Life is full of choices, particularly on the $300 range.

So I selected manual partitioning and chose ext3 for my first SSD. This is a screenshot from my first attempt. No, actually it was my second attempt. I screwed up the first one. Turns out I screwed up this one, too. See if you can spot why.


It may not be that obvious, but swap should not be on the first disk. If you are using swap at all (and I know many suggest that you don't), it would be a better choice to put it on the second disk. With 1GB of RAM, swap should be at least 1GB, which leaves only 3GB for Linux. The default installation needs something north of 2GB, so that leaves less than a gig to install software you use. Unless of course you are planning to play symlink tricks, in which case I should probably offer my condolences. Anyway, I ended up chopping 1GB with gparted from the second disk and redoing the installation again, but fortunately you won't have to.

What you should definitely do however, is put aside 16MB of space in the first disk for enabling Boot Booster in the BIOS again. This is an ASUS feature that persists the BIOS configuration on the disk, shaving a couple of seconds on each boot. It may not seem much, but think about how many times you are going to boot this thing. If you add them up, it starts to matter. Boot Booster needs only 8MB of disk to function, but I found that the installer's partition editor would only allocate one cylinder for 8MB, while we need two. Using 16MB did the right thing.

The regular customization I always do is to put /tmp and /var/tmp on tmpfs, so that many common operations don't touch the disks at all. Put the following lines in fstab to do that:


tmpfs /tmp tmpfs defaults,noexec,nosuid 0 0
tmpfs /var/tmp tmpfs defaults,noexec,nosuid 0 0

After editing /etc/fstab the proper way to remount these file systems is by following these steps:
  1. Log out of the Ubuntu desktop in order to minimize the number of open temp files.
  2. Press CTRL-ALT-F1 on your keyboard. You should see a login prompt. Login with your usual user/pass.
  3. Clear out the files in the existing /tmp and /var/tmp directories.
    sudo rm -rf /tmp/* /var/tmp/*
  4. Mount the filesystems using tmpfs.
    sudo mount /tmp
    sudo mount /var/tmp
  5. Reboot the EEE PC.
    sudo shutdown -r now
Next, from the boot-as-fast-as-you-can department comes another tweak. In /boot/grub/menu.lst set "timeout 0" in order to skip the kernel selection delay. If I ever need to boot a different kernel, I'll change it back to something larger, thankyouverymuch.

  1. Log out of the Ubuntu desktop. (Yes.. I said log out. This will go a long way to ensuring the fewest number of temp files are currently open.)
  2. Press CTRL-ALT-F1 on your keyboard. You should see a login prompt. Login with your usual user/pass.
  3. Clear out the files in the existing /tmp and /var/tmp directories.
    sudo rm -rf /tmp/* /var/tmp/*
  4. Mount the filesystems using tmpfs.
    sudo mount /tmp
    sudo mount /var/tmp
  5. Reboot the EEE PC.
    sudo shutdown -r now
Now in order to enable back Boot Booster follow these instructions for configuring the small partition we created earlier. When you reboot make sure to enable the option in the BIOS (it will be hidden when the proper partition isn't found).

Now you are all set. At least if you have no use for WPA-secured WiFi. Because if you do, you'll hit a bug with the included rt2860 driver. The details aren't that interesting, but the available options are two: either use the array.org kernel or patch the driver. The former solution used to be necessary with Intrepid, since lots of things were not working with the default kernel. However after trying out both, I don't see much need for Adam's kernel in Jaunty, unless you are terrified when encountering patch, make and their ilk. Since I'm on first name basis with them, I went with the second option, hoping that a future kernel update will bring along the fixes.

All in all, the process this time around was very easy. Almost easy peasy. My EEE PC with Ubuntu Netbook Remix makes me almost forget that I'm not on my beloved Mac.

Almost.

Thursday, April 23, 2009

An introduction to the GSS API

Application Programming Interface or API design is one of my favorite topics in programming, probably because it is both a science and an art. It is a science because there are widely accepted principles about how to design an API, but at the same time applying these principles within the constraints of a given programming language, requires the finesse of an experienced practicioner. Therefore it is with great pleasure that I'll try to explain the ins and outs of the GSS REST-like API, as promised before. As I've already mentioned, GSS is both the name of the source code project in Google Code, as well as the GRNET-sponsored service for the Greek research and academic network (although, it's official name after leaving the beta stage will be Pithos). Since anyone can use the open-source code to setup a GSS service, in this post I'll use generic examples, so anyone writing a client for the GRNET service should modify them accordingly.

When developing an application in a particular programming language, we are used to thinking about the APIs presented to us by the various libraries, which are invariably specified in that same language. For instance, for communicating with an HTTP server from a Java program we might use the HttpClient library API. This library presents a set of Java classes, interfaces and methods for interacting with web servers. These classes hide the underlying complexity of making the low-level HTTP protocol operations, allowing our mental process to remain constantly in a Java world. We could however interact with a web server without such a library, opting to implement the HTTP protocol interactions ourselves instead. Unfortunately, there is no such higher-level library for GSS yet, wrapping the low-level HTTP communications. Therefore this post will present a low-level API, in the sense that one has to make direct HTTP calls in his chosen programming language. The good news is that the following discussion is useful for programmers with any background, since there is support for the ubiquitus HTTP protocol in every modern programming language.

A RESTful API models its entities as resources. Resources are identified by Uniform Resource Identifiers, or URIs. There are four kinds of resources in GSS: files, folders, users & groups. These resources have a number of properties that contain various attributes. The API models these entities and their properties in the JSON format. There is also a fifth entity that is not modeled as a resource, but is important enough to warrant special mention: permissions.

Users · Users are the entities that represent the actual users of the system. They are used to login to the service and separate namespaces of files and folders. User entities have attributes like full name, e-mail, username, authentication token, creation/modification times, groups, etc. The URI of a user with username paul would be:

http://host.domain/gss/rest/paul/
The JSON representation of this user would be something like this:
{
"name": "Paul Smith",
"username": "paul",
"email": "paul@gmail.com",
"files": "http://hostname.domain/gss/rest/paul/files",
"trash": "http://hostname.domain/gss/rest/paul/trash",
"shared": "http://hostname.domain/gss/rest/paul/shared",
"others": "http://hostname.domain/gss/rest/paul/others",
"tags": "http://hostname.domain/gss/rest/paul/tags",
"groups": "http://hostname.domain/gss/rest/paul/groups",
"creationDate": 1223372769275,
"modificationDate": 1223372769275,
"quota": {
"totalFiles": 7,
"totalBytes": 429330,
"bytesRemaining": 10736988910
}
}


Groups · Groups are entities used to organize users for easier sharing of files and folders among peers. They can be used to facilitate sharing files to multiple users at once. Groups belong to the user who created them and cannot be shared. The URI of a group named work created by the user with username paul would be:
http://host.domain/gss/rest/paul/groups/work
The JSON representation of this group would be something like this:

[
"http://hostname.domain/gss/rest/paul/groups/work/tom",
"http://hostname.domain/gss/rest/paul/groups/work/jim",
"http://hostname.domain/gss/rest/paul/groups/work/mary"
]


Files · Files are the most basic resources in GSS. They represent actual operating system files from the client's computer that have been augmented with extra metadata for storage, retrieval and sharing purposes. Familiar metadata from modern file systems are also maintained in GSS, like file name, creation/modification times, creator, modifier, tags, permissions, etc. Furthermore, files can be versioned in GSS. Updating versioned files retains the previous versions, while updating an unversioned file replaces irrevocably the old file contents. The URI of a file named doc.txt located in the root folder of the user with username paul would be:
http://host.domain/gss/rest/paul/files/doc.txt
The JSON representation of the metadata in this file would be something like this:

{
"name": "doc.txt",
"creationDate": 1232449958563,
"createdBy": "paul",
"readForAll": true,
"modifiedBy": "paul",
"owner": "paul",
"modificationDate": 1232449944444,
"deleted": false,
"versioned": true,
"version": 1,
"size": 802,
"content": "text/plain",
"uri": "http://hostname.domain/gss/rest/paul/files/doc.txt",
"folder": {
"uri": "http://hostname/gss/rest/aaitest@uth.gr/files/",
"name": "Paul Smith"
},
"path": "/",
"tags": [
"work",
"personal"
],
"permissions": [
{
"modifyACL": true,
"write": true,
"read": true,
"user": "paul"
},
{
"modifyACL": false,
"write": true,
"read": true,
"group": "work"
}
]
}


Folders · Folders are resources that are used for grouping files. They represent the file system concept of folders or directories and can be used to mirror a client's computer file system on GSS. Familiar metadata from modern file systems are also maintained in GSS, like folder name, creation/modification times, creator, modifier, permissions, etc. The URI of a folder named documents located in the root folder of the user with username paul would be:
http://host.domain/gss/rest/paul/files/documents
The JSON representation of this folder would be something like this:

{
"name": "documents",
"owner": "paul",
"deleted": false,
"createdBy": "paul",
"creationDate": 1223372795825,
"modifiedBy": "paul",
"modificationDate": 1223372795825,
"parent": {
"uri": "http://hostname.domain/gss/rest/paul/files/",
"name": "Paul Smith"
},
"files": [
{
"name": "notes.txt",
"owner": "paul",
"creationDate": 1233758218866,
"deleted":false,
"size":4567,
"content": "text/plain",
"version": 1,
"uri": "http://hostname.domain/gss/rest/paul/files/documents/notes.txt",
"folder": {
"uri": "http://hostname.domain/gss/rest/paul/files/documents/",
"name": "documents"
},
"path": "/documents/"
}
],
"folders": [],
"permissions": [
{
"modifyACL": true,
"write": true,
"read": true,
"user": "paul"
},
{
"modifyACL": false,
"write": true,
"read": true,
"group": "work"
}
]
}

Working with these resources is accomplished by sending HTTP protocol requests to the resource URI with GET, HEAD, DELETE, POST, PUT methods. GET requests retrieve the resource representation, either the file contents, or the JSON representations for the resources specified above. HEAD requests for files return just the metadata of the file and DELETE requests remove the resource from the system. PUT requests upload files to the system from the client, while POST requests perform various modifications to the resources, like renaming, moving, copying, moving files to the trash, restoring them from the trash, creating folders and more. The operations are numerous and I hope to cover them in more detail in a future post.

One important aspect of every RESTful API is the use of URIs to allow the client to maintain a stateful conversation. For example, fetching the user URI would provide the files URI for fetching the available files and folders. Fetching the files URI would in turn return the URIs for the particular files and folders contained in the root folder (along with other folder properties). Returning to the parent of the current folder would entail following the URI contained in the parent property. This mechanism removes the state handling from the server and puts the burden on the client, providing excellent scalability for the service. Furthermore, since the URIs are treated opaquely by the client, the API allows client reuse across server deployments. A client can target multiple GSS services, as long as they speak the same RESTful API. Moreover, links from service A can refer to resources in service B without a problem (in the same authentication domain, e.g. the same Shibboleth federation). This is the same as using a single web browser to communicate with multiple web servers, by following links among them.

Monday, April 20, 2009

Reconciling Apache Commons Configuration with JBoss 5

There are many ways to configure a JavaEE application. Among the available solutions are DBMS tables, JNDI, JMX MBeans and even plain old files in a variety of formats. While we have used most of the above in various occasions, I find that plain files resonate with all types of sysadmins, when no other administrative interface is available. For such scenarios, Apache Commons Configuration is undoubtedly the best tool for the job. Recently, I came across an undocumented incompatibility when using Commons Configuration with JBoss 5 and I thought I should describe our solution for the benefit of others.

Usually we are storing our configuration files in the standard place for JBoss, which is JBOSS_HOME/server/default/conf, for the default server configuration. This has the disadvantage that is not as easy to remember as /etc in UNIX/Linux or \Program Files and \Windows in Windows systems, but it has the important advantage of being specified as a relative path in our code, making it more cross-platform without cluttering it with platform-specific if/else path resolution checks.

Commons Configuration can reload the configuration files automatically when changed in the file system, which helps prolong the server uptime. However JBoss 5 has introduced the concept of a virtual file system that caches all file system accesses performed through the context class loaders using relative paths. Unfortunately this generates resource URLs in the form vfsfile:foo.properties that Commons Configuration does not know how to deal with. Fixing this requires extending FileChangedReloadingStrategy, like we do in gss. Alternatively, one could patch Commons Configuration with the following change and use the standard FileChangedReloadingStrategy unchanged:


Index: FileChangedReloadingStrategy.java
===================================================================
--- FileChangedReloadingStrategy.java (revision 764760)
+++ FileChangedReloadingStrategy.java (working copy)
@@ -46,6 +46,9 @@
/** Constant for the jar URL protocol.*/
private static final String JAR_PROTOCOL = "jar";

+ /** Constant for the JBoss MC VFSFile URL protocol.*/
+ private static final String VFSFILE_PROTOCOL = "vfsfile";
+
/** Constant for the default refresh delay.*/
private static final int DEFAULT_REFRESH_DELAY = 5000;

@@ -161,7 +164,8 @@

/**
* Helper method for transforming a URL into a file object. This method
- * handles file: and jar: URLs.
+ * handles file: and jar: URLs, as well as JBoss VFS-specific vfsfile:
+ * URLs.
*
* @param url the URL to be converted
* @return the resulting file or null
@@ -181,6 +185,18 @@
return null;
}
}
+ else if (VFSFILE_PROTOCOL.equals(url.getProtocol()))
+ {
+ String path = url.getPath();
+ try
+ {
+ return ConfigurationUtils.fileFromURL(new URL("file:" + path));
+ }
+ catch (MalformedURLException mex)
+ {
+ return null;
+ }
+ }
else
{
return ConfigurationUtils.fileFromURL(url);


Neither of these solutions covers the case of storing configuration files in zip or jar containers, but since it is something I haven't found a use for yet, I can't test a fix for it. If anyone is interested in such a use case, I'd advise extending FileChangedReloadingStrategy, combining the logic in jar: and vfsfile: URL handling.

Monday, April 13, 2009

GSS architecture

Handling a large number of concurrent connections requires many servers. Not only because scaling vertically (throwing bigger hardware at the problem) is very costly, but also because even if an application can be designed to scale vertically, the underlying stack probably can not. Java applications for instance, like GSS, run on the JVM and although the latter is an excellent piece of engineering, using huge amounts of heap is not something it's tuned for. Big Iron servers with many cores and 20+ GB of RAM are usually running more than one JVM, since garbage collection is not all that efficient with huge heaps. And since running application instances with a 4-8 GB heap size can be done with cheap off-the-shelf hardware, why spend big bucks on Big Iron?

So having a large number of servers is a sane choice, but brings it's own set of problems. Unless one partitions users to servers (having all requests a particular user makes be delivered to the same server), all servers must have a consistent view of the system data, in order to deliver meaningful results. Assigning user requests to particular servers, usually requires expensive application layer load-balancers or customized application code on each server, so it would rarely be your first option. Having all servers work on the same data is a more tractable problem, since it can be solved by having the application state being replicated among server nodes. Usually, only a small part of the application state needs to be replicated, for each user, that is the part which concerns his current session. But even though session clustering solutions have been a well studied field and implementations abound, having no session to replicate is an even better option.

For GSS we have implemented a stateless architecture for the core server, that should provide us with good scalability in a very cost-effective manner. The most important part in this architecture is the REST-like API that moves part of the responsibility for session state maintenance to the client applications, effectively distributing the system load to more systems than the available server pool. Furthermore, client requests can be authenticated without requiring an SSL/TLS transport layer (even though it can be used if extra privacy is required), which would entail higher load on the application servers or require expensive load balancers. In the server side, API requests are being handled by servlets that enlist the services of stateless session beans, for easier transaction management. Our persistence story so far is JPA with a DBMS backing, plus high-speed SAN storage for the file contents. If or when this becomes a bottleneck, we have various contingency plans, depending on the actual characteristics of the load that will be observed.


The above image depicts the path user requests will travel along, while various nodes interact in order to serve them. A key element in this diagram is the number of different servers that can be found, effectively specializing in their own particular domain. Although the system can be deployed on a single physical server (and regularly is, for development and testing), consisting of a number of standalone sub-services instead of a big monolithic service is a boon to scalability.

This high-level overview of the GSS architecture should help those interested to find their way around the open-source codebase and explain some of the design decisions. But the most interesting part from a user's point of view would be the REST-like API, that allows one to take advantage of the service for scratching his own itch.

So that will be the subject of my next post.

Wednesday, April 8, 2009

Introducing GSS

During my recent work-induced blog hiatus, I've been working on a new software system, called GSS. I've been more than enjoying the ride so far and since we have released the code as open-source, I thought discussing some of the experience I've gained might be interesting to others as well.

GSS is a network, er grid, er I mean cloud service, for providing access to a file system on a remote storage space. It is the name of both a service (currently in beta) for the Greek research and academic community and the open source software used for it, that can also be used by others for deploying such services. It is similar in some ways to services like iDrive, DropBox and drop.io, but it can also be regarded as a more high-level Amazon S3. Its purpose is to let the desktop computer's file system meet the cloud. The familiar file system metaphors of files and folders are used to store information in a remote storage space, that can be accessed from a variety of user and system interfaces, from any place in the world that has an Internet connection. All usual file manager operations are supported and users can share their files with selected other users or groups, or even make them public. Currently there are four user interfaces available, a web-based application, a desktop client, a WebDAV interface and an iPhone web application, in various stages of development. Underlying these user interfaces is a common REST-like API that can be used to extend the service in new, unanticipated ways.


The main focus of this service was to provide the users of the Greek research and academic community with a free, large storage space that can be used to store, access, backup and share their work, from as many computer systems as they want. Since the available user base is close to half a million (although the expected users of the service are projected to the low ten thousands), we needed a scalable system, that would be able to accommodate high network traffic and a high storage capacity at the same time. A Java Enterprise Edition server coupled with a GWT-based web client and a stateless architecture were our solution. In future posts I will describe the system architecture with all the gory details. The exposed virtual file system features file versioning, trash bin support, access control lists, tagging, full text search and more.

All of these features are presented through an API for third party developers to create scripts, applications or even full blown services that will fulfill their own particular needs or serve other niches. This API has a REST-like design and though it will probably fail a formal RESTful definition, it sports many of the advantages of such architectures:

  • system resources such as users, groups, files and folders are represented by URIs
  • GET, HEAD, POST, PUT and DELETE methods on resources have the expected semantics
  • HTTP caching is explicitly supported via Last-Modified, ETag & If-* headers
  • resource formats for everything besides files are simple JSON representations
  • only authenticated requests are allowed, except for public resources

Users are authenticated through the GRNET Shibboleth infrastructure. User passwords are never transmitted to the GSS service. Instead GSS-issued authentication tokens are used by both client and server to sign the API requests after the initial user login. SSL transport can provide even stronger privacy guarantees, but it is not required, nor enabled by default.

The GSS code base is GPL-licensed and therefore anyone can use it as a starting point to implement his own file storage service. We have yet to provide binary downloads, due to the various dependencies, but the build instructions should be enough to get someone started. We are always interested in source or documentation patches, of course (did I mention it's open source?). Most importantly, the REST API will ensure that clients developed for one such service can be reused for every other one.

I will have much more to say about the API in a future post. In the meantime you can peruse the code and documentation, or even try it out yourself. I'd be very interested in any comments you might have.

Creative Commons License Unless otherwise expressly stated, all original material in this weblog is licensed under a Creative Commons Attribution 3.0 License.