Valence roadmap

A few days ago I sent the following note to the Firefox devtools mailing list. The point was to start a discussion about where we want to take Valence (née Firefox Developer Tools Adapter) in 2015. I am reproducing the message below in an effort to broaden the audience, but I would encourage anyone who wants to participate in the discussion to post to the mailing list. However, if that sounds too intimidating (it really shouldn't!), I'm always happy to respond to comments in this blog.


Hey fellow devtoolers!

I know that eggnog and Christmas trees is all that everyone is thinking about right now, but I'd like to take a minute before we all indulge in our holiday festivities, to talk about Valence. And 2015.

2015 will be the year when Valence becomes something more than a promising experiment. It is the year that will (hopefully) see a version 1.0 that has feature-parity with Firefox, for features where the Safari/Chrome debugging protocol aligns well with the Firefox RDP. This is a proposed roadmap of how we'll get there from where we are now.

There are two main themes in this roadmap: do more and do better, and I think we should address them in reverse order. Do-things-better contains what I think we need for cutting a 1.0 release:

Improve existing features (do better)

• Get the bundled ios_webkit_debug_proxy working on Windows & Linux
• Implement support for the network panel
• Handle format specifiers in console.log and friends (%s, %c, etc.)
• Display inferred names for functions
• Evaluate expressions in the selected frame when execution is paused
• Support source-mapped CSS
• Break on DOM events
• Fix some longstring-related bugs that we currently work around by using plain strings
• Fix every XXX and TODO comment in the code
• Implement every todoMethod in the actors

Most of these tasks have a well understood scope, so it is just a "small" matter of time and available resources. However, another angle on the do-better theme is not to just fix known problems, but also to make them stay fixed and provide a reliable experience to the user. These tasks belong in that category and should probably be done in parallel with the above:

Improve reliability (do better for realz)

• Deploy some testing infrastructure (probably based on ted's work)
• Port existing Firefox devtools tests and/or write new ones
• Fix stability bugs in Valence

I believe the above lists should be enough for a v1.0, but there are many more things we could do to provide value to our users. Here are some obvious features we could add for a v2.0:

Expand our feature set (do more)

• Support for the performance panel (profiler and timeline)
• Support for the storage panel
• Support for the canvas debugger
• Support for debugging embedded web views on iOS and Android, which would let us become a sort of unofficial development environment for Cordova/PhoneGap
• Support Firefox Developer Tools in the pipeline (memory profiler, worker debugger)

Not all of these tasks have been scoped or had a feasibility assessment, but there are some indications that we should be able to pull them off with some effort. These features would make the value proposition of Valence and (Firefox Developer Edition by extension) really enticing for many more developers.

But when you have all this, why stop there? The adaptability of Firefox Developer Tools that Valence affords can take us to a whole new level of debugging functionality that developers have always dreamed of:

Full-stack debugging (do way more)

• Support debugging server-side node.js applications, with console, scratchpad, debugger and profiler
• (Adding async stack support in Firefox Developer Tools would let us combine server- and client-side stack traces for debugging ecstasy)
• Research the framework ecosystems of languages that compile to JS and can run on node.js and assess the feasibility of supporting them directly via source maps
• Implement adapters for JDWP (to debug Java) and Xdebug (to debug PHP)

Only some preliminary research has gone into these tasks, so their feasibility and effort required is still TBD. Most of the above are further out from a priority standpoint and some may well be in the pie-in-the-sky domain, but I think we should at least investigate node.js debugging in 2015. It would be an exciting engineering accomplishment and could provide a concrete value proposition for Valence and Firefox Developer Edition.

Prioritized shortlist
Given all of the above, and taking difficulty and usefulness into account, I think a reasonable, prioritized, shortlist of Valence tasks for 2015 would be the following:

1. Get the bundled ios_webkit_debug_proxy working on Windows & Linux
2. Display inferred names for functions
3. Implement support for the network panel
4. Fix stability bugs in Valence
5. Deploy some testing infrastructure
6. Port existing Firefox devtools tests and/or write new ones
7. Support debugging server-side node.js applications
8. Support for debugging embedded web views on iOS and Android
9. Handle format specifiers in console.log and friends (%s, %c, etc.)
10. Evaluate expressions in the selected frame when execution is paused
11. Support source-mapped CSS
12. Break on DOM events

Of course, depending on broader organizational priorities, this list may be too short or not much of a shortlist, for an entire year. Not to mention that planning that far ahead is usually futile and an exercise in frustration. Still, this should hopefully provide a useful base for a discussion about what we want Valence to be and how we want to go about it.

Any and all feedback is welcome (before or while you enjoy your eggnog)!

Cheers,
Panos

JavaScript debugging updates

There have been a lot of interesting developments in the JavaScript debugging front since my last update, so let me give you a brief overview of the most important changes.

Let's begin with the eagerly awaited Firefox OS. Starting the debugger in Firefox OS has gotten way easier. There is a toggle in the settings app that will set or reset the devtools.debugger.remote-enabled pref, so that you don't have to mess with files and adb at all. At least for production builds that have marionette disabled by default. Developer builds will have to keep disabling marionette for a bit longer. You can find the Remote Debugging toggle in Settings -> Device Information -> More Information -> Developer. Note that you will have to reboot the device (or restart the b2g process) for the change to have effect. Also, as you can see in the screenshot, it's a good idea to disable out-of-process support while debugging, at least until the debugger can support it.

Firefox OS settings

While we are on the subject of debugging JavaScript on mobile, I shouldn't neglect to mention that Mihai Sucan has landed web console support for Firefox OS and Firefox for Android. There are still a few rough edges, particularly in the case of Firefox OS, but now you can connect your desktop Firefox to your mobile device and see the errors and logged messages, as well as evaluate JS code in the context of the mobile page.

Speaking of desktop Firefox, the amazing Victor Porof has landed an obscene number of patches in the last few weeks, with many fixes and tweaks for the debugger frontend. Of those I'd like to highlight the new functionality to search for variables in the current scope chain while the debugger is paused. Prepend your search with a star to search in the variables view. When you deal with large objects with long scope chains, finding the variable you are looking for can be like searching for a needle in a haystack. Now it's a piece of cake.

Searching for variables

And last, but certainly not least, chrome debugging support has appeared in Nightly this past weekend! This will make debugging add-ons or Firefox itself way easier than ever before. You will need to enable remote debuging (devtools.debugger.remote-enabled) and browser chrome debugging as well (devtools.chrome.enabled, the same pref you switch to get Scratchpad to run against chrome).

Chrome debugging preferences

This will get you a nice new Browser Debugger menu item in the Web Developer menu, which launches a separate debugger process to inspect your browser instance.

Browser Debugger menu item

There has never been a better time to dive into Firefox internals and become a valued contributor. Take a look at this bug, to see how simple debugging has just become. Help us make the web better, by improving the lives of more than 400 million users!

Debugging Firefox OS

A few months ago, long before the Firefox debugger had shipped, I had tried to see what it would take to get it to debug B2G, or as we call it now, Firefox OS. The hack proved successful and with time it grew into an important debugging target for us. I demoed it at JSConf last April, but at that time a lot of the debugger frontend functionality had not been available in nightlies. As of yesterday, however, nightly builds of Firefox can debug nightly builds of Firefox OS, both in the device and desktop versions. Let me give you a quick rundown of the necessary steps.

First of all you need to go to about:config on your desktop Firefox and turn devtools.debugger.remote-enabled to true. After restarting Firefox you will have a new Remote Debugger menu item in your Web Developer menu.

You need to change the same setting in Firefox OS as well, and that requires finding the prefs.js file. For desktop builds, this will be in gaia-repository/profile/prefs.js. For devices, you should connect it to the desktop through a USB cable, use adb shell to connect to the device, and find the name of the default profile in /data/b2g/mozilla/. The prefs.js file wil be in that subdirectory, which in my case is /data/b2g/mozilla/ngsweij3.default/prefs.js. You can get that file off the device using:

adb pull /data/b2g/mozilla/ngsweij3.default/prefs.js prefs.js
 
That will store a copy of that file in your current working directory. You need to modify this file and then send it back to the device with:

adb push prefs.js /data/b2g/mozilla/ngsweij3.default/prefs.js

The necessary changes to that file are the following two lines:

user_pref("devtools.debugger.remote-enabled", true);
user_pref("marionette.defaultPrefs.enabled", false);

They enable the debugger server and at the same time disable the Marionette server, to avoid a conflict that we haven’t resolved yet.

That is all that is necessary for the desktop Firefox OS case, but you need to do one more thing for debugging Firefox OS on the device. If you are going to debug over a USB connection (which will also provide helpful console logs via adb logcat) you will need to forward a local port to the device using adb:

adb forward tcp:6000 tcp:6000

Port number 6000 is the default and if you need to change it for some reason, you can do so with the following pref:

user_pref("devtools.debugger.remote-port", 7000);

Debugging over WiFi is also possible, although disabled by default, so you will need to change the following pref to do so:

user_pref("devtools.debugger.force-local", false);

In that case, you don’t have to do the port forwarding step, but you may change the port number in the same way.

In order for the new settings to take effect in the device you have to restart it with ‘adb reboot’.

That’s all folks! Now you just start your nightly Firefox and select the Remote Debugger menu item. A dialog will pop up asking for the remote address and port number to connect to. The defaults will work for the desktop Firefox OS or the device debugging-over-USB cases (unless you changed the port number of course!). If you opted for debugging over WiFi, you will need to provide the IP address of the device. You can find it by tapping on the connection name in the settings.

After the connection is established, you will see a list of the sources loaded on the device, giving you the option to inspect the code, set breakpoints, step through the execution or even modify the values of variables. Here is a screencast that shows the debugger in action:

A better introductory guide to debugging Firefox OS will be available soon on MDN. This is just the first step in supporting Firefox OS development with Firefox developer tools. Just around the corner are web console support and profiling, with more to follow in the near future. If you have issues or suggestions for these tools, please get in touch! Post a comment, file a bug or pop in #devtools for a chat. Let's make the web the best platform to develop on!

Debugging Firefox

Last week we had a Firefox Developer Tools team get-together at the Mozilla London office. These meetups are an extremely valuable tool for the project to tackle hard problems, because when you put all of the experts on a particular hard problem in the same room, hard problems tend to run away screaming.

We managed to attack a lot of hard problems this week: a profiler, source maps, the developer toolbox, web console remoting and chrome debugging, were just a few of them. And not only that, but we also landed a few exciting new features as well, with fullscreen scratchpad and markup panel preview being my personal favorites. I'm sure others will cover the rest in appropriate detail, but I'd like to say a few things about chrome debugging.

Last March, during the previous team meetup we had managed to get a chrome debugging proof-of-concept working. We were thrilled by it, even though it was a big hack, because the implications were enormous. Something that not many people know is that the Firefox frontend is built in JavaScript, CSS and XUL, which is an HTML derivative with additional capabilities. That is, we use the exact same technologies that power your favorite web applications, to build the browser that displays said web apps to the user. This similarity has always been front and center to our thinking when considering tools for web developers: how can we use these tools to make hacking on Firefox easier?

Without easy to use tools, fixing bugs is time-consuming and adding features demands a very disciplined approach to programming. It hampers our ability to move the project faster and innovate rapidly. Furthermore, having a non-profit foundation to back the development of an open-source browser means that you don't have the option to simply throw more money at the problem, hiring more engineers than everybody else to increase the pace of innovation in the product. This is a browser created by thousands of contributors to take care of their own interests. We have to make their work easier, by giving them the tools they need to work faster.

When we did the chrome debugging prototype last March, the Firefox Debugger had been in trunk for a short while, disabled by default. No visual styling had been done on it and only the most basic of functionality was there. Six months later we have shipped the first version and we keep iterating on it.

The proof-of-concept couldn't debug chrome window globals, only modules, and was hijacking the remote debugging protocol in a hackish way that was guaranteed never to make it into a Firefox release. This time we have both kinds of globals supported and a sound protocol design that was agreed upon a few months ago.

This new tool, which we dubbed "Browser Debugger", will be useful to add-on developers, too. In my demo last week I used it to debug Firebug. We have patches under review or close enough that implement a first draft of the experience we aim to present users with. I want to land that as soon as we can, enable the feature by default (but conditioned on the devtools.chrome.enabled pref like other tools) and keep iterating.

It was without a doubt a very rewarding week. Some might expect us to also treat these rare events as opportunities to hang out, get drunk and take embarrassing pictures of each other. I am not going to either confirm or deny such allegations. At least not until I'm presented with hard, photographic evidence. Preferably on Flickr.

JSConf 2012

This week I had the opportunity to attend JSConf. I've been trying to make it to one of these conferences in the past without success and I now realize I was missing a lot! Chris and Laura Williams deserve special credit for the excellent job they are doing here. Even though I missed all the parties (because jetlag), the venue, the speakers, the tracks, and the conversations were all top class. Highly recommended.

If I had to pick the highlights of the conference, an undoubtedly difficult undertaking due to the sheer number of awesome talks, my personal favorites would be these two:

The B2G phones given out by Mozilla to all attendees, a gesture that sparked the creativity of quite a few people, culminating in a few B2G apps after a few hours of hacking. Yes, it's that easy.

The chance to meet and listen to Dan Ingalls, in my humble opinion one of the greatest programmers of our time, giving his inspirational talk on the Lively Kernel. It would be a failure of our profession if the ideas in Lively Kernel do not find a way to reach mass market at some point. This system, like its predecessors Squeak and Dynabook, represent a more humane way of programming computers that could be a game changer.

My own Debugging B2G talk was well-attended, even though a malfunctioning video cable threatened to kill it before it even began. I received lots of comments afterwards and requests for access to the debugger build that I demoed, and even though I pointed people to the patches, I think that the best thing to do for those interested is to wait a few more weeks, until we get all of that work in nightlies. After getting back at the hotel I recorded a screencast of the talk for those who couldn't attend, but unfortunately you will miss me mumbling while frantically plugging and unplugging a video cable and cursing at thunderbolt ports. Hands down the highlight of the talk.




All in all it was a great opportunity to meat people, have conversations on the present and future of Firefox developer tools and ride a bull. Too bad I missed that last one.

Debugging mobile phones

Last week I posted an update on the progress we are making towards a functional JavaScript debugger in Firefox. We still have a ways to go before we get there, but the foundation this is based on, namely the Remote Debugging Protocol, provides a solid basis for debugging mobile browsers, and as it turns out, even mobile operating systems. Supporting these use cases is still farther down our list, but I decided to spend a few hours the other day trying to estimate how far we are from this goal. It turns out we are not that far, and I have a screencast to prove it.

I had done some prototyping work for Firefox for Android a while back, which was rather straightforward, so I was curious to know what would it take to tackle a more ambitious project: debugging Boot to Gecko. After a few hours of hacking and orienting myself around the B2G and Gaia code base, I got this far:



Not too bad I think for a few hours work. Don't expect this to be ready of course, before we have a working debugger in desktop Firefox. If, however, you'd like to follow along and maybe even lend a hand, you are always welcome!

Debugging JavaScript

tl;dr: Firefox nightlies now ship with an experimental JavaScript debugger. It's not ready for end-users yet, but we are feverishly working on filling in the missing bits.

Debugging is always hard, especially when debugging other people's code. Adding a bunch of console.log() statements, becomes tedious after a while and that's assuming that you are familiar enough with the code in question to know where to place them. Tools like Firebug are a tremendous help in this case. Having the ability to pause the execution at any point or inspect variables and stack frames, provides valuable insights into the runtime behavior of a program.

Firebug depends on the traditional JSD API in SpiderMonkey to perform its magic, but nowadays JSD is limiting in a number of ways. In order to overcome these limitations, the new JSDBG2 API was designed and implemented by Jim Blandy and Jason Orendorff, and the first version of this work landed in Firefox 8. Jim, being the kind of guy that has forgotten more about debuggers than I will ever know, designed a remote debugging protocol to separate the debugger from the browser, allowing for debugging desktop browsers, mobile browsers, mail readers and even more exotic software in the future. Dave Camp got a great protocol implementation up and running, but was assigned managerial duties (poor sap!) before having a chance to finish the job, so Victor Porof, Mihai Șucan and yours truly lined up to carry the torch. The culmination of this work landed in mozilla-central last week, including the remote protocol implementation and a prototype UI (disabled by default). It is now available in nightlies for adventurous souls to play with.

Let me be clear about that last point: what's in there right now is not useful for day-to-day work, even for experienced developers. There is no UI to add breakpoints besides the experimental Graphical Command Line Interface (whose awesomeness deserves a separate post), no stepping, no variable inspection besides call parameters and 'this', and numerous other limitations. In short: it's not ready, yet. In the course of the next weeks we will be furiously working to add the missing bits and improve the UI, so that it becomes a worthy addition to our existing suite of developer tools.

If, however, you are the kind of person who doesn't take no for an answer, here is what you need to do to play with it: type about:config in the awesomebar and set the preference devtools.debugger.enabled to true. Also, set devtools.gcli.enable to true in order to be able to set breakpoints. Restart your browser and select Tools -> Web Developer -> Script Debugger to open it. You will need to reload the page for the debugger to become aware of the scripts in the page (see? It's not ready!). Browse through the source scripts to find a place where you would like to add a breakpoint, then open the web console and type break add line <url> <line>. Don't worry about memorizing it, GCLI will guide you along the way, suggesting among the available options at each step. When the code hits the breakpoint, the debugger will display the stack frames and the variables in scope (not all of them, it's not ready, remember?) for inspection.

If you have better things to do with your time than play with experimental, uninished debuggers, but would still like to take a peek into what's in store already, here is a short screencast that should give you a broad idea. Note that the stepping functionality in the demo is not in the nightlies yet, because, you guessed it, it's not quite ready.



Ready or not, we would love to have any feedback you might have. Drop by #devtools for a chat, or file bugs to torturehelp us get things fixed. Rest assured, when it's done, debugging web apps will never be the same again.

Lessons from Startup Weekend

I had an exhausting but fun weekend at the Athens Startup Weekend a few days ago. Along with Christos I joined Yannis, Panagiotis Christakos and Babis Makrinikolas on the Newspeek project. When Yannis pitched the idea on Friday night, the main concept was to create a mobile phone application that would provide a better way to view news on the go. I don't believe it was very clear in his mind then, what would constitute a "better" experience, but after some chatting about it we all defined a few key aspects, which we refined later with lots of useful feedback and help from George. Surprisingly, for me at least, in only two days we managed to design, build and present a working prototype in front of the judges and the other teams. And even though the demo wasn't exactly on par with our accomplishments, I'm still amazed at what can be created in such a short time frame.

Newspeek, our product, had a server-side component that periodically collected news items from various news feeds, stored them and provided them to clients through a simple REST API. It also had an iPhone client that fetched the news items and presented them to the user in a way that respected the UI requirements and established UX norms for that device.

So, in the interest of informing future participants about what works and what doesn't work in Startup Weekend, here are the lessons I learned:

1. If you plan to win, work on the business aspect, not on the technology. Personally, I didn't go to ASW with plans to create a startup, so I didn't care that much about winning. I mostly considered the event as a hackathon, and tried my best to end up with a working prototype. Other teams focused more on the business side of things, which is understandable, given the prize. Investors fund teams that have a good chance to return a profit, not the ones with cool technology and (mostly) working demos. Still, the small number of actual working prototypes was a disappointment for me. Even though the developers were the majority in the event, you obviously can't have too many of them in a Startup Weekend.
2. For quick server-side prototyping and hosting, Google App Engine is your friend. Since everyone in the team had Java experience, we could have gone with a JavaEE solution and set up a dedicated server to host the site. But, since I've always wanted to try App Engine for Java and the service architecture mapped nicely to it, we tried a short experiment to see if it could fly. We built a stub service in just a few minutes, so we decided it was well worth it. Building our RESTful service was really fast, scalability was never a concern and the deployment solution was a godsend, since the hosting service provided for free by the event sponsors was evidently overloaded. We're definitely going to use it again for other projects.
3. jQTouch rocks! Since our main deliverable would be an iPhone application, and there were only two of us who had ever built an iPhone application (of the Hello World variety), we knew we had a problem. Fortunately, I had followed the jQTouch development from a reasonable distance and had witnessed the good things people had to say, so I pitched the idea of a web application to the team and it was well received. iPhone applications built with web technologies and jQTouch can be almost indistinguishable from native ones. We all had some experience in building web applications, so the prospect of having a working prototype in only two days seemed within the realm of possibility again. The future option of packaging the application with PhoneGap and selling it in the App Store was also a bonus point for our modest business plan.
4. For ad-hoc collaboration, Mercurial wins. Without time to set up central repositories, a DVCS was the obvious choice, and Mercurial has both bundles and a standalone server that make collaborative coding a breeze. If we had zeroconf/bonjour set up in all of our laptops, we would have used the zeroconf extension for dead easy machine lookup, but even without it things worked flawlessly.
5. You can write code with a netbook. Since I haven't owned a laptop for the last three years, my only portable computer is an Asus EEE PC 901 running Linux. Its original purpose was to allow me to browse the web from the comfort of my couch. Lately however, I'm finding myself using it to write software more than anything else. During the Startup Weekend it had constantly open Eclipse (for server-side code), Firefox (for JavaScript debugging), Chrome (for webkit rendering), gedit (for client-side code) and a terminal, without breaking a sweat.
6. When demoing an iPhone application, whatever you do, don't sweat. Half-way through our presentation, tapping the buttons didn't work reliably all the time, so anxiety ensued. Since we couldn't make a proper presentation due to a missing cable, we opted for a live demo, wherein Yannis held the mic and made the presentation, and I posed as the bimbo that holds the product and clicks around. After a while we ended up both touching the screen, trying to make the bloody buttons click, which ensured the opposite effect. In retrospect, using a cloth occasionally would have made for a smoother demo, plus we could have slipped a joke in there, to keep the spirit up.

All in all it was an awesome experience, where we learned how far we can stretch ourselves, made new friends and caught up with old ones. Next year, I'll make sure I have a napkin, too.

Spell-checking in JavaScript

I just heard of Atwood's Law a few days ago. Apparently Jeff Atwood first published his discovery a couple of years ago, which is like a century in Internet time, but I can't keep up with everything. The Law states that "any application that can be written in JavaScript, will eventually be written in JavaScript". As Atwood explains, it is based on Tim Berners-Lee's Principle Of Least Power:

Computer Science spent the last forty years making languages which were as powerful as possible. Nowadays we have to appreciate the reasons for picking not the most powerful solution but the least powerful. The less powerful the language, the more you can do with the data stored in that language.

I think there is a common theme between Atwood's Law, Zawinski's Law ("Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can") and Greenspun's Tenth Rule ("Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp"). The inevitability of the specified outcome makes the world seem like a much simpler place. There is no need to argue. Believe and ye shall be redeemed.

I might have been in this state of mind, when I came across Peter Norvig's "How to Write a Spelling Corrector", because I knew instantly I had to port the algorithm to JavaScript. The algorithm is quite simple and has already been ported to many different languages, so I seized the opportunity to study the differences in expressiveness, performance and style, between these languages and my latest affection, JavaScript.

After a few night's work I have it working and free (as in beer and speech) for anyone to use. Check out the speller project on GitHub. If you want to understand how the algorithm works, you should go and read Norvig's article, although you might get some hints from the comments in my code. The modestly condensed version of the code is 53 lines:

var speller = {};
speller.train = function (text) {
  var m;
  while ((m = /[a-z]+/g.exec(text.toLowerCase()))) {
    speller.nWords[m[0]] = speller.nWords.hasOwnProperty(m[0]) ? speller.nWords[m[0]] + 1 : 1;
  }
};
speller.correct = function (word) {
  if (speller.nWords.hasOwnProperty(word)) return word;
  var candidates = {}, list = speller.edits(word);
  list.forEach(function (edit) {
    if (speller.nWords.hasOwnProperty(edit)) candidates[speller.nWords[edit]] = edit;
  });
  if (speller.countKeys(candidates) > 0) return candidates[speller.max(candidates)];
  list.forEach(function (edit) {
    speller.edits(edit).forEach(function (w) {
      if (speller.nWords.hasOwnProperty(w)) candidates[speller.nWords[w]] = w;
    });
  });
  return speller.countKeys(candidates) > 0 ? candidates[speller.max(candidates)] : word;
};
speller.nWords = {};
speller.countKeys = function (object) {
  var attr, count = 0;
  for (attr in object)
    if (object.hasOwnProperty(attr))
      count++;
  return count;
};
speller.max = function (candidates) {
  var candidate, arr = [];
  for (candidate in candidates)
    if (candidates.hasOwnProperty(candidate))
      arr.push(candidate);
  return Math.max.apply(null, arr);
};
speller.letters = "abcdefghijklmnopqrstuvwxyz".split("");
speller.edits = function (word) {
  var i, results = [];
  for (i=0; i < word.length; i++)
    results.push(word.slice(0, i) + word.slice(i+1));
  for (i=0; i < word.length-1; i++)
    results.push(word.slice(0, i) + word.slice(i+1, i+2) + word.slice(i, i+1) + word.slice(i+2));
  for (i=0; i < word.length; i++)
    speller.letters.forEach(function (l) {
      results.push(word.slice(0, i) + l + word.slice(i+1));
    });
  for (i=0; i <= word.length; i++)
    speller.letters.forEach(function (l) {
      results.push(word.slice(0, i) + l + word.slice(i));
    });
  return results;
};

It may not be as succinct as Norvig's Python version (21 lines), or the record-holding Awk and F# versions (15 lines), but is much better than C (184 lines), C++, Perl, PHP, Rebol and Erlang. There is even a Java version with 372 lines. It must contain some sort of spell-checking framework in there, or something. The condensed version above, although correct, has terrible performance in most JavaScript engines, however. For real-world use you should pick the regular version which may be slightly longer, but performs much better.

Since this was a toy project of mine, I wanted to play with ServerJS modules as well, in order to run it as a shell script. I turned the code into a securable module, so you can run it from the command line, using narwhal. I have a couple of scripts to that end. Of course since this is JavaScript, you can try it from your browser, by visiting the demo page, courtesy of GitHub Pages. Be sure to use a modern browser, like Firefox 3.5, Safari 4 or Chrome 3 (beta), otherwise you won't be able to run the test suite, since I used the brand-new Web Workers to make the long-running tasks run in the background.

Norvig's Python implementation took 16 seconds for test 1 and the best I got was 25 seconds with Safari 4 on my Mac. Narwhal is using Rhino by default, so it is definitely not competitive in such tests (139 seconds), but I'm planning to fix support for v8cgi and give that a try.

And it goes without saying that I'd love to hear about ways to improve the performance or the conciseness of the code. If you have any ideas, don't be shy, leave a comment or even better fork the code and send me a pull request on GitHub.

GSS authentication

As I've described before, the GSS API is a REST-like interface for interacting with a GSS-based service, like Pithos. Using regular HTTP commands a client can upload, download, browse and modify files and folders stored in the GSS server. These interactions have two important properties: they store no client state to the server and they are not encrypted. I have already mentioned the most important benefits from the stateless architecture of the GSS protocol, namely scalability and loose coupling along the client-server boundary. In the same line of thought, SSL/TLS encryption of the transport was avoided for scalability reasons. Consequently, these two properties of the communication protocol, lead to another requirement: message authentication for each API call.

Since no session state is stored in the server, the client must authenticate each request as if it were the first one. Traditional web applications use an initial authentication interaction between client and server, that creates a unique session ID, which is transmitted with each subsequent request in that same session. While using this ID, the client does not need to authenticate again to the server. Stateless protocols, like WebDAV for instance, cannot rely on such an ID and have to transmit authentication data in each call. Ultimately the tradeoff is a minor increase in the message payload, in return for a big boost in scalability. The HTTP specification defines an Authorization header for use in such cases and WebDAV uses the HTTP Digest Access Authentication scheme. The GSS API uses a slight variation in that theme, blended with some ideas from OAuth request signing.

Essentially the standard HTTP Authorization header is populated with a concatenation of the username (for identifying the user making the request) and a HMAC-SHA1 signature of the request. The request signature is obtained by applying a secret authentication token to a concatenated string of the HTTP method name, the date and time of the request and the actual requested path. The GSS API page has all the details. The inclusion of the date and time helps thwart replay attacks using a previously sniffed signature. Furthermore, a separate header with timestamp information, X-GSS-Date, is used to thwart replay attacks with a full copy of the payload. The authentication token is issued securely by the server to the client and is the time-limited secret that is shared between the client and server. Since it is not the user password that is used as a shared secret, were the authentication token to be compromised, any ill effects would have been limited to the time period the token was valid. There are two ways for client applications to obtain the user's authentication token, and they are both described in detail in the API documentation.

In my experience, protocol descriptions are one thing and working code is a totally different one. In that spirit, I'm closing this post with a few tested implementations of the GSS API signature calculation, for client applications in different languages. Here is the method for signing a GSS API request in Java (full code here):

public static String sign(String httpMethod, String timestamp,
        String path, String token) {
  String input = httpMethod + timestamp + path;
  String signed = null;

  try {
    System.err.println("Token:" + token);
    // Get an HMAC-SHA1 key from the authentication token.
    System.err.println("Input: " + input);
    SecretKeySpec signingKey = new SecretKeySpec(
        Base64.decodeBase64(token.getBytes()), "HmacSHA1");

    // Get an HMAC-SHA1 Mac instance and initialize with the signing key.
    Mac hmac = Mac.getInstance("HmacSHA1");
    hmac.init(signingKey);

    // Compute the HMAC on the input data bytes.
    byte[] rawMac = hmac.doFinal(input.getBytes());

    // Do base 64 encoding.
    signed = new String(Base64.encodeBase64(rawMac), "US-ASCII");

  } catch (InvalidKeyException ikex) {
    System.err.println("Fatal key exception: " + ikex.getMessage());
    ikex.printStackTrace();
  } catch (UnsupportedEncodingException ueex) {
    System.err.println("Fatal encoding exception: "
                + ueex.getMessage());
  } catch (NoSuchAlgorithmException nsaex) {
    System.err.println("Fatal algorithm exception: "
                + nsaex.getMessage());
    nsaex.printStackTrace();
  }

  if (signed == null)
    System.exit(-1);
  System.err.println("Signed: " + signed);
  return signed;
}

Here is a method for signing GSS API requests in JavaScript, that uses the SHA-1 JavaScript implementation by Paul Johnston (full code here):

function sign(method, time, resource, token) {
  var q = resource.indexOf('?');
  var res = q == -1? resource: resource.substring(0, q);
  var data = method + time + res;
  // Use strict RFC compliance
  b64pad = "=";
  return b64_hmac_sha1(atob(token), data);
}

Here is a Tcl implementation by Alexios Zavras (full code here):

proc ::rest::_sign {id} {
  variable $id
  upvar 0 $id data
  set str2sign ""
  append str2sign $data(method)
  append str2sign $data(date)
  set idx [string first ? $data(path)]
  if {$idx < 0} {
    set str $data(path)
  } else {
    incr idx -1
    set str [string range $data(path) 0 $idx]
  }
  # append str2sign [::util::url::encode $str]
  append str2sign $str
  puts "SIGN $str2sign"
  set sig [::base64::encode [::sha1::hmac -bin -key $::GG(gsstoken) $str2sign]]
  set data(signature) $sig
}

I'd love to show implementations of the signature calculation for other languages as well, but since my time is scarce these days, I could use some help here. If you've written one yourself and you'd like to share, leave a comment and I'll update this post, with proper attribution of course.

Introducing GSS

During my recent work-induced blog hiatus, I've been working on a new software system, called GSS. I've been more than enjoying the ride so far and since we have released the code as open-source, I thought discussing some of the experience I've gained might be interesting to others as well.

GSS is a network, er grid, er I mean cloud service, for providing access to a file system on a remote storage space. It is the name of both a service (currently in beta) for the Greek research and academic community and the open source software used for it, that can also be used by others for deploying such services. It is similar in some ways to services like iDrive, DropBox and drop.io, but it can also be regarded as a more high-level Amazon S3. Its purpose is to let the desktop computer's file system meet the cloud. The familiar file system metaphors of files and folders are used to store information in a remote storage space, that can be accessed from a variety of user and system interfaces, from any place in the world that has an Internet connection. All usual file manager operations are supported and users can share their files with selected other users or groups, or even make them public. Currently there are four user interfaces available, a web-based application, a desktop client, a WebDAV interface and an iPhone web application, in various stages of development. Underlying these user interfaces is a common REST-like API that can be used to extend the service in new, unanticipated ways.


The main focus of this service was to provide the users of the Greek research and academic community with a free, large storage space that can be used to store, access, backup and share their work, from as many computer systems as they want. Since the available user base is close to half a million (although the expected users of the service are projected to the low ten thousands), we needed a scalable system, that would be able to accommodate high network traffic and a high storage capacity at the same time. A Java Enterprise Edition server coupled with a GWT-based web client and a stateless architecture were our solution. In future posts I will describe the system architecture with all the gory details. The exposed virtual file system features file versioning, trash bin support, access control lists, tagging, full text search and more.

All of these features are presented through an API for third party developers to create scripts, applications or even full blown services that will fulfill their own particular needs or serve other niches. This API has a REST-like design and though it will probably fail a formal RESTful definition, it sports many of the advantages of such architectures:

• system resources such as users, groups, files and folders are represented by URIs

• GET, HEAD, POST, PUT and DELETE methods on resources have the expected semantics

• HTTP caching is explicitly supported via Last-Modified, ETag & If-* headers

• resource formats for everything besides files are simple JSON representations

• only authenticated requests are allowed, except for public resources

Users are authenticated through the GRNET Shibboleth infrastructure. User passwords are never transmitted to the GSS service. Instead GSS-issued authentication tokens are used by both client and server to sign the API requests after the initial user login. SSL transport can provide even stronger privacy guarantees, but it is not required, nor enabled by default.

The GSS code base is GPL-licensed and therefore anyone can use it as a starting point to implement his own file storage service. We have yet to provide binary downloads, due to the various dependencies, but the build instructions should be enough to get someone started. We are always interested in source or documentation patches, of course (did I mention it's open source?). Most importantly, the REST API will ensure that clients developed for one such service can be reused for every other one.

I will have much more to say about the API in a future post. In the meantime you can peruse the code and documentation, or even try it out yourself. I'd be very interested in any comments you might have.

Programming for fun

I've been in a hiatus from blogging lately, mainly because free time has been in short supply. It's not just the very busy days at work, nights have been even busier. Instead of churning out long blog posts, I've been banging on code instead, day and night. Unfortunately I only get paid for the code I write during the day, but surprisingly (or not) I enjoy the code I write at night the most.

I have written code for banks, multinational corporations, the kind of places where the phrase "professional dress code" is not used strictly in jokes. I've helped build software that was resilient, scalable, secure and intuitive, although granted, not all those at the same time. This was mainly Custom Enterprise Software, or to put it more plainly, Software That Runs On Computers At Work. So I thought I'd try a different angle in my spare time: write software that people use at work, or at home. Or at a coffee shop. Or at the airport. Or wherever they feel like using a computer, anyway. This is sometimes called End-User Software, or to put it in layman's terms, Software People May Actually Like. After trying out some ideas that didn't seem worthwhile, I discovered that some of my buddies had similar thoughts. This is how FireStatus was born.

FireStatus is a Firefox extension that aims to be a swiss army knife for dealing with various social networks, right from your browser, without visiting any particular website. Facebook, Twitter and FriendFeed are currently supported, but more are in the pipeline. For starters, it allows you to simultaneously update your status to all or some of these services, so that all your friends see it, no matter what they are using. The notion of a status that is occasionally updated is familiar to Twitter and Facebook users, since the text field that asks 'What are you doing?' is prominent in their user pages. FriendFeed does not have the notion of a status, but its users can post short (or long) messages, just like Twitter's. FriendFeed also allows posting links to web pages, accompanied with a short description, something that many Twitter and Facebook users have been doing by constructing status updates that start with the description text and are followed by an appended link, usually shortened.

FireStatus can ease the task of posting these messages or status updates, by being always available, instead of needing to have the service pages open and without a large memory footprint, like other similar applications, since it takes advantage of the fact that most people nowadays always have one browser open. I know I do. Come to think of it, I don't know anyone who uses computers that does not keep a browser running most of the time. At least untile the darned thing crashes, which fortunately is something I haven't experienced in Firefox for quite some time.

Posting a message or status update

Clicking on the FireStatus icon, pops up a small toolbar window just above the status bar and below the window document. It is similar to the Firefox findbar that pops up when one searches for text in a page, albeit slightly larger. That was a deliberate design decision that aimed to imitate the success in the usability of the findbar. Lots of little details like this one have been carefully thought out and occasionally debated at length among the team and our beta testers:

• Enabling the spell checker in the status message field, for catching those typos when hastily typing a message.
  
• Having the URL inclusion unchecked by default, while the shortening checked, since most posts do not include URLs (and it might be embarrassing if done inadvertently), but those that do usually want them short.
• Adding a character counter to help guard against the maximum message length imposed by some services (e.g. Twitter).
• Tightly placing the available services in a way that allows for an unambiguous selection.
• Using the Escape and Enter keys as shortcuts for canceling and sending the update respectively.

As one reviewer in addons.mozilla.org so eloquently put it, "the simplicity & minimal design are the key components here".

Posting is not the whole story, though. These services provide continuous streams of updates from friends that we need to monitor. Sifting through the updates in every service, while very useful, becomes tedious after a while. It is like making a mental note to check your e-mail every once in a while for new messages. It's one way to do it for sure, but mail notifiers have been around for ages and provide more of a "live" experience.

FireStatus imitates their success in making e-mail conversations "alive", for conversations in the social network space. Every time a new incoming update is received, a notification popup appears, so I can continue using whatever application I was using, but being instantly aware of the news. Twitter notifications contain the name and the picture of the message author as well as the message sent. Low-priority or uninteresting messages can be just glanced at and then automatically dismissed, while important ones can be clicked on, so that the message can be viewed in the browser. FriendFeed notifications can be of various flavors, since FriendFeed aggregates updates from a large variety of online services.

Therefore FriendFeed updates display the service icon along with the author's name for quick identification of the type of update. Clicking on an interesting update opens a browser window to the link contained in the message. Facebook notifications currently contain new message counts, pokes and shares, but work for getting more data is underway.

Although the notifications appear instantaneous, as with e-mail notifications there is a polling process involved underneath. The polling frequency for each service can be separately tuned in the preferences window. The preference window can be opened by right-clicking on the FireStatus icon and selecting the appropriate option in the popup menu. Along with the time intervals between polling for updates, one can enable posting and/or receiving updates for the various services, as well as the authentication credentials where appropriate. For Facebook the user logs in in a Facebook-supplied browser window that pops up, similar to regular Facebook use. No credentials need to be stored separately by FireStatus. For FriendFeed the username and remote key have to be stored locally in the extension preferences. The remote key is not the same as the user password, but is provided by FriendFeed specifically for use by third-party applications, like ours. It can be found by clicking on the link displayed in the explanatory note. Twitter can work either way. If a username and password are entered in the preference window, they will be stored locally in the extension preferences and used for subsequent authentication. If, on the other hand, the fields are blank, FireStatus will consult the browser's Password Manager for any stored Twitter credentials. This will probably cause the master password dialog to pop up once, but the credentials will remain stored only in the Password Manager.

FireStatus preferences

Since we built this thing for our own personal use, pleasure and self-punishment, and as we don't plan to ever get rich out of it, we have released it from the beginning as open-source software through the permissive BSD license. You are most welcome to come by the project's home at Google Code, study the code, fix a bug (or two), help with the (embarrassingly outdated) documentation, contribute a review (otherwise it will remain an experimental extension forever) or just chat with us at the mailing list. Heck, if you feel like it, you can grab the whole code and use it in your own lucrative business and you don't owe us a dime.

With the shape the world economy currently is in, you probably couldn't afford it, anyway.

Murdered by Numbers

"JavaScript has a single number type. Internally, it is represented as 64-bit floating point, the same as Java's double. Unlike most other programming languages, there is no separate integer type, so 1 and 1.0 are the same value. This is significant convenience because problems of overflow in short integers are completely avoided, and all you need to know about a number is that it is a number. A large class of numeric type errors is avoided."

Douglas Crockford, JavaScript: The Good Parts

Bitscuits © Barney Livingston / CC

When I was twenty-something, I liked watching The Young Indiana Jones Chronicles. Besides the regular stuff that your average Indy fan loves, I was particularly fond of Indy's apparently insatiable appetite for traveling and learning different languages. So while I was contemplating what it would take for me to follow in his footsteps, I figured I'd better start with learning foreign languages. By that time I had already got my English certificate and I had a few years of learning French under my belt that had at least provided me with some means to court women (with little success, regrettably). So I started learning Italian, which everyone said was easy to pick up, especially if you had already mastered some other foreign language. That turned out to be true, and I managed to get a degree in Italian after two years of intensive studies. What made that period frustrating (and occasionally funny) however, were the times that I would inadvertently mix all three languages in the same sentence, creating my own version of Esperanto. Due to the similarities among them, I might be trying to speak Italian, but use an English noun while constructing the past tense of a verb in French. Sometimes I would even be oblivious to my mistake until someone else pointed it out to me. It all seemed very natural as I was doing it.

Lately I'm having a déjà vu, when I find myself coding in Java, JavaScript and C++, often in the same day. More than once I tried to initialize a Java object using a JavaScript object literal. Sadly, the compiler was not very accommodating. While writing a GWT-based front-end, I often transmitted POJOs without being aware that a long value was silently converted to a JavaScript Number, which essentially amounts to a Java double. Reading David Flanagan's "Rhino" book had already left me with the impression that JavaScript was rather flawed for not having separate types for integers, bytes, chars and all the other goodies that languages like C/C++ and Java spoil us with. But after getting a copy of Douglas Crockford recent, highly opinionated, "Good parts" book, his argument resonated with me: "A large class of numeric type errors is avoided." It's Richard Gabriel's "Worse Is Better" principle, or alternatively "Less Is More", in new clothes. Recent events made sure that the message was permanently bolted in my brain.

I've been writing a desktop application in Java that communicates with various Bluetooth and USB devices. The communication protocol is some sort of terminal-like commands and responses that initiate at the Java application and travel through a thin JNI layer down to the C/C++ device driver, and then to the actual device. The protocol documentation describes in... er, broad terms, the sequences of bytes that constitute the various requests and responses. Suffice it to say that my system administrator's experience in sniffing and deciphering network packets proved invaluable.

Sending the command was easy (or so I thought): fill a byte[] array with the right numbers and flush it through the JNI layer. There we get the jbyteArray and put it inside an unsigned char array, which is later send through the device driver to the actual device. When receiving responses the sequence was reversed. It all seemed to work fine for quite some time, until suddenly I discovered that one particular command caused the device to misbehave. I couldn't be sure if the device was faulty or my code was buggy, but since I had zero chances of proving the former, I focused on investigating the latter. A couple of days of debugging later I was still on square one, since as far as I could tell the command reached the device unscathed. Logic says that if a fine command reaches a fine device, then one would be entitled to a fine response. Since I wasn't getting it, I began questioning my assumptions.

I resisted the urge to blame the device, since I couldn't prove it conclusively, and started blaming the command. There was definitely something fishy about it, and to be honest, I had a bad feeling all along. The other commands were simple sequences, like:

0x14, 0x18, 0x1a, 0x3e

or

0x13, 0x17, 0x19, 0x26

This particular one however, was icky:

0x11, 0x15, 0x17, 0xf0

If you can't see the ickiness (and I won't blame you), let me help you.

Java's byte type is a signed 8-bit integer. C++'s unsigned char type is an unsigned 8-bit integer (at least in 32-bit Windows). Therefore we can represent values from -128 to 127 in Java and values from 0 to 255 in C. So, if you have a value between 128 and 255, ickiness ensues. 0xf0 is, you guessed it, between 128 and 255. It is 240 to be precise, if Windows Calculator is to be trusted.

Now, of course I am not that dumb. I knew that you can't assign 0xf0 to a byte in Java, so I had already made the conversion. You see, what is actually transmitted is a sequence of 8 bits. If you get the sequence right, it will reach it's destination no matter what. When you convert 0xf0 to bits you get 11110000. The first bit is the sign bit, which is causing all the trouble. If it was zero instead, we would be dealing with 1110000, or 0x70, or 112 if you're into decimal numbers.

So that's what I had done. I'd constructed the negative version of 112 and used that to fill my command buffer:

0x11, 0x15, 0x17, -112

Looking at it made me feel a bit uneasy, without being able to explain why. I used to think it was the mixed hex and decimal numbers. Yeah, I'm weird like that. However, the zillionth time I reread the Java's byte type definition, a lightbulb lit up over my head. I actually paid attention to the words in front of me: "The byte data type is an 8-bit signed two's complement integer".

Sure, it's 8 bits wide and yes it's signed, using the most common representation for negative binary numbers, two's complement. What's new here? I know how to negate in two's complem...

Whoa! Wait a minute. What I just described above isn't how you negate a number in two's complement. It's actually how you do it in the sign-and-magnitude variant. In two's complement you invert the bits and add one to the result:

11110000 -> 00001111 -> 00010000 or 16, or 0x10

Yep, 16, not 112. So the proper command sequence becomes

0x11, 0x15, 0x17, -16

and, yes, the device seems quite happy with that. As happy as devices get, that is.

So, basically, I wasted many many hours and suffered innumerable hair-pulling episodes for falling prey to a numeric type error. The kind Doug Crockford alludes to in his book. Now, don't get me wrong, I like mucking with bits as much as the next guy. But had I been living solely in JavaScript-land, with the single number type, I'd feel less tired and probably not hate my work as much. Though, granted, I might be needing a haircut right now.

Why you should go to conferences

One thing I like about working in a large corporation is that you get to meet all sorts of interesting people. You actually have to meet them, since you can't get anything done without coordinating with a whole bunch of them. But since this tends to get nothing done most of the time, I chose to work in a small company instead. Here if you want to do anything, you probably have to do it yourself (there are some interesting repercussions of this that I will leave for a future post). Now, if you work in a small company and want to meet lots of interesting people, you basically have three two options:

1. Use the net - blogs, mailing lists, newsgroups, twitter, friendfeed, etc. (done that)
2. Attend conferences.
3. Go to parties.

Now as you can personally attest to, I've got No.1 covered. So the last couple of days I worked on number 2 instead. The nice guys at ellak.gr organized the 3rd FLOSS conference and since it took place very close to where I work, I took the opportunity to avoid working meet some old friends and make some new ones.

Number One of course, is Wietse Venema. Since his work was an inspiration for me to pursue a PhD in computer security, I just had to see him in person once. He was more fun that I expected, to be honest. Security guys have a reputation of being a wee bit paranoid and itsy-bitsy abrupt on normal people, not to mention complete strangers. Wietse however was neither. This probably has to do with his academic background or maybe the fact that he is Dutch, who knows. He proved to be a warm, funny guy instead, who runs FreeBSD on his laptop (that always shows character in my book).

His talk was about Open Source and Security, both matters dear to my heart. The discussion of Postfix was a walk down memory lane for me. I can still remember the excitement around its initial release: "Hooray! No more sendmail exploits!". Wietse briefly discussed the architecture of postfix and how it relates to its security. He also showed some statistics comparing postfix to sendmail and qmail that I had seen in the past, but was curious to where everyone currently stands. Apparently postfix holds the second place in number of deployments (good), the first in number of lines of code (bad), but that is caused by having reached sendmail's feature list, without compromising the original architectural goals (good). For the problem of security at large, the most provocative suggestion was to make software development too hard for the laymen, so only experts could do it. Heh. As John Wayne so eloquently used to put it: "that'll be the day".

Lots of comparisons were presented in the talk by Diomidis Spinellis, titled A Tale of Four Kernels. The talk covered a code quality comparison of four major commercial and open-source operating systems, Windows, Solaris, Linux and FreeBSD. Diomidis is a well known author and FreeBSD commiter, and that was evident in the quality of his work and presentation. The room was packed, and the Q&A session lasted almost as long as the presentation itself. I suppose the sensitivity of the subject had something to do with it. Diomidis went through heaps of information so fast you could barely comprehend each slide, unless you had the forethought to study his paper before the conference, like your truly did. Although he was very delicate in his conclusions, stating that all systems had comparable quality, there were some members of the audience that seemed to have already made up their minds and didn't care for diplomacy.

Another great talk was by Jim Blandy. The talk was about Open Source Infrastructure for Software Development, which covered the evolution of version control systems (VCS), from SCCS, to the recent crop of distributed VCS. Being one of the people who gave us CVS and Subversion, his recollection of the history of this area was very exciting. Besides sharing some colorful historical tidbits, he also gave a meaningful comparison of the algorithms and data structures used by the various systems. He gave high marks to Subversion for being a fine choice in a centralized organization structure, but eloquently presented the fundamental paradigm shift that distributed VCS present. He said (as best as I can remember) that controlling access to the commit step (what all centralized VCS do) is the wrong (worst?) place to do it, since adding a change is not that big a deal. Instead the point of merging is what should be guarded, since that is when another changeset gets incorporated in our code. The fact that distributed VCS replicate whole repositories, not just working copies, makes allowing commits a natural thing, while making merge selection part of the release engineering process. I've read many pro-distributed-VCS arguments, but I found Jim's focus on the social aspect of the merge process a refreshing one. Luckily, most of my cohorts were present in this talk, so switching our team to Mercurial might not be just a dream after all.

Jim currently works on Mozilla, on the new ActionMonkey JavaScript engine. Since that work was outside the scope of this talk, I managed to corner Jim afterwards to find out more about its current status. The project attempts to integrate the current Mozilla JavaScript engine, SpiderMonkey, with the Tamarin engine contributed by Adobe. The plan was to replace various parts of the old engine with the new ones, like the garbage collector, the JIT compiler, etc. However doing so leads to performance regressions in some areas and Jim said that one lesson they have learned is that if you ship with worse performance, no matter how rare the circumstances that expose it, you will pay for it at some point. String manipulation appears to be one of those areas, since the old engine had been carefully optimized during all this time and achieving the same performance with the new one, requires work. I have been wondering for some time now whether SpiderMonkey uses some variation of the Ropes data structure for String representation, but Jim explained to me that was not the case.

He was also very excited about the experimental tracing JIT that they are working on, that appears to be a perfect fit for dynamic languages and is expected to boost JavaScript performance on future versions of the browser. I told him that I had already heard of it from Steve Yegge, and I've been meaning to read the literature Real Soon Now(TM), which I swear is the truth, the whole truth and nothing but the truth. After I finish this belated post of course. Jim said that Steve is a great story-teller, which means that Steve, if you ever find yourself in Athens and want a free dinner, just let me know. Explaining the benefits of tracing JIT led us to a detour into LISP-land and while Jim has written a Scheme compiler, I've only done LISP programming for a class as a student. Which is another way of saying I didn't understand a word he said, alas. Anyway, the cool things they are working on will probably come to us in firefox 4, so I still have time to get up to speed.

All in all, the conference was quite an enjoyable experience. Old friends, new friends, free food, stimulating discussions, what's more to ask?

What? To be on TV? Hey, that's easy. You just have to hang outside the main conference room chatting with random people, until you notice a cameraman and a journalist taking interviews from conference organizers and speakers. Then you make sure to be in the camera frame and voilà! You've got your 3 seconds of fame.