Microformats and Firefox 3 (for Users)

I'm now the official owner of microformats support in Firefox 3. Here is the preliminary design document.

At this point, we'll primarily be focusing on making microformats available to extension developers, with very little UI. The primary motivator behind this is that I don't think anyone has come up with a good user interface for microformats. I want to take some time to bring up some different UI paradigms and have some general discussion on these ideas just to see what people think. We're actually trying to have a UI discussion on the labs forums, but we're not getting much input so I thought I would try here.

So far there have been multiple extensions with multiple ways to interact with microformats:

  • Operator (toolbar, status bar button, toolbar button, right click on microformats)
  • WebCards (in page "cards" - notifcation ribbon on the bottom of the screen, right click on microformats)
  • Tails Export (sidebar)
  • Tails (icon on the status bar - window presenting content of the microformats)

Some of the other ideas that have been floated have been using the Firefox notification bar to inform about microformats, in page highlighting of microformats, notifications on the URL bar similar to RSS feeds and changing the mouse pointer. You can see some of these ideas at Alex Faaborg's blog.

The primary concern we have in creating a microformats UI is that it's not necessarily something that is going to be accessed a lot (like a back button for instance, or bookmarks), so how much screen real estate should it take?

With Operator, my original plan was to allow for many different UI paradigms, and I'm continuing that in the next release by allowing individual buttons for specific microformats (a Contacts toolbar button for instance). So now I'm putting the question to you:

What UI paradigm for microformats do you prefer? Is there a paradigm you wish was in Operator (like a sidebar for instance)? What do you think microformats should look like in Firefox 3?

Feel free to come up with any wild idea you want.

In my next post, I'll be talking about what we should do for developers.

Deploying Firefox in the Enterprise: Part 5 (Revisited)

Since my original post on setting up a Firefox update server, I realized that some of my information was not correct. This post is to correct that information.

My mistake stemmed from the fact that the format of the Firefox update URL changed between Firefox 1.5 and Firefox 2.

Here's the URL for Firefox 1.5:


https://aus2.mozilla.org/update/1/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/update.xml

Here's the URL for Firefox 2:


https://aus2.mozilla.org/update/2/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/update.xml

You'll notice two differences:

  1. The number after update was changed from 1 to 2 (this identifies which URL format to use)
  2. %OS_VERSION% was added to the URL

The important change here is the addition of OS_VERSION. Whereas before, the update URL provided information only about the browser, with OS_VERSION, the URL now contains information that changes based on the machine on which the browser is running. This makes providing updates a little trickier (although not that much).

(Incidentally, the reason that OS_VERSION was added was so that the update server could detect Windows98 and Windows95 and not send them updates that didn't work on those machines.)

Let's present a modified .htaccess file that handles this situation. I'm going to assume that in your enterprise deployment, you don't care about OS_VERSION, so we are just going to ignore it. In our original .htaccess file, we simply checked for the existence of update.xml in a path that corresponded to the URL. Because of the addition of OS_VERSION, that won't work anymore, since we would have to duplicate the update.xml file based on every possible OS_VERSION that could be sent to us. So we need to create a .htaccess file that ignores OS_VERSION. The way we do this, is simply redirect the URL to a new URL that doesn't contain OS_VERSION, and then use our old method of checking for the existence of a file.

RewriteEngine on

RewriteRule ^(.*)/release/(.*)/update.xml /$1/release/update.xml [R]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*) noupdate.xml

What this says is if the URL contains something between release and update.xml (OS_VERSION), remap it to remove the OS_VERSION. Then send it to the server again.

I'll leave updating the PHP script as an exercise for the reader.

I hope this clears up any issues with my original post.

Operator Action Architecture

One of the things I've been trying to finalize for the next release of Operator is the action architecture. Here's what things look like now.


ufJSActions.actions.corkd_search_tags = {
  description: "Find wines on Cork'd",
  icon: "http://corkd.com/favicon.ico",
  scope: {
    semantic: {
      "tag" : "tag"
    },
    url: "http://corkd.com"
  },
  /*
   * Perform an action
   *
   * @param semanticObject JavaScript representation of the semantic object
   * @param semanticObjectType Semantic object type as a string
   * @param domNode DOM Node associated with the semantic object
   * @return If you return a value, we attempt to open it as a url
   */
  doAction: function(semanticObject, semanticObjectType, domNode) {
    if (semanticObjectType == "tag") {
      if (semanticObject.tag) {
        return("http://corkd.com/tags/" + encodeURIComponent(semanticObject.tag));
      }
    }
  }
};

First let's talk about scope. Semantic scope is about indicating which microformats (and in the future other semantic data) this actions works with. On the left is the type of semantic data, on the right is the required value. So for instance, if you had an action that worked on hCalendar and required a dtstart, it would be "hCalendar" : "dtstart". You can use url to specify that an action should only be displayed when you are on a certain web page.

Next, let's talk about the action itself. When you return a value from an action, we assume it is a URL and open it. If you want to do your own thing (like export a vCard), you can do arbitrary JavaScript and just return. I'm also working on a "doActionAll" which will allow on action to act on all microformats of a given type on a web page.

What do people think of this overall design? Is it simple enough to encourage anyone to write actions? Am I missing anything obvious?

Firefox and the Windows Registry

So it ended up that it wasn't that easy to switch our installer to use "Product Version" to determine what version of Firefox was installed, so I was asked to provide information on how to use the Windows Registry to determine the installed version of Firefox. Since that information is useful to everyone, I thought I would post it here.

The following registry path is where Mozilla Firefox info is contained:

HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\Mozilla Firefox

(Note that Firefox 1.5 and below used to put information in HKEY_CURRENT_USER as well. Firefox 2 does not.)

Within this registry path, there is a key, CurrentVersion. This key indicates the installed version of Firefox as well as the locale. Don't confuse this with HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\CurrentVersion which is something else entirely.

If you need to locate the installation, there is a key under Mozilla Firefox with the same name as CurrentVersion. That key contains another key called Main that has a value for the install path (Install Directory).

Incidentally, in the screenshot, you might notice that other folder, Mozilla Firefox 2.0.0.3, which has subfolders called bin and extensions. That's there for posterity. A long time ago, we told people that to find the plugins directory, concatenate the product and version together and then look for the plugins key as a child of that key. If I remember correctly, that's how it was done even in the Netscape Navigator 2/3 days. You can read more detail about those keys on MDC.

Microformat Objects in JavaScript

Unfortunately for everyone out there, when I discover new things in JavaScript, it means I rewrite some parts of Operator, and I am in the middle of that right now. The thing that I discovered is the ability to have custom getters for objects in JavaScript.

One of the things I have wrestled with in the past is that my current model for accessing a microformat in JavaScript is clunky. You have two choices - you can either create the microformat in its entirety:

var hcard = createMicroformat(node, "hCard");

and access the properties like this:

alert(hcard.fn);
alert(hcard.adr.region);

or you can access an individual property like this:

var fn = getMicroformatProperty(node, "hCard", "fn");

The problem with the first approach is that you do a lot of work if you only want part of the microformat, and the problem with the second approach is that there is no caching of the data.

Enter custom getters:

var hcard = new hCard(node);
alert(hcard.fn);

and hcard.fn is not queried until you access it the first time and it is cached.

What I do now is that when the hCard is instantiated, I set the getter for each property to be a function that queries the property and then sets it (getMicroformatPropertyGenerator):

object.__defineGetter__(property, ufJSParser.getMicroformatPropertyGenerator(node, microformat, property, object));

getMicroformatPropertyGenerator looks like this:

getMicroformatPropertyGenerator: function(node, name, property, microformat)
  return function() {
    var result = ufJSParser.getMicroformatProperty(node, name, property);
    if (result) {
      microformat.__defineGetter__(property, function() {return result});
      return result;
    }
  };
}

The result allows for much cleaner code in an action, because the actions are now going to receive the microformat data object directly so they don't have to worry about DOM nodes and things like that. All an action has to do is access the individual properties. This will also allow for actions that aren't so dependent on microformats.

I'll still keep the old way around if people want to use the microformat parser in web pages. If anyone knows of a cross browser way to do what I am doing, or even a better way to do it, please let me know.

Firefox Internal Versioning

Updated with an additional column in the table (Size on disk). See comment after the table.

So we're running into a problem with our internal deployment of Firefox 2.0.0.3 that our software installer can't tell which is newer, Firefox 1.5.0.11 or Firefox 2.0.0.3. At the same time, I was asked by someone to create a list of files that could be used as "signature files" for Firefox. (For this purpose, a signature file is a file in a distribution with a unique file size that can be used to identify a particular version of the distribution.)

As a result of this effort, I came up with this lovely table.

  FIREFOX.EXE Values are taken from the
Version page of firefox.exe properties
Firefox Version Size Size on disk File version File Version Product Version
1.0 6621794 6623232 1.0.0.0 1 1.7.5: 2004110711
1.0.1 6626916 6627328 1.0.1.0 1.0.1 1.7.6: 2005022518
1.0.2 6627428 6631424 1.0.2.0 1.0.2 1.7.6: 2005031717
1.0.3 6631012 6631424 1.0.3.0 1.0.3 1.7.7: 2005041417
1.0.4 6631012 6631424 1.0.4.0 1.0.4 1.7.8: 2005051112
1.0.5 6636644 6639616 1.0.5.0 1.0.5 1.7.9: 2005071118
1.0.6 6636644 6639616 1.0.6.0 1.0.6 1.7.10: 2005071605
1.0.7 6637156 6639616 1.0.7.0 1.0.7 1.7.12: 2005091517
1.0.8 6639716 6643712 1.0.8.0 1.0.8 1.7.13: 2006041017
           
1.5 7162979 7163904 1.8.20051.11116 1.8: 2005111116 1.5
1.5.0.1 7166053 7168000 1.8.20060.11112 1.8.0.1: 2006011112 1.5.0.1
1.5.0.2 7171685 7172096 1.8.20060.30804 1.8.0.2: 2006030804 1.5.0.2
1.5.0.3 7172197 7176192 1.8.20060.42618 1.8.0.3: 2006042618 1.5.0.3
1.5.0.4 7177325 7180288 1.8.20060.50817 1.8.0.4: 2006050817 1.5.0.4
1.5.0.5 7183469 7184384 1.8.20060.6376 1.8.0.5: 2006071912 1.5.0.5
1.5.0.6 7183469 7184384 1.8.20060.7278 1.8.0.6: 2006072814 1.5.0.6
1.5.0.7 7190637 7192576 1.8.20060.25382 1.8.0.7: 2006090918 1.5.0.7
1.5.0.8 7191149 7192576 1.8.20061.2516 1.8.0.8: 2006102516 1.5.0.8
1.5.0.9 7200365 7200768 1.8.20061.20612 1.8.0.9: 2006120612 1.5.0.9
1.5.0.10 7206509 7208960 1.8.20070.21601 1.8.0.10: 2007021601 1.5.0.10
1.5.0.11 7206509 7208960 1.8.20070.31202 1.8.0.11: 2007031202 1.5.0.11
1.5.0.12 7209069 7213056 1.8.20070.50813 1.8.0.11: 2007050813 1.5.0.12
           
2.0 7604331 7606272 1.8.20061.1023 1.8.1: 2006101023 2.0
2.0.0.1 7620696 7622656 1.8.20061.20418 1.8.1.1: 2006120418 2.0.0.1
2.0.0.2 7633008 7634944 1.8.20070.21917 1.8.1.2: 2007021917 2.0.0.2
2.0.0.3 7633008 7634944 1.8.20070.30919 1.8.1.3: 2007030919 2.0.0.3
2.0.0.4 7637104 7639040 1.8.20070.51502 1.8.1.4: 2007051502 2.0.0.4
2.0.0.5 7644008 7647232 1.8.20070.5781 1.8.1.5: 2007071317 2.0.0.5
2.0.0.6 7644520 7647232 1.8.20070.6982 1.8.1.6: 2007072518 2.0.0.6

NOTE: I've updated this table to include a new field, Size on disk. This field is obtained from the General Properties page for FIREFOX.EXE. I got my original file sizes by looking at the file system from a command line and they correspond to Size, NOT Size on disk. I have no idea why, but I wanted to post both values here for posterity.

So first lets talk about the "signature file" problem. Unfortunately I quickly discovered that there are a few versions between which the file sizes for firefox.exe are EXACTLY the same. In particular:

  • 1.0.3 and 1.0.4
  • 1.0.5 and 1.0.6
  • 1.5.0.5 and 1.5.0.6
  • 1.5.0.10 and 1.5.0.11
  • 2.0.0.2 and 2.0.0.3

I investigated further and found that just looking at disk footprint, there is no way to tell those versions apart. I'm curious to know if other products besides our Tivoli products use file size to determine signature. If this is the case, it might be worth while to have a small signature file in Firefox that simply grows by a byte with each release. This would give us a file for which we could track versioning using file size.

The second problem is much more serious for us and revealed that despite the fixing of bug 286825, there is still a major problem with Firefox file versioning. In particular, take a look at the File version (not Version) of Firefox 1.5.0.11 and Firefox 2.0.0.3. You'll see that Firefox 1.5.0.11 is 1.8.20070.31202 and Firefox 2.0.0.3 is 1.8.20070.30919. So if an application uses File version to determine which is the latest version, it will claim that Firefox 1.5.0.11 is newer than Firefox 2.0.0.3. This is what is causing our software install problem.

Hopefully we can workaround the problem on our end, but at the same time, we need to get the file versioning in Firefox done properly. Otherwise, this is just yet another hurdle for the enterprise deployment of Firefox.

Operator 0.7 is available

Operator 0.7 is on addons.mozilla.org. I have a few things I wanted to share about this release. New features will be at the end of this post.

I'm not sure people realize there are other ways to interact with Operator besides the toolbar. Not only can you have Operator on the status bar or as a toolbar button, but you can right click on a microformat on a web page to access Operator functionality (Sometimes you can guess where they are, but if you want to see exactly where the microformats are, you can turn on Highlight Microformats in Options). If you want to access Operator from the status bar, go to Options and check the box that says Display icon in status bar. If you want to acesss Operator as a toolbar button, select View->Toolbars->Customize... There is an Operator toolbar button that can be dragged into the Firefox user inteface. Once you have done either of these, you can go to View->Toolbars and turn off the Operator Toolbar.

This release introduces a basic user interface for installing user scripts. Note that ability to add user scripts has always been there, I've just never had a good UI. User scripts can be used to add new microformats and new actions to Operator. As a part of the introduction of this UI, I'm moving some of what I call the non tier 1 microformats out of the core Operator and made them available as user scripts. My decision as to which ones to remove was mainly based on which microformats fit best into the Operator model; that is to say it is easy to provide actions that relate to the microformat. It is very important for you to realize that user scripts have FULL JavaScript privileges. Do NOT install Operator user scripts unless you trust the people that are providing them. I am going to make all my scripts available at http://www.kaply.com/weblog/operator-user-scripts/. If you want to write your own actions or microformats, feel free to look at my scripts, but please be aware that this API is very much in flux. I don't a good versioning mechanism right now, so at some point your scripts would cease to work. Note that I continue to support the original Operator action model, but not the original Operator microformats model. The microformats model is actually pretty close to final.

When I first created Operator, my stated goal was take the microformats that are on a web page and provide functionality up and above what was available in the web browser. Unfortunately, when I first started creating actions for the microformats, some of the actions I created were just a case of moving browser interaction into the Operator menus. (Go to web page being the best example). With the advent of a user interface for user scripts, I've taken the opportunity to move some of the function that shouldn't have been there in the first place out of the core. Note that no functionality has been removed; it is all available as user scripts. If you have a very strong opinion on this, one way or the other, please let me know.

All that being said, let me outline the new features in Operator.

  • User interface for managing user scripts
  • Operator updated when DOM nodes are added or removed
  • Export All available for contacts and events
  • 30 Boxes support has been greatly improved
  • Interaction problems with McAfee SiteAdvisor are fixed
  • Major performance enhancements
  • Native getElementsByClassName is used on Firefox 3
  • Operator wasn't working correctly with frames and iframes
  • New tag action for searching for videos on YouTube

In addition to those changes to the core, the following user scripts are now available:

  • Send to Bluetooth (Mac only)
  • hAtom
  • Simple GRDDL RDF export
  • XFN
  • species

Deploying Firefox 2 within the Enterprise: Part 6

I want to cover few more topics to wrap this things up.

The first thing you might be thinking is "man that seems like a lot of work. Isn't there something else I can do?" As I pointed out in the first post, Benjamin Smedberg's Firefox Release Repackager is great if you are a Mac user. If you are not using a Mac, or you want to add more than two extensions, or if you need to add files to the removed-files list we talked about, you can't use the release repackager. Here's how to manually modify a Firefox install.

First, download a Firefox setup executable. Unpack that file into a directory using 7z.

7z x "Firefox Setup 2.0.0.3.exe"

If you want to add files to the removed files list, do it now. Edit the file removed-files.log and add your files and directories following the syntax in the file. If you want to add extensions, you can follow the steps we outlined earlier, unzipping them in a temporary directory in nonlocalized/extensions and then renaming the temporary directory to the extension ID based on the information in install.rdf

If you are not interested in having a self extracting executable, this is actually enough. If you were to run setup.exe at this point, it would do the install, removing the files and directories you specified, and including the extensions you unzipped. If you want to create a self extracting executable, you would need to manually repackage the EXE. Unfortunately, I don't have those steps in front of me right now. I'll update this post if I find them.

Another problem that might arise is that you don't mind updates coming from Mozilla, but you'd like to determine when they get pushed to your users. For instance, you might want to do some internal testing. You can actually set preferences that prevent specific updates from being offered to your users. For instance, the preference app.update.never.2.0.0.1 tells Firefox to never offer the 2.0.0.1 upgrade. So you can use the CCK to set these preferences, and then update the CCK when you have verified a particular version.

What if Firefox can't upgrade your users because it is running at the wrong permissions level? If this is the case, you'll obviously need to create your own upgrade solution and turn off the Mozilla upgrades. To turn off the Mozilla upgrade checking, set the preference app.update.enabled to false.

Forgot one - if you want to do a silent/unattended install, invoke the installer (setup.exe or the full installer executable) with -ms.

Well, that's about all I have. I hope you've enjoyed this series of posts. If you have any questions about deploying Firefox in the enterprise, please let me know.

In addition, I'll take this moment to shamelessly plug IBM. If you are an enterprise and need help customizing and/or deploying Firefox, IBM can help. Contact me at mkaply@us.ibm.com

Deploying Firefox 2 within the Enterprise: Part 5

IMPORTANT! Some of the information in this post is not correct. I will be updating it soon.

Updated: See section on serving the update.xml file

OK, now that we have MAR files, how do we get them to our users?

Setting Up a Firefox Update Server

A Firefox update server is actually not that complex of a setup. Basically a Firefox update server takes a URL that describes a particular Firefox version, and if there is an update available, it returns an XML file that identifies the location of the update. So let's take a look at this URL.

If you are to go into about:config and search for app.update.url, you would see the following:


https://aus2.mozilla.org/update/1/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/update.xml

When Firefox 2.0.0.3 looks for an upgrade (English on Windows), the URL looks like this:


https://aus2.mozilla.org/update/1/Firefox/2.0.0.3/2007030919/WINNT_x86-msvc/en-US/release/update.xml

Most of this URL is self explanatory. The only part we need to explain is BUILD_ID. The build ID of a Firefox is a unique identifer that conveys the exact date down to the hour when a build was created. For the build above, that data is March 9, 2007 at 7:00 PM. You can see the build ID for any Firefox build by type about: in the URL bar. Note that selecting Help->About does not display the complete build ID. If you build a custom version of Firefox, you can see the build ID by looking in the file mozilla/obj/config/nsBuildID.h. If you see a build ID of 0000000000 in this file, it means that you forgot to set MOZILLA_OFFICIAL and BUILD_OFFICIAL before building.

If we look at the URL that is served, it ends in update.xml. So what do these update.xml files look like? Here's the update.xml to upgrade from Firefox 2.0.0.2 to Firefox 2.0.0.3. (https://aus2.mozilla.org/update/1/Firefox/2.0.0.2/2007021917/WINNT_x86-msvc/en-US/release/update.xml)

<?xml version="1.0"?>
<updates>
  <update type="minor"
          version="2.0.0.3"
          extensionVersion="2.0.0.3"
          buildID="2007030919"
          detailsURL="http://en-US.www.mozilla.com/en-US/firefox/2.0.0.3/releasenotes/">
    <patch type="complete"
           URL="http://download.mozilla.org/?product=firefox-2.0.0.3-complete&amp;os=win&amp;lang=en-US"
           hashFunction="SHA1"
           hashValue="bc64376c13867b5524d71e87f7ad98e727f2d3f4"
           size="7617299"/>
    <patch type="partial"
           URL="http://download.mozilla.org/?product=firefox-2.0.0.3-partial-2.0.0.2&amp;os=win&amp;lang=en-US"
           hashFunction="SHA1"
           hashValue="2555dcad94c9bc1c7327cb3699844f633888ea4a"
           size="270514"/>
  </update>
</updates>

Rather then go into a lot of detail here about this file, I'd rather point you to wiki.mozilla.org which has a great explanation of this file. (Incidentally, the wiki you can lots of other great information about the Software Update process) One thing you might be wondering is how to generate the hashValue. Microsoft has a tool available called FCIV that can be used to generate the hashes.

OK, we know the URL and we have the update.xml file, so how do we serve it to the browser?


Update: as Benjamin Smedberg pointed out, I am doing a lot of work for nothing. mod_rewrite can be used to accomplish this task. Create a file called "noupdate.xml" that contains <updates></updates>Then put this in a .htaccess file:

RewriteEngine on

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*) noupdate.xml

What this says is if you find the file, serve it, if you don't, serve the file that represents the empty update.


At first, it looks like we can just setup directories that correspond to the URL, but this won't work since we need to send an XML file that looks like:

<updates>
</updates>

If there are no updates available. So I'm going to use a simple PHP file to function as our update server. We'll need to do some changes to our web server configuration as well. Note I am using Apache.

First go into httpd.conf and make sure that mod_rewrite is available. Then go into your main web directory and add or edit the file .htaccess with this:

RewriteEngine on
RewriteRule ^/?update/(.*) /update.php/$1 [L]

What this rule says is whenever you see a URL that begins with "update" send it to update.php. The actual update.php file looks like this:

<?php
header("Content-type: text/xml");

$url_array=explode("/",$_SERVER["REQUEST_URI"]);
$noidea=$url_array[2];
$product=$url_array[3];
$version=$url_array[4];
$build_id=$url_array[5];
$build_target=$url_array[6];
$locale=$url_array[7];
$channel=$url_array[8];
$filename=$url_array[9];

$updatefile = "$noidea/$product/$version/$build_id/$build_target/$locale/$channel/$filename";
//echo "$updatefile";

if (file_exists($updatefile)) {
  $handle = fopen($updatefile, "r");
  $contents = fread($handle, filesize($updatefile));
  echo "$contents";
  fclose($handle);
} else {
  echo '';
  echo '';
}
?>

This PHP simply looks for the presence of an update.xml directory structure specified by the URL and if it is available, it returns it, otherwise it returns <updates></updates>

So how do we publish updates then? We simply create an update.xml file and place it in the appropriate subdirectory.

The last step is updating the browser to point to our new update URL. Your first thought might have been to simple update app.update.url in about:config, but that will not work. When app.update.url is queried by Firefox, it always looks for the "default preference" which means that even if a user changes the preference, it won't be used. For testing purposes, you can add a new preference called "app.update.url.override" that will work, but when you go to deploy your updates, you should set app.update.url in an extension like the CCK. That will cause it to become a "default preference."

One more installment to go. I'll try to address some general deployment issues, and if you have questions (any questions), PLEASE let me know.

Operator and DOM Mutation Events

This is working so well, I have to tell the world.

I just added support for DOM Mutation events to Operator. So now when DOM nodes are added and removed from the document, Operator will update appropriately.

But that's not the coolest part...

I added support for the DOMAttrModified event. So combined with Firebug, you can actually interactively modify your HTML to add/remove/modify a microformat and Operator will update live based on the changes you make. It's REALLY sweet.

New version coming soon....