Firefox Rapid Release Process
I've been writing a blog post about this in my head for a while, but after glazman's post, I definitely feel I need to weigh in.
My opinion of the new rapid release process depends on which hat I wear. So I'll offer three opinions.
As a Firefox developer, I love it. One of the things that frustrated me the most about Mozilla process in the pass is that security release only contained MAJOR fixes. So even if there was an important fix, it had to wait for the next release train which could be a year. This contributed to people complaining constantly about bugs even though they were fixed. So faster releases means faster fixes and less bugs for people to report.
As an add-on developer, I find it annoying but mostly a non-issue. The reason is simple: I target my add-ons for Firefox 3.6 and do basic testing on other Firefox versions. I know that seems strange, but Firefox 3.6 has 35% of the Firefox market share. So it would be silly for me to ignore millions of people. All these great new features for add-on developers are interesting, but it's unlikely I would ever use any of them until Firefox 3.6 had a < 10% share (which I don't think will happen any time soon, despite Mozilla dropping support). And the multiple versions of Firefox will make this problem even worse. I'm sure over time you'll see just about every Firefox version in use and you'll want to target the lowest common denominator.
As person involved in the corporate deployment of Firefox, I think it's a really bad idea. Companies simply can't turn around major browser updates in six weeks (and each one of these is a major update). With security releases, there was a reasonable expectation that web applications wouldn't break as a result of changes. With these releases, there is no such expectation. So a full test cycle needs to be run with every release. By the time this cycle is completed and the browser is piloted and deployed, another version of Firefox would already be released so they'd already be behind. And in the mean time, all of their browsers will be insecure, because all security updates are rolled into the major versions.
For corporate deployments, there has to be a stable branch. I bet someone is probably going to make a nice business out of creating and maintaining a stable branch...
Did you follow the LTS version discussion on mozilla.dev.planning ?
The idea was that for example one version a year would be selected to be a LTS version, with security bug backported until the next LTS version. Georg Maass suggested XULRunner would need more 36 monthes long LTS than one year.
My feeling is that Mozilla wasn't in principle opposed to such a solution, but that they don't have currently clients who request that, nobody who is ready to make it happen (by providing the required human resource for example), so that as long as there is nothing demonstrating the continued success of Firefox absolutely requires this to exist, it won't happen.
Who's definition of success? Firefox has had some success in the corporate market, but not a lot so this will be another hindrance to any success there.
One factor for us is that Firefox 4 (and presumably >4) doesn't work on RedHat 5, which is what most of our developers run on the desktop, being our primary development and deployment platform. Sure, it's not the current version, but updating to 6 isn't a trivial matter...
@Simon,
I have run FF4 on a CentOS 5.6 (which is basically RHEL 5.6, minus RH branding), without issues. If you don't count that it run slower than 3.6.x
and ate more RAM. So off it went and I'm back to 3.6.
Sure, if you mean RedHat 5.x from 1997-98 - I'm not surprised that FF4 doesn't run on it
The trouble is that the enterprise deployment world never successfully made itself visible for Mozilla. It's all "dark matter" for Mozilla.
No visibility on it means no visibility on what it could bring to engage the corporate world more, and no investment on it, Firefox happens to work in that context by accident but decision that change that can be taken at any moment.
Re corp install, I would think that a corp setup that takes more than 6 weeks to certify a browser needs someone that tells them how to do that.
Rather than claiming that Mozilla would need a LTS strategy, those companies need a rapid and agile testing strategy. Not because we want to ship every now and then, but because it's the right thing for them.
@Axel
The problem is that the current process requires a full time team who's only job at a company would be certifying browsers.
That might be able to certify it in less than 6 weeks, but they have to do it EVERY SIX WEEKS.
That's the problem.
That means full testing and deploying every six weeks. Compared to how it used to happen which was every six months to a year.
We're in 2011, there are automated test infrastructures for web apps all around. Making use of those will make those critical assets work better, and get the testing time from whatever you're thinking down to a handful machines doing things without a human looking over night.
Do that with the aurora builds as they come out, and you have 9-10 weeks to either complain to us, or plan the roll out of fixes you need locally.
Asking Mozilla to provide a time machine to go back to the days of IE6 is not helping those corps.
@Axel I think the core issue is that Mozilla envisions a world where everyone gets their browser updates directly from Mozilla every six weeks.
While that might work for the average user, it just doesn't fly in corporate environments, especially places like banks.
Testing or no testing, expecting a company to go through a full deployment cycle of their web browser every six weeks is simply ludicrous.
Maybe I'll be proven wrong, and I'm ok with that.
@Mike - I work at one of the largest financial services companies in the world (e.g. enterprise banking) - and I agree with Axel's comment above 100%.
All of the major browser manufacturers have gotten on board with the web standards movement - this means that in modern browsers the bulk of cross-browser compatibility issues developers once faced have been abstracted away.
Unless a web app is horribly designed - an app developed for Firefox 5 should "just work" with Firefox 6, 7, 8, 9...
The primary problem with is that auto tests runner needs to be recertified with the new browser revision as well. Worse, if you need to wait till a supporting version came out.
Also there are significant aspects to a web application that can`t be tested via automation. Manual testing is still needed. In case of IE there are 3-4 variants (depending on IE6 support), in case of Safari 2 (both windows and Mac, because of behavioral differences), in case of Chrome mostly 1. Now for Fx 3-4 instead of 1.
Basically this move increased the test cases that needs to be executed manually and via automation by at least 40%. No matter what you are saying corporate users can`t be forgotten. In case of large e-commerce sites or in case of internal LOB applications as well.
So in short, without a chance to use older versions (6-9 months should be enough) without security updates. Mozilla just created a new IE6 situation.
@Axel, @Mike:
See for instance the workflow used by openSUSE
Linux (a Linux distro targeting individual users, and distributed either by sale of boxed sets, or free-as-in-beer over the Net), as follows:
- "Component freeze" (with choice of products and versions, and known positive results to tests that the chosen kernel version and all the software work together on "most targeted machines") 3 months before the public release of the next distro version
- Until the following distro version (one version every 8 months) there are stability/security "Online Updates" but probably no minor version updates (Firefox 4.0 to 4.0.1 but probably not to 4.1) and IIUC almost certainly no "major version" updates (not 4.0 to 5.0). Note how the decision to release a new "major version" every six weeks made it more difficult to even consider upgrading as often as in the past.
(openSUSE 11.4, released March 10, currently includes Firefox 4.0.1. "Component freeze" for the next release, openSUSE 12.1, is foreseen for August 8.)
And BTW, this is why I get Mozilla products straight from Mozilla. For other products I include the openSUSE "Test updates" repository among the repositories polled by my "Online Updates" but it is not the default: it amounts to applying fixes (which may be e.g. crash fixes already fixed in the X.Org or GTK2 source) before the months of "baking" required for them to trickle down from the openSUSE "Test updates" repository to the public "Update" repository. See for instance the horror story (of a Firefox crash due to an Xorg bug) at https://bugzilla.novell.com/show_bug.cgi?id=423740
@Axel - Mike Kaply is completely correct. I have 500,000 corporate users on Firefox 3.6. We're just completing a test cycle of Firefox 4 on many thousands of internal business web applications. Many hundreds of application owners and their test teams have participated. We gave them several months to ready themselves. We worked with dozens of internal Add-On developers and product teams to prepare their add-ons for Firefox 4. We're poised to deploy Firefox 4.01 in 3Q when the corporate change freeze lifts. Education programs, documentation updates, communications all are planned. While several of us keep up with Aurora, I can't expect thousands of app owners to do the same. I applaud the effort to accelerate the pace of Web experience and I expect to chase version releases well into the future. The Firefox 4 EOL is a kick in the stomach. I'm now in the terrible position of choosing to deploy a Firefox 4 release with potentially unpatched vulnerabilities, reset the test cycle for thousands of internal apps to validate Firefox 5 or stay on a patched Firefox 3.6.x. By the time I validate Firefox 5, what guarantee would I have that Firefox 5 won't go EOL when Firefox 6 is released?
Right at the time 4 was published, the plan was to EOL it when 5 came along. And the plan is to EOL 5 when 6 comes along. Etc.
It seems this hasn't been communicated clearly enough.
The problem is that is a crazy plan, regardless of whether or not it is well communicated.
I see that Microsoft has offered to "Win Back IBM's business". Not a problem so long as they build IE for Linux and OSX.
Otherwise, maybe IBM can make a tidy profit out of forking Firefox and only doing one release a year (other than security fixes).
If they do, then maybe the Firefox version distributed by openSUSE will someday be stabler, and safer to use, than the one distributed by Mozilla (Reminder: zLinux [aka Linux for -IBM- System z] is completely opensource; IBM supports both Red Hat and SUSE Linux)
@Axel My university (Portmouth, UK) had a policy of only releasing major versions during the summer break, when their computers were mostly out of use and the IT staff had some time free from supporting users. In this environment it's not even possible to do a full test cycle of mission critical applications - one such application might be someone's dissertation project which the IT department would not be aware of.
Besides, if Firefox 4 has already been EOLed, it's not a six week test cycle that's needed, but a one day. If there was a critical fix applied to Firefox 5 tomorrow, that presumably wouldn't be available on Firefox 4. Even on my own computer I don't want to move to 5 yet because that would mean losing a couple of add ons for very little benefit.
I don't have a problem with the rapid release cycle, but I do have a problem with the software being EOLed so quickly. I think releases need to be supported for at least a year after the replacement version is released. If that means they only bump the major version once a year, and have x.1, x.2 and x.3 releases during the year that don't get supported then so be it.
I'm a developer in a large enterprise environment, and I agree with Axel.
This should be a wakeup call that it's time for enterprise IT admins to get on board with the modern practice of rapid, agile processes.
In the enterprise world there is more to consider than your own pages. Until about a month ago my company was still using IE 6. Locally I support only about 700 users and for our image I had IE 7 loaded as I was able to get it to work with everything except for about 6 users whom had a customer application that linked to IE 6 and wouldn't talk to IE 7. I had another couple of users who connect to a customer site that the thing checks explicitly for IE 6. MS had a tool that would report the useragent of 7 as 6 and it worked fine but when the company pushed out 8 it did not have the ability to report as 6 so I had to backlevel them again to 7. We've asked them to simply bump the check to at least 7 which we knew perfectly well works but they won't do it. John Waliki already stated how much work it is for an enterprise to do it with their own sites but it becomes exponentially harder when working with another company that couldn't care less about such upgrades.
It's not about it taking more than 6 weeks to certify a browser. It's about how many times you have to do that.
Mike - I've been wearing the same corporate hat all day and beating my head on the desk. For most corporations, the technology is a tool for accomplishing their core competency and the business drives the technology. Being faced with deciding which is more important: security updates or the critical production web application needed to manufacture your product is not a happy place to be. A more stable release is needed when you are looking at large corporations with millions of pages of web content (sites, applications, etc.)
Great article. I agree that branching for profit would be *great* way to go. I just hope that Mozilla decides to allow those patches on their site - perhaps two weeks behind "premium" customers that pay for immediate security patch release. That company would have at least 1,000,000 customers in corporate and government. If they charged a mere $10/person per year (*not* per installation!!) then they would entice **many, many** home users to pony up. Why doesn't Mozilla opt for this model? FF3 would be patched for 3 years, FF4 for 3, etc. Patch for pay... has a nice ring to it!
Simple: You don't keep up to date, you're insecure. Mozilla is at least open with bugs and security fixes: all the Linux distributions and so on also put out CVEs. If you're basing your internal corporate web infrastructure on one version of a web browser - you're possibly doing it wrongly.
Mozilla are moving to a more rapid release cycle - and fixing bugs in the process. If there's a critical bug, it'll be fixed with a point release.
You are a lot less likely to get your infrastructure owned by a third party if you are constantly up to date and patched. Notably in this respect, Microsoft don't release details of their patches particularly well, don't advertise widely what precise security vulnerabilites are addressed by each patch and how long they've been open/vulnerable for and tend to wait and roll up patches to a massive patch cycle once in a while.
Software deployment of a Web browser really isn't hard even to 500,000 users - notably, Mozilla has "check for updates and update me" built in (as do Opera/Chrome).
Rapid deployment does, however, make it significantly harder for people like the Linux distributions to keep up - though I note with interest that Debian has already released Iceweasel 5.0 ...
[Flameproof underwear]
If your enterprise really can't patch: firewall yourself off from the rest of the world. If you can't firewall yourself off from the rest of the world, fire your current testers and system administrators and get folk who know what to do, Not patching for significant periods of time in the name of extended testing of functionality is no longer acceptable.
If you can't afford to do this and require "five nines" reliability, be prepared to keep an in-house security team to scrutinise source, build on separate infrastructure, penetration test/"red team" and deploy. In the meantime, if your configuration lockdown isn't sufficient, your users will have found a way to sneak in the outside world's more up to date version.
[/Flameproof underwear]
No enterprise needs to accept Firefox as a browser in their organisation. Or in their customer base. If Firefox wants commercial users, it should court them. Apparently it doesn't want them and won't court them.
If it DID, then since evidently Firefox 5 is only Firefox 4.0.2 with a misplaced number increment, they could release "Firefox 4.0.2 LTS" for corporate customers, exactly the saMe software, different number.
But it doesn't.
I suppose anyone else could just take the code base and change the number.
The issue isn't one of not wanting to patch your web browser. The issue is one of having no choice but to accept every major feature change, as a side-effect of trying to keep up-to-date on security maintenance.
Take an example from the other side: Microsoft Internet Explorer 6 on Windows XP is STILL receiving security patches, alongside separate releases of newer versions such as IE7, IE8, and now IE9. So, if an enterprise has some in-house app which requires them to stay with IE6 due to dependence on certain quirks of IE6's rendering system, they have that option, while still keeping their browser patched against the latest security vulnerabilities as they are discovered. (At least, until the end of Windows XP's extended support life cycle in 2014. IE8 and IE9 are both guaranteed to continue receiving security patches until at least 2020.)
In Mozilla's own history, they continued issuing security patches for version N (and sometimes even N-1) for up to a year after the release of version N+1. This is not true anymore. There are vulnerabilities in Firefox 4, 5, and 6, which will never receive patches by any means other than upgrading to Firefox 7, along with all the in-house re-validation and testing which would have to go into such a migration.
By the time the in-house testing and validation on Firefox 7 is complete, Firefox 8 will probably have been released, leaving behind some permanently un-patched vulnerabilities in Firefox 7, requiring the whole process of testing and validating Firefox 8 to start all over again in a vicious cycle that never allows time to actually deploy ANYTHING.
As far as compatability of plugins between releases, Firefox should guarantee compatability of plugins and addons between major releases at the binary and API level. That should address concerns about doing major release upgrades. Firefox should also provide a test suite and gaurantee that those who code to the spec tested by the test suite can be gauranteed compatability between all releases.
As far as security updates for old versions, the release upgrades themselves are security updates. Manpower may be limited with this project and as above with Firefox gauranteeing plugin compatability, it is difficult to justify continueing to maintain older releases when a security update can be gotten by upgrading to a new release. Upgrading process is also not very large or significant in reality.
I think a lot of people, including the author, is missing the point.
It is not about "enterprise deployments". It is also about common users.
A three-month release cycle is a joke, no matter how big resources you have. You ship features, buy you also ship bugs and serious regressions. Because no matter how well you test, three months simply is *not* enough for any kind of serious QA. No security updates? You must be kidding me.
I am simply puzzled by all this. And it makes me sad because Firefox is still one of the flagship open source projects. This just reinforces the perception of haphazardness of open source projects. Release engineering? WTF? Engineering? What is that?
You make us all look bad.
He who ever thinks that it would be enough to focus on the consumer does not realize that the average user will take the SAME software that he is used from his OFFICE, not the other way round. Crazy strategy.
And who can benefit from new features every now and then? Web technologies cannot be used until most browsers support them, and the firefox-specific features are not really important.
Mozilla, please go back to the old versioning, including old version numbers. They were clear (new features or bugfixes) and helpful.
An an IT professional involved in software support, it's an easy call to predict the end of FF at many shops. There's no way most corporations are going to stick with it now that it's based on this alien update concept of RR plus early EOL. We require stability in our desktops, not some sort of goofball upgrade race.
This whole thing makes me really sad. Goodbye, FF.
I agree with foobar123 there will be lots of common users not happy with this as well, not all of us are happy to just update apps to new major versions and roll along with the new bugs that come our way, FF4 introduced some major gui changes, FF5 also has changes I have noticed, more than what a 4.0.2 would have had. A rapid release cycle in itself isnt a bad thing, but the quick EOL is the killer, they should be supporting major releas e branches at once, eg. new version every 3 months then support 4 versions at once for security giving a year for updating. Currently FreeBSD support both 7.x and 8.x. I am trying to understand the logic of whats going on and the only thing that comes to mind is its a knee jerk reaciton to what google have been doing. Guess what mozilla, some of us dont like chrome and rapid development.
Well that as they say is that for Firefox. I am going to look for an alternative and Opera(shudder), Safari and SeaMonkey look good. Although Sea Monkey is what I am using now. Works great. Better than the Mozilla suite used to work. My biggest Complaint was the Addons for Firefox Breaking and with SeaMonkey, I don't have to worry about that as much, esp since it's not doing RR BS. It's and OS Application, not a Web Application. Imagine if MS did this, OMG, what and Insanity that would create.
I just wanted to point out. These 2 qoutes from Asa before this RR started.
"I think we're going to see some serious bottom up pressure in the enterprise space... We'll even get some of the big boys in the Fortune 500... It's going to be a great year for Firefox, and it won't be limited to the home user."
"Even more exciting is how much progress Firefox is making [in enterprise usage] and how quickly. At nearly 20% and growing almost half a point per month, I don't think anyone can discount Firefox's potential to succeed in the enterprise as well as it is with consumers. Go Firefox!"
And these 2 after.
"Enterprise has never been (and I'll argue, shouldn't be) a focus of ours."
"Yes, I’m basically saying that I don’t care about making Firefox enterprise friendly."
Nothing like extremes. From 2 years for an updated product to now every 6 weeks. Asa Dotzler is the problem and the guy just is not stable himself, therefore neither is firefox going to be stable. Rapid release is the opposite of give the software time to be stable. I believe the guy just does not understand software complexity. Nothing mission critical in this world runs on microsoft code nor google code. They are in the entertainment business and they play with software, change it for no reason, and annoy millions of people because of it every day and Dotzler wants to do the same.
Rapid release has been very badly done. Users are very fed up with it, me included, and we don't see the point. Half of my add-ons are reported as incompatible at every release, and I really do not want to have to go and fix them every time, FF should be all about simplicity. Firefox is losing ground not because Chrome updates whenever there is a "day" in the day but because they are ignoring users.
I have used FF ever since it was born but now I'm off to try Chrome, as if Google didn't already own enough of me.
I have a great idea, how about instead of adding "new features", Mozilla concetrates on making a program that opens quickly, doesn't use much memory and allows the user to view web pages?
Developers and heavy-users on one side, wanting the newest cutting-edge web features on their desktop on one side; and on the other side, corporate, clueless users, or simply those wanting a more stable experience and annoyed by update dialogs popping up every couple weeks (disclaimer: like me).
This remember me of the old days of Mozilla Netscape/Phoenix (latter Firebird, latter yet Firefox) fork.
"Those who cannot remember the past are condemned to repeat it."
What else to say?
When Firefox stopped supporting 3.6.xxxx, they lost me. I'm glad, because after a lot of research, I found that Firefox isn't as good as everybody thinks it is. I'm going to try Opera and see how it goes.