Moving on from Mozilla
Today is my last day as a Mozilla employee, though I will always stay a Mozillian. Mozilla is the premier open source project and touching a small part of it has been both inspiring and humbling. Meeting all the smart and talented people I "knew" from reading Planet still blows my mind.
Leaving is very bittersweet for me. We accomplished so much in 2011 and 2012 looks to have even more on tap. Hunkering down and shipping Firefox 4, going through a transition like the new release process, shipping a mobile product, and questioning years of assumptions while figuring it out as we went along was exhilarating. No one (including many Mozillians) thought we could ship Firefox 5 a quarter after Firefox 4. Well guess what, we did. And then we shipped Firefox 6, 7, 8, and 9--all on time. Sure, there were hiccups. Transitions are hard. We laid the foundation for the future and 2012 is the year we take it to the next level.
During my interview, Mike Beltzner (@beltzner) asked me if I identified as a community member. I told him that I lurked around on Planet and Bugzilla, so not really. He looked me in the eye and told me he viewed lurkers as a valuable part of the community, making me feel like a member of the Mozilla family from day one. If you are a lurker, please take note. Of course, now that I know where I can effectively contribute I don't intend to stop...so look for me in bugs and participating in email threads about the correct shade of orange for the Firefox button (kidding!).
I want to publicly thank Johnathan Nightingale (@johnath), Senior Director of Firefox Engineering, for his guidance and advice. He has been both an amazing manager and mentor from day one. His analytical ability, managerial prowess, and technical chops have been inspiring and I hope you get a chance to experience them as I have.
Any release management work / questions should be directed towards Alex Keybl (@alexkeybl). Alex joined us this year from Apple (where he drove features for Mac OS X Lion) and has already made a large impact at Mozilla. I'm sorry that I am leaving but I know Firefox releases will go smoothly in his capable hands.
They say you are supposed to end posts like this with a quote for added effect. I think this quote accurately captures my feelings about moving on from Mozilla:
I'm so excited. I'm so excited. I'm so...scared. - Jessie Spano
Keep up the amazing work and thank you for the past couple of years.
Christian Legnitto - soon to be ex-Firefox Release Manager
Reminder: Firefox 3.6.23 → 7.0.1 advertised update scheduled for Thursday
Just a friendly reminder that we are planning to offer the Firefox 3.6.23 → 7.0.1 advertised update this Thursday, 2011-10-6.
What does the advertised update mean?
- We will be offering an advertised update (also sometimes called a "major update") for Firefox 3.6.x -> Firefox 7.0 on 2011-10-06 (1 week and 2 days after Firefox 7's release)
- The advertised update has no bearing on support levels. It does not mean Firefox 3.6 is end-of-life
- The advertised update has no bearing on the Firefox ESR proposal
- This is part of the old release process and is unchanged from what we have been doing for the past 3+ years
- The prompt is opt-in...if users decline, they stay on 3.6.x and get security updates until the next time we decide to prompt
- We did this for 3.0.x -> 3.5, 3.5.x -> 3.6, 3.6.x -> 4.0 and 3.6.x -> 5.0...nothing new
- If enterprise deployments have advertised/"major" updates disabled in rollouts their users will not see the prompt but they will see Firefox 3.6.24 when it is released on 2011-11-08
This is mainly to move regular Firefox users who don't know there is an update up to a newer/better version if they are so inclined.
Based on previous advertised update offers we expect to see a significant percentage of users installing the new version. We are, of course, watching the data closely to see what happens and to make sure there are no unexpected issues.
As always, feel free to contact me with any questions or concerns.
Why are there no Firefox 5.0.1 and 3.6.19 automatic updates for Windows and Linux?
This newsgroup discussion inspired me to talk about Firefox 5.0.1 and 3.6.19, the situation / trade-offs encountered, and our ultimate decision process.
First, some background. We test pre-release versions of Apple's latest OS, Mac OS X Lion (10.7). Our users also test on their own and we see their crash reports in our crash reporting system. Apple seems to have introduced a bug in their ATS framework in a previous Lion seed. The bug makes Firefox (all versions) extremely unstable. We reported it to Apple but there are only so many bugs that can be fixed and unfortunately this one didn't make the cut for their GM seed. Because Apple had announced "July" as the release of Lion we knew we couldn't just wait for Firefox 6's release (August 16th) to fix it. At the same time, it came to our attention that the latest Java update released by Apple on Mac OS X Snow Leopard (10.6) broke Java for Firefox 5 users. Clearly we had to create out-of-band updates to address both issues.
Immediately, we questioned what platforms we should ship the updates to. It seemed odd to release an updated Firefox to all users on all platforms when the bugs only affect the Mac population.
There are some distinct downsides to releasing across all platforms in this situation:
- We know update fatigue is very real...last we checked over 7% of users don't update specifically because they are annoyed about too many update offers (see Gerv's post as well). To subject Windows and Linux users to the heavyweight update process, potentially annoying them enough to not update to Firefox 6 in August is a big concern
- The updates aren't trivially small. Profile-guided optimization on Windows makes it so the binaries aren't exactly the same for a rebuild. Additionally, the code fixes aren't conditionally compiled so the resulting binaries would be even more different. Though binary patch / delta updates reduce the impact, we still don't feel comfortable having our entire Windows and Linux populations download data that did nothing for them
- More QA / qualification is required and there is more opportunity for issues to crop up (such as automation failures, packaging errors, etc)
Of course, there are some downsides to releasing only for Mac users in this situation:
- User confusion. Windows and Linux users may see the update announcements and think their version is insecure / there is something wrong with automatic updates
- Supporting tools may expect all platforms to have the same supported version
- Our update automation only supports patching to the last update. When we go to release Firefox 6 and Firefox 3.6.20 all non-Mac users will get the full update rather than a binary patch update (unless we try to do manual hackery)
- The Mozilla.com website only supports specifying a "latest version". It is not broken down by platform. We have to offer all platforms for download on Mozilla.com or only offer the older version, exposing new Mac users to the two fixed problems until they update (which may not happen if they keep crashing in the middle of the download)
- Due to the way our update server works, we have to pop up a window offering Firefox 5 to the freshly updated Firefox 3.6.19 Mac users or make it so "Check for updates" on 3.6.19 doesn't offer Firefox 5--both options not ideal
This is why release management is hard. Rarely is there a good option and often it takes too long to get the proper data needed to make an informed decision. Just gathering the above information from various stakeholders took a fair amount of time. To make matters more frantic, we needed to get the updates out as soon as possible to release before Lion's ship date (which is still unknown but thought to be soon).
I looped in product management, as they are the ultimate product "owners" and know how to reason about the assorted tradeoffs and their impact on both our users and Firefox in general. After conferring with product management we decided the downsides of releasing for all platforms outweighed the ones for releasing just for Mac.
We verified with supporting system owners that releasing only for Mac would not cause issues (crossing off #2 above), chose to offer all platforms on the website (largely mitigating #4), and made a call to turn off manual update offers to Firefox 5 for 3.6.19 users (#5 above). We also have time to figure out how to manually hack around patch packages to mitigate #3 but made it clear we are willing to live with the current situation when Firefox 6 / Firefox 3.6.20 comes along.
This decision and process could definitely have been messaged more clearly. We were moving quickly and I was at a team offsite that stole more attention than anticipated. I'm happy with the decision we took but realize there was no "correct" decision in this situation. Both ways had tradeoffs and if we chose releasing to all platforms I imagine I would be writing a similar post explaining why.
Important dates for the Firefox rapid release process
This has come up on assorted mailing lists so I figured I'd blog about it...
Please do note the rapid release process is still new and might need to be adjusted as we go through a couple.
If you are.....
A Developer
- You need to focus on the fact that the next merge from mozilla-central to mozilla-aurora is 2011-05-24 (any code for Firefox 6 needs to be in by that date)
- You need to know any major issues you find in code landed before 2011-04-12 should be brought to the attention of release drivers to put into beta or aurora, wherever is appropriate
A Localizer
- You need to focus on the fact that you only have until 2011-05-17 to finish localizing Firefox 5 content in mozilla-aurora (time left for current work)
- You need to focus on the fact that the next merge from mozilla-central to mozilla-aurora is 2011-05-24 (time the work for the next cycle starts so you can plan)
Add-on Developer
- You only have until 2011-05-17 to test your add-on with Firefox 5 content in mozilla-aurora and mark your add-on as 5.0* compatible.
You will be able to test for 6 more weeks on beta but our beta audience has over 2 million people...you probably don't want to field their support questions and should just focus on getting it compatible during the aurora timeframe.
- You should start testing your add-on with Firefox 6 content in mozilla-aurora on 2011-05-24. You will have 6 weeks to test on aurora and another 6 weeks to test on beta
Anyone Else
- Firefox 5 will be released on 2011-06-21
I hear there is a developer calendar somewhere and intend to add the assorted merge dates. I have also been saying it every week in the platform, MoCo, channel, and product meetings. We're working on getting it added to the wiki in the appropriate places (thanks to Matt Brubeck for adding it on the RapidRelease wiki page).
Additionally, there are dates in the specifics document as well as a quick python program to spit out your very own merge schedule. Please note we are still working out the mechanics of merges so they may be +- a day or so.
Come join the new Firefox release team at Mozilla!
Here at Mozilla we release a fair amount of software, most recently Firefox 4:
Photo by Bob Moss
With the new Firefox rapid release process managing the details of those releases is even more critical as we absolutely cannot slip the schedule. Because the schedules are set in stone we need to manage a lot more up front. Additionally, with releases shipping on a faster cadence there is less time for mistakes to be caught. When one adds in new platforms (smartphones, tablets), new products (Firefox Mobile, Firefox Home), and new channels (Nightly, Aurora) there is clearly a growing need for project and release management.
To meet this growing need Mozilla is hiring for a new release management team!
Is this you?
We currently have two positions available:
Associate Engineering Project Manager - Releases: We're looking for someone at the beginning of their career who can learn the ropes and grow to be an experienced member of the release management team. Problem solving, great communication skills, and technical chops are a must!
Engineering Project Manager - Releases: We're looking for someone with tons of experience managing software releases, hopefully at internet scale. This person will hit the ground running, taking the lead on many releases as well as driving process improvements.
What is Release Management?
Wikipedia has a somewhat decent description of what release management actually is. The main takeaway is the fact that it is a project management function, not development (writing the code for the bits that are eventually shipped), QA (testing the bits that are eventually shipped), or Release Engineering (managing the systems that build, package, and deliver the bits). All these groups have differing priorities and worldviews yet need to interact on the same project--that's where release management comes in.
Don't worry, release management is not just playing with schedules, sending emails, and organizing meetings. There is ample opportunity to create tools (examples aren't hard to find) , help define the direction of the product (by driving features or hashing out new processes), and make decisions that directly impact hundreds of millions of people worldwide.
Darn, I don't live in the San Francisco Bay Area
While our preference would be to have the employees work out of our Mountain View, CA office, it is not explicitly required. You don't have to live in the San Francisco Bay Area, California, or even the United States! We have offices all over and for the right candidate / timezone we might even consider 100% remote (though your effectiveness will likely be diminished as a large part of the job is interaction with other people).
If you do want to move to the San Francisco Bay Area, we can investigate how to get you here.
Sweet, sign me up!
Great! We want to fill these positions quickly, so get your resume in order and apply via the jobs website:
Version numbers on mozilla-central changing next week
This is a heads up that mozilla-central will be bumped to the new versioning scheme early next week PDT. It will either be Monday (and bump to 5.0a1) or Tuesday (and bump to 6.0a1). Please see bug 648653 and the development specifics document for details.
Please comment in bug 648653 if there are urgent items that need to be taken care of before this happens or if there is a concrete reason Monday or Tuesday appears to work better.
Code freeze for Firefox 3.6.14 and 3.5.17 non-blocking bugs is TONIGHT
As posted in the schedule and discussed in various meetings, code freeze for Firefox 3.6.14 and 3.5.17 non-blocking bugs is tonight.
This freeze is for non-blocking bugs. Blocking bugs have an additional week.
If you have a bug that needs to land and don't have commit privs, please hop in #developers on irc.mozilla.org and ask someone else to push for you.
After today non-blocking patches will get their approvals recinded and will need to be renominated for a later release.
If there are any questions or concerns, please feel free to contact me in irc (LegNeato) or send an email (clegnitto@mozilla.com).
Looking back at 2010, forward at 2011
2010 was an interesting year for me professionally. Inspired by similar lists online, I present what I did (or can remember at least):
- Released Mac OS X 10.6.2 and a bunch of other Apple-related stuff
- Left Apple and came to Mozilla in March
- Shipped out-of-process plugins in Firefox 3.6.4
- Got the idea for Pulse, started a prototype in my spare time
- Gave a talk about 3.6.4 and Pulse at the Mozilla summit
- Released Firefox 3.0.19
- Released Firefox 3.5.9 through 3.5.16
- Released Firefox 3.6.3 through 3.6.13
- Became a Bugzilla contributor and committer
- Released a Bugzilla extension for integration with a message broker (bugzilla-push)
- Created a GitHub service hook to integrate GitHub with a message broker like Pulse
- Released Firefox 4 beta 6 through beta 8
- Obtained commit access to Mozilla's repositories, though I haven't pushed anything yet
- Moved Pulse out of a prototype and into a Mozilla quarterly goal (which I believe was met)
- Took over Jeff's bztools project
- Wrote a MediaWiki extension to integrate wiki.mozilla.org with Pulse (to be released soon)
I'm pretty happy with what I have done so far and am looking forward to doing even more.
New technologies
It was also fun to play around some hot new technologies (and some old but new to me). These are new the ones I became familiar with in 2010:
- RabbitMQ (and by by proxy a little bit of Erlang)
- AMQP and STOMP
- Node.js and Socket.io
- git and GitHub
- CouchDB and Riak
- Mercurial
- orbited
- WebSockets
2011 Goals and Plans
2011 is shaping up to be awesome! This is the general plan for the year I am working off at the moment:
- I intend for the mechanics of a Firefox release to become 100% automated. This is critical and must happen for 2011. There are too many competent people babysitting scripts and communicating via email and IRC during releases. Luckily, Pulse was brought up in 2010 and will help enable this
- In general I intend to make Pulse a cornerstone of Mozilla's tools and processes, eliminating polling across the organization and adding information transparency. AMQP / enterprise messaging is the future for Mozilla's tools. The first higher-level tool I intend to convert once all the current instrumentation is rolled out is TinderBoxPushLog
- I will bring up a release management system like I had at Apple. It will become the single source of truth for releases and will be integrated with the build system via Pulse
- I want to standardize release and project management processes across all client software that Mozilla ships. This will be critical in 2011 as I am sure there will be many simultaneous releases
- I will fix this Bugzilla bug, which should help Mozilla's Bugzilla workflow immensely
- I will fix this Bugzilla bug, which should help anyone who gets bugmail (that's everyone, right?)
- I will fix this Bugzilla bug, so that Ehsan can stop including the functionality in his Firefox extension and I can fix a bug from 1999 (bragging rights, whoo!)
- I will release my finished MediaWiki extension that ties MediaWiki to a message broker. It's done, it just needs to be polished before I can attach my name to it and release it on GitHub
- I will publish a video walkthrough detailing how I think Mozilla should structure their repos and approach project and risk management in an accelerated release world
- Hopefully my proposal will be accepted and implemented. If not, it was an extremely useful mental exercise and will guide some small improvements in the future
- I plan to quickly tackle managing the quarterly Firefox releases and will try to make them as smooth as possible
- I'll release my top secret Mozilla-related project I've been working on in my "spare time" for the last couple of months. Probably not until the end of the year though
- I will take ownership of extension/plugin blacklisting if no one wants it. I've been resisting doing so in 2010, but I think it needs an owner and the likely owners are swamped. It can also benefit from some new processes and clear operating guidelines
- I really want to blog more about my experiences at Apple, comparing and contrasting with Mozilla. I feel I have a unique position to talk about the release technology and project management sides but haven't found time in 2010 to do so. Hopefully 2011 will be different and I'll blog before I forget everything
All three lists have a lot of Pulse/Pulse-related bits in them. This isn't by chance. Everything I set out to improve at Mozilla needed a system like Pulse and I needed that foundation before I could build higher-level tools and processes. Now that it is in place and people are starting to "get the religion", I hope I'm going to see a rapid increase in awesomeness. Special thanks to David Dahl and Margaret Leibovic for smiling, nodding, and feigning excitement when I excitedly showed them Pulse messages scrolling by in a terminal every five minutes for the past year. It has to be annoying sitting by me, but they both seemed to take it in stride...or perhaps it was the booze at their desks.
Code freeze for Firefox 3.6.13 and 3.5.16 is TOMORROW
Just a friendly reminder that code freeze for Firefox 3.6.13 and 3.5.16 is tomorrow (Thursday, 2010-11-18) @ 11:59 pm PST. Here's where we currently stand:
Blocker report for Firefox 3.5.16 and 3.6.13 (as of 2010-11-17 16:32:53-08:00)
Overview
3.5.16
- blocking: 32
- fixed: 17
- open: 15
3.6.13
- blocking: 37
- fixed: 22
- open: 15
Open Blocker Assignees
Blake Kaplan: 3
Robert Sayre: 3
Mats Palmgren: 2
Josh Aas: 2
Steven Michaud: 2
Johnny Stenback: 1
Ms2ger: 1
Jason Orendorff: 1
Nicholas Nethercote [:njn]: 1
Igor Bukanov: 1
Tons of Mozilla/Firefox releases today
Crazy release day today!
For those following the Firefox 4 betas we released the fifth one today. Check out the post for details but there is some hot graphics acceleration action on Windows and an audio API that enables awesomeness.
For those not willing to run pre-release software, we released Firefox 3.6.9 today for users on the stable branch. As always there are great crash fixes and security improvements, but we also snuck in a little hardening. Firefox 3.6.9 adds support for the X-FRAME-OPTIONS HTTP header. Check out Michael's blog post about it for details. Big hat tip to Brandon Sterne for working on it and making people more secure. Taking an aggressive hardening approach on stable branches is happening more and more (like enabling the last bits of ASLR in Firefox 3.6.7). I like it!
Because good things generally come in threes, we also released Firefox 3.5.12. Again, more security and stability fixes are always a welcome sight. At this point you should probably be running Firefox 3.6.9 though, as it is way better...trust me!
With this many releases, RelEng and QA had to work overtime. They did amazing work! Thanks to everyone, and thanks especially to Anthony Hughes...he's an update testing machine.
Did I say good things in threes? There was actually even more release shenanigans today. Both Thunderbird and SeaMonkey had releases today as well, though I was less involved and can't speak to all the goodness they bring.