Moving on from Mozilla
Today is my last day as a Mozilla employee, though I will always stay a Mozillian. Mozilla is the premier open source project and touching a small part of it has been both inspiring and humbling. Meeting all the smart and talented people I "knew" from reading Planet still blows my mind.
Leaving is very bittersweet for me. We accomplished so much in 2011 and 2012 looks to have even more on tap. Hunkering down and shipping Firefox 4, going through a transition like the new release process, shipping a mobile product, and questioning years of assumptions while figuring it out as we went along was exhilarating. No one (including many Mozillians) thought we could ship Firefox 5 a quarter after Firefox 4. Well guess what, we did. And then we shipped Firefox 6, 7, 8, and 9--all on time. Sure, there were hiccups. Transitions are hard. We laid the foundation for the future and 2012 is the year we take it to the next level.
During my interview, Mike Beltzner (@beltzner) asked me if I identified as a community member. I told him that I lurked around on Planet and Bugzilla, so not really. He looked me in the eye and told me he viewed lurkers as a valuable part of the community, making me feel like a member of the Mozilla family from day one. If you are a lurker, please take note. Of course, now that I know where I can effectively contribute I don't intend to stop...so look for me in bugs and participating in email threads about the correct shade of orange for the Firefox button (kidding!).
I want to publicly thank Johnathan Nightingale (@johnath), Senior Director of Firefox Engineering, for his guidance and advice. He has been both an amazing manager and mentor from day one. His analytical ability, managerial prowess, and technical chops have been inspiring and I hope you get a chance to experience them as I have.
Any release management work / questions should be directed towards Alex Keybl (@alexkeybl). Alex joined us this year from Apple (where he drove features for Mac OS X Lion) and has already made a large impact at Mozilla. I'm sorry that I am leaving but I know Firefox releases will go smoothly in his capable hands.
They say you are supposed to end posts like this with a quote for added effect. I think this quote accurately captures my feelings about moving on from Mozilla:
I'm so excited. I'm so excited. I'm so...scared. - Jessie Spano
Keep up the amazing work and thank you for the past couple of years.
Christian Legnitto - soon to be ex-Firefox Release Manager
Why are there no Firefox 5.0.1 and 3.6.19 automatic updates for Windows and Linux?
This newsgroup discussion inspired me to talk about Firefox 5.0.1 and 3.6.19, the situation / trade-offs encountered, and our ultimate decision process.
First, some background. We test pre-release versions of Apple's latest OS, Mac OS X Lion (10.7). Our users also test on their own and we see their crash reports in our crash reporting system. Apple seems to have introduced a bug in their ATS framework in a previous Lion seed. The bug makes Firefox (all versions) extremely unstable. We reported it to Apple but there are only so many bugs that can be fixed and unfortunately this one didn't make the cut for their GM seed. Because Apple had announced "July" as the release of Lion we knew we couldn't just wait for Firefox 6's release (August 16th) to fix it. At the same time, it came to our attention that the latest Java update released by Apple on Mac OS X Snow Leopard (10.6) broke Java for Firefox 5 users. Clearly we had to create out-of-band updates to address both issues.
Immediately, we questioned what platforms we should ship the updates to. It seemed odd to release an updated Firefox to all users on all platforms when the bugs only affect the Mac population.
There are some distinct downsides to releasing across all platforms in this situation:
- We know update fatigue is very real...last we checked over 7% of users don't update specifically because they are annoyed about too many update offers (see Gerv's post as well). To subject Windows and Linux users to the heavyweight update process, potentially annoying them enough to not update to Firefox 6 in August is a big concern
- The updates aren't trivially small. Profile-guided optimization on Windows makes it so the binaries aren't exactly the same for a rebuild. Additionally, the code fixes aren't conditionally compiled so the resulting binaries would be even more different. Though binary patch / delta updates reduce the impact, we still don't feel comfortable having our entire Windows and Linux populations download data that did nothing for them
- More QA / qualification is required and there is more opportunity for issues to crop up (such as automation failures, packaging errors, etc)
Of course, there are some downsides to releasing only for Mac users in this situation:
- User confusion. Windows and Linux users may see the update announcements and think their version is insecure / there is something wrong with automatic updates
- Supporting tools may expect all platforms to have the same supported version
- Our update automation only supports patching to the last update. When we go to release Firefox 6 and Firefox 3.6.20 all non-Mac users will get the full update rather than a binary patch update (unless we try to do manual hackery)
- The Mozilla.com website only supports specifying a "latest version". It is not broken down by platform. We have to offer all platforms for download on Mozilla.com or only offer the older version, exposing new Mac users to the two fixed problems until they update (which may not happen if they keep crashing in the middle of the download)
- Due to the way our update server works, we have to pop up a window offering Firefox 5 to the freshly updated Firefox 3.6.19 Mac users or make it so "Check for updates" on 3.6.19 doesn't offer Firefox 5--both options not ideal
This is why release management is hard. Rarely is there a good option and often it takes too long to get the proper data needed to make an informed decision. Just gathering the above information from various stakeholders took a fair amount of time. To make matters more frantic, we needed to get the updates out as soon as possible to release before Lion's ship date (which is still unknown but thought to be soon).
I looped in product management, as they are the ultimate product "owners" and know how to reason about the assorted tradeoffs and their impact on both our users and Firefox in general. After conferring with product management we decided the downsides of releasing for all platforms outweighed the ones for releasing just for Mac.
We verified with supporting system owners that releasing only for Mac would not cause issues (crossing off #2 above), chose to offer all platforms on the website (largely mitigating #4), and made a call to turn off manual update offers to Firefox 5 for 3.6.19 users (#5 above). We also have time to figure out how to manually hack around patch packages to mitigate #3 but made it clear we are willing to live with the current situation when Firefox 6 / Firefox 3.6.20 comes along.
This decision and process could definitely have been messaged more clearly. We were moving quickly and I was at a team offsite that stole more attention than anticipated. I'm happy with the decision we took but realize there was no "correct" decision in this situation. Both ways had tradeoffs and if we chose releasing to all platforms I imagine I would be writing a similar post explaining why.
Come join the new Firefox release team at Mozilla!
Here at Mozilla we release a fair amount of software, most recently Firefox 4:
Photo by Bob Moss
With the new Firefox rapid release process managing the details of those releases is even more critical as we absolutely cannot slip the schedule. Because the schedules are set in stone we need to manage a lot more up front. Additionally, with releases shipping on a faster cadence there is less time for mistakes to be caught. When one adds in new platforms (smartphones, tablets), new products (Firefox Mobile, Firefox Home), and new channels (Nightly, Aurora) there is clearly a growing need for project and release management.
To meet this growing need Mozilla is hiring for a new release management team!
Is this you?
We currently have two positions available:
Associate Engineering Project Manager - Releases: We're looking for someone at the beginning of their career who can learn the ropes and grow to be an experienced member of the release management team. Problem solving, great communication skills, and technical chops are a must!
Engineering Project Manager - Releases: We're looking for someone with tons of experience managing software releases, hopefully at internet scale. This person will hit the ground running, taking the lead on many releases as well as driving process improvements.
What is Release Management?
Wikipedia has a somewhat decent description of what release management actually is. The main takeaway is the fact that it is a project management function, not development (writing the code for the bits that are eventually shipped), QA (testing the bits that are eventually shipped), or Release Engineering (managing the systems that build, package, and deliver the bits). All these groups have differing priorities and worldviews yet need to interact on the same project--that's where release management comes in.
Don't worry, release management is not just playing with schedules, sending emails, and organizing meetings. There is ample opportunity to create tools (examples aren't hard to find) , help define the direction of the product (by driving features or hashing out new processes), and make decisions that directly impact hundreds of millions of people worldwide.
Darn, I don't live in the San Francisco Bay Area
While our preference would be to have the employees work out of our Mountain View, CA office, it is not explicitly required. You don't have to live in the San Francisco Bay Area, California, or even the United States! We have offices all over and for the right candidate / timezone we might even consider 100% remote (though your effectiveness will likely be diminished as a large part of the job is interaction with other people).
If you do want to move to the San Francisco Bay Area, we can investigate how to get you here.
Sweet, sign me up!
Great! We want to fill these positions quickly, so get your resume in order and apply via the jobs website: