This newsgroup discussion inspired me to talk about Firefox 5.0.1 and 3.6.19, the situation / trade-offs encountered, and our ultimate decision process.
First, some background. We test pre-release versions of Apple’s latest OS, Mac OS X Lion (10.7). Our users also test on their own and we see their crash reports in our crash reporting system. Apple seems to have introduced a bug in their ATS framework in a previous Lion seed. The bug makes Firefox (all versions) extremely unstable. We reported it to Apple but there are only so many bugs that can be fixed and unfortunately this one didn’t make the cut for their GM seed. Because Apple had announced “July” as the release of Lion we knew we couldn’t just wait for Firefox 6’s release (August 16th) to fix it. At the same time, it came to our attention that the latest Java update released by Apple on Mac OS X Snow Leopard (10.6) broke Java for Firefox 5 users. Clearly we had to create out-of-band updates to address both issues.
Immediately, we questioned what platforms we should ship the updates to. It seemed odd to release an updated Firefox to all users on all platforms when the bugs only affect the Mac population.
There are some distinct downsides to releasing across all platforms in this situation:
- We know update fatigue is very real…last we checked over 7% of users don’t update specifically because they are annoyed about too many update offers (see Gerv’s post as well). To subject Windows and Linux users to the heavyweight update process, potentially annoying them enough to not update to Firefox 6 in August is a big concern
- The updates aren’t trivially small. Profile-guided optimization on Windows makes it so the binaries aren’t exactly the same for a rebuild. Additionally, the code fixes aren’t conditionally compiled so the resulting binaries would be even more different. Though binary patch / delta updates reduce the impact, we still don’t feel comfortable having our entire Windows and Linux populations download data that did nothing for them
- More QA / qualification is required and there is more opportunity for issues to crop up (such as automation failures, packaging errors, etc)
Of course, there are some downsides to releasing only for Mac users in this situation:
- User confusion. Windows and Linux users may see the update announcements and think their version is insecure / there is something wrong with automatic updates
- Supporting tools may expect all platforms to have the same supported version
- Our update automation only supports patching to the last update. When we go to release Firefox 6 and Firefox 3.6.20 all non-Mac users will get the full update rather than a binary patch update (unless we try to do manual hackery)
- The Mozilla.com website only supports specifying a “latest version”. It is not broken down by platform. We have to offer all platforms for download on Mozilla.com or only offer the older version, exposing new Mac users to the two fixed problems until they update (which may not happen if they keep crashing in the middle of the download)
- Due to the way our update server works, we have to pop up a window offering Firefox 5 to the freshly updated Firefox 3.6.19 Mac users or make it so “Check for updates” on 3.6.19 doesn’t offer Firefox 5–both options not ideal
This is why release management is hard. Rarely is there a good option and often it takes too long to get the proper data needed to make an informed decision. Just gathering the above information from various stakeholders took a fair amount of time. To make matters more frantic, we needed to get the updates out as soon as possible to release before Lion’s ship date (which is still unknown but thought to be soon).
I looped in product management, as they are the ultimate product “owners” and know how to reason about the assorted tradeoffs and their impact on both our users and Firefox in general. After conferring with product management we decided the downsides of releasing for all platforms outweighed the ones for releasing just for Mac.
We verified with supporting system owners that releasing only for Mac would not cause issues (crossing off #2 above), chose to offer all platforms on the website (largely mitigating #4), and made a call to turn off manual update offers to Firefox 5 for 3.6.19 users (#5 above). We also have time to figure out how to manually hack around patch packages to mitigate #3 but made it clear we are willing to live with the current situation when Firefox 6 / Firefox 3.6.20 comes along.
This decision and process could definitely have been messaged more clearly. We were moving quickly and I was at a team offsite that stole more attention than anticipated. I’m happy with the decision we took but realize there was no “correct” decision in this situation. Both ways had tradeoffs and if we chose releasing to all platforms I imagine I would be writing a similar post explaining why.