Status colors

First up, a minor yet helpful addition. When showing bug data in a table, the rows are now colored by their status. Spiffy.

Screenshot of colors

And the code:

A couple of notes:

• mediawiki-bugzilla adds CSS classes with the status values on the table automatically
• The CSS can be changed by the mediawiki administrator by editing the css directly (not ideal) or adding in another css file at [extension location]/web/css/custom.css
• The “lock” icons and such come from mediawiki. I’m working on removing them (by stopping medawiki from processing the extension’s ouput) but if you use a custom theme it shouldn’t be an issue
• For this example, remember that type=”table” is the default…that’s why it’s rendered as a table

Configurable fields/columns

Once people started using the extension the first question was “can I choose which fields to show and hide?” With this change you specify fields in the “include_fields” setting of BZ REST API options as you normally would. Mediawiki-bugzilla will then a) only fetch those fields and b) display those columns.

Custom fields selected and columns shown

And the code:

A couple of notes:

• Columns are displayed in the order they are listed in “include_fields”
• Not specifying “include_fields” gives the default fields
• Not all fields have custom templates yet. This means some fields (such as assigned_to) won’t show up properly. I’m going to be writing a bunch but feel free to submit a patch!
• Only “include_fields” is supported. mediawiki-bugzilla won’t do anything with “exclude_fields“
• “_default” isn’t supported as a field, so list the fields you want explicitly

Javascript tables

Large lists of bugs on a wiki page can get a bit unwieldy. It was requested to be able to sort by columns, search through the results, and paginate to keep vertical space to a minimum. I beefed up the support for client side jquery/Datatable rendering.

Administrators can turn on client-side jquery table support

And the code:

A couple of notes:

• This is currently enabled by the mediawiki admin / all or nothing. Future plans will make it configurable on a per-table basis
• Pagination is done client-side…it will not make the page load faster
• Yes, the default skin is ugly. I haven’t taken a crack at it yet

Pie chart support

You can currently make bar graphs by setting type=”count”, display=”bar”, and including Bugzilla REST API /count parameters. I actually had support for other chart types stubbed out and decided to implement pie charts.

Pie charts of three different sizes

And the code:

A couple of notes:

• You can specify a chart size by using size=”small/medium/large”. The default size is large. The size attribute doesn’t work for bar charts yet
• I want to embed the numeric data in an accessible format for our screen reader friends. Right now, it is just an image
• Yes, the default color scheme is ugly. I haven’t taken a crack at it yet

Unordered list support

We can make tables, we can make bar charts, and we can make pie charts…why not a simple list? While I was adding support for pie charts I added a simple unordered list display.

Bugs can be displayed in a list

And the code:

A couple of notes:

• The field/column support I talked about above works here too!
• Field data is separated by ” – “. I haven’t taken time to add niceties like “(none)” when there isn’t data
• The odd spacing in the screenshot is caused by mediawiki’s default css rules

Better error reporting

It’s not perfect, but I’ve put in a little more support for detecting errors. If the JSON is invalid or your “display” and “type” combination are invalid you will get a message saying so.

Invalid JSON input now shows an error

And the code:

A couple of notes:

• PHP doesn’t give me the exact JSON parse error so I can’t be more helpful. This example has a trailing ‘,’, which isn’t allowed in JSON
• Errors from the Buzilla REST API timing out or PHP using too much memory are not caught / reported

Also, when heard about the blog I couldn’t help but think of this amazing video that was making the rounds when we started the new release process:

http://www.youtube.com/watch?v=kq8s-Rubs5w

Happy Friday!

I am currently at Facebook and recently graduated bootcamp. I’m on the Release Engineering team with Chuck Rossi working on release related things. Right now I help out with the Facebook.com web releases but I’ll be moving into mobile (Android, iOS, mobile web, etc) shortly. It’s pretty cool to see how a company the size of Facebook deals with releasing how often they do, the scope of their frontend and backend systems, and the tools and processes they have in place. Once I start working on mobile, I am absolutely giddy I get to work with Mike Shaver again.

So what does this mean for Mozilla? A lot! I’ve been missing-in-action, waiting for legal approval to work on Mozilla stuff. Now that I have that approval, I should be around more and more. Here’s a list off the top of my head of what I intend to work on in my “spare” time:

1. Making it so source migration is as easy as pushing a button. I started this work before I left but based on the time it took Alex and I to struggle through the last migration it’s clear that it isn’t nearly good enough. I intend to expand my hacky merge tool to make it solid
2. I have Bugzilla improvements I have committed to. Some even have patches. I have promised to finish them and I will do so. I know the benefit to Mozillians is great but quite honestly it is hard to motivate myself to work on Bugzilla without feeling the benefit personally. Plus, I have no desire to subject myself to Perl anymore…and that’s coming from a guy who works in PHP now (ugh). That being said, I will finish what I started and help the Mozilla community in this way as much as I can
3. Usable, generic release management tools. Every company I know has tools to track what is/isn’t in a branch, overlay interesting events on the repo graph, perform cherry picks / transplant between release branches, etc. I intend to write some reusable components I can use at Facebook as well as Mozilla
4. Help with mediawiki-bugzilla. I started the project as a proof of concept and it’s been great to see Lawrence and Jake Maul get it actually used in production. The project is so useful and so simple I’ll probably hack on it in my spare time to let off some steam. Plus, Facebook has a lot of PHP so I guess I’ll feel a little bit more at home working on this codebase. Any web developer looking to contribute to Mozilla / open source should take a look at this project…high impact (a lot of open source projects use MediaWiki and Bugzilla), super simple code base, accessible language with no real external dependencies, and very little development red tape (just a pull request on GitHub!)
5. Channel triage. I actually enjoy triage and I want to take the load off of Alex and others so they can do “real” work during their day. I have some ideas for tools and processes to help out but I plan to triage at least 3 times a week or so…it helps me go to sleep when my mind is racing. First, I need to make sure I have all the proper Bugzilla access but then I will be acting as the Triage Fairy™ as much as possible
6. Mozilla Pulse. I fumbled the handoff (sorry guys!) but I intend to push Pulse forward as much as I can

I won’t be working on Mozilla stuff in any official Facebook capacity, but I’ll be around as much as I can in my personal time. I’m excited!

Leaving is very bittersweet for me. We accomplished so much in 2011 and 2012 looks to have even more on tap. Hunkering down and shipping Firefox 4, going through a transition like the new release process, shipping a mobile product, and questioning years of assumptions while figuring it out as we went along was exhilarating. No one (including many Mozillians) thought we could ship Firefox 5 a quarter after Firefox 4. Well guess what, we did. And then we shipped Firefox 6, 7, 8, and 9–all on time. Sure, there were hiccups. Transitions are hard. We laid the foundation for the future and 2012 is the year we take it to the next level.

During my interview, Mike Beltzner (@beltzner) asked me if I identified as a community member. I told him that I lurked around on Planet and Bugzilla, so not really. He looked me in the eye and told me he viewed lurkers as a valuable part of the community, making me feel like a member of the Mozilla family from day one. If you are a lurker, please take note. Of course, now that I know where I can effectively contribute I don’t intend to stop…so look for me in bugs and participating in email threads about the correct shade of orange for the Firefox button (kidding!).

I want to publicly thank Johnathan Nightingale (@johnath), Senior Director of Firefox Engineering, for his guidance and advice. He has been both an amazing manager and mentor from day one. His analytical ability, managerial prowess, and technical chops have been inspiring and I hope you get a chance to experience them as I have.

Any release management work / questions should be directed towards Alex Keybl (@alexkeybl). Alex joined us this year from Apple (where he drove features for Mac OS X Lion) and has already made a large impact at Mozilla. I’m sorry that I am leaving but I know Firefox releases will go smoothly in his capable hands.

They say you are supposed to end posts like this with a quote for added effect. I think this quote accurately captures my feelings about moving on from Mozilla:

I’m so excited. I’m so excited. I’m so…scared. – Jessie Spano

Keep up the amazing work and thank you for the past couple of years.

Christian Legnitto – soon to be ex-Firefox Release Manager

This has already become pretty valuable to me for looking at overall feedback, positive/negative for a particular version or query, etc.

It’s a bit rough / hacky, there may be bugs, but you can find it here:

http://hg.mozilla.org/users/clegnitto_mozilla.com/release_tools/file/default/scrape_input.py

The tool requires:

• Python (v2.6+, which is used on Mac 10.6+)
• The module “requests“. Install it with ‘sudo easy_install requests‘ or ‘sudo pip install requests‘
• The module “argparse” (should be in python 2.6+). Install it with ‘sudo easy_install argparse‘ or ‘sudo pip install argparse‘

The tool currently dumps the CSV to stdout, so you’ll most likely want to redirect output to a file.

Input now supports OR queries

When writing the tool I realized I needed to support searching for multiple terms. For example, when trying to get feedback on hangs just searching for “hang” wouldn’t give me the complete picture. I needed to search for something more like “hang OR freeze OR responding”. I supported this in my tool with queries for each search term and aggregating the results.  I quickly realized there were duplicates in the aggregated counts (input doesn’t give me unique ids to de-dupe) so I would only get an upper bound.

I hopped into #input on irc.mozilla.org and asked about OR queries. Dave Dash said it’d probably be easy to support, disappeared for a bit, and then BAM, input supported OR queries (using ‘|’ in the web UI, ‘,’ in my tool)! Thanks Dave!

Pivoting on Firefox version (–version)

Details:

• Show input types and ratios
• Allows you to see general “quality”…(praise/issues) is essentially a quality index
• You can restrict the analysis to a particular search if you want by using ‘–search’
• Beta and alpha versions are folded into the main version number

Example usage:

• python scrape_input.py –version 8.0
• python scrape_input.py –product mobile –version 8.0
• python scrape_input.py –version 7.0 –search memory
• python scrape_input.py –version 6.0 –search hang,freeze,responding

Pivoting on input type (–type)

Details:

• Track input over the life of a product
• Breaks down input by version and overall
• You can restrict the analysis to a particular search if you want by using ‘–search’
• Beta and alpha versions are folded into the main version number

Example usage:

• python scrape_input.py –product mobile –type issues
• python scrape_input.py –type all
• python scrape_input.py  –type issues –search hang,responding,freeze

Manually graphing in Numbers

1. Open the CSV file
2. Select the first row and make it a header row
3. Select the date column and other columns to graph
4. Choose “Share X values” after selecting the data in the gear menu
5. Click the chart button then scatter chart button
6. Open the inspector (View > Show Inspector)
7. Click the chart tab
8. Click the series subtab
9. Data symbol -> None
10. Connect points -> Straight

Have fun!

Proposal

That’s it, pretty simple. Examples:

…etc

Why?

• “#” denotes a keyword that doesn’t widely apply across the project
1. Humans and tools can skip over easily
2. Trivial bugzilla patch to make it sort at the bottom / last (I’ll attach it to the bug in a bit). We can even filter them out of the autocomplete entirely if desired
3. Trivial to make a watchdog script that enforces states and makes sure only certain people are adding / removing certain groups of keywords (which can be eventually expanded to a bz extension)
4. Easy to let anyone with editbugs create them with a bugzilla patch (if desired)

• “#” denotes a keyword in this system
1. Tools can rely on the format and process it, unlike current keywords (is ‘-’ part of the keyword or a namespace/separator? Depends!)

• Hierarchical structure of tags
1. Tools can query for keywords lists at the proper depth easily
2. Humans can easily group keywords at a glance
3. Humans can query easily through the existing bugzilla interface (e.g “show me everything Release Management is triaging”)

• Keyword subsections can use whatever rules they like, only “#” in the 1st position and “/” anywhere is special (spaces obviously don’t work)

• (ab)using the current keyword system. This is essentially a small step to moving keywords into a tree model, rooted via component. I need something NOW and can’t wait for bugzilla improvements (and already have enough of them assigned to me).

Why not use the whiteboard?

The whiteboard is junk! I can’t do historical queries for when a whiteboard substring was removed. The bugmail isn’t explicitly clear and I have to take time to scan the string, fun regex stuff in scripts, etc.

If bugzilla had a decent keyword or tag system the whiteboard would go away…it’s a kludge.

Future plans?

• Make anyone able to grab a “#” namespace and then create as many keywords as they want inside without having the current heavy-weight keyword access. This should actually be easy to do in bugzilla
• Release Management may migrate this sort of stuff to our web system outside Bugzilla, but for now we’re (ab)using current tools
• We need to replace Bugzilla’s keyword system. Yesterday.

Please let me know what you think and if you or your group would find such a change useful. The goal would be for it to be invisible to anyone who isn’t using it and wildly useful to those who do.

There was a problem with the patch though. Boris Zbarsky had the following comment in the bug:

someone who knows this code better than I should evaluate risk…

When Boris defers to someone else, I get worried. He knows everything. If he doesn’t feel confident commenting on the risk it is likely in an area that requires crazy black magic…and likely risky.

The very next comment was Robert Longson saying:

The fix is clearly right [...] Looks like pretty good value for a 1 line change to me.

Now, I had seen Robert’s name previously but I wasn’t as intimately familiar with his areas of expertise. I remembered Mozillians.org launched recently and decided to see if it could shed some light on Robert to give his signoff additional weighting.

So I searched. Luckily, I got a hit!

Not too bad for a website that launched recently. I checked out Robert’s profile:

Looks like Robert is a SVG peer! If there was any doubt (don’t know why there would be) I have Daniel vouching for the information. Because I know Daniel, this is about as solid a confirmation I could get.

So, now I have a heavy-hitter (Boris) writing the patch and a peer in the area of the patch (Robert) saying it is correct and safe. That makes taking the regression patch a no brainer.

Mozillians.org helped to turn a fuzzy product decision into a clear one, reducing risk and making us more confident in a Firefox change. I also can’t wait for the tool to have timezone support so I know when community members I need patches from are available or not.

Please add yourself if you haven’t already so we can make this resource even more valuable.

I apologize for any churn this may have caused.

What does the advertised update mean?

• We will be offering an advertised update (also sometimes called a “major update”) for Firefox 3.6.x -> Firefox 7.0 on 2011-10-06 (1 week and 2 days after Firefox 7′s release)
• The advertised update has no bearing on support levels. It does not mean Firefox 3.6 is end-of-life
• The advertised update has no bearing on the Firefox ESR proposal
• This is part of the old release process and is unchanged from what we have been doing for the past 3+ years
• The prompt is opt-in…if users decline, they stay on 3.6.x and get security updates until the next time we decide to prompt
• We did this for 3.0.x -> 3.5, 3.5.x -> 3.6, 3.6.x -> 4.0 and 3.6.x -> 5.0…nothing new
• If enterprise deployments have advertised/”major” updates disabled in rollouts their users will not see the prompt but they will see Firefox 3.6.24 when it is released on 2011-11-08

This is mainly to move regular Firefox users who don’t know there is an update up to a newer/better version if they are so inclined.

Based on previous advertised update offers we expect to see a significant percentage of users installing the new version. We are, of course, watching the data closely to see what happens and to make sure there are no unexpected issues.

As always, feel free to contact me with any questions or concerns.

If you are running Firefox Beta, you already have the version that was released today. One of the benefits of being on the cutting edge! Want an even earlier look at Firefox? Try Firefox Aurora.

What is mediawiki-bugzilla?

It is a server-side MediaWiki extension that provides read-only access to the Bugzilla REST API. It’s basically a working proof of concept at this point but I will be beefing it up in the coming weeks.

Why did I write my own MediaWiki extension?

I deal with wiki pages, bugs, and triage daily. It seemed stupid to be copying data and sometimes I wanted to embed up-to-date bug information in various wiki pages. I looked for solutions and found the excellent Bugzilla Reports extension. Unfortunately, it didn’t fit my needs:

• Bugzilla Reports needs direct access to the Bugzilla database. Yikes! Not only would I not get that access that would mean we would have to security audit the whole thing. Also, it was unclear that the assorted groups and permissions could be preserved across queries
• The specification/query language was homegrown. In addition to wiki markup one had to take time to learn how to tune and use a specification unique to Bugzilla Reports
• I’m not the best PHP programmer in the world but the code seemed overly complex and was difficult to dive into

I actually hacked together a proof-of-concept backend to have Bugzilla Reports query the Bugzilla REST API instead of the database but it didn’t work quite right and wasn’t maintainable.

I wanted to make an extension that is painless to use, easy to develop, and has no possible way of exposing restricted bugs. I didn’t want to create my own query specification / format, so I piggybacked off Gerv’s excellent Bugzilla REST API. The extension is basically an easy way to make REST API calls from MediaWiki so the docs can be used for the extension as well.

How to use mediawiki-bugzilla

You essentially use this extension in this way:

For example, the following would be used to get all P1 bugs in the Bugzilla product:

Notice that this is the same example used in the Bugzilla REST API documentation. Any query you can do via the API you should be able to embed in a wiki page using this extension.

I also stubbed out support for charting using the Google Charts API:

If you put both of the above examples into a wiki page you get something like this:

I know the tables aren’t styled yet. I’m going to be using jquery UI + the datatable plugin, with a pref to fall back to standard HTML tables without javascript.

All this info (and more!) can be found in the README.

Next steps

The major thing I am going to add is caching of the queries every 5 minutes or so while allowing manual cache overriding. This should make it so 50 people loading a wiki page with 10 queries at the same time doesn’t overwhelm Bugzilla.

After that I will beef up the charting and add some docs. Then we’ll get this rolled out on wiki.mozilla.org!

As always, patches welcome. The code is very simple and it should be very easy to add new features as there aren’t really any right now.

First, some background. We test pre-release versions of Apple’s latest OS, Mac OS X Lion (10.7). Our users also test on their own and we see their crash reports in our crash reporting system. Apple seems to have introduced a bug in their ATS framework in a previous Lion seed. The bug makes Firefox (all versions) extremely unstable. We reported it to Apple but there are only so many bugs that can be fixed and unfortunately this one didn’t make the cut for their GM seed. Because Apple had announced “July” as the release of Lion we knew we couldn’t just wait for Firefox 6′s release (August 16th) to fix it. At the same time, it came to our attention that the latest Java update released by Apple on Mac OS X Snow Leopard (10.6) broke Java for Firefox 5 users. Clearly we had to create out-of-band updates to address both issues.

Immediately, we questioned what platforms we should ship the updates to.  It seemed odd to release an updated Firefox to all users on all platforms when the bugs only affect the Mac population.

There are some distinct downsides to releasing across all platforms in this situation:

1. We know update fatigue is very real…last we checked over 7% of users don’t update specifically because they are annoyed about too many update offers (see Gerv’s post as well). To subject Windows and Linux users to the heavyweight update process, potentially annoying them enough to not update to Firefox 6 in August is a big concern
2. The updates aren’t trivially small. Profile-guided optimization on Windows makes it so the binaries aren’t exactly the same for a rebuild. Additionally, the code fixes aren’t conditionally compiled so the resulting binaries would be even more different. Though binary patch / delta updates reduce the impact, we still don’t feel comfortable having our entire Windows and Linux populations download data that did nothing for them
3. More QA / qualification is required and there is more opportunity for issues to crop up (such as automation failures, packaging errors, etc)

Of course, there are some downsides to releasing only for Mac users in this situation:

1. User confusion. Windows and Linux users may see the update announcements and think their version is insecure / there is something wrong with automatic updates
2. Supporting tools may expect all platforms to have the same supported version
3. Our update automation only supports patching to the last update. When we go to release Firefox 6 and Firefox 3.6.20 all non-Mac users will get the full update rather than a binary patch update (unless we try to do manual hackery)
4. The Mozilla.com website only supports specifying a “latest version”. It is not broken down by platform. We have to offer all platforms for download on Mozilla.com or only offer the older version, exposing new Mac users to the two fixed problems until they update (which may not happen if they keep crashing in the middle of the download)
5. Due to the way our update server works, we have to pop up a window offering Firefox 5 to the freshly updated Firefox 3.6.19 Mac users or make it so “Check for updates” on 3.6.19 doesn’t offer Firefox 5–both options not ideal

This is why release management is hard. Rarely is there a good option and often it takes too long to get the proper data needed to make an informed decision. Just gathering the above information from various stakeholders took a fair amount of time. To make matters more frantic, we needed to get the updates out as soon as possible to release before Lion’s ship date (which is still unknown but thought to be soon).

I looped in product management, as they are the ultimate product “owners” and know how to reason about the assorted tradeoffs and their impact on both our users and Firefox in general. After conferring with product management we decided the downsides of releasing for all platforms outweighed the ones for releasing just for Mac.

We verified with supporting system owners that releasing only for Mac would not cause issues (crossing off #2 above), chose to offer all platforms on the website (largely mitigating #4), and made a call to turn off manual update offers to Firefox 5 for 3.6.19 users (#5 above).  We also have time to figure out how to manually hack around patch packages to mitigate #3 but made it clear we are willing to live with the current situation when Firefox 6 / Firefox 3.6.20 comes along.

This decision and process could definitely have been messaged more clearly. We were moving quickly and I was at a team offsite that stole more attention than anticipated. I’m happy with the decision we took but realize there was no “correct” decision in this situation. Both ways had tradeoffs and if we chose releasing to all platforms I imagine I would be writing a similar post explaining why.

• Bug 585802 – Change the cc/user autocomplete (and backend) usermatching to ignore spaces / search on space separated names

Previously, searching for “Christian L” returned all users with “Christian” in the first or last name as well as all users with “L” in the first or last name. As you can imagine, this was a long list to search through (lots of names have an “L” somewhere) and made searching by name useless–especially when you didn’t know how to spell someone’s name.

Most people work around this bug by searching just for a IRC nick using “:[ircnick]“.  IRC nicks are generally unique which produces a smaller list, and inserting the colon assures that there are little to no substring matches.

The search behavior really annoyed me so I dug in, patched Bugzilla, and changed the behavior:

Notice the only people shown match "Christian L" rather than everyone with an "L" in their first or last name

I now find myself searching for irc nicks less and less and just using a person’s name. This is one less mental step for me as I no longer have to map from a person to an irc nick in my head.

This was my first Bugzilla patch and the first improvement I wrote that got rolled out on bugzilla.mozilla.org. Special thanks to Max for guiding me through the patch / Bugzilla codebase.

I hope that this small change made Bugzilla a little more usable for everyone.

Code freeze for Firefox 3.6.18 is THIS MONDAY (2011-06-06) @ 11:59 pm PDT!

If you have a bug in the blocking list, you are on the hook to have it done (or let branch-triage@mozilla.org know why it can’t be done).

Thanks!

Please do note the rapid release process is still new and might need to be adjusted as we go through a couple.

If you are…..

A Developer

• You need to focus on the fact that the next merge from mozilla-central to mozilla-aurora is 2011-05-24 (any code for Firefox 6 needs to be in by that date)
• You need to know any major issues you find in code landed before 2011-04-12 should be brought to the attention of release drivers to put into beta or aurora, wherever is appropriate

A Localizer

• You need to focus on the fact that you only have until 2011-05-17 to finish localizing Firefox 5 content in mozilla-aurora (time left for current work)
• You need to focus on the fact that the next merge from mozilla-central to mozilla-aurora is 2011-05-24 (time the work for the next cycle starts so you can plan)

Add-on Developer

• You only have until 2011-05-17 to test your add-on with Firefox 5 content in mozilla-aurora and mark your add-on as 5.0* compatible.

  You will be able to test for 6 more weeks on beta but our beta audience has over 2 million people…you probably don’t want to field their support questions and should just focus on getting it compatible during the aurora timeframe.

• You should start testing your add-on with Firefox 6 content in mozilla-aurora on 2011-05-24. You will have 6 weeks to test on aurora and another 6 weeks to test on beta

Anyone Else

• Firefox 5 will be released on 2011-06-21

I hear there is a developer calendar somewhere and intend to add the assorted merge dates. I have also been saying it every week in the platform, MoCo, channel, and product meetings. We’re working on getting it added to the wiki in the appropriate places (thanks to Matt Brubeck for adding it on the RapidRelease wiki page).

Additionally, there are dates in the specifics document as well as a quick python program to spit out your very own merge schedule. Please note we are still working out the mechanics of merges so they may be +- a day or so.

Photo by Bob Moss

With the new Firefox rapid release process managing the details of those releases is even more critical as we absolutely cannot slip the schedule. Because the schedules are set in stone we need to manage a lot more up front. Additionally, with releases shipping on a faster cadence there is less time for mistakes to be caught. When one adds in new platforms (smartphones, tablets), new products (Firefox Mobile, Firefox Home), and new channels (Nightly, Aurora) there is clearly a growing need for project and release management.

To meet this growing need Mozilla is hiring for a new release management team!

Is this you?

We currently have two positions available:

Associate Engineering Project Manager – Releases: We’re looking for someone at the beginning of their career who can learn the ropes and grow to be an experienced member of the release management team. Problem solving, great communication skills, and technical chops are a must!

Engineering Project Manager – Releases: We’re looking for someone with tons of experience managing software releases, hopefully at internet scale. This person will hit the ground running, taking the lead on many releases as well as driving process improvements.

What is Release Management?

Wikipedia has a somewhat decent description of what release management actually is. The main takeaway is the fact that it is a project management function, not development (writing the code for the bits that are eventually shipped), QA (testing the bits that are eventually shipped), or Release Engineering (managing the systems that build, package, and deliver the bits). All these groups have differing priorities and worldviews yet need to interact on the same project–that’s where release management comes in.

Don’t worry, release management is not just playing with schedules, sending emails, and organizing meetings. There is ample opportunity to create tools (examples aren’t hard to find) , help define the direction of the product (by driving features or hashing out new processes), and make decisions that directly impact hundreds of millions of people worldwide.

Darn, I don’t live in the San Francisco Bay Area

While our preference would be to have the employees work out of our Mountain View, CA office, it is not explicitly required. You don’t have to live in the San Francisco Bay Area, California, or even the United States! We have offices all over and for the right candidate / timezone we might even consider 100% remote (though your effectiveness will likely be diminished as a large part of the job is interaction with other people).

If you do want to move to the San Francisco Bay Area, we can investigate how to get you here.

Sweet, sign me up!

Great! We want to fill these positions quickly, so get your resume in order and apply via the jobs website:

• Associate Engineering Project Manager – Releases
• Engineering Project Manager – Releases

There is some discussion as to what “merge on 2011-04-12” means. It is generally open to two interpretations:

1. Developers can land fixes up to 11:59 pm PDT on 2011-04-12 and the actual merge will happen on 2011-04-13 PDT
2. Developers can land fixes up to the merge point, which happens sometime on  2011-04-12

We are going with #2 above and intend to do the merge @ ~6:00 am PDT.

How will you know the merge is done / mozilla-central is tracking for Firefox 6 instead of 5?

There will be a version bump on mozilla-central to 6.0a1 (see bug 648653). Anything before that will generally be on track for the next Firefox version, anything after will likely wait for a future release.

What happens if the version bump hasn’t happened and it is after 6:00 am PDT?

We’ve decided to give some more time, you can still land for the upcoming Firefox release (likely for < 24 hours).

Please use best judgement when pushing late tonight and tomorrow. We will NOT be closing the tree…normal mozilla-central tree rules apply.

Please comment in bug 648653 if there are urgent items that need to be taken care of before this happens or if there is a concrete reason Monday or Tuesday appears to work better.

• Blocking bugs marked [softblocker] (betaN or final) NO LONGER have automatic approval
• To land a softblocker on mozilla-central you can do one of the following:

  If the risk / reward ratio is close to zero….

  1. Add justification to the bug
  2. Toggle the “approval2.0″ flag on the patch to “?” (aka ask for approval on the patch itself)

  -OR-

  If you think the bug is wrongly categorized and needs to block the release of Firefox 4…

  1. Remove “[softblocker]” from the whiteboard
  2. Add justification to the bug
  3. Toggle the “blocking2.0″ flag to “?” (aka nominate it for hardblocking)
• This means the ONLY bugs with automatic approval are hardblockers (betaN or final)
• We will be setting the “blocking2.0″ flag to “-” shortly (aka making them not blocking)
• We will not be going through and removing [softblocker] from the whiteboard

Please let me know if you have any questions or concerns.

• The betaN softblockers will be punted out of Firefox 4 very soon
• Please look over the betaN softblockers by THIS FRIDAY, 9:00 am PST, looking for bugs that should be upgraded to hardblocking status
• If you find a softblocker that needs to make Firefox 4, do the following:
  1. Remove “[softblocker]” from the whiteboard
  2. Add justification to the bug
  3. Toggle the “blocking2.0″ flag to “?” (aka nominate it for blocking)

Details

The betaN softblockers are bugs that have been previously identified as:

1. really nice to have for Firefox 4 but won’t stop us from shipping (the “softblocking” part)
2. needing beta coverage to make sure the fix is correct and doesn’t cause new issues (the “betaN” part)

Regardless if beta 12 is the last beta, we are rapidly approaching the point where there will be no more betas. Once we freeze for the last beta, non-blocking bugs that require beta coverage are implicitly no longer approved.

We ask that interested developers make sure our soft/hardblocking and beta coverage decisions are sound before we freeze for the final Firefox 4 beta. Additionally, we understand that new information may have come to light (such as reproducibility, impact, etc) and want to make sure we are making decisions with the correct data.

Please take a moment to look over the betaN softblocker list by THIS FRIDAY, 9:00 am PST, looking for bugs that should be upgraded to hardblocking status.

If you find a softblocker that needs to make Firefox 4, do the following:

1. Remove “[softblocker]” from the whiteboard
2. Add justification to the bug
3. Toggle the “blocking2.0″ flag to “?” (aka nominate it for blocking)

To be warned, the bar for a hardblocker is extremely high at this point, especially for bugs that have been marked softblocking so long.

After this Friday at 9:00 am PST we reserve the right to remove the blocking status of the betaN softblockers (and thus automatic approval) at any time. Once we do so, approvals will be on a case-by-case basis.

Beta 11

We are pushing to get these bugs in by EOD today (PST):

• https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocking2%3Abeta%20sw:b11

If you are on the hook for any of those (including reviews and feedback) they are your number one priority.

If you are landing bugs NOT in that list, please think about the risk to beta 11. If anything is even slightly risky, please hold off a day until we branch for beta 11 before landing on mozilla-central. We will not close mozilla-central and instead trust developers to act responsibly with risk.

Once we have all / the the majority of the bugs in the query above we will call a build using the last green changeset.

Beta 12

The current plan is to build when the remaining betaN hardblockers are done. This is heavily dependent on fix and blocker creation rate. I will be doing some analysis to give people a possible date range, but as far as development is concerned everything is the same (fix as many betaN hardblockers as quickly as possible).

As usual, feel free to contact me on IRC or via email if there are questions/concerns.

What prompted me to look specifically at input data was a mention of bug 628872 on Twitter. It seemed like a bug that should block Firefox 4 but I wanted to know the extent of the problem. Rather than try to reproduce I went to input and saw the following:

Though likely not statistically significant, there has definitely been an uptick in negative feedback containing “iplayer”. This graph even helps to narrow down a regression timeframe. Recognizing the usefulness of this approach, I did a bunch more queries that came to mind.

I first decided to see if the YouTube player had a similar graph:

From the graph it is easy to tell that YouTube feedback has been relatively consistent, spiking every time we release.

In weekly Mozilla meetings there has been talk about a bug on Hotmail’s side causing problems for Firefox 4 beta users. Searching for “hotmail” I got:

Clearly users are feeling the pain and letting us know. Similarly, one of the top issues discussed in support reports has been copy and paste. Searching for “copy paste” gave me:

We’re on the case (bug 613915) and need to fix the issue before final.

Next I thought about bug 626016 (which is about Facebook chat) so I searched for “chat”:

The two spikes are interesting. My completely uninformed guess about the spike on August 12 is Facebook (or some other chat) going down or a server-side website release that went wrong and was quickly rolled back. The spike on the right is likely bug 626016.

Up next I looked at “netflix”:

This is another interesting graph. The left spike was likely due to the known issue of bad user-agent detection on Netflix’s side (bug 522957). The increased displeasure on the right is likely due to bug 598406. From that known issue “hulu” had similar sniffing problems which look to have been resolved:

This method can also be used to gauge general user sentiment. I knew the removal of the status bar was contentious, so I pulled up the graph for “status”:

Clearly you can see the initial displeasure when the change landed in a release and the resulting dropoff. Of course, there is still a level of sustained feedback which has prompted some additional product changes.

Finally, I searched for “apple” with no bug particularly in mind. I was pretty surprised to find this graph:

The spike seemed to spike and fade too suddenly to be a Firefox issue. I did a quick Google search for “apple october 21” and immediately saw what was going on. On the 23rd Apple reported their earnings. Such an event wouldn’t normally impact Firefox in any way, but Apple live-streams their earnings report. Because Apple is heavily invested in H.264 they streamed it using that technology. Firefox doesn’t support H.264, Firefox users couldn’t access the stream and thus were complaining. The complaints were only relevant while the live stream was relevant and disappeared the next day. Fascinating!

I found this sort of analysis interesting and thought provoking and am glad I have a tool like Firefox Input available to me (and the world!).

The Updated Beta Plan

• If all betaN hardblockers are done by this Friday afternoon PST (2011-01-28) we will build beta 11 at that time. This is essentially staying on the previously posted plan
• If on Friday there are remaining betaN hardblockers we will not build and instead defer the decision until Tuesday (2011-02-01)
• If all betaN hardblockers are done by next Tuesday afternoon PST (2011-02-01) we will build beta 11 at that time. There will be no planned beta 12
• If on Tuesday there are remaining betaN hardblockers we will make a judgement call
• If on Tuesday the remaining betaN hardblockers are on track to land before Friday (2011-02-04), we will not build on Tuesday and instead wait for Friday. There will be no planned beta 12
• If on Tuesday the remaining betaN hardblockers look at risk to land by Friday (2011-02-04) we will create beta 11 on Tuesday and plan on having a beta 12. Beta 12 will not have a fixed schedule and will instead be built when the remaining betaN hardblockers are at zero

Bug Fix Priorities

[risky] betaN hardblockers > betaN hardblockers > final hardblockers > betaN softblockers > final softblockers

Again, feel free to contact me with any questions or concerns.

Summary

1. We’ll take the most recent green changeset from mozilla-central this Friday, 2011-01-21 at 2:00 pm PST and call that beta 10
   • The tree will close for < 15 minutes for this (if at all)
   • We’ll freeze nightly updates at or around the changeset we branched from (likely only for the weekend)
   • Any problems with the beta will be dealt with on the relbranch
2. We will release beta 10 as soon as possible during the week of the 24th
3. We’ll take the most recent green changeset from mozilla-cental on Friday, 2011-01-28 at 2:00 pm PST (or sooner if the remaining betaN hardblockers are finished) and call that beta 11
   • The tree will close for < 15 minutes for this (if at all)
   • We’ll freeze nightly updates at or around the changeset we branched from (likely only for the weekend)
   • Any problems with the beta will be dealt with on the relbranch
4. We will release beta 11 as soon as possible during the week of the 31st

Details

Both frontend and platform managers have determined that their betaN hardblockers will not all be finished by this Friday. Additionally, greater than 318 bugs have been marked fixed since beta 9. We need beta coverage for most of those fixes but also need beta coverage for the remaining betaN hardblockers. To control risk for the RC the plan is to do two betas (10 and 11) rather than hold beta 10 for all betaN hardblockers. This will give us a week of beta coverage for the 318 bug fixes while still allowing beta coverage for the remaining betaN hardblockers.

Code freeze for beta 10 and 11 will also be a bit different. Previous betas closed the tree to stabilize and reduce beta risk. At this point in development, every day counts–we cannot shut down mozilla-central over the weekend as we have done before. Instead we’ll freeze what’s in the tree and run with it, dealing with any beta issues on the relbranch. Instead of metering what goes in before a freeze we will be driving to get as many fixes as possible so they can get more beta coverage.

To enable additional testing of the bits we intend to ship to beta while keeping the tree open for development, we will freeze mozilla-central nightlies to the changeset we branched from (likely only for the weekend). Developers can still use hourlies to test and will able to integrate their work with others without waiting a weekend for the tree to reopen.

For those curious, joduinn has provided the the specific plan for the nightly changeset freeze:

1. RelEng will create a nightly ~2pm PST Friday, with same changeset as the beta10 builds
2. RelEng will create the nightly build fri+sat+sun night as usual, but will force the updates to a hidden location. A human can download+install those nightly builds if they want (for regression ranges / reproduction), but no-one will get updated to these nightlies
3. On Monday, release drivers will send the “ok to reenable normal nightly operation”. RelEng will revert changes and trigger a mid-day Monday nightly so people on Friday afternoon nightly will directly update to Monday afternoon (skipping the fri/sat/sun night nightlies)
4. We will then move snippets generated over weekend back to usual locations, and cleanup
5. The change is a small, and we believe safe, change to how updates are created

Explicit Bug/Release Dependencies

• Beta 10: None, time-based release to test what we have done since beta 9
• Beta 11: BetaN hardblockers
• RC: Final hardblockers

I’ll keep the various mailing lists and planet updated if anything changes. Again feel free to shoot me an email (or find me on IRC) with any questions you have.