What is bugzilla-amqp?

A server-side Bugzilla extension that sends messages to a message broker via AMQP whenever a Bugzilla object (bug, keyword, component, etc) is created or modified.

Why?

It enables push notifications for interesting events in Bugzilla! This is a big deal. Tools no longer have to poll the various APIs when dealing with bug data…instead they can sit back and get notified! Want to know when you are CC’d? Easy! Want to know when a new bug is written? No problem! Take a look at the quick demo video (webm, theora…warning, large!)

Because it talks AMQP, tools interested in the Bugzilla messages/events can be written in just about any language you want for any platform you want.

The impetus for writing this extension came from the desire to integrate Mozilla Pulse (running RabbitMQ) with bugzilla.mozilla.org, having push messages end-to-end.

Sounds awesome! I want this on bugzilla.mozilla.org now!

It won’t be rolled out on bmo for a bit yet. All these need to happen:

1. There are some features that need to be added first (like, uh, security)
2. After that, because there is a fair amount of code (as far as Bugzilla extensions go), it will likely need to go through a security review
3. Performance testing needs to happen so that it doesn’t bring down bmo inadvertently
4. The server running Mozilla Pulse needs to get beefier and the traffic expectations with IT have to be revisited (I promised them it was a prototype after all…)

I have filed bug 589322 to track putting the extension into production on bmo.

Ok, still sounds awesome…where do I get the code?

I’ve put it at http://github.com/LegNeato/bugzilla-amqp. Let me know if you use it and/or find any issues and feel free to fork away!

Are you some hardcore Bugzilla hacker?

Nope, I’m a Firefox release manager . The Bugzilla extension system is pretty easy…I highly suggest you take a look if you ever wished Bugzilla did something differently or wanted a feature added.

If you have any bugs in these queries, we need your attention on them ASAP:

• Firefox 3.6.9 blocking bugs that do not have an approved patch
• Firefox 3.5.12 blocking bugs that do not have an approved patch

If you don’t think one of your bugs should be blocking (or you think it won’t make code freeze), please say so in the bug or email me directly.

If you have any bugs in these queries, the patch needs to be landed or it will not make the release:

• Firefox 3.6.9 approved bugs not marked as fixed
• Firefox 3.5.12 approved bugs not marked as fixed

Projects in order of importance

1. Patch for Bugzilla to add rich bug relations.
   This will give us greater confidence that bugs aren’t missed/overlooked. It will also help development by organizing bugs consistently and allow for richer tools, processes, and progress reporting
2. Create a new “release management” system that will manage all aspects of a release.
   This will allow consistent processes between different teams/releases, make sure nothing is missed, add checking tools, and generally become the “truth” when it comes to release metadata (schedule, state, status, etc)
3. Get Mozilla Pulse (http://pulse.mozilla.org) solid, usable, and useful.
   Pulse has the opportunity to make all systems at Mozilla better. I hit some bumps with RabbitMQ but will be working on ironing them out and providing a scalable, HA system that can later be handed over to another team
4. Create a triage reporting tool.
   This will make the approval processes more open by publishing triage notes and results. It will also be a place to put action items so that they are acted upon before the next triage session
5. Finish new release note framework.
   I started this when I first came but got tied up in other projects. The way we do release notes involves a lot of copy and paste, it is difficult for QA to create automated tests, etc. I intend to fix this
6. Patch for TabCandy to support user-defined rules.
   The response to TabCandy has been great, but I think having this feature will be essential for power users and those who don’t want to manually organize tab groups. I’ve looked at the TabCandy code a bit and this shouldn’t be too hard to hack in. I may work on this a bit as a breather from the above projects (but probably not)

If you have any ideas about these systems (or others you think release management needs), please let me know!

I’ve been running into issues with RabbitMQ (the erlang message broker that runs on pulse). I griped a little on Twitter and got some responses, so I decided to write a more in-depth description of what I am running into. I’m not going to explain any message broker specific terminology, so feel free to skip this post if you don’t know what I am talking about. None of this should be important if you just want to use pulse in the future.

The general idea of using a message broker at Mozilla is to make useful tools on top of infrastructure, with the infrastructure (producers) being loosely coupled from the tools (consumers). Because of this, I came up with this configuration for an initial prototype:

Exchanges

org.mozilla.exchange.bugzilla (topic)

• All Bugzilla messages are routed in here. Bugzilla is the producer, with permissions of “.*bugzilla” “.*bugzilla” “.*bugzilla”. That is, the Bugzilla producer can do anything to the Bugzilla exchange
• The message routing key hierarchy looks like bug.added, bug.changed.[field], etc
• The plan was to add more, sticking logic in the producer (that is, bug.changed.resolution when the message data is CLOSED should be elevated to bug.closed instead, etc)
• The message rate is very high-volume for Mozilla’s Bugzilla, as you can imagine

org.mozilla.exchange.hg (topic)

• All hg.mozilla.org messages are routed in here. HG is the producer, with permissions of “.*hg” “.*hg” “.*hg”. That is, the HG producer can do anything to the HG exchange
• The message routing key hierarchy looks like hg.mozilla.central.repo.[opened/closed], hg.releases.mozilla.1.9.2.[commit/push], etc
• The message rate is not that high-volume, though when watching all repositories it could be a bit bursty

org.mozilla.exchange.build (topic)

• All build.mozilla.org messages are routed in here. Buildbot is the producer, with permissions of “.*build” “.*build” “.*build”. That is, the Buildbot producer can do anything to the build exchange
• This is currently experimental and the routing keys haven’t been figured out to provide the most value
• Very high-volume, though less so than the Bugzilla exchange

Consumers

These were my general goals for consumers:

1. Be as simple as possible so people can start playing with pulse, proving the idea and getting some momentum
2. I do not want to be the bottleneck for experimentation, so no user accounts or administration tasks necessary to just consume messages
3. Users writing consumers should not need to learn about any of the underlying message broker terminology or technology
4. Users could be running consumers on their local machines, and when they reconnect all the messages they missed should be there waiting (they could clear the old messages or process them depending on their needs)

Because of those, I came up with the following plan:

1. Create a user named public with a password of public and permissions of “” “” “.*”, which as far as I know means the user can read from anything but not write or create. The public user can still write and create server-created resources, which means when it asks for the foo queue, the server will create it if it doesn’t exist and public will then only have access to read from it
2. Create a trivial shim library in python on top of carrot to abstract out the message broker bits and help Mozilla-specific consumers get up and running quickly
3. Make sure people testing set a unique string for their applabel, which means their queue will be unique and message delivery will not fall back to round-robin between different people

So, seemed like a good plan, right? And it worked! Until…

Issues

Deleting unused queues

It became clear people (myself included) created some queues and then later changed to a different queue. The old queues were sitting there accumulating messages which would never be consumed. I went to delete the queues and…..rabbitmqctl doesn’t have a delete queue command. Darn. Ok, I have the BQL plugin installed, so not a huge deal to pop in and delete them through that, but it seems odd this functionality is missing.

Running out of memory with old persister

There were some bugs in the Bugzilla producer which caused messages to be extremely throttled. I fixed them and immediately the broker ran out of memory and fell over. This was because there were 10 or so queues that weren’t having messages actively consumed, each with ~1000 messages. I didn’t see this in testing because all my testing consumers were running and consuming the messages that were sent without any buildup. Additionally, the server is running on a VM (it’s a prototype after all) which doesn’t have a bunch of memory to begin with.

I tried to connect to the queues with a python consumer (using carrot) to drain them, but everything just hung. I could not drain the queues and unblock the server, which meant I couldn’t write an administration script that removed 500 messages out of any queue with > 500 un-acked messages.

Reading around, a lot of people are running into this problem. The good news is that the new persister is supposed to fix it, though it isn’t quite done yet. It looks like the new persister is in QA and many people on the mailing lists are running it, so I decided to take the plunge on this prototype system.

Incompatibilities between RabbitMQ 1.7.x and 1.8.x

The prototype pulse system was running RabbitMQ 1.7.x and everything was working well (except for the out of memory bit above). To get the new persister, I had to update to 1.8 (as the latest persister branch is 1.8 based). I decided to upgrade to 1.8 release and make sure everything else still worked before adding the additional layer of pre-release code on top. This is what I did:

1. Downloaded rabbitmq-public-umbrella
2. Compiled, installed, and then activated some plugins

I deleted the old persister log, started the server, and immediately found an issue.

The public user couldn’t seem to create queues anymore. Darn, that meant people wouldn’t be able to use my shim lib. Reading around, it looked like it could be caused by having a 1.7.x data directory with 1.8.x, so I deleted the whole data directory and let RabbitMQ recreate it. I then built up the exchanges, users, and permissions exactly as before. The problem was still there.

So, it looks like the RabbitMQ change to the new AMQP semantics in 1.8 broke what I was doing. Apparently, it is no longer possible to have a read-only user create a queue. I guess this makes sense, though it was my (naive) understanding that automatic queue creation was built into the AMQP spec. That is, the read-only user is requesting it, and if it exists it is handed back to the user, otherwise the server creates it on their behalf. Perhaps this is a bug?

In any case, I opened up the permissions for the public user (this is a prototype system with no real users remember).

Running out of memory with new persister

I decided to take the plunge and make sure the new persister fixed my memory issue before pursuing the permissions issue. This is roughly what I did to upgrade:

1. Downloaded rabbitmq-public-umbrella
2. Downloaded the new persister branch
3. Replaced rabbitmq-server in rabbitmq-public-umbrella with the persister branch
4. Compiled, installed, and then activated some plugins

I then created some queues, started up the Bugzilla producer, and sent thousands of messages through. RabbitMQ fell over again, as far as I can tell with the same problem. I deleted the whole data directory and let RabbitMQ recreate it. I then built up the exchanges, users, and permissions exactly as before. And it still ran out of memory.

Questions

1. Are people successfully running the new persister for RabbitMQ?
2. Do I need to explicitly turn on the new persister when using the new persister branch? If so, how? There are (understandably) no docs that I can find.
3. Am I setting up the exchanges, queues, and vhosts wrong? As far as I can tell everything was working great before the OOM stuff and the 1.8 semantic changes.
4. Is there a better way to structure what I want to do?
5. Is my use-case not supported by RabbitMQ? That would be odd, as this seems like the exact use case that message brokers were made to solve. Do other brokers support what I want?

Because of the release, we moved the previous 1.9.2.6 blocking and nomination flags to 1.9.2.7, as the next released version of Firefox will be 3.6.7.

Code freeze for 3.6.7 and 3.5.11 was Friday, which means we are now frozen. There are still a bunch of open blockers! PLEASE, please take a look at these lists and see if you are on the hook. Even though we are frozen, we will be approving fixes for blockers (mostly security fixes) on an ad-hoc basis if they come in early enough in the week.

• Open blockers in Firefox 3.6.7
• Open blockers in Firefox 3.5.11

If you have any bugs in these queries, we need your attention on them ASAP (if you aren’t in the critical path for Firefox 4 beta of course):

• Firefox 3.6.6 blocking bugs that do not have an approved patch
• Firefox 3.5.11 blocking bugs that do not have an approved patch

If you don’t think one of your bugs should be blocking, please say so in the bug or email me directly.

If you have any bugs in these queries, your patch needs to be landed:

• Firefox 3.6.6 approved bugs not marked as fixed
• Firefox 3.5.11 approved bugs not marked as fixed

These bugs have the checkin-needed keyword and it would be great for other people to land them:

• Firefox 3.6.6 checkin-needed
• Firefox 3.5.11 checkin-needed

Ehsan and Reed may beat you to those checkin-needed bugs though!

• Bug 567134 – Use ASLR in NSS if it’s available

This bug (combined with Bug 559133 – Use ASLR in NSPR if it’s available) will go a long way to hardening Firefox!

If we could get a patch landed for Firefox 3.6.6 (code freeze is tonight!) it would be AMAZING.

Thank you to everyone that worked so hard over the last couple of months to make this release happen. I know OOPP will be a huge win for Firefox users and will make browsing better all around.

Community testers, release engineering, QA, Firefox and platform engineering, and others did amazing work to make this release solid and release it into the hands of our users.

There were a lot of “firsts” for this release and I believe we learned a lot. I’ll be giving a talk at the Mozilla Summit summarizing what we learned and how we can apply it to future releases, so if you are interested in the nitty-gritty check it out!

Thanks again to everyone for their hard work!

Firefox 3.6.4 (get the release candidate!) has a platform version of 1.9.2.4. The version number 1.9.2.5 is currently being used by Fennec. We’ll be taking fixes above and beyond that version, so the next platform version Firefox will use will be named 1.9.2.6. We will keep the version numbers coherent by naming it Firefox 3.6.6 (essentially skipping over 3.6.5).

I will be moving the blocking flags and status flags as appropriate, so if you see activity in bugs don’t worry. Also, If you have approvals for 1.9.2.5 they are now for 1.9.2.6 (and should still be landed on mozilla-1.9.2 default).

This is nothing more than a name change, everything still refers to the same Firefox release.

We originally announced a beta of Firefox 3.6.4 on April 20th, 2010. We then let everyone know there was an updated beta on May 4th, 2010. Today is May 27th. What’s been happening since the 4th? Lots!

1. In total we built (and mostly released) five builds of Firefox 3.6.4 for our beta users
2. We have been analyzing crash reports, making sure there are no new issues and quantifying the benefits of out-of-process-plugins. Out-of-process-plugin hangs / crashes (or OOPsies) are now easier to see thanks to a new version of Socorro that rolled out a couple weeks ago. You can see an example by looking at this report. The “hang |” signatures are reports that are sent when a plugin misbehaves. Note that there is still more work to do to filter out OOPsies in aggregate graphs and counts
3. We have been working with antivirus and firewall vendors to make sure the new “plugin-container.exe” process used by OOPP isn’t flagged as malware. If you work for such a vendor and have not heard from us please get in touch
4. We have started creating the sixth build with a fix to the only known blocker. Hopefully build number six will meet our high quality standards and become Firefox 3.6.4 final (after a beta period of course). I’ll do another blog post letting everyone know when the release candidate build is out. In the meantime, you can help out by installing and testing the latest beta from http://www.mozilla.com/firefox/all-beta.html

As always, the most up-to-date release plan can be found on the wiki.

We know out-of-process-plugins will make a huge difference for our user base and we are trying to get Firefox 3.6.4 released as soon as possible. Hang tight and help us test!

Though a rapid security update would normally be viewed as entirely negative, 3.6.3 had a couple of positive points for me:

1. It was my first release, which made me more aware of what was going on. The experience should help releases going forward.
2. The release fixed a public security issue without regressions. It made Firefox users safer without compromising stability. The no regressions bit is very important, as we want users to trust updates enough to apply them quickly. If you see a regression please let us know.
3. We beat our Pwn2Own response time from last year. Last year it was 10 days (which is great), this year it was 8 days (which is amazing).
4. We beat other companies to a fix and received recognition from the press for it. Though security is not a race, it’s nice to be first (without sacrificing stability of course).
5. Beating Apple to a publicly released fix gives me the opportunity to gloat to my old team. I enjoy gloating immensely.

Thanks everyone!

The bug is “fallout” from bug 553073. It should not affect any release version of Firefox and should only affect testers who auto-updated from a build without bug 553073.

There is a simple workaround. In Terminal, run:

touch /Applications/Minefield.app

Of course, change the location to where you actually have the nightly if it isn’t in /Applications.

If you are not comfortable with the Terminal, simply moving the application to the desktop and back should fix the issue as well.

As a consequence of fixing bug 553073, the default browser choice should now stick if you run Firefox and Minefield side by side.

This is me

My name’s Christian Legnitto (LegNeato on IRC/Twitter) and I started working at Mozilla last week. I love it so far! I joined Mozilla as the new release driver for Firefox security and stability releases, so you’ll be hearing a lot about updates from me.

Before Mozilla I worked at Apple doing (surprise) security and stability releases for Mac OS X. If you use Mac OS X and have installed any security update or point release since 10.4.6, you’ve seen my handiwork.

So what does a release driver do? A lot! First and foremost, I define the scope and schedule for the deliverable/update. Once defined, I then drive the project forward. This usually involves a lot of status checking and bugging people for fixes and/or testing. At the end of it all I’m the person that decides to push the big red button and send an update out to millions of users. And if anything breaks because of the update? It’s my job to take responsibility, figure out why it happened, and make sure it doesn’t happen again.

I’ll also be using my past release experience to suggest process and tool improvements at Mozilla. And because I am a programmer at heart, I will likely write a bunch of tools myself when I have downtime.

My Mozilla-related posts should show up on planet.mozilla.org and planet.firefox.com, but I’ll be talking about other stuff as well, so feel free to check out http://christian.legnitto.com/blog or follow me on twitter (@LegNeato). Also, all this tweeting and blog posting is fairly new to me…so feel free to give me some pointers!