Another update on Pulse
With Firefox 4 beta 8 and Firefox 4 beta 3 for mobile out (whew!) it's time to give an update on pulse.mozilla.org.
View Pulse messages live from your browser!
I set up a page to show the messages flowing through the system. You can view it at http://pulse.mozilla.org/live. The page makes for a nifty (barebones) status dashboard of work going on around Mozilla.
The technology behind the page is fairly interesting. I am using Node.js to connect to RabbitMQ via AMQP and serve the messages over Socket.io (which uses long polling or WebSockets) to client code served by Django via WSGI to the browser. That's a lot of alphabet soup but it was surprisingly easy to set up. It's pretty exciting and fun to use the cutting edge technologies everyone is talking about.
A future blog post will detail exactly what I did, but you can see the Node.js code at http://hg.mozilla.org/users/clegnitto_mozilla.com/pulsewebsite/file/default/node/browser_amqp.js. Simple yet powerful.
General
- I no longer consider Pulse a prototype. I now consider it in beta and will be doing more evangelism to get people writing tools against the system. Geo in QA has a prototype system written against Pulse and has already suggested some great improvements
- The system moved to a new, beefier VM in the Phoenix data center. Hopefully the days of running out of disk space and memory are over!
- I have documented (with video!) how to go from our stock RHEL 5 VM image to a running Pulse instance. I will be writing it up and finishing up the video in the coming weeks
Website
- The website is finally in Mercurial. It is basically the old static site stuck into a Django template, but at least it lays the groundwork for future work
- Added the live view functionality as mentioned above
Scrapers
- I converted the scrapers/publisher shims from Celery to cron. This removes one moving piece, adds a lot more control, and I have yet to see the scrapers hang
- The new ftp scraper script has been checked in and is running on production
Messages
- There is now a heartbeat message sent every minute. This message lets people playing around with the system know their code is working and makes it so the web view doesn't have periods of inactivity. To see the messages you can use "PulseTestConsumer" from the python helper library or you can connect to the "org.mozilla.exchange.pulse.test" exchange via a standard AMQP client. There may be other messages sent through that exchange, so if you want to only listen for the heartbeat messages listen for "heartbeat"
- One of the RelEng buildbot masters is now publishing build messages into Pulse (see bug 614576). To see the messages you can use "BuildConsumer" from the python helper library or you can connect to the "org.mozilla.exchange.build" exchange via a standard AMQP client
- James Socol and Jeff Balogh have set up some of their Mozilla GitHub repos to publish commit events into Pulse using the GitHub service hook I created. To see the messages you can use "CodeConsumer" from the python helper library or you can connect to the "org.mozilla.exchange.code" exchange via a standard AMQP client. To only listen to GitHub messages filter on "github.#". If you listen for "#" you will also get messages from Mozilla's Mercurial repositories
Code
- The python helper library now defaults to temporary/non-durable queues. This should make experimentation easier for folks and will lessen the resource requirements on the server
- The python helper library now specifies its requirements in such a way that easy_install and pip will automatically download necessary dependencies
- I have started to put example code into Mercurial so new users don't need to copy and paste from the website. The repository is at http://hg.mozilla.org/users/clegnitto_mozilla.com/pulsequickstart/. I intend to expand it a fair amount, add other language examples, etc.
There is also exciting work going on to instrument assorted systems so I can retire the shim/scraper scripts. I'll likely have more to report about that in the coming weeks though.
If anyone has graphic skills and would like to help me out with the website, a logo, or create a cool dashboard using the data flowing though Pulse get in touch! I've been playing around with interesting ways to visualize the data and hope to have more to show in the coming quarter.
Lots of Pulse changes, out of prototype mode this week!
I logged off IRC, chat, and email Friday afternoon to push Pulse forward. I made a bunch of progress.
Changes
- New, beefier VM, all set up (bug 609956 and bug 614029)
- The new VM setup supports publishing and consuming messages via websockets / Socket.io
- The 'org.mozilla.exchange.hg' exchange has been renamed to 'org.mozilla.exchange.code'. This is because we'll be publishing messages from non-hg sources into it as well (like GitHub using http://christian.legnitto.com/blog/2010/11/23/github-amqp-integration-service-hook-live/), bitbucket, svn, etc). Please note the previous change will likely break your existing python scripts until you update the mozillapulse helper library!
- The mozillapulse python helper library has been updated. I added transparent support for the above exchange change. If your scripts are using HgConsumer they will automatically use the proper exchange so there is no need to change them. In the future you should migrate from HgConsumer to CodeConsumer though
- The (lame) website has been put into HG. It uses Django. Currently it it just one static template, but I intend to make it more dynamic very soon
- A new scraper / shim that polls ftp and sends interesting build release/generation events. I'll be documenting this in the coming week
- I finished up a MediaWiki extension so we can get wiki.mozilla.org events via Pulse. I'll be releasing it next week
Coming this week
- DNS switchover from the old VM to new VM. The switch will happen on Tuesday during scheduled IT downtime
- Documentation about the new ftp scraper / shim
- General improvements to the website and documentation (likely leveraging the websocket functionality to show live Pulse messages)
- Instructions (with video!) how to set up and configure the server-side of Pulse on a stock RHEL 5 VM
- Pulse will be marked as BETA rather than an unsupported prototype
Coming in the near future but likely not next week
- Rolling out extensions on assorted systems so that I can retire the shims/scrapers that poll
- Hooking up to a 2nd VM that listens to all Pulse messages in all exchanges and stores them in Redis or CouchDB
- Interesting (web?) visualizations to encourage others to write tools against Pulse
For more information, sign up for the mailing list.
What’s up with Pulse?
Armen had asked about the current state of pulse on the mailing list and suggested I blog about it. There hasn't been much said publicly since the summit so I agreed it would be good to update everyone.
First, pulse has become a quarterly goal for Bob Moss' team! This is huge and means we'll get some talented people and additional momentum behind the project. They are tasked with moving it from a concept to something that can stand up on its own. I will still likely be heavily involved, though in the end I would love to just be a consumer of the system.
Recently we had a meeting to discuss the current state of pulse and how to hand it off. The slides from that discussion are can be found on my people account, though I am not sure they make a lot of sense without my explanations.
Current state / happenings:
- I've turned the scrapers back on so there are real messages currently flowing through pulse. The two scrapers running are the Bugzilla API and HG webpage message scrapers (used by BugzillaConsumer and HgConsumer in the python helper library). Side note: I need write a maintenance script so unacknowledged bugzilla messages in user queues don't fill up the disk space on the VM
- The bugzilla extension (bugzilla-push) I wrote enabling bugzilla to publish directly to pulse is done
- I rolled out bugzilla-push on landfill.bugzilla.org for testing. The bug tracking bugzilla.mozilla.org rollout is bug 58932
- Bugzilla-push is going through security review soon (bug 599979)
- My bugzilla refactoring patch that enables comment messages to be published into pulse finally got approved yesterday! (bug 590334)
- I wrote a quick and dirty hg hook to allow push and changeset messages to be published directly into pulse (hg-broker). The bug tracking hg.mozilla.org rollout is bug 603029
- I am currently writing a mediawiki extension so all wiki.mozilla.org changes can be published into pulse
- I have started to revamp the documentation/website on pulse.mozilla.org
- We've talked with the RabbitMQ guys a bit and may join WebDev in having them come in for a consult so I can voice my needs/concerns
- I've started playing around with elasticsearch so we can have storing and searching of all messages (doesn't work quite right yet)
I'll try to make it a habit to blog about pulse more. For the latest news feel free to join the mailing list.
Side projects for me in the coming weeks
I am going to focus on these side projects in the coming weeks, in addition to driving the Firefox security releases. If you see anything missing or something that needs my attention, please let me know.
Projects in order of importance
- Patch for Bugzilla to add rich bug relations.
This will give us greater confidence that bugs aren't missed/overlooked. It will also help development by organizing bugs consistently and allow for richer tools, processes, and progress reporting - Create a new "release management" system that will manage all aspects of a release.
This will allow consistent processes between different teams/releases, make sure nothing is missed, add checking tools, and generally become the "truth" when it comes to release metadata (schedule, state, status, etc) - Get Mozilla Pulse (http://pulse.mozilla.org) solid, usable, and useful.
Pulse has the opportunity to make all systems at Mozilla better. I hit some bumps with RabbitMQ but will be working on ironing them out and providing a scalable, HA system that can later be handed over to another team - Create a triage reporting tool.
This will make the approval processes more open by publishing triage notes and results. It will also be a place to put action items so that they are acted upon before the next triage session - Finish new release note framework.
I started this when I first came but got tied up in other projects. The way we do release notes involves a lot of copy and paste, it is difficult for QA to create automated tests, etc. I intend to fix this - Patch for TabCandy to support user-defined rules.
The response to TabCandy has been great, but I think having this feature will be essential for power users and those who don't want to manually organize tab groups. I've looked at the TabCandy code a bit and this shouldn't be too hard to hack in. I may work on this a bit as a breather from the above projects (but probably not)
If you have any ideas about these systems (or others you think release management needs), please let me know!
Firefox 3.6.4 is still chugging along
If you haven't called into Mozilla's platform or release meetings you may be wondering what the heck is going on with Firefox 3.6.4.
We originally announced a beta of Firefox 3.6.4 on April 20th, 2010. We then let everyone know there was an updated beta on May 4th, 2010. Today is May 27th. What's been happening since the 4th? Lots!
- In total we built (and mostly released) five builds of Firefox 3.6.4 for our beta users
- We have been analyzing crash reports, making sure there are no new issues and quantifying the benefits of out-of-process-plugins. Out-of-process-plugin hangs / crashes (or OOPsies) are now easier to see thanks to a new version of Socorro that rolled out a couple weeks ago. You can see an example by looking at this report. The "hang |" signatures are reports that are sent when a plugin misbehaves. Note that there is still more work to do to filter out OOPsies in aggregate graphs and counts
- We have been working with antivirus and firewall vendors to make sure the new "plugin-container.exe" process used by OOPP isn't flagged as malware. If you work for such a vendor and have not heard from us please get in touch
- We have started creating the sixth build with a fix to the only known blocker. Hopefully build number six will meet our high quality standards and become Firefox 3.6.4 final (after a beta period of course). I'll do another blog post letting everyone know when the release candidate build is out. In the meantime, you can help out by installing and testing the latest beta from http://www.mozilla.com/firefox/all-beta.html
As always, the most up-to-date release plan can be found on the wiki.
We know out-of-process-plugins will make a huge difference for our user base and we are trying to get Firefox 3.6.4 released as soon as possible. Hang tight and help us test!