With Firefox 4 beta 8 and Firefox 4 beta 3 for mobile out (whew!) it’s time to give an update on pulse.mozilla.org.
View Pulse messages live from your browser!
I set up a page to show the messages flowing through the system. You can view it at http://pulse.mozilla.org/live. The page makes for a nifty (barebones) status dashboard of work going on around Mozilla.
The technology behind the page is fairly interesting. I am using Node.js to connect to RabbitMQ via AMQP and serve the messages over Socket.io (which uses long polling or WebSockets) to client code served by Django via WSGI to the browser. That’s a lot of alphabet soup but it was surprisingly easy to set up. It’s pretty exciting and fun to use the cutting edge technologies everyone is talking about.
A future blog post will detail exactly what I did, but you can see the Node.js code at http://hg.mozilla.org/users/clegnitto_mozilla.com/pulsewebsite/file/default/node/browser_amqp.js. Simple yet powerful.
- I no longer consider Pulse a prototype. I now consider it in beta and will be doing more evangelism to get people writing tools against the system. Geo in QA has a prototype system written against Pulse and has already suggested some great improvements
- The system moved to a new, beefier VM in the Phoenix data center. Hopefully the days of running out of disk space and memory are over!
- I have documented (with video!) how to go from our stock RHEL 5 VM image to a running Pulse instance. I will be writing it up and finishing up the video in the coming weeks
- The website is finally in Mercurial. It is basically the old static site stuck into a Django template, but at least it lays the groundwork for future work
- Added the live view functionality as mentioned above
- I converted the scrapers/publisher shims from Celery to cron. This removes one moving piece, adds a lot more control, and I have yet to see the scrapers hang
- The new ftp scraper script has been checked in and is running on production
- There is now a heartbeat message sent every minute. This message lets people playing around with the system know their code is working and makes it so the web view doesn’t have periods of inactivity. To see the messages you can use “PulseTestConsumer” from the python helper library or you can connect to the “org.mozilla.exchange.pulse.test” exchange via a standard AMQP client. There may be other messages sent through that exchange, so if you want to only listen for the heartbeat messages listen for “heartbeat”
- One of the RelEng buildbot masters is now publishing build messages into Pulse (see bug 614576). To see the messages you can use “BuildConsumer” from the python helper library or you can connect to the “org.mozilla.exchange.build” exchange via a standard AMQP client
- James Socol and Jeff Balogh have set up some of their Mozilla GitHub repos to publish commit events into Pulse using the GitHub service hook I created. To see the messages you can use “CodeConsumer” from the python helper library or you can connect to the “org.mozilla.exchange.code” exchange via a standard AMQP client. To only listen to GitHub messages filter on “github.#”. If you listen for “#” you will also get messages from Mozilla’s Mercurial repositories
- The python helper library now defaults to temporary/non-durable queues. This should make experimentation easier for folks and will lessen the resource requirements on the server
- The python helper library now specifies its requirements in such a way that easy_install and pip will automatically download necessary dependencies
- I have started to put example code into Mercurial so new users don’t need to copy and paste from the website. The repository is at http://hg.mozilla.org/users/clegnitto_mozilla.com/pulsequickstart/. I intend to expand it a fair amount, add other language examples, etc.
There is also exciting work going on to instrument assorted systems so I can retire the shim/scraper scripts. I’ll likely have more to report about that in the coming weeks though.
If anyone has graphic skills and would like to help me out with the website, a logo, or create a cool dashboard using the data flowing though Pulse get in touch! I’ve been playing around with interesting ways to visualize the data and hope to have more to show in the coming quarter.