LegNeato! Christian Legnitto's blog about Mozilla, Apple, technology, and random stuff

Bugzilla activity visualization

Here's a cool visualization of bug activity on bugzilla.mozilla.org over the past 2 days:

Downloads:
http://people.mozilla.org/~clegnitto/videos/bugs.mp4
http://people.mozilla.org/~clegnitto/videos/bugs.webm
http://people.mozilla.org/~clegnitto/videos/bugs.ogv

And here's the same activity, zoomed in on the "Core" cluster/branch:

Downloads:
http://people.mozilla.org/~clegnitto/videos/bugs_core.mp4
http://people.mozilla.org/~clegnitto/videos/bugs_core.webm
http://people.mozilla.org/~clegnitto/videos/bugs_core.ogv

I sped up the videos a bit so that every 30 seconds in the video is equal to a day of activity in reality.

Background

I saw David Humphrey's post using gource to visualize source repositories. I looked at it and thought "That's neat, it would be cool to have it in real-time on the screens in the Mozilla offices!"

Implementation

Luckily gource supports reading a simple pipe-delimited format from stdin, which makes integration with outside tools trivial. I wrote a 20 line python script that uses my pet-project pulse to pump hg.mozilla.org push events into gource. It wasn't super exciting though, as there isn't a ton of pushes happening (even with the try repository).

I thought about it a little bit and realized that pulse also has Bugzilla data flowing through it. I decided it'd be really cool to (ab)use gource to visualize bug activity.

First, I determined the mappings from bug activity to repository activity:

• File added -> new bug ("bug.new" for the pulse routing key)
• File modified -> bug changed ("bug.changed.#" for the pulse routing key)
• File deleted -> closed bug (searching for certain states in "bug.changed" messages)
• Committer -> user creating or changing the bug
• File path -> /[Bug's product]/[Bug's component]/[Bug's id]

Constructing the "path" in this way makes bugs cluster in a coherent way. Realtime still wasn't super exciting (for pulse reasons I will not go into here), so I let the tool run for a couple days, dumped the resulting messages to a file, and pointed gource at it.

If you want to play around with it (gource is interactive), I've uploaded the data here. The user/pass is nospam (there are email addresses in the file so I didn't want to leave it wide open). I used these gource options to make the video:

gource --log-format custom --hide bloom,filenames --user-scale .5 -s 30 /path/to/data.txt

Future plans

I would love for someone to write a visualization tool using canvas or WebGL. I don't have time to do it with all the Firefox release work. If you want to try to tackle this, I can provide ample help getting it hooked up to pulse's data stream though.

I also looked a bit at code swarm, as I think it would provide better visuals for bugs. Rather than committers I would focus on bugs, with the change types (cc added, comment added, fields changed, etc) as the different colored dots. If I get time I'll run the same data through cod swarm and see what looks better. I won't be able to do realtime with code swarm though, as it uses an XML file format.

As an aside, this is why I am so excited about pulse. Having the data in an easily consumable stream unlocks the potential for tools we haven't even thought of, generally with minimal development work.