Here’s a cool visualization of bug activity on bugzilla.mozilla.org over the past 2 days:
[external-vfe name=”bugs”]
And here’s the same activity, zoomed in on the “Core” cluster/branch:
[external-vfe name=”bugs_core”]
I sped up the videos a bit so that every 30 seconds in the video is equal to a day of activity in reality.
Background
I saw David Humphrey’s post using gource to visualize source repositories. I looked at it and thought “That’s neat, it would be cool to have it in real-time on the screens in the Mozilla offices!”
Implementation
Luckily gource supports reading a simple pipe-delimited format from stdin, which makes integration with outside tools trivial. I wrote a 20 line python script that uses my pet-project pulse to pump hg.mozilla.org push events into gource. It wasn’t super exciting though, as there isn’t a ton of pushes happening (even with the try repository).
I thought about it a little bit and realized that pulse also has Bugzilla data flowing through it. I decided it’d be really cool to (ab)use gource to visualize bug activity.
First, I determined the mappings from bug activity to repository activity:
- File added -> new bug (“bug.new” for the pulse routing key)
- File modified -> bug changed (“bug.changed.#” for the pulse routing key)
- File deleted -> closed bug (searching for certain states in “bug.changed” messages)
- Committer -> user creating or changing the bug
- File path -> /[Bug’s product]/[Bug’s component]/[Bug’s id]
Constructing the “path” in this way makes bugs cluster in a coherent way. Realtime still wasn’t super exciting (for pulse reasons I will not go into here), so I let the tool run for a couple days, dumped the resulting messages to a file, and pointed gource at it.
If you want to play around with it (gource is interactive), I’ve uploaded the data here. The user/pass is nospam (there are email addresses in the file so I didn’t want to leave it wide open). I used these gource options to make the video:
gource --log-format custom --hide bloom,filenames --user-scale .5 -s 30 /path/to/data.txt
Future plans
I would love for someone to write a visualization tool using canvas or WebGL. I don’t have time to do it with all the Firefox release work. If you want to try to tackle this, I can provide ample help getting it hooked up to pulse’s data stream though.
I also looked a bit at code swarm, as I think it would provide better visuals for bugs. Rather than committers I would focus on bugs, with the change types (cc added, comment added, fields changed, etc) as the different colored dots. If I get time I’ll run the same data through cod swarm and see what looks better. I won’t be able to do realtime with code swarm though, as it uses an XML file format.
As an aside, this is why I am so excited about pulse. Having the data in an easily consumable stream unlocks the potential for tools we haven’t even thought of, generally with minimal development work.
I would love to see something like this in canvas/WebGL. Is there an easy way to pulse information in the javascript?
Totally. Previously I used http://jmesnil.net/stomp-websocket/doc/, as I have the stomp plugin installed on pulse and put orbited in front to provide push. Currently the orbit piece is down but I can get it back up in the next couple days or so. There’s also https://github.com/dansimpson/amqp-js, which I haven’t played with.
What happens if a component or product includes a slash (‘/’) as part of its name? 🙂
Good catch :-). I now encode ‘|’ and ‘/’ so they don’t mess with the data format
Awesome 🙂 And thanks for providing .ogvs and .webms 🙂
Gerv
Wow, that’s really awesome, having things like flicker around realtime in Mozilla offices would really make a nice impression (even if it means that things happen way slower, but maybe we’ll just need some generic visualization of all pulse data (and a really large screen) in the end to make it provide full awesomeness 😉
Have you seen the App store wall from WWDC? (http://www.youtube.com/watch?v=ZCW26uzaGLM). It’s in their lobby now. I have plans 😉