A modest Bugzilla keyword format proposal
I just sent an email to dev-planning proposing a new keyword format. Nothing earth-shattering, but I think the upside is huge. I'll reproduce it here, but feel free to comment in the dev-planning thread.
Proposal
#(namespace)/..../(leafnode keyword)
That's it, pretty simple. Examples:
#relman/triage/needs-info #relman/triage/defer-to-group #ux/papercuts/user-control #ux/papercuts/useful-error-messages #crashkill/watching #crashkill/blocklist-candidate #crashkill/3rd-party/adobe #crashkill/3rd-party/adobe/contacted #gfx/bugkill/too-old #gfx/bugkill/bitrot
...etc
Why?
- "#" denotes a keyword that doesn't widely apply across the project
- Humans and tools can skip over easily
- Trivial bugzilla patch to make it sort at the bottom / last (I'll attach it to the bug in a bit). We can even filter them out of the autocomplete entirely if desired
- Trivial to make a watchdog script that enforces states and makes sure only certain people are adding / removing certain groups of keywords (which can be eventually expanded to a bz extension)
- Easy to let anyone with editbugs create them with a bugzilla patch (if desired)
- "#" denotes a keyword in this system
- Tools can rely on the format and process it, unlike current keywords (is '-' part of the keyword or a namespace/separator? Depends!)
- Hierarchical structure of tags
- Tools can query for keywords lists at the proper depth easily
- Humans can easily group keywords at a glance
- Humans can query easily through the existing bugzilla interface (e.g "show me everything Release Management is triaging")
- Keyword subsections can use whatever rules they like, only "#" in the 1st position and "/" anywhere is special (spaces obviously don't work)
- (ab)using the current keyword system. This is essentially a small step to moving keywords into a tree model, rooted via component. I need something NOW and can't wait for bugzilla improvements (and already have enough of them assigned to me).
Why not use the whiteboard?
The whiteboard is junk! I can't do historical queries for when a whiteboard substring was removed. The bugmail isn't explicitly clear and I have to take time to scan the string, fun regex stuff in scripts, etc.
If bugzilla had a decent keyword or tag system the whiteboard would go away...it's a kludge.
Future plans?
- Make anyone able to grab a "#" namespace and then create as many keywords as they want inside without having the current heavy-weight keyword access. This should actually be easy to do in bugzilla
- Release Management may migrate this sort of stuff to our web system outside Bugzilla, but for now we're (ab)using current tools
- We need to replace Bugzilla's keyword system. Yesterday.
Please let me know what you think and if you or your group would find such a change useful. The goal would be for it to be invisible to anyone who isn't using it and wildly useful to those who do.
Announcing mediawiki-bugzilla, a MediaWiki extension to embed bug data
Do you use wiki pages to track things? Tired of copy and pasting bugzilla queries and bug information? Me too. I got fed up one day and decided to do something about it and the result is mediawiki-bugzilla.
What is mediawiki-bugzilla?
It is a server-side MediaWiki extension that provides read-only access to the Bugzilla REST API. It's basically a working proof of concept at this point but I will be beefing it up in the coming weeks.
Why did I write my own MediaWiki extension?
I deal with wiki pages, bugs, and triage daily. It seemed stupid to be copying data and sometimes I wanted to embed up-to-date bug information in various wiki pages. I looked for solutions and found the excellent Bugzilla Reports extension. Unfortunately, it didn't fit my needs:
- Bugzilla Reports needs direct access to the Bugzilla database. Yikes! Not only would I not get that access that would mean we would have to security audit the whole thing. Also, it was unclear that the assorted groups and permissions could be preserved across queries
- The specification/query language was homegrown. In addition to wiki markup one had to take time to learn how to tune and use a specification unique to Bugzilla Reports
- I'm not the best PHP programmer in the world but the code seemed overly complex and was difficult to dive into
I actually hacked together a proof-of-concept backend to have Bugzilla Reports query the Bugzilla REST API instead of the database but it didn't work quite right and wasn't maintainable.
I wanted to make an extension that is painless to use, easy to develop, and has no possible way of exposing restricted bugs. I didn't want to create my own query specification / format, so I piggybacked off Gerv's excellent Bugzilla REST API. The extension is basically an easy way to make REST API calls from MediaWiki so the docs can be used for the extension as well.
How to use mediawiki-bugzilla
You essentially use this extension in this way:
<bugzilla> (JSON REST API query key/value pairs) </bugzilla>
For example, the following would be used to get all P1 bugs in the Bugzilla product:
<bugzilla> { "product": "Bugzilla", "priority":"P1" } </bugzilla>
Notice that this is the same example used in the Bugzilla REST API documentation. Any query you can do via the API you should be able to embed in a wiki page using this extension.
I also stubbed out support for charting using the Google Charts API:
<bugzilla type="count" display="bar"> { "product": "Bugzilla", "priority": "P1", "x_axis_field": "severity" } </bugzilla>
If you put both of the above examples into a wiki page you get something like this:
I know the tables aren't styled yet. I'm going to be using jquery UI + the datatable plugin, with a pref to fall back to standard HTML tables without javascript.
All this info (and more!) can be found in the README.
Next steps
The major thing I am going to add is caching of the queries every 5 minutes or so while allowing manual cache overriding. This should make it so 50 people loading a wiki page with 10 queries at the same time doesn't overwhelm Bugzilla.
After that I will beef up the charting and add some docs. Then we'll get this rolled out on wiki.mozilla.org!
As always, patches welcome. The code is very simple and it should be very easy to add new features as there aren't really any right now.
A small Bugzilla improvement
With the recent Bugzilla 4.0 rollout on bugzilla.mozilla.org comes a small but useful usability improvement. It's so small you may not have noticed (I wrote it but didn't notice for weeks!), so I decided to blog about it. The bug I filed and fixed was:
- Bug 585802 - Change the cc/user autocomplete (and backend) usermatching to ignore spaces / search on space separated names
Previously, searching for "Christian L" returned all users with "Christian" in the first or last name as well as all users with "L" in the first or last name. As you can imagine, this was a long list to search through (lots of names have an "L" somewhere) and made searching by name useless--especially when you didn't know how to spell someone's name.
Most people work around this bug by searching just for a IRC nick using ":[ircnick]". IRC nicks are generally unique which produces a smaller list, and inserting the colon assures that there are little to no substring matches.
The search behavior really annoyed me so I dug in, patched Bugzilla, and changed the behavior:

Notice the only people shown match "Christian L" rather than everyone with an "L" in their first or last name
I now find myself searching for irc nicks less and less and just using a person's name. This is one less mental step for me as I no longer have to map from a person to an irc nick in my head.
This was my first Bugzilla patch and the first improvement I wrote that got rolled out on bugzilla.mozilla.org. Special thanks to Max for guiding me through the patch / Bugzilla codebase.
I hope that this small change made Bugzilla a little more usable for everyone.
Firefox 4 “softblocking” bugs no longer have automatic approval
Quick note about softblocking bugs:
- Blocking bugs marked [softblocker] (betaN or final) NO LONGER have automatic approval
- To land a softblocker on mozilla-central you can do one of the following:
If the risk / reward ratio is close to zero....
- Add justification to the bug
- Toggle the "approval2.0" flag on the patch to "?" (aka ask for approval on the patch itself)
-OR-
If you think the bug is wrongly categorized and needs to block the release of Firefox 4...
- Remove "[softblocker]" from the whiteboard
- Add justification to the bug
- Toggle the "blocking2.0" flag to "?" (aka nominate it for hardblocking)
- This means the ONLY bugs with automatic approval are hardblockers (betaN or final)
- We will be setting the "blocking2.0" flag to "-" shortly (aka making them not blocking)
- We will not be going through and removing [softblocker] from the whiteboard
Please let me know if you have any questions or concerns.
Whoops, I caused a bunch of bug churn
You may have noticed a bunch of bug churn in any bug marked blocking2.0:beta9+ (and their related bugs).
I'll blog more about what happened later, but the short story is an automated script blanked fields not in the "_default" fieldset when querying the Bugzilla REST API (specifically See Also, CCs, Blocks, Depends On, Keywords). The script was tested, but only against test bugs that didn't have data in those fields. I watched the first couple of production bugs getting changed, didn't see a problem, and let it run. I was confirming everything acted as I expected when I saw the issue halfway through script execution (before people started hollering at me) so there are some bugs in that list that had the correct changes applied (that is, the only data that changed was blocking2.0:beta9+ to blocking2.0:betaN+).
I manually went through each bug and I believe I have fixed them all up. Of course, when going through ~116 bugs manually there is potential for error. Please send me an email if I missed anything or a bug looks suspect to you.
The most annoying part (other than looking like an idiot) is in the time I spent developing the script, a cleanup script, and manual checking I could have patched the missing feature I was working around into Bugzilla proper.
Sorry for the bugspam.
Python library for Bugzilla’s REST API
I am now maintaining a python library (originally written by Jeff Balogh) for interacting with Bugzilla via the the REST API. You can get it from:
https://github.com/LegNeato/bztools
Check out the README for a simple example of how to use the library. You can also take a look at the scripts at http://hg.mozilla.org/users/clegnitto_mozilla.com/release_tools for some more advanced uses.
Some features:
- Supports authentication (with credentials stored in the system keychain with a config file fallback)
- Supports querying the API via agents with optional query options, so you can include/exclude what you need
- Support for Bugs, Users, Attachments, Comments, Changes, Changesets, Flags, Keyword. Dependency support will be working shortly as well
- Supports common set and equality operations for objects. For example, looping through a list of Bug objects:
for bug in buglist: print bug
Checking if a Bug object is in a list of Bugs:
if bug in buglist: print "Found!"
Adding sets of bugs together into one large set:
all_bugs = buglist1 + buglist2 + buglist3
Checking if two User objects are the same:
if bug.assigned_to == bug.reporter: print "Assigned to the reporter!"
The library has really cleaned up my scripts and has been insanely useful. Thanks to Jeff for creating such a nice library and letting me take it over and improve it.
Bugzilla activity visualization
Here's a cool visualization of bug activity on bugzilla.mozilla.org over the past 2 days:
Downloads:
http://people.mozilla.org/~clegnitto/videos/bugs.mp4
http://people.mozilla.org/~clegnitto/videos/bugs.webm
http://people.mozilla.org/~clegnitto/videos/bugs.ogv
And here's the same activity, zoomed in on the "Core" cluster/branch:
Downloads:
http://people.mozilla.org/~clegnitto/videos/bugs_core.mp4
http://people.mozilla.org/~clegnitto/videos/bugs_core.webm
http://people.mozilla.org/~clegnitto/videos/bugs_core.ogv
I sped up the videos a bit so that every 30 seconds in the video is equal to a day of activity in reality.
Background
I saw David Humphrey's post using gource to visualize source repositories. I looked at it and thought "That's neat, it would be cool to have it in real-time on the screens in the Mozilla offices!"
Implementation
Luckily gource supports reading a simple pipe-delimited format from stdin, which makes integration with outside tools trivial. I wrote a 20 line python script that uses my pet-project pulse to pump hg.mozilla.org push events into gource. It wasn't super exciting though, as there isn't a ton of pushes happening (even with the try repository).
I thought about it a little bit and realized that pulse also has Bugzilla data flowing through it. I decided it'd be really cool to (ab)use gource to visualize bug activity.
First, I determined the mappings from bug activity to repository activity:
- File added -> new bug ("bug.new" for the pulse routing key)
- File modified -> bug changed ("bug.changed.#" for the pulse routing key)
- File deleted -> closed bug (searching for certain states in "bug.changed" messages)
- Committer -> user creating or changing the bug
- File path -> /[Bug's product]/[Bug's component]/[Bug's id]
Constructing the "path" in this way makes bugs cluster in a coherent way. Realtime still wasn't super exciting (for pulse reasons I will not go into here), so I let the tool run for a couple days, dumped the resulting messages to a file, and pointed gource at it.
If you want to play around with it (gource is interactive), I've uploaded the data here. The user/pass is nospam (there are email addresses in the file so I didn't want to leave it wide open). I used these gource options to make the video:
gource --log-format custom --hide bloom,filenames --user-scale .5 -s 30 /path/to/data.txt
Future plans
I would love for someone to write a visualization tool using canvas or WebGL. I don't have time to do it with all the Firefox release work. If you want to try to tackle this, I can provide ample help getting it hooked up to pulse's data stream though.
I also looked a bit at code swarm, as I think it would provide better visuals for bugs. Rather than committers I would focus on bugs, with the change types (cc added, comment added, fields changed, etc) as the different colored dots. If I get time I'll run the same data through cod swarm and see what looks better. I won't be able to do realtime with code swarm though, as it uses an XML file format.
As an aside, this is why I am so excited about pulse. Having the data in an easily consumable stream unlocks the potential for tools we haven't even thought of, generally with minimal development work.
bugzilla-amqp is now buzgilla-push, supports the STOMP protocol
The Bugzilla server-side extension I released previously has been renamed to bugzilla-push. It can now be found at http://github.com/LegNeato/bugzilla-push. The main impetus for the name change was that it now supports STOMP in addition to AMQP. It seemed silly to keep "amqp" in the name when it supports multiple protocols.
The reason for adding STOMP support is to keep my message broker options open. While RabbitMQ is pretty nice, it may not meet the needs for pulse.mozilla.org once it gets out of prototype mode. All open source brokers (HornetQ, Apache ActiveMQ & Qpid, Redhat MRG, etc) have pledged to support AMQP eventually, but many have not implemented it. Most have implemented STOMP though, as the protocol is both stable and simple. The extension now gives the Bugzilla administrator the option of choosing which protocol to use based on their requirements.
Notable changes since I last blogged:
- Pluggable backends with optional CPAN dependencies. If you don't want to use AMQP, you don't need to have those dependencies installed
- Added simple support for message security. This was a major hurdle for getting bugzilla-push rolled out on bugzilla.mozilla.org. I intend to beef it up more in the coming weeks as well
- Fixed a bug where false values were being sent as "0" in the JSON messages instead of JSON's false
- Fixed support for using vhosts that are not "/" (the AMQP default)
Planned in the next week:
- Supporting YAML for message encoding
- Supporting Python's pickle for message encoding
- Get the extension rolled out on landfill.bugzilla.org
- Test, test, test
- Ask for security review from Mozilla's web security team
Push notifications for Bugzilla!
I've had some downtime between Firefox releases and chose to work on a pet project on-and-off for the past week. I'm announcing it today as bugzilla-amqp.
What is bugzilla-amqp?
A server-side Bugzilla extension that sends messages to a message broker via AMQP whenever a Bugzilla object (bug, keyword, component, etc) is created or modified.
Why?
It enables push notifications for interesting events in Bugzilla! This is a big deal. Tools no longer have to poll the various APIs when dealing with bug data...instead they can sit back and get notified! Want to know when you are CC'd? Easy! Want to know when a new bug is written? No problem! Take a look at the quick demo video (webm, theora...warning, large!)
Because it talks AMQP, tools interested in the Bugzilla messages/events can be written in just about any language you want for any platform you want.
The impetus for writing this extension came from the desire to integrate Mozilla Pulse (running RabbitMQ) with bugzilla.mozilla.org, having push messages end-to-end.
Sounds awesome! I want this on bugzilla.mozilla.org now!
It won't be rolled out on bmo for a bit yet. All these need to happen:
- There are some features that need to be added first (like, uh, security)
- After that, because there is a fair amount of code (as far as Bugzilla extensions go), it will likely need to go through a security review
- Performance testing needs to happen so that it doesn't bring down bmo inadvertently
- The server running Mozilla Pulse needs to get beefier and the traffic expectations with IT have to be revisited (I promised them it was a prototype after all...)
I have filed bug 589322 to track putting the extension into production on bmo.
Ok, still sounds awesome...where do I get the code?
I've put it at http://github.com/LegNeato/bugzilla-amqp. Let me know if you use it and/or find any issues and feel free to fork away!
Are you some hardcore Bugzilla hacker?
Nope, I'm a Firefox release manager
. The Bugzilla extension system is pretty easy...I highly suggest you take a look if you ever wished Bugzilla did something differently or wanted a feature added.
