I’ve been looking at Aakash’s awesome Firefox Input site to try to figure out how we are doing with Firefox 4. Unlike others who have sifting through individual reports and writing bugs, I’ve been trying to look at the data in aggregate. I did some adhoc queries over the lifespan of the Firefox 4 betas and thought the results were interesting. Of course, this is not scientific in any way but I think the insights are still valuable. I think some sort of triggers / alarms doing this analysis automatically should be added to input (bug coming soon).
What prompted me to look specifically at input data was a mention of bug 628872 on Twitter. It seemed like a bug that should block Firefox 4 but I wanted to know the extent of the problem. Rather than try to reproduce I went to input and saw the following:
Though likely not statistically significant, there has definitely been an uptick in negative feedback containing “iplayer”. This graph even helps to narrow down a regression timeframe. Recognizing the usefulness of this approach, I did a bunch more queries that came to mind.
I first decided to see if the YouTube player had a similar graph:
From the graph it is easy to tell that YouTube feedback has been relatively consistent, spiking every time we release.
In weekly Mozilla meetings there has been talk about a bug on Hotmail’s side causing problems for Firefox 4 beta users. Searching for “hotmail” I got:
Clearly users are feeling the pain and letting us know. Similarly, one of the top issues discussed in support reports has been copy and paste. Searching for “copy paste” gave me:
We’re on the case (bug 613915) and need to fix the issue before final.
Next I thought about bug 626016 (which is about Facebook chat) so I searched for “chat”:
The two spikes are interesting. My completely uninformed guess about the spike on August 12 is Facebook (or some other chat) going down or a server-side website release that went wrong and was quickly rolled back. The spike on the right is likely bug 626016.
Up next I looked at “netflix”:
This is another interesting graph. The left spike was likely due to the known issue of bad user-agent detection on Netflix’s side (bug 522957). The increased displeasure on the right is likely due to bug 598406. From that known issue “hulu” had similar sniffing problems which look to have been resolved:
This method can also be used to gauge general user sentiment. I knew the removal of the status bar was contentious, so I pulled up the graph for “status”:
Clearly you can see the initial displeasure when the change landed in a release and the resulting dropoff. Of course, there is still a level of sustained feedback which has prompted some additional product changes.
Finally, I searched for “apple” with no bug particularly in mind. I was pretty surprised to find this graph:
The spike seemed to spike and fade too suddenly to be a Firefox issue. I did a quick Google search for “apple october 21” and immediately saw what was going on. On the 23rd Apple reported their earnings. Such an event wouldn’t normally impact Firefox in any way, but Apple live-streams their earnings report. Because Apple is heavily invested in H.264 they streamed it using that technology. Firefox doesn’t support H.264, Firefox users couldn’t access the stream and thus were complaining. The complaints were only relevant while the live stream was relevant and disappeared the next day. Fascinating!
I found this sort of analysis interesting and thought provoking and am glad I have a tool like Firefox Input available to me (and the world!).