Solving the Bugzilla Backlog

I’ve been doing a lot of complaining lately in directly violation of my first post of the year.

Going forward, I’m going to try harder to offer solutions instead of just doing a lot of complaining.

So, lately with Tyler Downer’s post, there’s been a lot of talk about Bugzilla and triage and other things. Yesterday, I was debugging a problem and discovered a bug that had been open for 5+ years with no fix (even though the component it was in had been rewritten in the mean time). (And before you ask, yes I posted a patch.)

The fact is that Bugzilla is a mess, not just with unconfirmed bugs, but with bugs that are no longer relevant, bugs that have already been fixed and more.

So here’s my thought:

Stop all non critical activity on the Mozilla project for one week. All development, all test, etc. Have everyone in the community (and I mean everyone) attack Bugzilla and get it as cleaned up as possible. Have awards for the people that resolve the most bugs.

Thoughts?

Please note: I reserve the right to delete comments that are offensive or off-topic.

Leave a Reply

Your email address will not be published. Required fields are marked *

37 thoughts on “Solving the Bugzilla Backlog

  1. I think we can’t do something like this, because solving bugs is more important.
    However, I think we can add some automatic triaging to bugzilla, for example (not in the advanced form) when it detects some hot words in the bug (like “memory”, “cpu”, “addon”, and things like these) bugzilla could automatically advice the user to enable safe mode and asking support.mozilla.com before actually filing the bug.
    This could reduce the number of bugs filed every day, and so if each deveoper triage a pair of bugs every day, the absolute number of bugs will be reduced too.

  2. I actually think that once we get a good triage process set up (like an UNCO bug product) this would be excellent to do long with the transfer of UNCO bugs from Firefox into the new product. I know that a dedicated effort like this (with plenty of planning before hand) would probably make alot of things better on BMO, and would help us start “the new mozilla” with less problems on BMO. I know if a dedicated effort like this were to happen, I would jump right into it.

    • Yes, definitely the triage process needs to be improved first, then the big push can start. I also love the idea of triage flags (https://wiki.mozilla.org/User:Tmyoung/Bugzilla_Triage), so different triagers don’t have to keep reading the same bug report over and over again to figure out what needs to be done. It’s sad though that there have been calls to reform the triage process for *years* and they never seem to go anywhere… Volunteers’ time isn’t free.

      • Unfortunately, devs felt they could afford to put triage improvements on the backburner for years. And that is what has got us to where we are today. It wasn’t until just about 6 months ago that Mozilla hired people to try to improve BMO itself (re-writing the guided bug entry form, etc.) I hope it doesn’t take that long to get people working on improving triage as well.

  3. Having “the whole community” stop and do this for a week would be mostly spent teaching the majority of us how to do it and a fraction of the time maybe getting stuff done.

    A better strategy would be to find a dedicated and enthusiastic team of 20-30 highly proficient triagers and do this in a series of short sprints — maybe every Mon/Tue/Wed for 3 weeks or something.

    Just throwing a giant blob of people at the problem, regardless of their individual knowledge or experience, would just be falling into the mythical man month trap. A focused, skilled and dedicated team with a clear mission and the time to do it? Much, much more effective.

    • > Having “the whole community” stop and do this for a week would be mostly spent teaching the majority of us how to do it and a fraction of the time maybe getting stuff done.

      In this case, I don’t think so. If the goal was to get experienced people on this, yes. But our goal here is just to get eyeballs on it.

      We could certainly have an “advanced team” who would get the bugs when the regular folks couldn’t figure it out.

      I’m not talking about unconfirmed bugs here. I’m talking about the many thousands of bugs sitting in Bugzilla that haven’t been touched for years.

      I don’t have a ton of experience with Bugzilla, but I can look at a random list of 10 bugs in Bugzilla and usually route 1 or 2 to the right place.

    • >A better strategy would be to find a dedicated and enthusiastic team of 20-30 highly proficient triagers

      I think this also falls into the same mythical man month trap. A generic “triager” isn’t going to have any idea whether a bug titled “refactor such and such code” is still valid or not. Only developers in the relevant component would know.

      When I did triage, I settled in on one area I was comfortable with and that seemed neglected (printing crashes/hangs and making testcases for them). By doing this I gained experience in the area and could generally work more effectively, e.g. I learned what to look out for and what common bugs were duplicates. Having every triager work on generic lists of bugs seems really inefficient. Specialization helps.

      >and do this in a series of short sprints — maybe every Mon/Tue/Wed for 3 weeks or something.

      I do like this idea. Maybe having developers do triage sprints of *confirmed* bugs in their own components would work effectively, or could at least be tried.

      This would also make developers aware of many old valid bugs they may never have seen; it’s likely some could be easily fixed. I think for now some emphasis should be shifted over to bug fixes from adding the latest and greatest new features and incremental performance improvements. Every project needs to do this now and then. What good is having the browser start up 20ms faster if the user spends an hour trying to work around a bug or can’t get the browser to do what he wants? (Tyler had a good post arguing for this approach; it was largely rejected.)

  4. I’m concerned that we’re already talking about ways to solve that problem before we’re sure that there indeed is a problem. The fact is there are many unconfirmed bugs, and it would certainly be nice if more of them were looked at, but then again there is plenty of other things that are worth doing too.

    I can only speak of what I know of, which are Graphics and WebGL bugs, and there I think we’re doing quite OK, all new bugs are looked at by at least a couple people and consequently most of the unconfirmed, never-replied-to bugs are probably junk. Not all of it, but an overwhelming majority, making it not worth interrupting our normal tasks just to triage them all.

    I’d also add that Mozilla _does_ sometimes take time to interrupt normal work and go into huge triaging sessions. At the all-hands before the Firefox 4 release, we spent at least 1 day in a room to triage important bugs. This was not about triaging random unconfirmed bugs, the bugs were preselected interesting-but-no-fix-yet bugs and I think that’s what made it worth it.

    I don’t know that we have a big problem of important bugs staying ignored. There are important bugs that don’t get fixed, but most of the time that’s not because they’re ignored, rather the important bugs attract a lot of discussion and sometimes no clear fix emerges from that.

    • Again, to be clear here, I’m not mainly talking about the triage problem.

      I’m talking about the fact that Bugzilla is filled with tens of thousands of open bugs that have been lost in history and as a result, something should be done with them. It makes searching through Bugzilla a pain and it leaves the perception that Mozilla products have thousands of unfixed “bugs”

      > all new bugs are looked at by at least a couple people and consequently most of the unconfirmed, never-replied-to bugs are probably junk.

      That hasn’t been my experience. And even if it is true, to look at a bug, disregard it and leave it as unconfirmed is bad practice. The least you can do is make an effort to do something with the bug, even if it is just to say “we won’t fix that”

      Go to Bugzilla. Pick some random word. Do a search. Watch the results.

      • > The least you can do is make an effort to do something with the bug, even if it is just to say “we won’t fix that”

        I think it’s a bad idea, as user may have suggested a good enhancement which may get WONTFIX for some reason (the description was not good enough and triager though it was something else than what was actually meant, or just he didn’t like the idea, although the idea is awesome, etc.), which is actually not reason enough to WONTFIX the suggested thing.

        My suggestion is to build a system based on the priority of bugs by the number of votes for them.
        Users should have a limited number of votes.
        There also should be a system of ranks, where users get more votes if they do some good actions, like file new bugs (they should gain votes only for the bugs which were accepted), create patches, provide feedback (triagers/mozdevs should have some kind of an ability to reward users who provided useful feedback, like they tested something on particular platform, helped to figure out the some important details, etc.).
        And the main idea is that once, let us say a week (or more/less frequently) the mozdev team has to start fixing N most popular bugs.

        In that kind of a rank system, where users actually gain more importance for the Mozilla Development Process, I believe that users will become more polite (as they will hunt for mozdevs’/triagers’ rewards for providing important info), they will hunt for new bugs (as filing the valid bugs grants them some kind of a reward measured in their authority) and you may even control the crowd to sort the bugs: users will sort bugs (by adding dependencies, signalizing that the bug is invalid etc) just to get a reward.

        Many users play RPG games, and this becomes just another game for them, but the difference is that they don’t gain the authority in a virtual world anymore, they gain it in the real world, they start affecting the decisions inside the devepolment process of a REAL and awesome product.

        Seriously, I believe it will work out!

        I also ask anyone significant (in Mozilla team) to speak out this suggestion on any next meeting, maybe that suggestion will get positive feedback…

        • Oh, forgot to mention, that users should be able to give multiple votes for the same bug, so the amount of votes would play a role of some kind of money here.

          And by the way about money: if that system gets a life, you may built into bugzilla the ability to donate particular developers, so developers will be also motivated to fix the bugs. That kind of a rewarding system may even bring some new 3rd party developers, who’ll be just bounty hunting for real money.

          You may ask me: “why can’t users donate now? there are a lot of possibilities to donate!”
          The answer is simple: users may have HUGE interests in fixing particular bugs. In my case donations will stimulate fixing the real and _particular_ bugs.

          And I’d like to immediately criticize my idea:
          1. That may lead to the wrong system of motivation. Developers become interested in having a buggy product, as new bugs mean more money. (+ there will be “unpopular” bugs that will live for years, but I think it’s better than having “popular” bugs living for years, like it is now).
          2. Mozilla is a non-profit organization and I don’t know how should be organized the donations in that case (where to send money in case the bug was fixed by a 3rd party developer? how to pay taxes in that case? etc.).

          So I’m not completely sure if the idea with money is good.
          But maybe some of it’s parts may be used: for example, MoFo may create an internal rewarding system, to reward most disruptive traigers/developers. And in case they want to “thank” some 3rd party contributor, who fixed some top-wanted bug which had some bounty reward – they can just use the donated money to thank this 3rd party contributor by sending him T-shirts/baseball caps/etc. with Mozilla/Firefox/BMO logos.

          • And to explicate a bit more: votes should also be given to users for their donations. So rich users could actually specify the bugs they want to get fixed immediately. If they are willing to pay for this – why not?

  5. I’m a web developer whose been trying to fix as many bugs as possible (just cause i get board working on work stuff) and i find it hard to identify easy bugs vs very hard bugs without going 25+minutes into the code to find out. Any way to make that easier might make the little bugs easier to fix and knock those ones out.

  6. What I did yesterday was look at the backlog of unconfirmed bugs that I have made in the past. Some were probably over two years old, from when I first started using Bugzilla. Now that I have a lot more experience with bugzilla and the development of everything at Mozilla, I was able to mark many of the bugs as duplicates, invalid, etc.

    If we could ask everyone to look at their old unconfirmed bugs and determine which ones could be resolved, I think we could cut unconfirmed bugs by maybe even 10%. Even if it’s not that much, I’m sure it would be a big help.

    • Possibly an automated email to everyone who has old bugs on BMO saying “Bug 123456, 234567 and 345678 haven’t been tested in X days. Please go through and retest your bugs, close them as appropriate, or mark that they are still valid. Those bugs that don’t have a response within X days will be auto-closed.”

      • Gerv did something like that some years ago (in 2005 maybe?). A first comment on the bug, then a little later if nothing else had changed in the bug, RESOLVED EXPIRED. A few old bugs were cleaned out but AFAICT it didn’t have any lasting effect.

  7. I’m gonna sound like a heretic here. I view user-filed bug reports as kind of a statistical thing. Not every one will be triaged. Not every one will be fixed. I don’t see that changing. Nonetheless, big problems are likely to bubble up to the surface and be addressed anyway.

    It’s unfortunate that this isn’t clear to individual filers, who might be puzzled/offended if their bug report isn’t acted on.

  8. Boo-hoo. A 5 year old bug that’s likely existed since the Netscape days (and, AFAIK, can’t be triggered through stock Firefox). In all that time, 3 people have reported it, the Firefox project got started, and half a billion people have carried on. Yes, it is absolutely is something to fix. But it’s also an extremely minor bug with very low impact.

    Any large software project will build up a backlog of of bugs. If you know of such a project that hasn’t, I’ve love to hear about it so we can figure out how they do it.

    Anyway, I think the worst thing we could do right now is to have the entire community sink a whole week into Bugzilla. [Weeks, really, if you want a significant impact.] Because after that we’d be right back where we started in no time. It’s not the first attempt we (or I) have done towards cleaning up Bugzilla, only to be overwhelmed again by an unsustainable incoming rate… I really liked Tyler Downer’s blog post ideas for improving things. That would be a fine start. Let’s fix that, and then look at what to do with crufty old bugs.

    • That’s a really sad attitude to have. My point was simply that if the bug is not going to be fixed, close it. What’s the point of having open bugs if they don’t matter? It doesn’t benefit anyone.

      And the reason that no one reports the bug is because they see the bug mentioned in the documentation for confirmEx and don’t report it again.

      It is a bug. Either fix it or mark it as WONTFIX. Is there another option I’m missing?

      Or heck, close every single bug in Bugzilla that’s over a year old and be done with it.

      • A bug should absolutely stay open if it describes a real issue and module owners agree should be fixed. That’s _completely_ orthogonal to the issue of if there are resources to fix it.

        Why do I get the feeling that if we did close bugs like that, you’d just end up complaining that people can’t find open issues or that Mozilla is trying to hide bugs?

        • A bug should absolutely stay open if it describes a real issue and module owners agree should be fixed. That’s _completely_ orthogonal to the issue of if there are resources to fix it.

          +1

    • I agree with Mike, the bug isn’t serving any useful purpose by being open, and if it is (like if it is a nice place to mark dupes to), then it needs to be fixed. There is no reason to have bugs left open and clutter search results up.

      I also agree that right now, putting a week or two into cleaning BMO up won’t have a long term impact. But, if we can take a serious look at improving triage, hiring more people, making guidelines that everyone abides by, Then go and have a cleanup week, it will have a significant long-term benefit.

      • I agree completely with Dolske: we have thousands of things we could work on. Having an open bug for these things is reasonable. But most of these things are not important enough to our core community to make them active priorities. So they should remain as open bugs (an indication that it is a change that the module owner would accept).

        This is obviously different from the UNCONFIRMED bugs issue. I feel like njn may be right about the statistics there, but I agree that there is an issue with outside perception thinking that we’re actively going to deal with each incoming bug when we don’t currently have a plan to do that.

        • “Dealing with” doesn’t necessarily mean fixing. A bug RESOLVED WONTFIX or INVALID is certainly “dealt with”, and so are INCOMPLETE bugs as long as the reporter doesn’t make the problem clear enough to determine if the latest nightly suffers from it or not, and in what manner.

          Every time I see a newly reported bug in SeaMonkey::General, I try to “deal with it” in some way, even if it is by as little as deciding whether it belongs in MailNews Core or in Toolkit.

    • Ken, I am actually talking to a few other triagers with other companies, and it turns out that everyone seems to to have the same problem more or less. There were a few comments to that degree on my blog a few days back.

  9. Hey Mike. I’ve had some conversations with Tyler and I’ve also been involved with bug-tracking in the Mozilla community since 2001. I actually know pretty well exactly how Mozilla could truly fix this problem. I’d be happy to come and give a talk on it at Mozilla HQ sometime; I live in Mountain View, so it wouldn’t be that hard.

    -Max

    • Max, I’m curious on what your solution for the problem is. Would you mind filling in the details? (It may be something we talked about in the past, if so just refresh me).

      • Hey Tyler. It’s not “a solution,” it’s not some one magical thing that will resolve the whole problem. It’s just a series of practices and some discussions to be had about the system that I don’t really want to write up in a blog comment here, but I’d be happy to give a presentation on in person.

  10. I see quite a lot of bug verification going on (marking FIXED bugs VERIFIED). Maybe all that effort would be better spent triaging unfixed bugs?

      • I agree, but I wouldn’t totally rule out that valid bugs can be found in the verification process. Rarely do verifiers reopen bugs anymore, they usually file a new one for a special case or subset of the original, larger issue–unless things are so severely broken that the whole issue needs to be revised and revisited.

        But there would likely be little harm in asking all or most verifiers to focus on open bug triage for a week and see what happens. The only real danger in that is that many of them may not have a lot of background in the more involved work of open bug triage–though having more helping hands will certainly make any solution be realized more quickly!