Bug 200000 Filed in Bugzilla
Monday March 31st, 2003
At 13:11 Pacific Time today, the 200,000th bug report was filed in mozilla.org's Bugzilla database. Bug 200000, reported by Chris Hofmann, represents another milestone in the Mozilla project and was coincidentally filed on the fifth anniversary of the release of the source code. Note that just because there are 200,000 bugs in the database, it does not mean that there are 200,000 bugs in Mozilla. Bug reports are used for everything from major crashes to spelling mistakes in alerts. Not all bug reports are for real bugs either: Bugzilla is also used for such varied purposes as feature requests and tracking the creation of new CVS accounts. In addition, bug reports don't go away when they are fixed and around half of all new bugs filed are either duplicates of existing reports or otherwise invalid.
#8 Re: I wonder...
Tuesday April 1st, 2003 8:51 AM
You are replying to this message
Although such a list would be marginally interesting, it would suffer from several problems.
First of all, the "open bugs" at any given point in time will not necessarily enumerate all of the crashes - you'd expect that some would remain hidden for some period of time until someone stumbled across them. I've seen it happen that even crashers in widely-used programs might remain hidden for years (!) until somebody happens to run into just the right sequence of events.
Secondly, this doesn't reflect the frequency or severity of each bug. If a crasher happens every time the program runs, that's obviously going to be more of a problem than if it only affects 1/100 of 1% of the users. (Unless perhaps the crasher only happens on shutdown, when it looks messy but might not affect anything else).
Thirdly, when a crash is discovered, it may take some considerable effort to determine exactly when the crash first became possible, which is necessary to know if you're going to try to retroactively figure out which bugs were "active" at each point in time. Incorrect code might be lying around for some time but the crasher might be inaccessible because program flow never set up a situation where the bug led to a crash. Perhaps the code was never executed or perhaps it was executed but didn't lead to a crash until something else was changed.
Fourthly, when you find a crash and do figure out which version(s) it applies to, you have to consider that in the nature of things you'll always have more experience with the older builds and so would have a more complete list of the crashers in the older versions. Crashes in the newer builds might not be known yet simply because of less experience with the build. This point is related to #1 but it isn't the same thing - #1 deals with crashes that might remain hidden for some time before being discovered, this one considers the fact that crashes that were introduced shortly before a particular build will be more likely to be known at any given point in time for older builds than for newer builds.
Some kind of MTBF (Mean Time Between Failures) measurement is better than the raw number of crashers but these can be difficult to obtain from users in the field - usually reliable numbers would only be possible from in-house testing.
Obtaining accurate metrics of this kind is not an easy problem.