Mozilla 0.9.6 Released
Tuesday November 20th, 2001
mozilla.org today made available for download binaries of the Mozilla 0.9.6 Milestone. The builds are available on the releases page, or you can get them directly from the ftp site.
#160 Re: Where it's at
Friday November 23rd, 2001 12:35 PM
You are replying to this message
> When nearly 200 individual developers set target milestones indicating that they'd like to fix some set of bugs for a particular milestone and don't get 100 percent of them fixed then our project is in jeopardy?
Yes, I am saying you should treat any significant shortfalls in bug fixing for milestones as a risk to the project. It presents quite a few risks, actually, but probably the most important one is that it may be concentrating harder bugs into later milestones through a natural selection process by which the easier ones get whacked while later ones get pushed off. This multiplies the risks late in the project, since harder bugs are more likely to be destabilizing to fix. Another problem is that routine shortfall has a negative psychological effect, leading developers to view targets as fluid rather than as commitments. There are other problems, with which a seasoned project manager would already have experience.
The rest of the message mostly just says "trust me, I know things are going great." You've been saying that for years now. Even situations which in retrospect you admit were problematic, like the crash landings, you insisted were perfectly all right at the time. You're like the boy who wouldn't cry wolf if his leg was already halfway down the beast's gullet. You've positioned yourself as a cheerleader and that makes it hard to take your "trust me" very seriously.
> You can talk about the rising lines and the different curves but you won't convince me that we're not getting better all the time.
Everyone acknowledges that the browser is getting better, so this is a straw man argument on your part. The question is not whether it is improving but whether it is improving in a way that will allow a release of reasonable quality within a reasonable time. Because the bug database is not under control, and because the number of outstanding bugs is still large and getting larger, and because the developers don't meet their bug fixing targets, there's no way to know for sure whether it is on track or not, but that confusion in itself suggests a high risk situation.
> I've got the mtbf numbers, the user reviews and comments and the direct experience to back that up.
I'll take those three things in order. I've discussed with you before why MTBF is a weak quality metric. A system could never crash and yet still be completely unusable -- if you drove the MTBF number to infinity, you could still have a very poor quality product. Crashes are just one class of defect. You never responded to that, and you never responded to my request for links to the MTBF numbers so I could see them for myself. (It later turned out there aren't even any real MTBF numbers for my platform, the Mac, because the talkback program isn't usually installed on that platform.) As for user reviews, they remain very mixed, and I've documented in the past how mixed or negative reviews have been falsely presented as positive here. Your direct experience does not carry a lot of weight with me because of your relentless boosterism, and it also is not a good metric of the stability risk presented by outstanding bugs.
That the browser is improving does not mean that it will be of sufficient quality to ship in a successful way any time soon. There are 3560 bugs known to be targeted for releases between now and 1.0. If even one percent of those bugs causes the kind of crash landing that the obscure little bug about submitting non-displayed form elements did, which destabilized nightlies for over a week, then you have thirty-five weeks or more of seriously regressed builds to look forward to. I think one percent is likely to be a very optimistic estimate, since it does seem as if harder bugs may be building up, but even that would mean as much as eight months of extra delay, above and beyond ordinary bug fixing. And yes, this whole time, as a trend, the browser would be improving overall, but it wouldn't be shipping in a form that end users would accept.
Now, there are about nine thousand bugs not targeted for 1.0. Some of them have been reviewed, but you keep telling us that a lot of them are junk which hasn't been properly reviewed because you're short-handed. Any of these thousands of unreviewed bugs might turn out to be significant. Any of them might require crash landings. They form a tremendous pool of potential but untracked risk. What's the actual risk there? No one can say. Your experience that the browser is increasing in quality does not really bear on the question of how many of the tracked and untracked bugs will cause this kind of destabilizing impact. Recent experiences and the disturbing "reopened" line, though, suggest that the potential for further problems of this type is very real.
Beyond the raw number, one has to look at the line slopes. The open and assigned line is continuing to go up, which means that reviewed and validated bugs are continuing to be reported faster than they're being fixed. That means that at this rate, at the time you expect to be able to ship 1.0, there will actually be more reported bugs than there are now. The risk of crash landings will actually have increased, and you will be looking at shipping a product with many thousands of outstanding bugs. End users will not accept a project that buggy. To get towards a decent quality level, this bug line must go down -- bugs must be fixed faster than they are reported to wind up with a product of reasonable quality. There's no way around it.