Update on 0.9.4
Friday September 7th, 2001
We asked Asa for a quick status on 0.9.4's status, which was for release this weekend, and here's what he had to say:
#103 Re: you obviously haven't been paying attention
Tuesday September 11th, 2001 8:21 PM
You are replying to this message
I have been paying attention. The project has run way short on bug-fixing goals with most milestones. Even after a mass postponing of targeted bugs to the next milestone, the milestone is often closed with more than a hundred targeted bugs still unfixed. Saying this isn't a problem is just not right. No one expects every targeted bug to be fixed, but the documented performance is not even close.
Number of blockers is not a good quality metric since the bug queue is largely unevaluated. If duplicates can't even be pulled out, and there are thousands of bad bugs in the queue (as you keep telling us when you say the bug queue doesn't matter), that means no one knows how many bugs should have been considered blockers, because many of the bugs have not been evaluated yet.
Average severity of incoming bugs is a good but limited metric. It's limited because, again, incoming bugs are not checked, so their severities aren't reliable; and because bugs need to drop across all severity levels, not just the most serious, in order to create a professional-quality product. A product with no crashers but thousands of cosmetic problems is still a low-quality product.
Mean time between failures, where failures means crashes that were reported, is not a good metric at all. The system shouldn't even be crashing often enough to make this a relevant number once you get to beta. It's a very blunt metric that only covers a certain kind of defect. You could take it to infinity without having a shippable product.
(Incidentally, where are these metrics listed? I just searched for mean time between failures on mozilla.org, and went through a ton of QA FAQs and related documentation, without finding them.)
Again, citing a bunch of "products" that have gone nowhere in the market is irrelevant. How satisfied do you think Netscape is with the ~0% market share on Netscape 6.x? The fact is that no Mozilla-based product has yet been taken up by end users in any significant number. The reason for that is that Mozilla isn't ready yet. You acknowledge that yourself by declining to vet any version so far as 1.0.
I've discussed regression rate before. Try this: <http://bugzilla.mozilla.o…&links=1&banner=1>
QA is not fun. It's hard work. Your QA process, which is based on the proposition that QA is fun, is failing. This is demonstrated by the fact that by your own description, you can't evaluate your defect curve because there aren't enough people to look at the incoming bugs.
You could fix that by having AOL throw a little money and a few resources at the problem -- by my estimate it would take about three months and five hundred thousand dollars to get the bug queue to a reasonably evaluated state. Not a huge expenditure for a critical task for a major software project. But you can't even consider that maybe the open source method isn't working here. That's an article of faith that must never even be questioned.
We did pay to fix bugs in Mozilla. Lots of them. We got absolutely nothing out of it, because Mozilla support is still, a year later, totally irrelevant to the market. Dirty little secret about open source: it subsists on handouts, and gives nothing back. More and more companies are finding that out. IBM may finally find a way to do something with the model, but no one else has. All open source companies except a few mom-n-pop consultancies are losing money. And again, there's only so long AOL is going to continue to fund you if Netscape 6.x doesn't start to gain some market share. It's expensive to pay for all those developers and AOL is in an ongoing cost-cutting regime.