> Keep in mind that not every program is available on the daily basis as Mozilla is. > I'm sure that in the development process many programs have prominent features > broken by new changes -- we just don't get to SEE them because most companies > don't release their programs in such a way.

I'm going on the basis of my own professional experience over the last twenty years, which has included many years of experience in alpha and beta cycles of software with nightly builds. If major features are dropping in and out on a nightly basis, you're nowhere near the stability that will be required to ship the software. It's characteristic of pre-alpha or early-to-mid-alpha software. Beta software needs to be highly solid -- very few changes should cause regressions. Instead, Mozilla experiences changes that cause major regressions every two or three days at the most. That means current schedule estimates which predict only about another four months in beta are probably optimistic, since alpha is stretching over more than a year, and that some process changes will be required in order to get this very complex system to market at all.

> I do agree, though, that modificiations need to be evaluated THOROUGHLY before being checked in.

Which, of course, is very hard to do for software with such a huge feature set. Developers don't have time to check each of a hundred web pages and a hundred user-accessible features for every change. This is where automatic testing could become critical. Changes could be made on a submission basis, and then run through automatic testing before being approved for checkin. Of course, this itself will take time and money -- there would need to be a fairly large server farm, or else the automatic testing would become a process bottleneck, and creation of the automated tests would be labor-intensive. But without it, the software may never attain stability.

> Another thing -- you? Too positive? ;-)

Sometimes, yes. I was recently quite conciliatory on the issue of the software having made a lot of progress, based on my use of the 0.9.6 milestone build, and I was willing to go along with Asa on the issue of forward progress being constant. When people in the forums started observing, almost unanimously, that the builds had been crap for weeks, then I realized I was being overly positive. Mozilla seems to take one step back for every two steps forward, and the ratio may be worse than that. Destabilizing changes that set back the program for weeks still seem to happen once or twice a month, and there's no apparent end in sight to these "crash landings." That's not really a recipe for forward progress.

Again, please bear in mind that I am noting these risks and raising these issues because I think mitigating the risks and addressing the issues is critical to the success of this program. It is unfortunate that some people seem to be able to see expressed concerns and potential solutions only as nay-saying. This sort of feedback is critical to the success of any large software program. Cheerleading approaches that casually dismiss concerns and risks lead to massive crash-and-burns like Apple's Copland -- which during its late builds, was in a similar state of stability as Mozilla is today.

> I'm pretty sure when 1.0 arrives there won't be many very critical > bugs which weren't reported before unlike many closed source > projects. on thing worries me though. it's that the bug count doesn't decrease :( i've seen some posts saying that mozilla team fixes 1000-1500 bugs in every milestone, but i guess most of these bugs are regressions.

i like the latest nightly builds on my laptop :)

> Yet you seem to think that your analysis of Mozilla progress is somehow more accurate than people who are intimately involved with Mozilla on a daily basis.

Nope, I'm basing my conclusions very closely on tyhe reports of people who do run the nightly builds. They report that the nightlies have been extremely unstable all month, and almost every day, they have reported major features dropping in and out. I'm very familiar from my software development career with what that means about the overall stability of a project.

As for you, I'm getting more and more sure that the loudest advocates of the Mozilla open source process have never before been involved with shipping a large software project, and do not have the background or training to understand what they are seeing.

> You have frequently suggested a need for root cause analysis, but you have never to my knowledge explained what you think it would accomplish.

Root cause analysis is a standard quality technique which applies equally well to any project, and is not confined to software. It is an attempt to analyze problems to find out which components of the overall system they have in common. Once a set of data on problems is built up, it becomes possible to find through statistical methods which components are tending to cause the most problems. The reasons for the instability caused by that component can then be analyzed in detail by QA engineers, and they can formulate proposals for changes to prevent or minimize further problems from that component.

> BTW, nightly builds are part of the live debugging process and thus are not expected to be stable on a regular basis.

Again, that's just not how it works. If new checkins break existing features on a regular basis, there's a serious problem with system fragility. A system of this type will probably never achieve the stability needed to ship. The problem can't be solved piecemeal by fixing more bugs, since every bug fix carries with it a large probability of creating a new problem. Fragility has to be solved on a system level.

Maybe we can create a project to develop a new Netscape jacket. It may be flawed and require years of development but it will not be too small because it will grow and grow until it is the biggest jacket ever. Maybe we can add chat features and other things not necessary in a jacket. Maybe we can make something that can be used to form any article of clothing anybody could want. Then we can add themes so people can make our jacket look like other peoples' jackets. People may criticize us at sites like jacketquest.com but we can retaliate at jacketquestquest.com. If we are unable to turn our article into an actual jacket then we can just call it a beta version and sell the jacket to AOL.

Regressions are just a fact of life at all stages of development in a software project of this size.