MozillaZine

Tree Closes Tonight for Mozilla 0.9.9

Tuesday February 19th, 2002

mozilla.org will begin the process to release Mozilla 0.9.9 tonight at 11:59 pm pacific by closing the tree to free checkins. Mozilla 0.9.9 is the last major milestone prior to 1.0, and includes numerous bugfixes in composer, history, and other areas. Along with this, likely new features that will be in the milestone include a new full screen window mode, set image as wallpaper, and composer publishing.

Following the tree close tonight, expect up to a week of tree closure for stability and regression fixes, then a branch to be cut some time next week. As soon as the branch is cut, the trunk will reopen for Mozilla 1.0 checkins. mozilla.org is shooting for a March 1st release date, but due to this being the last milestone before 1.0, and because there is a large amount of code that will be landed tonight prior to tree close, delays may occur.


#137 Re: crash data link

by asa <asa@mozilla.org>

Monday February 25th, 2002 10:14 PM

You are replying to this message

I'll try to make it real simple for you.

Assume 4 users test a nightly build. They all use the builds for 6 hours or less (it's a nightly build and that seems a reasonable guess about how long they'll browse with it before getting the next nightly build). Four of them use builds that don't have talkback, they opt out of talkback in the install routine, or they decide to hit the "NO" button when talkback offers to send in the crash report. Two of them use builds that are talkback enabled and are willing to submit incidents if they arise so they're the only ones we can really talk about . One of the testers with a talkback build crashes once at startup and again a few hours later in the 6 hours he uses the build and he opts to click "OK" on the talkback reports that are generated, sending them in to our servers. The other person running the talkback build never crashes in the 6 hours of use so no talkback agent comes up and she has no talkback report to send in so we have no data on her 6 hours of uptime without a crash.

Now, what would talkback data tell us. It would tell us that we had an MTBF of 3 hours. Is that an accurate representation of the experience of our 4 users? No. Is it even an accurate representation of the two users that enabled talkback in their builds? No, it's not.

So what do we do about this, you ask (or should have)? It's pretty simple. We have Milestone releases where people use the builds for more than a day. See, if you replace your build you get a new talkback agent and no data from your previous build is ever sent in and calculated. This is why t's important that we get people using the browser for long periods of time since many people don't crash for long periods of time. So with a Milestone, in the first few days we get a lot of reports from the early crashers but no reports from the people that haven't yet crashed. Given enough time most of the testers will find a crash but it takes several weeks before we start seeing incidents from enough of the sample to generate meaningful MTBF stats. We gotta wait for all those people who don't get crashes in the first few hundred hours of browsing so we can average them in too. It simply wouldn't be fair to exclude the people with exceptional uptime unless you were willing to exclude the people with exceptional failure rates.

So to look at a daily MTBF number is to look at a small piece of the story which doesn't count the many testers that don't crash in the first day (and we know there are plenty based on the numbers of milestone talkback incidents where the first crash doesn't occurr till well after a day's worth of uptime). You're only seeing the people who did manage to crash before they updated their build. I'll bet that many here will attest to replacing a nightly build without having crashed it. I do this all the time and even your misunderstanding of talkback data and your misinformation in this forum only gives me a slight itch to to intentionally crash my browser right before I update to a new build just get my 14hours of crash-free browsing logged in the system. But I don't and there's no reason to play with the data like that just to spite a couple of people that don't know what they're talking about.

Daily talkback data is useful for identifying the begin and enddate of a new crash signature in the nightly builds. It helps us to predict some of the topcrash bugs that become more apparent when we have hundreds of thousands of talkback incidents for a single build rather than hundreds. Daily MTBF is so close to useless that it's not even worth analyzing. We don't have talkback in the daily builds for calculating MTBF. We have it for identifying the most common stack signatures and tackling the most visible crash bugs as soon as possible.

--Asa

I've put far too much time into trying to explain this to you. I'm taking a break. Feel free to speak freely from ignorance for a few days while I put my time into actually working on making Mozilla better.