So How Does Mozilla Development Work?

As most of you all know, mozilla.org makes "builds" from the Mozilla source code several times a day. These daily (or nightly) builds are the result of a "pull" of sourcecode from the server cvs.mozilla.org. Regular nightly builds are pulled from the "tip" of the CVS "trunk". This means that these builds include all of the most recent changes "checked in" to the CVS tip. We stop development each morning and do these builds to test the previous day's changes, verify that check-ins actually fixed bugs and identify any new problems created in the last day's work.

About every 5 weeks drivers@mozilla.org steps in to moderate risk for a few days on the development trunk and works to create a short-lived code "branch" from the trunk. All check-ins to the trunk during this "freeze" require approval from drivers@mozilla.org. As soon as we have a build that is stable and we have a handle on the handful of bugs that we've identified as most important to the Milestone, a branch is created. This code branch can be thought of as an independent copy of the code on the trunk at the moment of branching. As soon as the branch is "cut", the trunk is opened to development so developers no longer have to ask drivers for approval to check in changes. A few developers working with drivers@mozilla.org try to get fixes for the Milestone show-stopper bugs and get them checked in to the branch. During this work the majority of developers have returned to focus on the tip of the trunk. When drivers are satisfied that the Milestone branch is ready for consumption, a release tag is made on that branch. This tag is a marker on all of the files that need to be used in making the Milestone build. A build is made from that tag and released and the few people that were working on that Milestone branch return to work on the trunk. And that's the basics of Mozilla trunk and branch development.

How Does All This Relate To Commercial Releases?

A vendor can do work in a couple of ways, in private or in public. Both processes have advantages and disadvantages.

In the private process a vendor can pull a tree from cvs.mozilla.org, do work on this tree behind closed doors and (to meet the requirements of the code licenses) make available the changes to Mozilla files when they release their product. I think that the primary advantage to this kind of development is that vendors can use their own development process rather than mozilla.org's. This gives them the freedom to make changes that might not be allowed into the Mozilla codebase. There are several disadvantage of this method. First, the vendor does not get to use mozilla.org resources like tinderbox, Mozilla nightly builds, and community testing. Second, the vendor may make changes to its code that make it incompatible with the Mozilla trunk. This makes it difficult for the vendor to use fixes that happen on the trunk and it makes it difficult for mozilla.org to use fixes that that happen in the vendor's tree. Vendors use this method to varying degrees. Some vendors do this "fork" with each Mozilla Milestone, or as frequently as possible, trying to keep their tree similar to the Mozilla tree. This makes it easier to share fixes between the two code bases. OEone's Penzilla development is a good example of this kind of fork. They worked with code from an earlier Milestone for a while and then merged that work onto a more recent Milestone code tree. Other vendors do a one-time fork and take the code off in radical directions. Active State's Komodo development is a good example of this kind of fork. In either of these methods the vendor is required to make any changes to Mozilla files publicly available when they release their product. When forking is extreme and the vendor makes radical changes to files it can make integrating desirable changes back into Mozilla very difficult and sometimes impossible.

In the public process a vendor makes its changes in the Mozilla tree. These changes must go through the mozilla.org review, super-review, and during freezes, approval processes. These changes can be landed on the Mozilla trunk or on a Mozilla branch. Either way the change is made in the publicly available cvs.mozilla.org repository. Mozilla Milestones branches are appealing for vendors because some of the stabilization work has already happened on the branch. This makes it easier for a vendor to do a stable commercial release from a branch. Some vendors take code at the Mozilla Milestone release tag, make a few minor modifications locally and release a distribution. A good example of this is Beonex Communicator. Other vendors need to make a lot of changes or several far-reaching changes so they continue work in public on the Mozilla Milestone branch after mozilla.org is finished with that branch. An example of this method is Netscape's work on Netscape 6.1. Netscape continued to check into the Mozilla 0.9.2 public CVS branch after mozilla.org was finished with the Milestone. This development effort on the branch eventually resulted in the 0.9.2.1 CVS tag, which was the base for the Netscape commercial release. The first major benefit from this development model is that vendor's changes are publicly available before the vendor release and the changes are progressively (much easier) integrated into the public branch. The second benefit is that other vendors can participate and take advantage of the improvements made to the branch in their own releases. For example, Red Hat was able to include a build from 0.9.2.1 (Mozilla tag corollary to Netscape 6.1) in one of it's its release. Releases from this tag were considerably more stable and polished than the earlier 0.9.2 Milestone.

So What's Netscape Doing?

Netscape continues to participate in this second and more public development process. They will have developers working on both the Mozilla trunk and the 0.9.4 branch. Netscape could just as well grab the 0.9.4 tag and do all that work on it's own tree out of sight (a small fork) but that would be a loss to Mozilla. If they did a private fork it would be painful work for mozilla.org to merge any changes to Mozilla files back onto the 0.9.4 branch making them useful to other vendors and developers. They would be required to make those changes available but the additional work would probably mean that the changes wouldn't get integrated back into cvs.mozilla.org. In addition to the vendors like RedHat and OEone, there are other developers that would like to be able to build plug-ins and other add-ons like themes or sidebars that will work with major commercial releases. Having access to a Mozilla tag that matches the Mozilla base in that commercial release makes much of this development easier.

The Mozilla licenses and development processes allow for many different approaches to building Mozilla-based products. Keeping track of bugs and development work on multiple branches can be difficult and our processes and tools will have to evolve to support this kind of development, but is a clear win for the community of developers and vendors when this work happens in public and in cvs.mozilla.org. Netscape has a commercial need to do stabilization work on Mozilla code before releasing Netscape 6.x and they have and continue to do as much of that work as possible on publicly available branches.

I Have More Questions

I hope that this helps to explain some of what's going on with Mozilla branches. If you have questions feel free to post them and I'll do what I can to answer.