DOM Level 2 Conformance TestMonday November 20th, 2000Bob Clary writes, "I have written a DOM Core Level 2 Conformance test which can be run with both Mozilla/Netscape 6 and Internet Explorer 5.5. It uses only the DOM methods and does not depend upon the HTMLDocument or the XML Parser. The test coverage is not yet complete but is nevertheless extensive. I think you will find the results interesting. "The test generates two sets of reports: one by test case and one by API name. It is completely interactive and contains links to the relevant parts of the DOM Core 2 Recommendation as well as links to bug reports in Bugzilla. Of course there are no links to reports of Internet Explorer bugs available. "If you run all of the tests it will use 50-60M and may take several minutes depending on your processor so be patient. If you do not have enough memory to run all of the tests at once, you can choose a subset of the test cases to run. "If you have any problems, questions or suggestions please feel free to contact me." UPDATE: The host for the conformance test is taking down the site due to heavy bandwidth usage. We'll let you know when the site's back online. That isn't a very good abstract of the situation, because the conclusions were not revealed. What are they? I looked at the test results in both browsers; it seemed that both Mozilla (from the 16th) and IE 5.5 were reasonably compliant, but with a significant number of flaws. IE seemed to be slightly more compliant. There were no 'overall' numbers ("of 1000 tests, 650 were passed")... that's the kind of information we need to make sense of this. (Obviously the specific information about which methods don't work is extremely useful to JS programmers too.) Anybody counted the results or something? --sam I left it to you to make the comparisons. I will give a brief summary of my opinion if you wish. Essentially it appears that IE 5.5 (on Win32 at least) supports a good deal of the DOM 1 Recommendation which is what is claimed from their hasFeature. They do not implement DOMException at all and throw their own Exceptions which accounts for their failures of the Exception tests. In addition, IE does not implement portions of the DOM API which in practice are not used. For example, Comment does not have children and IE's support for that portion of Comment's interface is lacking. Mozilla supports the DOM 1 and the DOM 2 Core to a greater extent than IE 5.5 on Win32. It has not implemented all of the Namespace related methods however and has other Namespace related problems. Both IE and Mozilla report null strings instead of null values for several attributes. As to your comment that there are no "overall" numbers and no "specific" information about the methods, I point you to the first row in the API report where the over all numbers are reported. Each test case is listed where you can click on the link and view the entire test case and it's results. In addition, clicking on the + sign in from of the API Names in the API report will display a list of specific tests which can be displayed for that API Name. Links to the DOM Core 2 Recommendation as well as links to Bugzilla are provided as well. This was described in the documentation of the Conformance tests in general and specicially in the DOM Core 2 tests. Was the explanation not clear ? Do you have suggestions on what additions I can make to make this clearer? Bob
Bob, great work; this obviously took a _great_ deal of work to create, and it shows that you really care about web standards. Thanx for this great material! Brad Neuberg Good luck with your project, it shows that some people out there really do care about making the internet as accessable and as user-friendly as possible. No one knows this stuff yet. How did you learn it? What kinda of references did you use to know which objects existed? The DOM spec is scary as hell :) Thanks, Mike Mostly I used the W3C Docs. The first thing I did ( a long time ago in the spring ) was to implement a version of the DOM 2 Core entirely in Javascript! It really helped to understand what was going on. From there I built a parser in Javascript. I don't necessarily recommend this approach however. ;) My goal was (is) to build a cross browser middle level API to the DOM that can be used transparently by Mozilla, IE or other browsers that come close enough to supporting the DOM. To do that I needed to understand not only the DOM but how well Mozilla and IE support it. Thus the conformance testing. What you see is still an unfinished project, but one that has gone through many stages, redesigns and rewrites. Several people have made suggestions on improving it. The change from the *instance* view of testing to the *api* view of testing is one such change. As for the W3C DOM Documentation, they aren't so bad when you get used to them. They describe the objects (interfaces) and how to create them, how the methods and properties behave under every circumstance, essentially everything you need to know. I really wish I could document my code to the same standard of excellance. Basically all you need is a Document to begin with. This is the W3C DOM Core Document *not* the HTML Document. In Mozilla you can create a Document using the document.implementation.createDocument interface. In IE it is more difficult since their default document isn't really an HTMLDocument descended from Document. For IE, you must use the MSXML ActiveX control to create a Document. Once you have a Document (and therefore a DOMImplementation) you can create any instance of any object you need. Next, I found experimentation and trial and error to be quite rewarding. Hope that helps and thanks! Bob I must have missed those overall numbers - sorry. Kind of embarassing since I viewed it twice, once each browser... guess I looked at the bottom of the table, not the top. As for making it clearer - I think it's clear enough (especially the 'by method' bit) and will be a useful resource for DOM programmers who might happen to come across the bugs and incompatibilities. All I was really missing was the overall summary of current test results, rather like what you wrote here in fact - what the significant holes in compliance are for each browser. I think something like that would be useful, maybe for people who just want an 'executive summary' rather than to use it for programming. Admittedly, you do then run the risk of being seen as partisan, or having your tests used as nothing more than a pissing contest between the two major browsers. --sam i tested it in Netscape 6 and the result is; passed 2002, fail 306, skipped 1151, total 3459. os is Win2000. I did not test IE 5.5 coz I don't use IE. I didn't post the results before since I was hoping to stimulate a discussion, but will go ahead and give the totals for IE 5.5 on Win2k. Passed 1057, Failed 511, Skipped 1891, Total 3459 Comparing the totals is not that informative since a number of the tests are repeated for each of the test cases. For example IE's lack of support for DOMException, IE and Mozilla's problem with null strings vs. null values, Mozilla's problem with ownerDocument etc. The totals give a rough idea, but the true results are found by inspecting the test cases and individual API names for where each fails. Tests can be skipped due to a lack of support for the feature or due to the test not being appropriate for the context. For example, testing NamedNodeMap against a Comment's attributes property. Inspecting the individual tests should make it clear as to why a test was skipped. The basic idea was to provide developers with a means of testing the browsers and not to provide a review or comparitive summary. I felt that was better left to others... like the readers of mozillaZine! ;) I have tested too NS6 I have been seeing reported errors: most of them are minor errors in no common use case. I think NS6 is satisfactory DOM-2 compliant Examples: * Attr.insertBefore (9 errors) Is very unusual have Text children inside a Attr object Idem with Attr.removeChild and Attr.replaceChild * Properties returning "" against returning null * Document.appendChild Doesn't throw a exception when trying an anormal use * Document.createAttributeNS Doesn't implemented. HTML-only DOM implementations do not need to implement this method (OK XML is important today) * Many errors are due not thrown exceptions in no valid operations ...
Test Results IE5.5 (Windows 2000) Passed Failed Skipped Total 1057 525 1877 3459 NS6 Passed Failed Skipped Total 2002 306 1151 3459 -IE5.5 only pass 30% of tests -NS6 pass 57% -IE5.5 skips 54% of tests (100*1877/3459) because it doesn´t implement related objects and methods -NS6 skips only 33% -IE5.5 fails 50% of done tests (100*525/(1057+525)) -NS6 fails ONLY 15% (100*306/(2002 + 306)) No words. I have a *free* web site from my ISP that is bandwidth limited to 225M per month. In the past two days I have used 240M and my ISP is threatening to disable my site until December. It was still available this morning EDT, but I don't know for how much longer. I will try to make other arrangements for hosting the site as soon as possible. I apologise for not being able to handle the load. I thank you for the interest you have shown in the test and the comments you have made here. Bob Clary #14 Re: DOM 2 Core Conformance Test to be taken off lineby NIKITA Saturday November 25th, 2000 3:28 AM Why not host it on SourceForge? I'm sure they have the bandwidth. The W3C DOM Core Level 2 Conformance Test is now available at http://bcTestFrame.damowmow.com/. Thanks to Ian (Hixie) Hickson for hosting the test and thanks to all of the other people who offered to host it as well. Several people have expressed interest in the source for the tests. I am currently working on a more permanent hosting solution and intend to release the source under an open source license in the near future and possibly provide CVS access if that is possible. The test framework used in developing the DOM Core test suite is applicable to any Javascript API. The DOM HTML Level 2, DOM CSS Level 2, etc are all possibilities. Perhaps this test framework can provide the basis for an entire set of test suites for Browser Conformance. Wouldn't that be nice? Would you like to help? Thanks again and I hope you continue to find the tests useful. Please let me know if you have any problems. For example, Internet Explorer has problems running the tests through JunkBuster so just exclude the site using IE's connection settings. You can reach me by email at bob@bcTestFrame.damowmow.com or BobClary@Mindspring.com for the time being. ;) Bob Clary A failure rate of 10% may make for a B+ on a test at school, but for someone attempting to write a significant JavaScript-based web page, it means that there will be DOM problems every single hour of development. |