| Compliance Advisor |
|---|
This page is a ``Call for Participation'' in the ``Compliance Advisor'' project. We give a brief overview of the project, suggest the development timeline, and request interested parties to contact us.
Nothing on this page is set in stone. This is just a proposal. The final project shape will be determined by project participants.
1. Compliance Advisor overview
1.1 Scope
1.2 Advising versus certifying
1.3 Test cases
1.4 Support and Development
1.5 Availability
1.6 Current Status
2. Timeline
3. Call for Participation
Compliance Advisor is a software package that tests compliance with Internet standards and good-practice norms. Our first (and possibly the last) target would be to test HTTP/1.1 compliance and related norms. Furthermore, we will start with the tests for various HTTP intermediaries such as caching proxies, origin server accelerators, and L7 switches.
We suggest to concentrate on HTTP because that protocol is the current foundation of the Web operation, and because HTTP specs are publicly available. We target HTTP intermediaries because, compared to the end-user agents and origin servers, intermediaries are both much easier to test and much harder to bypass (increasing the scope and side-effects of non-compliance bugs).
The Advisor will primarily target ``common-sense'' compliance rather than formal protocol compliance. The tool will test many HTTP MUSTs, but will also have tests that target good-practice and de-facto standard behavior. See the next section for more details. |
Achieving formal HTTP compliance is often impractical. Compliance Advisor points out potential bugs in particular HTTP implementation, but is not designed to certify product compliance. The tool will be primarily useful for developers, network administrators, and system integrators, not marketing departments.
The development of a formal compliance test is likely to become a victim of inter-vendor marketing conflicts and will be slowed down by various protocol committees that may want to define the scope and specifics of the tests. Instead, we suggest to allow for inclusion of virtually any practical test. The users will be able to specify a subset of test cases they want to apply. A failure or success of a particular test is FYI only. That is, Advisor simply points out that some requirements (either protocol-based or third-party-invented) have been violated or met.
The ``advisory'' mode will also make competitive use of the tool difficult. There probably be no industry-wide compliance competitions, but we can held non-competitive compliance test-offs. Reduced competition may ease marketing pressure on commercial vendors and allow for better cooperation of all parties.
As indicated above, we suggest to allow for inclusion of virtually any practical test. That is, the Advisor will work with cases that its users find ``interesting''. Most likely candidates include many of HTTP's MUSTs as well as tests that check for well-known HTTP pitfalls. |
Some of the tests may look like the ones proposed at http://www.mnot.net/http_compliance/list.
A ``core group'' will be built from vendors and other interested parties. The members make major test design decisions and get access to early software releases. Cumulative membership fee should cover the project needs for the first 6 months with a commitment of further support. Commercial vendors, working groups, various committees and consortiums are all welcome to join the core group.
The Measurement Factory owns the tool and is responsible for software development and support. We will coordinate the activities and may organize face-to-face meetings if needed. We have the resources and expertise necessary to build a decent compliance test suite.
As many of your know, we prefer to be very open about our work and enjoy input from the community. We hope to preserve that style while working on Compliance Advisor.
Compliance Advisor will be available to core members at no additional charge, in source code, including early/beta releases.
Given core group consensus, the tool is likely to be available free of charge for non-commercial purposes.
Commercial licenses will be available at a price that core
members are comfortable with. Since core members take the initial
risk of financing the software, we need to make sure their
competitors are not getting an advantage of them (i.e., that the
members are not sponsoring their competitors).
1.6 Current Status
TMF is currently evaluating a compliance tester provided to us by one of the caching vendors. The tool currently contains 256 test cases and is designed to work with caching proxies. While more work is needed to make the tool publicly available, we believe that it can work as a prototype and can help to bootstrap the development.
Here is the ``totals'' statistics of the prototype application to Squid caching proxy:
Passed Tests : 93 Failed Tests : 163 Aborted Tests : 0 Total Tests : 256Note that many of the failed tests are due to compatibility problems and other bugs. As we said, the tests are not ready for production use yet.
We propose the following schedule.
- January - February
A core support group is built. Overall tool design is discussed and agreed upon. Membership fees get collected. A working tool prototype is made available to the members. - February
First software releases are made publicly available. - March - April
Stable software version is released. - April
First round of non-competitive tests can be held if there is sufficient interest.
To summarize, we want to hear from all interested parties who are ready to form the core group and seed the development. Given sufficient support, we should be ready to produce first results very soon.
Please contact us, preferably by e-mail.