This page discusses possible licensing terms. The actual terms are formulated elsewhere.
1. Principles
2. Current practice
2.1 The purpose of the certification
2.2 Eligibility requirements
2.3 Enforcing the requirements
2.4 Violations detection
2.5 Other
3. Choices to make
3.1 The purpose of the certification
3.2 Eligibility requirements
3.3 Enforcing the requirements
3.4 Violations detection
3.5 Other
4. Putting it all together
We believe that the usage terms must be based on the following principles
In a market where not all products are tested, and where new products appear quite often, a wider use of the certification mark increases both the value of public tests and the value of the mark itself, benefiting all test participants. In this environment, a product carrying the certification mark should appeal to a customer as a ``better'' product, all other factors being equal.
To design Polygraph certification program, we have surveyed a few existing programs with similar principles:
We were pleased to find a lot of common terms in the existing certification programs, many of which can be adopted for Polygraph certification. We will use these ``best practices'' to illustrate our choices below.
All certification programs have a clear ``big-picture'' principle: In customers' perception, certified products or companies are ``better'' than not certified ones. Thus, the certification is attractive to vendors as a way to gain market advantage over uncertified competitors.
We believe that, in the context of our discussion, two distinct certification purposes make sense. The certification confirms (and relays to the user) one of the two statements:
The distinction between the two is paramount for the certification program design. Assuming that known characteristics are better than unknown ones, both approaches imply a ``better'' product.
The two statements above are independent; one does not, in general, imply the other. ``Better'' characteristics do not imply that the characteristics are ``known'': a customer may have little idea what chemicals comprise an organic tomato, but the same customer knows that the organic tomato is better for her health than a traditionally grown vegetable. Same argument goes for the ``certified for MS Windows'' mark.
On the other hand, a PC with an Intel processor may be slower than a PC with an AMD chip, but it is better for the customer to know that a given PC has an Intel processor than to have to guess what processor it actually has. Similarly, a UL certified product does not necessarily have better characteristics than its uncertified counterpart, but a customer benefits from knowing the list of components of a UL certified product.
In practice, the certification program can poses both qualities, of course.
All surveyed certification programs have similar eligibility requirements. A product must pass a test or inspection before it can be awarded a certificate. For the purpose of our discussion, let's treat ``inspections'' as ``tests''.
While basic eligibility requirements are clear, eligibility still presents a major design challenge. In real life, products change: bugs get fixed, features are added, hardware upgraded, prices adjusted. The product that passed a test a few month ago, may look and perform differently now even if sold under the same name.
Most certification programs we surveyed allow minor revisions of certified products to carry the certification mark. While several licenses make an attempt to define ``minor revision'' precisely, the practical meaning usually boils down to an informal ``appears to be similar and would pass the certification with the same results'' approach.
Note that some licensing criteria and tests are quite rigorous and expensive. For example, Microsoft certification program includes a series of tough tests that may take a few weeks to complete at $40,000 - $60,000 price tag. Clearly, such a program has to allow minor variations of certified products to carry the certification mark.
[Ed.Note: Interestingly enough, the current revision of the Microsoft certification program was made significantly tougher and noticeably more expensive following vendor complaints that the first version made too many vendors eligible. On the other hand, Microsoft has enough power to enforce its judgment calls regarding ``minor'' versus ``major'' revisions; something that would be very difficult to do in a consensus-driven environment that TMF fosters. ]
An example of the case where the vendor is certified (based on a few sample tests and general criteria) is the ``CNET Certified Merchant'' program. CNET certification demonstrates that even a basic requirements are sometimes met only by a relatively small group of merchants, making the certification worthwhile for those companies.
Virtually all certification programs surveyed have surprisingly similar and relatively powerful enforcement clauses. After the certification is issued, typical licensing terms require
Some licenses (e.g., UL) even require random checks of the product manufacturing facilities. The decision to perform a check is usually made by the license provider and the provider alone. The costs of random tests are usually not covered by the licensee.
No explicit provisions are made regarding making the results of the unannounced tests public. This ``secrecy'' is, however, natural because the results of the tests are usually not detailed enough and/or useful enough for an average customer (just seeing the certification logo is sufficient). For example, most grocery shoppers do not care how many phosphates are in an organic apple; they are just happy to see the ``organic'' label that indicates that the product is ``healthier'' (under current medical assumptions).
If a random or on-demand check fails, the product usually loses the privilege of carrying the certification mark.
Surveyed certification programs are usually vague when it comes to methods for discovering license violations. One common requirement is for a licensee to report all suspected violations of the license by the third party to the license provider.
Many licenses require that details of the certified product is reported on the license provider Web site. This requirement, were applicable, makes it relatively easy for a customer to verify if a particular product is indeed certified.
Most certification licenses have a limited term, typically 2-5 years. Certification logo distribution requirements are nearly identical for all programs surveyed: no or very limited modifications of original art are allowed.
This section identifies the choices that we have to make in order to build the Polygraph certification program. As usual, no existing certification license can meet our requirements completely. However, in most cases we do not need to invent new approaches, we just need to pick-and-choose from the most applicable ideas available in existing certification programs.
Since this is a discussion document, we will enumerate possible choices, but will not make the final decision in most cases. For the specific proposed terms, look elsewhere.
It is important to note that the choices below are, unfortunately but naturally, interconnected. To better understand what each option means and to make judgment calls, consider reading the entire section first.
We have seen that a ``better'' product may mean ``better characteristics'' and/or ``known characteristics''. It is our strong opinion that Polygraph should stay away from certifying the relative ``quality'' of tested products.
PolyMix-3 results are based primarily on performance. While pricing information is reported and basic HTTP compliance is assumed, Polygraph does not and cannot test many other factors that affect overall product quality. Moreover, ``better'' is often impossible to define performance-wise because performance has many orthogonal measurements. While we can say that product A has better throughput that product B, we should not be in business of saying that product A's overall performance is better than B's.
Finally, it would be wrong to say that a product that passed a PolyMix test is always better, performance-wise, than a product that was never tested under PolyMix. We can only speculate what PolyMix performance of untested products is.
We believe that Polygraph certification should make major product characteristics ``known'' or ``open''. In other words, we should not judge product characteristics, we should report them.
An alternative would be to claim that PolyMix-3 certification is a sign of ``good'' product quality rather than just open information about the product.
Our eligibility-related choices are as follows.
The first option is more-or-less straightforward. It is relatively easy to define an ``exact clone'' as the product that matches tested configuration as it was reported in the test results. However, this option is probably too rigid, especially for vendors that offer the same software on a variety of (similar) hardware platforms (e.g., Inktomi, Microsoft, and Squid). It would be infeasible for such a vendor to certify all (or even majority) of their products.
Option (2) is less rigid, but suffers from the necessity to define what a ``minor revision'' is. There is no clear and formal definition that will be reasonable for all possible situations. For example, should two different hardware platforms be considered a minor modification for software-only solutions? If not, then the problem with option (1) remains in option (2). If yes, then there are virtually no limits to ``minor'' modifications. Thus, TMF will have to make judgment calls when assigning the certification mark. Sooner or later, judgment calls lead to ugly conflicts and controversies. On the bright side, the scope of the certification mark will be focused on products that were actually tested.
The last option removes the defects of the first two because it is the vendor that is, essentially, gets certified, not a particular product: Products that have never been tested are allowed to carry the certification mark as long as their vendor has at least one certified product. In other words, a certified vendor can designate products that can be publicly tested if needed without going into trouble of actually testing those products; a certified vendor would essentially say ``Look, I have nothing to hide so if somebody cares about performance of my products, please come to TMF, test them, and make the results public; I would only benefit from you doing so!''.
The major downside of option (3) is that somebody (TMF, a buyer, or a vendor-competitor) needs to initiate and pay for the tests of products that carry the certification mark, but have not been tested yet. Thus, if there is not sufficient motivation and resources for testing a product, its performance will remain unknown. Proliferation of the certification mark can also be considered as a problem, but will probably not become a big issue considering the current number of uncertified vendors. Moreover, one could argue that a wide spread of this kind of certification is actually a positive thing (other kinds of certification marks loose their value if almost every product carries them).
Enforcing comes into play when one needs to verify that a particular product that carries a certification mark meets certification requirements. Such verification may be needed in several cases, depending on the certification requirements options discussed above. The fundamental question is between these orthogonal options:
Most products currently on the market have user licenses that prohibit public benchmarking. Thus, cache-offs and vendor-initiated tests at TMF are currently the only general ways to generate official Polygraph results.
If TMF is not allowed to test products at-will (or on-demand), the certification license should be designed so that such tests are not needed. This will necessarily make the terms of the license more rigid and the scope more narrow.
If on-demand tests are allowed, the certification terms can be relaxed because virtually any conflict can be resolved by testing the product in question. If the product license prohibits public benchmarking, the prohibition is effectively removed (for TMF only) by using the Polygraph certification mark. Needless to say, TMF will make reasonable efforts to engage the vendor in the tests before executing the tests on its own. Vendor cooperation is desired to be able to produce the best results possible for the product. Such cooperation would be required by the license.
Since our goal is to provide details about product characteristics and not to judge the quality of those characteristics, a reasonable vendor should not, in most cases, object to tests of the products carrying the certification mark. In a sense, such a test is unlikely to ``fail'' because in most cases, a product will be able to handle at least ``light'' workload and will comply with basic external requirements. This can be considered as a loophole for mediocre products to carry the certification mark. However, it is only a loophole if we assume that a vendor does not care that the published results are mediocre relative to vendor's competitors. If the assumption holds, any performance-based certification mode will have a similar loophole.
The common violation detection practice seems to be applicable to Polygraph certification. TMF should require signed licenses for all versions of the products that carry the certification mark. The license will include configuration and price details. The submitted information becomes public and searchable on TMF Web site, providing buyers and competitors with a tool to verify that a given product has the right to carry a certification mark.
Technology permitting, certification logos can carry special ``tags'' to simplify the search, matching the logo with a group of known-to-be-certified products.
A vendor expressed a concern that a logo with an embedded ``www.measurement-factory.com'' text provides ``free advertisement'' for TMF on behalf of vendors and is not appropriate. We do not have strong preferences regarding this issue and are looking for more feedback to make the final choice:
Given the above observations and choices, we have come up with a few versions of licensing terms. The simple table below compares the versions and provides links to detailed descriptions (where available).
| ||||||||||||||||||||||||||
Terms with no links are not practical and are provided for completeness only.
Please note that the above table contains imprecise terms and descriptions. Refer to the actual terms pages for more accurate and detailed information.