These slides were presented at the W3C Workshop on Quality Assurance. Please also see Measurement Factory position paper submitted to the workshop.

Outline

• HTTP testing is important
• HTTP testing is difficult
• Current status
• What's next

Background

• complex Web protocols
• variety of environments and communicating products
• developers cannot fully test their implementations
• integrators cannot test product cooperation
• unverifiable marketing claims

Core problems

• HTTP is ``too complex'' (about 345 MUSTs, 255 SHOULDs, 180 MAYs)
• nobody knows what the bugs are
• not enough people care enough

How can we improve the situation?

• make a tool to expose most of the problems

Why this is hard: XML vs. HTTP testing

*ML	HTTP
static environment: text with all included and referenced items	dynamic environment: sequence of message exchanges with a partially controlled agent
lying on a sofa and reading a ``French Riviera'' travel guide	asking a Sophia local for directions and following them
XML validator	HTTP agent validator

HTTP testing

• Goals:
  • put the agent into a certain well-known state and
  • then ask questions to verify compliance
• Method:
  • sending HTTP messages

HTTP testing: complications

• compliant request => many compliant responses
• compliant request => many incompliant responses
• volatile states depend on time and other external factors
• some states are unreachable for some agents

HTTP testing: more complications

• compliance is not enough
• both client and server side implementations may be required
• thousands of interesting states
• dozens of agents

But it is possible!

• success with Web Polygraph benchmark
• research studies on HTTP compliance
• working prototype
• demand

Co-Advisor features

• automation of QA tasks
• domain-specific test description language
• sophisticated compliance and compatibility tests
• a large, dynamic collection of test cases
• detailed human-oriented test descriptions and logs

Development model

• core software (driver) plus collections of tests
• membership-based seed funding
• free, open-source versions
• community input on tests scope and design
• borrow best practices from our Squid caching proxy and

Current status

• working prototype implementation
• 256 HTTP/1.1 compliance test cases
• preliminary results:

  	Squid cache	Commercial caches
  Passed	61%	45% -- 75%
  Failed	39%	25% -- 55%

  
  

• building core membership group

Lessons learned

• formal HTTP compliance is not enough
• HTTP may not be enough (RTSP, ICAP, WCIP, ...)
• extended scope: caching proxies, L4/7 SLB, origin servers
• automated engineer- and executive-oriented test analysis

What we need

• more project members
• important test cases
• grass-root demand for compliant and well-behaving products
• W3C QA cooperation
• full-blown implementation

Cooperation with W3C

• W3C could endorse and even own/maintain a
• leave the core driver development to us
• test case description (or programming) language as a glue

More information

• http://www.measurement-factory.com/products/co-advisor/
• < rousskov at measurement-factory.com >