VoiceXML Conformance Report
Conformance Test Suite
The next step in my study was to test VoiceXML conformance across a range of VoiceXML browsers using only the VoiceXML 2.0 and Speech Recognition Grammar Specification (SRGS) as guidelines for creating the test source code. I did not refer to any VoiceXML or SRGS documentation from any of the platform providers.
The purpose of this test was to determine how many platform providers that claim VoiceXML 2.0 support are actually able to run compliant code without requiring additional modifications.
- Code that was created was platform independent
- Code was validated against the official DTDs
Test Source Code
For the test, I developed a minimal VoiceXML application that consisted of:
- One VoiceXML form to gather a social security number
- One SRGS XML DTMF grammar
VoiceXML Source Code
<?xml version="1.0"?> <!DOCTYPE vxml PUBLIC "-//W3C//DTD VOICEXML 2.0//EN" "http://www.w3.org/TR/voicexml20/vxml.dtd"> <vxml version="2.0" xmlns="http://www.w3.org/2001/vxml"> <form id="ssn"> <field name="ssn_number"> <grammar src="ssn_dtmf.grxml" mode="dtmf" type="application/srgs+xml"/> <prompt bargein="true">Please enter your social security number</prompt> <filled> <prompt>You entered <value expr="ssn_number"/> </prompt> <clear namelist="ssn_number"/> <goto next="#ssn"/> </filled> </field> <catch event="nomatch noinput"> <reprompt/> </catch> </form> </vxml>
SRGS XML Grammar Source Code
<?xml version="1.0"?> <!DOCTYPE grammar PUBLIC " -//W3C//DTD GRAMMAR 1.0//EN" "http://www.w3.org/TR/speech-grammar/grammar.dtd"> <grammar mode="dtmf" version="1.0" xml:lang="en-US" root="ssn" xmlns="http://www.w3.org/2001/06/grammar"> <rule id="ssn" scope="public"> <ruleref uri="#digit"/><ruleref uri="#digit"/> <ruleref uri="#digit"/><ruleref uri="#digit"/> <ruleref uri="#digit"/><ruleref uri="#digit"/> <ruleref uri="#digit"/><ruleref uri="#digit"/> <ruleref uri="#digit"/> </rule> <rule id="digit" scope="private"> <one-of> <item>1</item><item>2</item><item>3</item> <item>4</item><item>5</item><item>6</item> <item>7</item><item>8</item><item>9</item> <item>0</item> </one-of> </rule> </grammar>
The results of the conformance test for the 6 platforms are listed below. The good news is that 3 out of 6 platforms executed the code. The bad news is that 3 of the 6 platforms didn't.
While Browser 4 and Browser 6 didn't execute the code, the changes required to make it work were minimal. However, for the sake of the test, the code either worked or it didn't. To be fair, I did go to the trouble of troubleshooting what needed to change to allow the code to run. This information is detailed below.
To make the code work on Browser 4, I had to change the DTD reference from W3C to one provided by the vendor.
This is a minor change that is acceptable when you want to use extra browser extensions, however, it should still be capable of running generic VoiceXML code that uses the default W3C DTD.
The second change that I had to make was to change the mime type attribute of the <grammar> element to:
This is forgivable because the VoiceXML specification only provides an example of what the mime type might be rather than stating what it must be.
Browser 5 was more difficult. I gave up troubleshooting the problem after spending an hour trying to figure it out.
Like Browser 4, Browser 6 required a different DTD.
Also, as with Browser 4, the mime type attribute of the <grammar> element needed to be changed to:
The third and final change was to remove the SRGS grammar DTD. It took me a while through the process of elimination to discover the solution to this particular problem.
VoiceXML DTD Problems
During the testing process, I noticed that several code checking tools offered by the platform vendors consistently complained about the W3C DTD referenced in the VoiceXML test program. One of the VoiceXML contributors later confirmed that the DTD listed in the specification contained errors, which would be fixed soon. This may or may not have contributed to the fact that Browser 4 and Browser 6 required a different DTD since some XML parsers would not have been able to validate VoiceXML source code using the W3C DTD.
Testing Tool Validation Problems
One thing I noticed as I was testing the various platforms is that the source code valuators offered by the vendors often gave false positive results meaning that when I tested a VoiceXML program that I had intentionally broken, the majority of the tools often reported the code to be valid even though it would not work when I dialed into the application. This made the troubleshooting process all the more difficult. Browser 2 and Browser 3 were the only platform code valuators that accurately identified problems in the source code.
I spent about 60 minutes troubleshooting each of the three platforms that didn't run the VoiceXML test program and I was only able to figure out how to fix the problem on two of them. The fact that debugging output was not very helpful most of the time meant that I had to resort to fixing problems through the process of elimination, which is very time consuming. These code valuators need to do a better job of inspecting element data and attributes in addition to validating the code against a DTD.
From my perspective, it would be better to use a proprietary standard that was supported by a wide range of vendors whose platforms achieved interoperability and conformance than to use an "open standard" in which implementations were inspired by the standard rather than conforming to it.
Unless ALL VoiceXML platforms are able to run compliant code, VoiceXML will not be portable, will not meet customers expectations, and will therefore not be very useful. If this test of 6 browsers is an general indication that only 50% of the available platforms are VoiceXML compliant, then customers need to be careful to test platforms for compliance before making a final decision.
In the future, I plan on extending the VoiceXML 2.0 test script to exercise the rest of the specification and also plan to expand the number of platforms that will be tested. If you have ideas or recommendations on what the test script should contain or would like to recommend VoiceXML gateways that you'd like to see tested, please send me an email with that information.
About Jonathan Eisenzopf
Jonathan is a Senior Partner of The Ferrum Group, LLC which provides speech IVR consulting, training, and voice user interface design. Feel free to send an email to firstname.lastname@example.org regarding questions or comments about this or any article.
Page 2 of 2