March 2, 2021
Hot Topics:

It's Only Natural: Evaluating Natural Language Dialogs

  • By Jonathan Eisenzopf
  • Send Email »
  • More Articles »

What is Feasible?

As we've already discussed, some applications are naturally suited to become directed dialogs. Others that are open or conversational may require a mixed initiative dialog. In some cases, it may simply be impossible to practically create the application using one of the styles. This should be identified early on in the process. If that happens, then the decision is clear and it becomes a matter of whether the cost and effort will justify the end result.

Examine the Difference in Effort

In terms of measuring the difference in effort between natural dialogs and directed dialogs, we actually need to think about several different factors.

The average mixed initiative natural dialog will take several orders of magnitude longer to develop and maintain than a directed dialog. The skills required to develop a natural mixed initiative dialog are also steep and require some knowledge of linguistics and speech recognition.

The four areas of development that we can compare are:

  • grammars
  • prompts
  • error handling
  • maintenance

A directed grammar will contain a rather limited number of possible utterances. For example, when you ask a caller for their credit card type and give them a list of their options, there are only a handful of possible responses. However, if you ask the caller a more open ended question like, "How would you like to pay for this?", the number of possibilities goes up quite dramatically. Writing a grammar for an open ended question requires us to represent all of the possible answers that we might get from the caller. Even for such a simple question, this is no small task. In my humble opinion, however, programming natural dialogs is more about how you handle recognition errors than actually focusing on catching every possible utterance. In either case, natural language grammars will always be larger and thus, will take more time to develop.

Prompts in directed dialogs should be clear enough to eliminate most ambiguities. Doing so limits the number of error handlers you have to write. However, in a natural dialog, you will have to write error handlers and prompts to go along with each possible utterance. This will of course require more programming time and more time in the recording studio to record the prompts.

Maintaining a natural mixed initiative dialog will also require a higher degree of maintenance, because the grammars and prompts must be regularly tuned to account for utterances that haven't already been accounted for. Directed dialogs will also need the same maintenance, but not as frequently and won't require as much work.

What Value in Natural Dialogs?

Sure, natural dialogs are cool compared to touch-tone or directed speech menus, but coolness is not a final measure of whether a natural dialog should be employed. Yes, people who are not familiar with speech recognition may have an initial wow factor, but that inevitably wears off and then the question is, why and when is a natural dialog better?

The measure I use is this: If a natural dialog isn't better from a usability standpoint (which translates into fewer bail-outs) or faster (callers can get the job done quicker, reduces call time) when compared to a directed dialog, then go with the directed dialog, which is quicker, easier and cheaper to build.


The differences in directed dialogs vs. natural mixed initiative dialogs can easily be an order in magnitude of 3 or more, so care should be taken in making your decision, especially where cost and time are concerned. This article should give you some ideas on where to look in evaluating which approach makes sense in your case. If you're still not sure which approach to take after reading this, or if you still have more questions, send me an email, eisen@ferrumgroup.com.

About Jonathan Eisenzopf

Jonathan is a member of the Ferrum Group, LLC  which specializes in Voice Web consulting and training. Feel free to send an email to eisen@ferrumgroup.com regarding questions or comments about this or any article.

Page 2 of 2

This article was originally published on November 9, 2002

Enterprise Development Update

Don't miss an article. Subscribe to our newsletter below.

Thanks for your registration, follow us on our social networks to keep up-to-date