VoiceTracking a FedEx Package with an HTML Scraper

Tracking a FedEx Package with an HTML Scraper

One way to develop powerful VoiceXML applications is to leverage existing
Web applications by simulating a Web browser and scraping the results via a
programmable HTTP Web agent. This example allows users to track a Federal
Express package, using the fedex.com Web site.

Overview

One of the selling points of VoiceXML is that it can work with your
existing Web infrastructure. Applications can be developed using the same tools, run on
the same server over the same Internet connection. So it makes sense that VoiceXML
applications should be able to leverage existing code.

Hopefully many of your Web application have been componentized into re-usable
libraries, so that other interfaces can be developed using the same code. In reality,
most Web applications are flat scripts. Others are not even available to you at all.
One way of leveraging Web applications in the latter two examples is to write a layer
of code between the voice interface and an existing Web or wireless interface. This
exercise involves emulating the intended client, submitting form input, and parsing
results into a form that can be played back by a VoiceXML interface.

The example that we’re going to look at is a Perl script that uses the Federal
Express Web tracking application available at http://www.fedex.com/us/tracking/.
I will be referring to line numbers in the source code throughout
this article, so you might want to open the example in another browser or print it out for reference.
You can also download the code without line numbers.

The application is totally self-contained. There aren’t any other external files or programs
needed to run the script other than a Perl interpreter and the LWP and CGI modules. The script
will run on Windows and Unix servers and on any Web server that supports CGI.

How it Works

An example of a typical dialog is below:


Computer: Please enter your federal express tracking number 
now.
Customer: 1232343
Computer: Package 1232343 is as follows. Left FedEx Sort 
Facility NEWARK NEW JERSEY. The last update for this package 
occurred on August six at twelve thirty nine.

The first time the script is run, the user is prompted for their Fedex tracking number. If the
user doesn’t enter a number, they are given additional instructions and prompted for the number
again. When the user enters a number on their telephone keypad, it’s sent back to the script
as the track_num form field. This number is appended to the Fedex tracking URL and sent
using an LWP agent. LWP is a Perl library for simulating a Web browser agent. The page that
comes back from the Fedex site is parsed for the tracking information. We also check for an
error message. If an error occurred, the user is prompted again for a tracking number, otherwise,
the tracking information is played and the call is ended.

fedex_tracker.pl

This Perl script utilizes the LWP libraries, which allow us to send queries to a remote
Web server. This library, along with the CGI library, are both loaded on lines 1 and 2.
On line 10, we check for the existence of the track_num parameter. If it exists,
we go ahead and send it to the Fedex Web site by executing the &parse_html function
on line 11. If an error occurs, meaning that the tracking number was invalid,
we output some VoiceXML content that plays an error message and re-runs the script. If
the request was successful, the code on lines 26 to 54 is executed, which plays the
tracking information that was retrieved.

If the track_num form field was not passed to the script, the VoiceXML content
on lines 55 to 79 is sent back to the VoiceXML gateway. This is the dialog that asks
the user for the tracking number. So if a number is not sent, this is the dialog that
gets executed first.

Basically, the script will send back the proper dialog for the following states:

  • The user is running the script for the first time; the user is prompted for a number
  • The user has just entered a number; the VoiceXML form submits the number back to the script, which submits
    the number to the Fedex site and plays the results.

  • The user entered an incorrect number; the user hears an error message and plays the error message.

    The guts of the script are in the parse_html subroutine starting on line 81.
    This is where we send the tracking number to the Fedex Web site and parse the results.
    This is also where we need to do a little regular expression fancy footwork.
    On line 83, we build the full contents of the URL that we will be submitting
    the information to (including the tracking number). Fortunately, the Fedex
    script allows us to submit data via the GET method (instead of POST), so
    we can simply call the get function of the LWP library, passing
    the URL we just built. The function passes the resulting HTML back, which
    is stored in the $page variable.

    So now we have a real big text string that contains an HTML page. How
    do we get that information out of all that text? Well, by finding a pattern
    to the results and using that as a mark for where the data starts. To figure
    this out, I used the Fedex application from a Web browser and viewed the
    HTML source. I found that a unique block of HTML occurs just before the
    tracking data begins. The result are the regular expression on line
    91 which says, find this text CLASS=”resultstableheader” followed
    by zero or more characters .*? followed by a </tr> tag,
    followed by a whole bunch of text (.*?) up to a </table> tag.
    This last bit of text up to the </table> tag is where our data
    is located. We want to grab that and save it on line 92.

    Now that we have the glob of text containing the data, we have to
    parse it with more regular expressions. To accomplish this in one
    fatal swoop, I created a compound regular expression on lines 94 and 95
    that finds a row of data and splits it into pieces. Lines 97 to 112
    split the rows into records and saves the information in an array of
    hashes. Each array item contains the message, date, time, and notes
    that are related to the record. Fedex lists not only the current status,
    but the whole history of a package. While we’re only playing back the
    first record, which is the latest tracking information, you can reuse
    this data structure and set of regular expressions to play the entire
    history.

    Once the data structure has been built, it returns the array back to
    the caller. You can see where we reference the first record of this
    data structure on lines 27 to 31, where we’re creating local copies,
    which are included in the VoiceXML output.

    Conclusion

    Well, we just created a pretty useful VoiceXML application in not
    much more than 100 lines of code. That’s impressive I think. The
    opportunity to voice enable any number of Web applications is at
    hand, and here’s some code you can use to get started.


    About Jonathan Eisenzopf


    Jonathan is a member of the Ferrum Group, LLC based in Reston,
    Virginia that specializes in Voice Web consulting and training. He
    has also written articles for other online and print publications
    including WebReference.com
    and WDVL.com. Feel free to send an
    email to eisen@ferrumgroup.com
    regarding questions or comments about the VoiceXML Developer series,
    or for more information about training and consulting
    services.

  • Get the Free Newsletter!
    Subscribe to Developer Insider for top news, trends & analysis
    This email address is invalid.
    Get the Free Newsletter!
    Subscribe to Developer Insider for top news, trends & analysis
    This email address is invalid.

    Latest Posts

    Related Stories