Testing IVR with Speech Recognition

Question

I was hoping I could make use of the voice recognition part of call confirmation while testing our IVR system. However, just having the system call my phone extension to test how well the system recognizes specific words has been unsuccessful. I’ve experimented with having it recognize the words OK, yes, one and transfer and have made sure to speak very clearly into the phone and have tried saying the word several times in succession, but have had no luck. Do you have any suggestions?

Answer:

Speech recognition over the phone is not accurate, especially with microsoft speech recognition engine that is builtin to the gateway. The best suggestion we have is to design the command differently. For example, instead of saying ‘yes’ or ‘no’, use ‘the system is working’ vs. ‘no’. The longer phrase is easier to recognize against a short word.

The other suggestion is to have your IVR system product key press like #1, #2 if it can.

Follow up:

I was hoping that I could avoid human intervention and have the software recognize the word OK, which the automated IVR says when the test is successful. Even if I used “call till confirmed” to have it try up to three times before reporting a failure. I do have a backup plan, but this would have been ideal. Regardless, this software is by far the best that I’ve found for conducting these tests.

More follow up:

I am happy to say that I got the speech recognition working for testing our company’s IVR system. My objective was to get Broadcast by Phone to recognize *something* at a certain part of the IVR phone test to determine if the test was successful or if it failed. Hopefully, these suggestions will help others that are attempting to accomplish the same thing.

1. Like Voicent support suggested, have the software recognize a phrase rather than a single word.
2. Programming your message to search for several different phrases will increase your chances of success. I have it looking for three different phrases and am currently experiencing a 100% confirmation rate.
3. If you are using a phone hooked up to your modem to help you design your responses, hang up the phone when the software is supposed to confirm a voice response. Any background noise can throw it off.
4. To keep the line open long enough to confirm the response, I have a10 second wave file of just silence. That way there is no noise to interfere with the speech recognition. If you are obtaining voice response from an actual person, you can create your message file and then have an additional silent wave file at the end of that message to keep the line opened while the software looks for the correct response.

More to share:

Hello, I thought I would update you on my progress for automating our IVR tests using Broadcast by Phone. Maybe some of your other customers will find this information useful. I have several phone tests successfully set up to be tested hourly using Broadcast By Phone. Here is a synapse of the process:

Dial the number, work through the IVR prompts and enter an account number. After the system verifies the account number I use speech recognition to listen for the appropriate response from the IVR. If the test is not successful I have the software try again up to two more times (fault tolerance because speech recognition isn’t perfect). Once the test is complete I have an Excel spreadsheet automatically updated with which call was made, date/time of call, number of attempts, and whether I obtained confirmation. When the test is complete I have an automated pop-up sent to one of the consoles in our data center specifying which test was conducted and that it was successful. If the test was not successful then I have an automated pop-up sent to several consoles in our data saying the test wasn’t confirmed and a manual test is required.

To accomplish all this I had to learn Visual Basic Script (VB script). If your customers don’t know vb script they just need to google it. There is tons of free information out there and it was pretty easy to learn.

The first thing I did was set up the Voicent Gateway as “Most Aggressive” because I want it to always think it is talking to a live agent. This is because I am using speech recognition to obtain a confirmation for a successful phone test.

The second step is to set up the Broadcast By Phone file. Each of my files consists of just one phone number. The phone number field consists of the phone number, whatever additional numbers and pauses are necessary to navigate through the IVR menu, and the test account number that will trigger the response I want to test from the IVR.

Next I used message designer and set it up to use a different message for answering machine and human. Since I have the gateway set up to think it is talking to a human I don’t really care about the answering machine message. I have mine set up to play a two second wave file of just silence. For the human response message I have the software play a 10 second wave file of just silence (you can use Microsoft voice recorder without a microphone plugged in). This is important because I want complete silence so that nothing will interfere with the speech recognition engine and also to keep the phone line opened long enough to obtain voice confirmation. I then set up the voice response to confirm the call. It was helpful for me to have several multiword statements set up to confirm against to increase my odds of successful confirmation. For example, I get to a part of the phone call where the IVR says, “are you calling to report no power or some other condition….please say no power or some other condition….” So I set up my confirmation to listen for “no power”, “some other condition”, “are you calling to report no power or some other condition”. For each of these I selected “yes” to confirm. If you want to find out which statement the voice engine is picking up you can choose a different choice for each one, like “yes” for no power, “no” for some other condition, etc. I found that the same statement does not always trigger the speech recognition. The next step is to test, test, test until you are able to continuously receive a confirmation from your test. It doesn’t have to be 100%. I probably get a confirmation 75% of the time, which is why I have fault tolerance built into my VB script. Also, it is important to auto save the file after 1 call (in broadcast options).

Once you’ve got your Broadcast By Phone files set up then you can set up your VB Script. This is how I have mine set up. First, I state my variables (of course), then I set up a loop that will run up to three times (this is for fault tolerance. It is also call till confirmed using speech recognition). I then use the command line interface to launch my BBP file. I use –cleanstatus to clear the file and –startnow to start the dialing process. I put my VP Script to sleep for 110 seconds while I wait for the BBP software to finish the phone call. After the sleep cycle is done the script shuts down the BBP job. Then I call the file back in, this time as a text file. This is so that I can verify confirmation. I have the script look for the word, “confirmed” in the text file. If it finds it then the loop ends and the script continues. If it doesn’t find it then I have it go back to the beginning of the loop and try up to 2 more times (you can have it be 5, 17, 32, as many times as you want).

The next thing the script does is open an excel spreadsheet that I already have set up. The scripts goes to the first blank row in the file and logs the call. I have it log which call was made (I have 4 different tests that I do), date/time, number of attempts, and whether it obtained confirmation. (call till confirmed with voice recognition).

If I am able to verify confirmation I use the “net send” command to send a message to one of the consoles in our datacenter stating that “[name of test] call was successful.” If I was unable to obtain confirmation then I send a message to several consoles, “[name of test] was not confirmed. Please test manually.”

The final step is to schedule the phone tests. I am using Microsoft Task Scheduler, which comes free with Windows XP, to schedule my scripts. I have each one set up to go off once an hour every hour in 10 minute intervals.

Now, I ran into one other problem that I should mention. I found that the speech recognition would work well for a while, but the more test calls I make the less effective it was at obtaining confirmation. I would start out with near 100% success rate and by the next day I was only getting a 25% success rate if I was lucky. Fortunately, I found a free solution to correct this problem. If you go to Microsoft System Tools there is a program called disk cleanup. This can be launched through command line interface (cleanmgr.exe /sagerun:11) so I also have that in Microsoft Task Scheduler to go off once an hour at 3 minutes before the hour. I probably don’t need to run it that often, but it only takes a few seconds to run, so why not?

Well, I think that’s everything. Thank you for making such a versatile software package that allowed me the freedom to accomplish so much. I am now a hero at work because I was able to automate a mundane process that we were doing manually. I hope others find this information helpful.

Thanks,
Bob

Great work, Bob, and thank you so much for sharing.