Wednesday, September 05, 2012
Answer Devices Releases Mobile Assistant Benchmark
Our friends at Answer Devices have released a very useful data set for anyone developing or testing a mobile virtual assistant like Siri or CallMom. The "1000 questions every mobile assistant should be able to answer" includes sample inputs that cover a range of topic areas including apps, contacts, games, knowedge, dialing, SMS, maps, search. Borrowing from published transcripts with popular virtual agents, Youtube videos, CallMom log files and even Apple TV commercials, the sample data is designed to test both speech recognition and response accuracy.
An Apple critic, Gene Munster of Piper Jaffray, published a study of Siri in June in which he tested the app with 1600 sample inputs in both quiet and noisy background environments. Although the reported result was about 65% accuracy, it was difficult to verify because Munster did not make the sample data public. As far as we know, the Answer Devices data set is the only public benchmark test suite available to compare mobile virtual assistants.
Pandorabots has offered to test any virtual assistant that has an API with the Answer Devices data. If you are developing a virtual assistant and would like Pandorabots to run the test for you, please contact firstname.lastname@example.org.