Channels ▼
RSS

Mobile

Windows Phone 8 App Development: Using Voice Commands


The query string parameters are very useful when you work with more complex voice commands. I'll dive deeper on how to take advantage of additional query string parameters in more complex scenarios in the next article. It is still necessary to review a few additional things related to the VCD files.

When you executed the app for the first time, the code in the Application_Launching event handler called the Windows.Phone.Speech.VoiceCommands.VoiceCommandService.InstallCommandSetsFromFileAsync method with an asynchronous execution:

await VoiceCommandService.InstallCommandSetsFromFileAsync(
      new Uri("ms-appx:///VoiceCommandDefinition.xml"));

This method receives a URI of the XML VCD file to register with the voice commands. In this case, the "ms-appx:///" prefix URI scheme indicates that the file is located in the app's install folder within the file structure of the final XAP package. The method validates all the CommandSet elements included in the specified XML VCD file. However, the method installs and activates only the CommandSet whose xml:lang attribute value matches the specified language for the GSE.

Working with Multiple Languages

In this example, I don't want the Recipes app to be limited to English (United States) voice commands, so I need to expand the XML VCD file to include two additional languages. The following lines for the VCD file define the previously explained voice commands that allow the user to create a new recipe in three languages:

  • English (United States). The command set is defined with xml:lang="en-US"
  • Italian (Italy), shown as Italiano in the phone UI. The command set is defined with xml:lang="it-IT"
  • Spanish (Spain), shown as Español (España) in the phone UI. The command set is defined with xml:lang="es-ES"

<?xml version="1.0" encoding="utf-8"?>

<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.0">
  <!-- English - US (en-US) Command Set -->
  <CommandSet xml:lang="en-US">
    <CommandPrefix>Recipes</CommandPrefix>
    <Example>create new recipe</Example>

    <Command Name="CreateNewRecipe">
      <Example>create new recipe</Example>
      <ListenFor>create [a] [new] [recipe]</ListenFor>
      <ListenFor>add [a] [new] [recipe]</ListenFor>
      <Feedback>Creating a new recipe... </Feedback>
      <Navigate Target="/AddNewRecipePage.xaml"/>
    </Command>
  </CommandSet>

  <!-- Italian - Italy (it-IT) Command Set -->
  <CommandSet xml:lang="it-IT">
    <CommandPrefix>Ricette</CommandPrefix>
    <Example>creare una nuova ricetta</Example>

    <Command Name="CreateNewRecipe">
      <Example>creare una nuova ricetta</Example>
      <ListenFor>creare [una] [nuova] [ricetta]</ListenFor>
      <ListenFor>aggiungere [una] [nuova] [ricetta]</ListenFor>
      <Feedback>Creando una nuova ricetta... </Feedback>
      <Navigate Target="/AddNewRecipePage.xaml"/>
    </Command>
  </CommandSet>

  <!-- Spanish - Spain (es-ES) Command Set -->
  <CommandSet xml:lang="es-ES">
    <CommandPrefix>Recetas</CommandPrefix>
    <Example>crear nueva receta</Example>

    <Command Name="CreateNewRecipe">
      <Example>crear nueva receta</Example>
      <ListenFor>crear [una] [nueva] [receta]</ListenFor>
      <ListenFor>añadir [una] [nueva] [receta]</ListenFor>
      <Feedback>Creando una receta... </Feedback>
      <Navigate Target="/AddNewRecipePage.xaml"/>
    </Command>
  </CommandSet>

</VoiceCommands>

Build and start debugging the app. After the emulator launches the Recipes app, go to Settings | Speech | Speech language and select the desired active speech language for the GSE in the emulator. For example, if you select Italiano (Italia), you will be able to use GSE and say any of the following phrases to create a new recipe with your voice. I leave it to you to research the accurate pronunciation of the next phrases in Italian:

  • Ricette aggiungere
  • Ricette aggiungere una nuova ricetta
  • Ricette aggiungere nuova
  • Ricette aggiungere nuova ricetta
  • Ricette creare
  • Ricette creare una nuova ricotta
  • Ricette creare nuova

Ricette (Recipes in Italian) is the Italian name for the app, defined as the CommandPrefix for xml:lang="it-IT." Any of the phrases will match one of the two ListenFor elements for the voice command defined for the Italian language:

  • creare [una] [nuova] [ricetta]
  • aggiungere [una] [nuova] [ricetta]

When the GSE matches the phrase said by the user with the registered CreateNewRecipe command for xml:lang="it-IT," Windows Phone will display the app icon, its name, and the feedback text defined for the voice command in Italian. In this case, the feedback is "Creando una nuova ricetta…" and you will hear the phone's voice reading back the text in Italian (see Figure 16). Then, the debugger will show the same exception that appeared when you tested with English (United States) because there is no AddNewRecipePage yet.

Windows Phone 8 App Development Part 2
Figure 16: Windows Phone starting the Recipes app and navigating to the AddNewRecipePage as a result of an Italian voice command.

Of course, there is more work to be done to make an app that uses voice commands support multiple languages and cultures with all the necessary adjustments. However, with simple changes to the XML VCD file, you can provide basic support for multiple languages for the voice commands. Then, you have to make the additional changes to the contents of the app to support each different culture.

Working with Phrase Lists

So far, the Recipes app has been using a very simple voice command to create a new recipe. However, I want the Recipes app to allow the users to use voice commands to search for a particular type of cooking recipe with a specific ingredient. I'm going to go back to US English as the single target language in order to keep things simple.

VCD files allow you to define collections of words or phrases with the PhraseList element. Each PhraseList element requires a value for the Label attribute to provide it an identifying name. Then, you can reference the PhraseList in both the ListenFor and Feedback elements enclosing the value of the Label attribute in curly brackets ({}). For example, "search {recipeTypes}" will match both "search" followed by any value specified in the PhraseList whose Label attribute is equal to recipeTypes. The following lines for the VCD file define three PhraseList elements with many words:

  • recipeTypes
    • Salads
    • Sandwiches
    • Desserts
  • recipeConnectors
    • of
    • with
    • for
    • about
  • recipeElements
    • tomatoes
    • potatoes
    • chicken
    • cheese

In addition, the VCD file defines the new SearchRecipe command that uses the three PhraseList elements:

<?xml version="1.0" encoding="utf-8"?>

<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.0">
  <!-- English - US (en-US) Command Set -->
  <CommandSet xml:lang="en-US" Name="RecipesVoiceCommands">
    <CommandPrefix>Recipes</CommandPrefix>
    <Example>create new recipe</Example>

    <Command Name="CreateNewRecipe">
      <Example>create new recipe</Example>
      <ListenFor>create [a] [new] [recipe]</ListenFor>
      <ListenFor>add [a] [new] [recipe]</ListenFor>
      <Feedback>Creating a new recipe... </Feedback>
      <Navigate Target="/AddNewRecipePage.xaml"/>
    </Command>

    <Command Name="SearchRecipe">
      <Example>search salads with tomatoes</Example>
      <ListenFor>[search] {recipeTypes} {recipeConnectors} {recipeElements}</ListenFor>
      <Feedback>Searching {recipeTypes} {recipeConnectors} {recipeElements}</Feedback>
      <Navigate Target="/SearchRecipesPage.xaml"/>
    </Command>

    <PhraseList Label="recipeTypes">
      <Item>Salads</Item>
      <Item>Sandwiches</Item>
      <Item>Desserts</Item>
    </PhraseList>

    <PhraseList Label="recipeConnectors">
      <Item>of</Item>
      <Item>with</Item>
      <Item>for</Item>
      <Item>about</Item>
    </PhraseList>

    <PhraseList Label="recipeElements">
      <Item>tomatoes</Item>
      <Item>potatoes</Item>
      <Item>chicken</Item>
      <Item>cheese</Item>
    </PhraseList>

  </CommandSet>
</VoiceCommands>

Build and start debugging the app. After the emulator launches the Recipes app, go to Settings | Speech | Speech language and select English (United States) as the desired active speech language for the GSE in the emulator. Now, you can activate GSE and say any of the following phrases. Remember that "Recipes" is the CommandPrefix:

  • Recipes salads with tomatoes
  • Recipes sandwiches with cheese
  • Recipes search sandwiches with chicken
  • Recipes search sandwiches with tomatoes

Any of these phrases will match the ListenFor element for the new voice command, "[search] {recipeTypes} {recipeConnectors} {recipeElements}." The recipeConnectors list makes sure the voice command works when the user uses other words instead of "with" and the phone still understands the search. As a general rule, users don't have perfect grammar when they speak to the phone, and therefore, you should take this into account when you design your voice commands.

When the GSE matches the phrase said by the user with the registered SearchRecipe command, Windows Phone will display the app icon, its name, and the feedback text defined for the voice command. In this case, the feedback is dynamic because it uses the three PhraseList elements as fields. The feedback will use the recognized words to replace the PhraseList elements. For example, if you say, "Recipes search sandwiches with tomatoes," the feedback text will be "Searching sandwiches with tomatoes," and you will hear the phone's voice reading back the text (see Figure 17). Then, the debugger will show an Exception because there is no SearchRecipesPage yet.

Windows Phone 8 App Development Part 2
Figure 17: Windows Phone starting the Recipes app and navigating to the SearchRecipesPage as a result of the "Recipes search sandwiches with tomatoes" voice command.

The URI that the GSE generates includes additional query string parameters that allow you to determine which values of the PhraseList have been recognized in the voice command. I'll dive deeper on this in the next article and explain how to update the PhraseList elements to dynamically add elements to them within the app. This way, the user will be able to search recipes with more elements than the initial elements included in the VCD file.

In this article, I provided a detailed overview of the new speech experience and started developing a sample app that takes advantage of the new extensible voice commands in Windows Phone 8. In the next article, I'll explain how to retrieve parameters when it is necessary to execute the voice commands, and show how to update phrase lists within the app. In addition, I'll explain how to use the additional Windows Phone 8 speech features to make the app interact with the user.


Gaston Hillar is a frequent contributor to Dr. Dobb's.

Related Articles

Windows Phone 8 App Development: Getting Started

Using Speech APIs in Windows Phone 8


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video