Channels ▼
RSS

Mobile

Using Speech APIs in Windows Phone 8


In the previous article in this series on Windows Phone 8 app development, I explored the improved speech experience introduced in Windows Phone 8 and started developing an app that takes advantage of new extensible voice commands. In this article, I explain how to retrieve parameters when it is necessary to execute the voice commands and how to update phrase lists within the app. I also explain how to use speech recognition and text-to-speech features. This way, you will be able to combine voice commands, speech recognition, and text-to-speech to enable a voice-driven UX.

Checking Parameters and Executing Voice Commands

The Recipes app discussed in the first two parts of this series recognizes voice commands, but fails when it has to navigate to the target pages because the pages don't yet exist in the project. It is time to add the pages for each of the voice commands. I'll start with the simple AddNewRecipePage page that the app must display when executing the CreateNewRecipe voice command. To keep the sample app simple and focused on speech-related features, I won't use a Model-View-ViewModel (MVVM) approach — I'll just add the necessary pieces of code to the code-behind files. However, take into account that a MVVM approach is definitely suitable for these kinds of apps.

To add a new portrait page, right-click on the Recipes project in Solution Explorer, select Add | New Item… and select Windows Phone Portrait Page (Figure 1).

Windows Phone 8 App Development Part 3
Figure 1: Adding a new Windows Phone Portrait Page to the project.

Then, enter AddNewRecipePage.xaml in the Name textbox and click Add. Visual Studio 2012 will create and display the preview for a new portrait page (Figure 2).

Windows Phone 8 App Development Part 3
Figure 2: The initial contents for the new Windows Phone Portrait Page with the ContentPanel control selected.

A StackPanel, also known as the title panel, contains the name of the application and the page title. Obviously, I don't want the recipes app to display "MY APPLICATION" and "page name" in the page that adds a new recipe. You will notice the following lines that define the title panel don't use resources to retrieve the app's title. Thus, if you change the app's title in the app manifest, nothing will change in the pages you added to the project, because they will still display "MY APPLICATION:"

<!--TitlePanel contains the name of the application and page title-->
<StackPanel Grid.Row="0" Margin="12,17,0,28">
    <TextBlock Text="MY APPLICATION" Style="{StaticResource PhoneTextNormalStyle}"/>
    <TextBlock Text="page name" Margin="9,-7,0,0" Style="{StaticResource PhoneTextTitle1Style}"/>
</StackPanel>

I have to add another page to handle the other voice command, so I want to use resources to define the localized app name. Double click on AppResources.resx within the Resources folder. You will notice the value for ApplicationTitle is "MY APPLICATION" (Figure 3). You can change it to your desired apps's title and use the static resource in the different pages. In this case, it is necessary to set "RECIPES" as the value for ApplicationTitle.

Windows Phone 8 App Development Part 3
Figure 3: The resources defined in AppResources.resx.

Add a new resource named AddNewRecipePageTitle, set its value to "add a recipe" and change the lines that define the title panel in AddNewRecipePage.xaml to make the two TextBlock controls use the localized string resources and follow best practices:

<!--TitlePanel contains the name of the application and page title-->
<StackPanel Grid.Row="0" Margin="12,17,0,28">
    <TextBlock Text="{Binding LocalizedResources.ApplicationTitle, Mode=OneWay, Source={StaticResource LocalizedStrings}}" Style="{StaticResource PhoneTextNormalStyle}"/>
    <TextBlock Text="{Binding LocalizedResources.AddNewRecipePageTitle, Mode=OneWay, Source={StaticResource LocalizedStrings}}" Margin="9,-7,0,0" Style="{StaticResource PhoneTextTitle1Style}"/>
</StackPanel>

Now, add a Loaded event handler named AddNewRecipePage_Loaded to the AddNewRecipePage page. Then, add the following code to the event handler and establish a breakpoint at the if statement:

private void AddNewRecipePage_Loaded(object sender, RoutedEventArgs e)
{
    //// Check whether a voice command launched this page
    if (this.NavigationContext.QueryString != null 
        && this.NavigationContext.QueryString.ContainsKey("voiceCommandName"))
    {
        //// A voice command launched this page and I want to retrieve the command name
        //// included in QueryString
        var commandName = NavigationContext.QueryString["voiceCommandName"];
        
        //// Now, you can execute code according to the commandName value
        //// and retrieve values for additional parameters required by the commandName
    }
}

Build and start debugging the app in the emulator. After the debugging session executes the app, activate GSE and say "Recipes create new recipe." The phone should recognize the command, launch the Recipes app, and navigate to the AddNewRecipePage. Thus, the AddNewRecipePage_Loaded event handler will be fired and the debugger will stop at the breakpoint. Inspect the value for the NavigationContext.QueryString Dictionary<string,string> (see Figure 4). Visual Studio displays two key-value pairs for the dictionary, as shown in Table 1:

Key Value

voiceCommandName

CreateNewRecipe

reco

Recipes create new recipe

Table 1: Key-value pairs.

Windows Phone 8 App Development Part 3
Figure 4: Inspecting the NavigationContext.QueryString Dictionary<string,string> for the AddNewRecipePage page after the voice command launches the app.

In this example, the phone navigated to the page as a result of the execution of a voice command. However, the same page might be loaded in other scenarios where voice commands aren't involved, so you cannot assume that NavigationContext.QueryString is always going to have values when the Loaded event is fired. That's why the previously shown code uses an if statement to check whether NavigationContext.QueryString is not null and contains the "voiceCommandName" key. If that's true, it means that a voice command made the app navigate to this page.

In our example, there is just one voice command that can navigate to the AddNewRecipePage. However, it is possible to have many voice commands that can navigate to the same page and perform different actions in the page. That's why it is important to check the value for the "voiceCommandName" key. In addition, you can check the complete recognized speech by retrieving the value of the "reco" key. Depending on the value of the "voiceCommandName" key, you might need to retrieve additional parameters whose values are included in additional key-value pairs for the dictionary.

Retrieving Parameter Values for the Voice Commands

The SearchRecipe voice command defined in the previous article is an excellent example of a voice command that requires you to retrieve many parameters to perform the search. Thus, it is necessary to add the page that this command requires. Right-click on the Recipes project in Solution Explorer, select Add | New Item… and select Windows Phone Portrait Page. Enter SearchRecipesPage.xaml in the Name textbox and click Add. Visual Studio 2012 will create and display the preview for a new portrait page.

As done with the previously created page, I want to use resources in this page. Double-click on AppResources.resx within the Resources folder. Add a new resource named SearchRecipesPageTitle, set its value to "search recipes." Change the lines that define the title panel in SearchRecipesPage.xaml to make the two TextBlock controls use the localized string resources and follow best practices:

<!--TitlePanel contains the name of the application and page title-->
<StackPanel Grid.Row="0" Margin="12,17,0,28">
    <TextBlock Text="{Binding LocalizedResources.ApplicationTitle, Mode=OneWay, Source={StaticResource LocalizedStrings}}" Style="{StaticResource PhoneTextNormalStyle}"/>
    <TextBlock Text="{Binding LocalizedResources.SearchRecipesPageTitle, Mode=OneWay, Source={StaticResource LocalizedStrings}}" Margin="9,-7,0,0" Style="{StaticResource PhoneTextTitle1Style}"/>
</StackPanel>

Now, add a Loaded event handler named SearchRecipesPage_Loaded to the SearchRecipesPage page. Then, add the following code to the event handler and establish a breakpoint at the if statement:

private void SearchRecipesPage_Loaded(object sender, RoutedEventArgs e)
{
    //// Check whether a voice command launched this page
    if (this.NavigationContext.QueryString != null
        && this.NavigationContext.QueryString.ContainsKey("voiceCommandName"))
    {
        //// A voice command launched this page and I want to retrieve the command name
        //// included in QueryString
        var commandName = NavigationContext.QueryString["voiceCommandName"];

        //// Now, you can execute code according to the commandName value
        //// and retrieve values for additional parameters required by the commandName
        if (commandName == "SearchRecipe")
        {
            //// The command name is SearchRecipe, and therefore, it is necessary to retrieve additional parameters
            string recipeType;
            string recipeConnector;
            string recipeElement;
            this.NavigationContext.QueryString.TryGetValue("recipeTypes", out recipeType);
            this.NavigationContext.QueryString.TryGetValue("recipeConnectors", out recipeConnector);
            this.NavigationContext.QueryString.TryGetValue("recipeElements", out recipeElement);

            //// Display a MessageBox with the retrieved parameters for the search
            this.Dispatcher.BeginInvoke(() =>
                {
                    MessageBox.Show(
                        String.Format(
                            "I will search {0} {1} {2}",
                            recipeType ?? string.Empty,
                            recipeConnector ?? string.Empty,
                            recipeElement ?? string.Empty));
                });
        }
    }
}

Build and start debugging the app in the emulator. After the debugging session executes the app, activate GSE and say "Recipes search sandwiches with tomatoes." The phone should recognize the command, launch the Recipes app and navigate to the SearchRecipesPage. Thus, the SearchRecipesPage_Loaded event handler will be fired and the debugger will stop at the breakpoint. Inspect the value for the NavigationContext.QueryString Dictionary<string,string> (Figure 5). Visual Studio displays five key-value pairs for the dictionary, as shown in Table 2:

Key Value

voiceCommandName

SearchRecipe

Reco

Recipes search Sandwiches with tomatoes

recipeElements

tomatoes

recipeConnectors

with

recipeTypes

Sandwiches

Table 2. Five key-value pairs.

Windows Phone 8 App Development Part 3
Figure 5: Inspecting the NavigationContext.QueryString Dictionary<string,string> for the SearchRecipesPage page after the voice command launches the app.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video