Microsoft & .NET.NETA Guide to Running Google Assistant on Android Things

A Guide to Running Google Assistant on Android Things

Developer.com content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

In partnership with Google, AIY Projects recently launched a Raspberry Pi-based Voice Kit that provides an interface to Cloud services, such as the Cloud Speech API and new Google Assistant SDK. On their part, Google released a special Android Things Developer Preview 3.1 build for Raspberry Pi 3 to support the Voice Kit. Now, developers can run Android Things on the Voice Kit with full functionality, including integration with the Google Assistant SDK. In today’s tutorial, we’ll learn how all of these various components fit together by building the Google Assistant API sample for Android Things.

What You’ll Need

Don’t be intimidated by the long list. Most items are standard for Android Things development. The main hardware item for this tutorial is the AIY Projects Voice Kit, shown in Figure 1.

The finished kit
Figure 1: The finished kit

  • A flashed Raspberry Pi 3 B development board: Raspberry Pi 3 Model B is the latest iteration of the world’s most popular single-board computer. It provides a quad-core 64-bit ARM Cortex-A53 CPU running at 1.2GHz, four USB 2.0 ports, wired and wireless networking, HDMI and composite video output, and a 40-pin GPIO connector for physical interfacing projects.
  • AIY Projects Voice Kit: The Voice Kit originally shipped out to all MagPi Magazine subscribers on May 4, 2017. The parts list, assembly instructions, source code, as well as suggested extensions are also available on AIY Projects Web site. The complete kit is (or should be shortly) also for sale at over 500 North American Barnes & Noble stores, as well as UK retailers WH Smith, Tesco, Sainsburys, and Asda.
  • Micro-USB power adapter: To power the Raspberry Pi board.
  • MicroSD card reader
  • Ethernet cable
  • Android Studio 3.0+: The Official IDE for Android, Android Studio provides the fastest tools for building apps on Android devices. Features include code editing, debugging, performance tooling, a flexible build system, and a build/deploy system.
  • Google API Console Project with Google Assistant API enabled.

Assembling and Connecting to the AIYProjects Voice Kit

The AIYProjects Web site has detailed instructions for assembling the Voice Kit, so I won’t go over that again here.

The Android Things Launcher shows information about the board, including the IP address. Connect to this IP address using the Android Debug Bridge tool:

$ adb connect <ip-address>
connected to <ip-address>:5555

Raspberry Pi broadcasts the hostname “Android.local” over Multicast DNS. If your host platform supports MDNS, you also can connect to the board by using the following command:

$ adb connect Android.local

See the Android Things Dev site for more information on connecting to your board.

Download the Source Code

Here’s how to download and import the Google Assistant API sample for Android Things into Android Studio:

  1. Download the master.zip file from Github.
  2. Extract the downloaded zip file.
  3. Open Android Studio and select Import project from the menu.
  4. Select the “sample-googleassistant-master” directory from the extracted zip file location:Identifying the sample-googleassistant-master directory
    Figure 2: Identifying the sample-googleassistant-master directory

Code Walk-through

Now that we have the project loaded into Android Studio, let’s take a look at some of the relevant source code.

Open the VoiceHatDriver.java file in the editor. It’s the class responsible for implementing and registering the audio user drivers. Both the audio input and output drivers implementations are private AudioInputDriver subclasses named AudioInputUserDriver and AudioOutputUserDriver, respectively.

Public functions instantiate the classes and pass them along to the UserDriverManager’s registerAudio[Input|Output]Driver method:

private class AudioInputUserDriver extends AudioInputDriver {
   // Implement driver...
}

public void registerAudioInputDriver() {
   Log.d(TAG, "registering audio input driver");
   mAudioInputDriver = new AudioInputUserDriver();
   UserDriverManager.getManager().registerAudioInputDriver(
      mAudioInputDriver, 
      mAudioFormat, 
      AudioDeviceInfo.TYPE_BUILTIN_MIC, 
      BUFFER_SIZE
   );
}

private class AudioOutputUserDriver extends AudioOutputDriver {
   // Implement driver...
}

public void registerAudioOutputDriver() {
   Log.d(TAG, "registering audio output driver");
   mAudioOutputDriver = new AudioOutputUserDriver();
   UserDriverManager.getManager().registerAudioOutputDriver(
      mAudioOutputDriver, 
      mAudioFormat, 
      AudioDeviceInfo.TYPE_BUILTIN_SPEAKER, 
      BUFFER_SIZE
   );
}

Audio data is read and written by using the native Android Things PIO APIs over an I2S (Inter-Integrated Circuit) protocol.

The AssistantActivity class makes the converse gRPC method call, which initiates or continues a conversation with the embedded Google Assistant service. Hence, it sends one ConverseRequest message with a ConverseConfig object containing the initial audio configuration. Then, multiple ConverseRequest messages are sent with the audio data of the user voice request:

// Initial request with audio
config mAssistantRequestObserver.onNext(
   ConverseRequest.newBuilder().setConfig(
      ConverseConfig.newBuilder()
         .setAudioInConfig(ASSISTANT_AUDIO_REQUEST_CONFIG)
         .setAudioOutConfig(ASSISTANT_AUDIO_RESPONSE_CONFIG)
         .build()
   ).build()
);

// Send audio commands/queries.
mAssistantRequestObserver.onNext(
   ConverseRequest.newBuilder()
       .setAudioIn(ByteString.copyFrom(audioData))
       .build()
);

Responses from the Google Assistant API are received in the form of a StreamObserver of ConverseResponse objects. Each ConverseResponse may contain exactly one of these fields:

  • EventType with a conversation event
  • ConverseResult with structured data
  • AudioOut with the assistant audio data
  • Error with a google.rpc.Status message that specifies the error for the operation.

Each of these response types are handled by the onNext() method of the StreamObserver class by using a switch statement:

// Receive ConverseResponse stream.
StreamObserver<ConverseResponse> mAssistantResponseObserver = 
   new StreamObserver<ConverseResponse>() {
      @Override
      public void onNext(ConverseResponse value) {
         switch (value.getConverseResponseCase()) {
            case EVENT_TYPE:
               // Handle events.
               break;
            case RESULT:
               // Handle query result.
            case AUDIO_OUT:
               // Handle audio data of the assistant's answer.
               break;
            case ERROR:
               // Handle the converse response error.
               break;
         }
      }
   };

Conclusion

In the next instalment, we’ll configure the Google API Console Project and learn more about the Google Assistant API.

About the Author

Head shot

Rob Gravelle resides in Ottawa, Canada, and has built Web applications for numerous businesses and government agencies. Email him for a quote on your project.

Rob’s alter-ego, “Blackjacques,” is an accomplished guitar player, who has released several CDs. His band, Ivory Knight, was rated as one of Canada’s top hard rock and metal groups by Brave Words magazine (issue #92).

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Latest Posts

Related Stories