Maestro Audio Framework  v 1.0
NXP Semiconductors
Getting started with Maestro framework

Introduction

This document describes basic usage of Maestro Audio Framework. The intent of this framework is to enable chaining of basic audio processing blocks (called "elements"). These blocks then form stream processing object ("pipeline"). This pipeline can be used for multiple audio processing use cases. The processing blocks can include (but are not limited to) different audio sources (for example file or microphone), decoders or encoders, filters or effects and audio sinks.

Framework overview is depicted on the following picture:

Supported examples

  • maestro_demo
  • maestro_usb_mic
  • maestro_usb_speaker

The examples can be found in the audio_examples folder of the desired board. The demo applications are based on FreeRTOS and use multiple tasks to form application functionality.

Streamer setup

In order to setup the audio framework properly, it is necessary to create a streamer with streamer_create api. It is also essential to setup the desired hardware peripherals using the functions described in streamer_pcm.h. The Maestro example projects consist of several files with reference to audio framework. The initial file is main.c with code to create multiple tasks. For features including SD card (in the maestro_demo example, reading a file from SD card or writing to SD card is currently supported), the APP_SDCARD_Task is created. The command prompt and connected functionalities are handled by APP_Shell_Task. User commands are handled by functions in cmd.c. Five main commands are:

  • version: prints component version
  • file: performs audio file decode and playback
  • record_mic: records MIC audio and either:
    • perform voice recognition (VIT)
    • playback on a codec
    • store samples to a file
  • usb_mic: record MIC audio and playback to USB port as an audio 2.0 microphone device
  • usb_speaker: play data from USB port as an audio 2.0 speaker device

One of the most important parts of the configuration is the streamer_pcm.c where can be found the initialization of the hardware peripherals, input and output buffer management. For further information please see also streamer_pcm.h

In the Maestro USB examples (maestro_usb_mic and maestro_usb_speaker), the USB configuration is located in the usb_device_descriptor.c and audio_speaker.c files. For further information please see also usb_device_descriptor.h and audio_speaker.h.

With creation of streamer object, the message thread is also created. The message thread is placed in app_streamer.c file, reads message queue and reacts on following messages:

  • STREAM_MSG_ERROR
  • STREAM_MSG_EOS
  • STREAM_MSG_UPDATE_POSITION
  • STREAM_MSG_CLOSE_TASK

The STREAM_MSG_UPDATE_POSITION prints info about current stream position, STREAM_MSG_EOS stops streamer, STREAM_MSG_CLOSE_TASK closes streamer thread and STREAM_MSG_ERROR stops streamer and closes thread.

Commands

file

File command is only supported in the maestro_demo example. The command creates a streamer pipeline described in function STREAMER_file_Create in file app_streamer.c. The crucial part in this is defining pipeline type as STREAM_PIPELINE_FILESYSTEM or STREAM_PIPELINE_EAP. Depending on whether the EAP_PROC is defined and on the command option, that may be either <audio_file>, all or eap. This sets up the processing chain from file source to audio sink with decoder and potentially other processing in between. Playback itself is started in STREAMER_Start function. The elements that pipeline consists of are:

  • TYPE_ELEMENT_FILE_SRC
  • TYPE_ELEMENT_DECODER
  • TYPE_ELEMENT_EAP (if EAP_PROC is defined and eap option is used)
  • TYPE_ELEMENT_AUDIO_SINK

Other possible element types could be found in streamer_api.h.

Each of the elements have several properties that can be accessed using streamer_get_property. These properties allow user to change the values of the appropriate elements. The list of properties can be found in streamer_element_properties.h. See the example of setting property value in the following piece of code from app_streamer.c:

eap_args.preset_num = *eap_par;
POST_PROCESS_DESC_T eap_proc = {EAP_Initialize_func, EAP_Execute_func, EAP_Deinit_func, &eap_args, 0};
prop.prop = PROP_AUDIOSINK_ADD_POST_PROCESS;
prop.val = (uintptr_t)&eap_proc;
streamer_set_property(handle->streamer, prop, true);
prop.prop = PROP_FILESRC_SET_CHUNK_SIZE;
prop.val = 960;
streamer_set_property(handle->streamer, prop, true);
Element Property structure.
Definition: streamer_api.h:477

Some of the predefined values can be found in streamer_api.h.

record_mic

Record_mic command is only supported in the maestro_demo example. The command creates pipeline described in function STREAMER_mic_Create in file app_streamer.c. The pipeline is selected with STREAM_PIPELINE_PCM, STREAM_PIPELINE_MIC2FILE or STREAM_PIPELINE_VIT in pipeline_type parameter. Depending on the command option, that may be either audio, file or vit. This configuration takes samples from microphone input and send it to audio sink (speaker(s) or filesystem) with optional processing in between. The elements that pipeline consists of are:

  • TYPE_ELEMENT_AUDIO_SRC
  • TYPE_ELEMENT_AUDIO_SINK or TYPE_ELEMENT_FILE_SINK or TYPE_ELEMENT_VIT_SINK

Other possible element types and some of the predefined values could be found in streamer_api.h.

usb_mic

Usb_mic command is only supported in the maestro_usb_mic example. The command creates pipeline described in function STREAMER_mic_Create in file app_streamer.c. The pipeline is selected with STREAM_PIPELINE_PCM in pipeline_type parameter. This configuration takes samples from microphone input and send it to audio sink (USB port). The elements that pipeline consists of are:

  • TYPE_ELEMENT_AUDIO_SRC
  • TYPE_ELEMENT_AUDIO_SINK

Other possible element types and some of the predefined values could be found in streamer_api.h.

usb_speaker

Usb_speaker command is only supported in the maestro_usb_speaker example. The command creates pipeline described in function STREAMER_speaker_Create in file app_streamer.c. The pipeline is selected with STREAM_PIPELINE_PCM in pipeline_type parameter. This configuration takes samples from USB port and send it to audio sink (speaker output). The elements that pipeline consists of are:

  • TYPE_ELEMENT_AUDIO_SRC
  • TYPE_ELEMENT_AUDIO_SINK

Other possible element types and some of the predefined values could be found in streamer_api.h.

Configuration options

User can change the pipeline type when creating the streamer object. Currently tested options are: STREAM_PIPELINE_PCM, STREAM_PIPELINE_FILESYSTEM, STREAM_PIPELINE_MIC2FILE, STREAM_PIPELINE_VIT and STREAM_PIPELINE_EAP. There are more pipelines defined, but supporting them is planned for future.

Supported features

Current version of the audio framework supports some optional features. These can be limited for some MCU cores or boards variants.

Codecs

The maestro_demo can play multiple files with the file option (STREAM_PIPELINE_FILESYSTEM). The opus (as standalone or as a part of ogg encapsulation) and mp3 codecs are supported for now. For detailed code handling this file extension please check cmd.c file and shellFile() function. Supported codecs and its option are:

Decoder Sample rates [kHz] Number of channels Bit depth
MP3 8, 16, 24, 32, 48, 11.025, 22.050, 44.1 1, 2 (mono/stereo) 16
opus 8, 16, 24, 48 1, 2 (mono/stereo) 16

VIT

VIT can be enabled for the record_mic command using the vit option. In the STREAMER_mic_Create function in the app_streamer.c file, the VIT_PROC definition ensures the creation of pipeline with VIT (STREAM_PIPELINE_VIT). More details about VIT are available in the VIT package, which is located in middleware\vit\{Cortex_M7, HIFI4}\Doc\(depending on the platform).

EAP

EAP can be enabled for the file command using the eap option. In the STREAMER_file_Create function in the app_streamer.c file, the EAP_PROC definition ensures the creation of pipeline with EAP (STREAMER_PIPELINE_EAP). More details about EAP are available in the EAP package, which is located in middleware\EAP\Doc\.