-
Notifications
You must be signed in to change notification settings - Fork 2
AudYoFlo: Tutorial I: Step IV: Adding support for development and profiling in Matlab
In this part of this tutorial, we will modify our algorithm node to be run and profiled in Matlab. In case development is ongoing, a Matlab port is a useful way to analyse possible algorithm problems and to realize and verify C ports of algorithmic modifications.
Starting point for this part of the tutorial is the latest version of the code in step III. We will modify the code and finalize step IV of our tutorial.
Given a full release of the AudYoFlo platform compiled with options for Matlab support, in the runtime release folder, there will be a generic start script denoted as start-jvx-mat.bat,

In order to procede, we can double-click on the start script to run Matlab,

When doing so, additional Matlab paths are added which are required to start the graphical Matlab host. Also, two optional first commands are shown.
We start the host by using the second of the two options:
jvxHost('off-audio', '--config', 'C:\audyoflo\audio-win64\release\runtime/config.jvx', '--jvxdir', 'C:\audyoflo\audio-win64\release\runtime/jvxComponents');
Once open, we can start to configure the host to involve our new algorithm:

At first, we select the new algorithm component AudYoFlo Starter Project:

Then, we setup an input file source and some processing parameter:
- Select a file source in field
Input Specification- in this case we engagewavs/music_stereo_48000Hz.wav. - Setup two output channels for
Output. - Setup processing parameter for
Common Parameter(sample rate: 48000 Hz, in/outputchannels: 2...).
With this setup, we can run the processing by clicking start in the host UI:

Once running, a progress bar will review the curent processing progress:

Once processing is complete, we can review the output signal in the Data Management part of the UI:

Here, we can ..
- export the results to a variable in workspace,
- save the output in a wav file,
- plot the waveform in a larger plot window,
- start and stop playing audio

Now, we can store the current configuration by using the Save button in the Processing section. This will write a file named ./config.jvx. If we restart the host again, the settings in the configuration file (file source, channel, settings) will be considered right from the start.
So far, we have started the Matlab host and were able to process input data into output data. Often, however, this is not sufficient when developing algorithms with AudYoFlo. In the remainder of this section, we will step-by-step extend the source code to
- allow generic property updates in our audio node,
- specialize the project to drive our audio node,
- accessing properties using generated property access functions,
- generate a start script to start the Matlab host with a custom configuration,
- run a Matlab reference of our code in Matlab,
- involve and verify the core algorithm implementation realized in C
- run the algorithm in Matlab and c and
- allow profiling of our algorithm.
Once our audio node is loaded in the Matlab host, we can access the algorithm properties via Matlab command. Principally, there is a single function that allows to set and get all properties which is used as follows:

global jvx_host_call_global;
>> [p0 p1] = jvx_host_call_global('get_property_descriptor', 'JVX_COMPONENT_AUDIO_NODE', '/volume');
>> [p0 p1] = jvx_host_call_global('set_property_descriptor', 'JVX_COMPONENT_AUDIO_NODE', '/volume', 0.1);
>> [p0 p1] = jvx_host_call_global('get_property_descriptor', 'JVX_COMPONENT_AUDIO_NODE', '/volume');
Unfortunately, the syntax of this type of generic command is rather complex. Therefore we will customize the project to get better control of our algorithm next.
In the next step, we will at first split the algorithm to expose the functionality into two versions of our algorithm:
- The audio node without any Matlab extension and
- the audio with Matlab extensions and options for verification.
In order to do so we enter the CMakeLists.txt file in the ayfAuNstarter project and make the following modifications:

In the first highlighted part, we start to introduce a namespace environment for our audio node (namespace AYFSTARTER) by setting an additional preprocessor define.
set(LOCAL_COMPILE_DEFINITIONS_COPY "${LOCAL_COMPILE_DEFINITIONS}")
set(LOCAL_COMPILE_DEFINITIONS "${LOCAL_COMPILE_DEFINITIONS_COPY};JVX_PROJECT_NAMESPACE=AYFSTARTER")
In the second highlighted, we add lines to create a secondary copy of the component
if(JVX_USE_PART_MATLAB)
# Setup namespaces
set(LOCAL_COMPILE_DEFINITIONS "${LOCAL_COMPILE_DEFINITIONS_COPY};JVX_PROJECT_NAMESPACE=AYFSTARTERM")
# Activate the secondary Matlab project
JVX_ACTIVATE_VERSION_MATLAB(${PROJECT_NAME} JVX_EXTERNAL_CALL_ENABLED)
endif()
By using the macro JVX_ACTIVATE_VERSION_MATLAB, we generate build steps for a second version of the audio node with special settings for the Matlab integration. Note that, in this case, the namespace deviates from the namespace in the previous version.
In the source, we need to make modifications in the header file CayfAuNStarter.h and in the implementation CayfAuNStarter.cpp to introduce the namespaces by adding the lines
#ifdef JVX_PROJECT_NAMESPACE
namespace JVX_PROJECT_NAMESPACE {
#endif
as the header and
#ifdef JVX_PROJECT_NAMESPACE
}
#endif
as the footer,

Also, we have to extend the component allocation entries in componentEntry.cpp to allow support for two versions to be distinguished by the preprocessor define `` and to account for the introduced namespace,

After building the AudYoFlo system, the two versions of the audio node can be listed in the Matlab host,

We will use the version with the postfix matlab in the Matlab host for all development, verification and instrumentation while we will use the other version for all projects in which Matlab is not involved.
In the next step, we will generate access functions to set and get property values from within Matlab. These functions are based on the pcg and pcgf files given at build time. As a conclusion, we add the following line in the CMakeLists.txt file:
if(JVX_MATLAB_PROPERTY_GENERATOR)
jvx_genMatProperties(${JVX_TARGET_NAME} "JVX_COMPONENT_AUDIO_NODE" "node" "${LOCAL_PCG_FILES}" "${LOCAL_PCG_FILE_OPTIONS}")
endif()
On running the install target, an additional code generation step will be run to generate the access functions. These can be found in folder
matlab/m-files/audioProperties/+ayfAuNStarter in the AudYoFlo release-runtime folder:

Once the Matlab host is running and the algorithm is selected, we can set and get the property as follows:

Note that we need to address the correct namespace ayfAuNStarter to call the corresponding property functions.
In the next step, we will genrate a dedicated start script for our algorithm. We do so by adding the following lines in the CMakeLists.txt file:
set(JVX_MATLAB_ORIGIN_PATH "${CMAKE_CURRENT_SOURCE_DIR}/../../jvxLibraries/jvx-matlab-testscripts")
set(LOCAL_START_SCRIPT_MATLAB_SCRIPT_NAME "ayfStarter-mat")
set(LOCAL_START_SCRIPT_MATLAB ${CMAKE_CURRENT_SOURCE_DIR}/scripts/${JVX_OS}/start_mat)
In addition, a start script file template start_mat.bat.in is provided which is located in the folder scripts/windows,

When running the install build target, this modification yields the generation of the startscript start-ayfStarter-mat.bat,

When double-clicking this start script, Matlab will be started with a dedicated pre-configuration:

This configuration contains project specific presets such as, e.g., the specific configuration file ayfstarter-mat.jvx.
At this moment, it is a good idea to configure the host and store it in the specified configuration file.
In the next step, we create a Matlab reference version of the new algorithm. For this purpose, we add another folder matlab-in in our audio node project with the file jvx_init_callback.m.in,

In addition, we add another entry in the CMakeLists.txt file:
set(JVX_INSTALL_MATLAB_MINIMAL_FRAMEWORK TRUE)
When running the install build target, additional entries occur in the release-runtime folder at matlab/m-files/subprojects-audionode/+ayfAuNStarter,

When running the Matlab host, we select the following project JVX mex call Audio node to allow to choose the newly created Matlab implementation,

Once activated, we select the option ayfAuNStarter in the field with label Subproject. Then, we save the configuration file, close and afterwards restart the host.
On the restart, we see an important message at startup:

Another path is added to the Matlab path which points to the entry to our Matlab reference which will be involved during processing,
sources/Components/AudioNodes/ayfAuNStarter/matlab-local
In that folder, we find 4 Matlab functions:

These functions are involved in the processing loop as shown in the following Figure:

- The function
jvxStartOffline_localis called at processing startup, - the function
jvxStopOffline_localis called at processing stop, - the function
jvxProcessOffline_localis called for every single frame that shall be processed and - the function
jvxBeforeProcessOffline_localis called beforejvxProcessOffline_localbut only in the context of the very first frame.
This function receives all processing parameters as arguments and returns a handle which is typically used to hold algorithm specific data.
function [ jvx_handle ] = jvxStartOffline_local( ...
jvx_bsize_in, jvx_srate_in, jvx_num_in, jvx_format_in, ...
jvx_bsize_out, jvx_srate_out, jvx_num_out, jvx_format_out, ...
jvx_is_offline, precision, jvx_add_hints)
This function receives the handle and may postprocess all data that was involved during processing.
function [ ] = jvxStopOffline_local( jvx_handle )
This function is involved in the main processing loop and is called for the very first frame only before the first call to jvxProcessOffline_local. Compared to the call to jvxStartOffline_local, jvxBeforeProcessOffline_local is called as part of the processing loop. Therefore, runtime information is available that may be used to pre-allocate large matrices and other data that requires knowledge of, e.g., the data length (number of frames). It receives the handle jvx_handle and may modify data in that handle to finally return it.
function [jvx_handle] = jvxBeforeProcessOffline_local(jvx_handle)
This function is called on every step to process a buffer of input data into a buffer of output data. The data is passed to the callback as jvx_in_frame, and the task of the process function is to return variable jvx_out_frame. Audio buffer are moved around with a dimension of NxM with N as the channel number and M as the buffersize. If a wrong format is returned processing stops and an error message is displayed on the Matlab console.
In addition to the input data also the previously allocated handle jvx_handle is passed to the function and returned once the processing is done. In jvx_handle, typically, algorithm data can be forwarded from frame to frame processing, e.g., to hold and modify filter states as the memory of the algorithm.
function [jvx_handle, jvx_out_frame] =jvxProcessOffline_local(jvx_handle, jvx_in_frame)
While a frame is being processed, runtime meta data can be obtained and used via global variables. Two such variables are tpically in use:
global inProcessing;
global jvx_host_call_global;
While inProcessing holds processing loop specific data such as the overall number of frames to be processed and the value of a frame counter, jvx_host_call_global is the reference to host which may be used to, e.g., set and get property values while processing is active.
In the previous section, we have implemented the new algorithm purely in Matlab. However, the C/C++ implementation is not involved at this moment since we have activated the component JVX mex call Audio Node as a generic gateway to principally run functions in Matlab.

If we switch back to the audio node for this project, AudYoFLo Starter Project - matlab, the options to select the Matlab processing functions disappear and we can only run the C/C++ implementation.

Therefore, in the next step, we will activate the feature to involve Matlab code within our algorithm.
For the functionality to involve Matlab code when operating our audio node, at first, we need to make the following modifications in the CMakeLists.txt file of the project ayfAuNStarter:
At first, we need to add library jvx-component-templates-mex to our project. Note that this is a header-only template library, therefore, only the include folder has to setp,

In the next step, we modify the code to derive from another base class,

Instead of deriving from class CjvxBareNode1io, we involve class CjvxMexCallsProfileTpl<CjvxBareNode1io> to add another functionality layer between our class and the original base class.
Since we have derived from a new class, we need to renamce the following functions:
process_buffers_icon -> local_process_buffers_icon
prepare_connect_icon -> local_prepare_connect_icon
postprocess_connect_icon -> local_postprocess_connect_icon
in the file CayfAuNStarter.h. Of course, this change has to be done also in CayfAuNStarter.cpp. There, the code must be slidely simplified as the local_* functions are called in a template environment that allows to remove the base class invocations,

By doing so, the implementations of the functions process_buffers_icon, prepare_connect_icon, and postprocess_connect_icon from the new base class CjvxMexCallsProfileTpl<CjvxBareNode1io> will be called at first. From there, the local versions local_process_buffers_icon, local_process_buffers_icon, and local_postprocess_connect_icon can be called in a specific context.
By introducing the new base class and renaming the functions, we can change the function call hierarchy from

to

With these modifications, we can start the Matlab host again. This time we can select our audio node project (1) and activate the ayfAuNStarter sub project.

We need to restart the host and can run our algorithm in Matlab. By checking the checkbox Engage Matlab we can involve or not involve the Matlab version of out algorithm,

So far, the audio node class implementation provides a container in which the core algorithm is linked in as a library realized in C. Often, this library in C is another implementation of a core algorithm which is also available in Matlab. In that context, it is a good idea to allow the direct comparison of a Matlab implementation and the C based implementation of the core application within Matlab. This approach is supported by AudYoFlo by providing code generation tools and methods to define very specific side calls to dedicated exposed functions. This will be explained and demonstrated in the following.
Typically, all access to the audio node is via the AudYoFlo APIs. That includes, e.g., the call of functions to switch component states, to process buffers or to produce configuration entries to be stored in a config file. If we desired to invoke a very specific function such as a core processing function realized in a C library, it may be rather difficult to involve that function alone - without the overall system around.
In the Matlab host implementation a feature has been integrated that allows the definition and calls to functions that do not comply with the main APIs. This is demonstrated by the following Figure:

In that Figure, the Matlab console is used to control the host and, in that context the audio node via the AudYoFlo APIs. However, there is one path in the interaction graph highlighted in red color. In order to realize this kind of function, we require
- A core data processing function realized in C - in our audio node this is the function
ayf_starter_processwhich is part of theayfstarterlib, fileayfstarterlib.clocated in foldersources/Libraries/ayfstarterlib/src/ayfstarterlib.c. - A way to let Matlab know that an additional function is exposed besides the given high level API functions in AudYoFlo
- A dedicated member function that links Matlab calls to the core library function.
In AudYoFlo, we can use the Matlab Code Generation Tool(MCG) to define non-API functions that can be exposed via Matlab. The following code modifications are required:
At first, we need to add the files CayfAuNStarter_mex.cpp and exports_project.mcg to the list of source files in CMakeLists.txt for the additional part where the Matlab version of our algorithm is specified,

The entry
set(JVX_MCG_LOCAL_FILE ${CMAKE_CURRENT_SOURCE_DIR}/mcg/exports_project.mcg)
refers to the file for the definition of all Matlab entry functions and the entry
set(LOCAL_SOURCES ${LOCAL_SOURCES} ${CMAKE_CURRENT_SOURCE_DIR}/src/CayfAuNStarter_mex.cpp)
adds a new file CayfAuNStarter_mex.cpp in which all Matlab specific functions are implemented.
File exports_project.mcg is located in folder sources/Components/AudioNodes/ayfAuNStarter/mcg and contains entries to define entry functions to be addressed from within Matlab,
SECTION MATLAB_EXPORTS
{
REFERENCE_CLASS = "CayfAuNStarter";
OUTPUTFILE_NAME = "CayfAuNStarter_external";
SECTION ayf_starter_lib_process
{
DESCRIPTION = "Simple example function to call core processing function";
ACCEPT_INPUT_TYPES = { "JVX_DATAFORMAT_DATA"};
DIMENSION_INPUT_FIELD = {"2D"};
DESCRIPTION_INPUT_PARAMETERS = { "Input signal, NxM"};
ACCEPT_INPUT_NUMBER_MIN = 1;
ACCEPT_INPUT_NUMBER_MAX = 1;
PRODUCE_OUTPUT_TYPES = {"JVX_DATAFORMAT_DATA"};
DIMENSION_OUTPUT_FIELD = {"2D"};
DESCRIPTION_INPUT_PARAMETERS = { "Output signal, NxM"};
ACCEPT_OUTPUT_NUMBER_MIN = 1;
ACCEPT_OUTPUT_NUMBER_MAX = 1;
INPUT_OUTPUT_CROSS_REFERENCE_X = {0};
INPUT_OUTPUT_CROSS_REFERENCE_Y = {0};
};
};
At this place, a function ayf_starter_lib_process is defined that expects 1 input and 1 output argument, each N x M matrices with N as the number of channels and M as the number of samples (buffersize).
The header file for our audio node needs to be modified to hold the function declarations resulting from the code generator,

The includes are conditional due to the pre-processor define JVX_EXTERNAL_CALL_ENABLED which is only set in the Matlab version of our node, ayfAuNStarterm. Besides the generated header file, we need to involve two functions
void initExternalCall();
void terminateExternalCall();
to register and unregister the new functions.
The new file CayfAuNStarter_mex.cpp is added to our project and provides the required implementations to realize the Matlab specific functions,

The two function at the beginning are required to expose the new function via the host whereas the function ayf_starter_lib_process``contains the actual implementation of the exposed function. In the code, the core library ayfstarterlibwith the provided functionsayf_starter_init, ayf_starter_prepare, ayf_starter_postprocessandayf_starter_lib_process` is obviously linked.
In order to use the functions, the project is built and we start Matlab using the start script.
Once the host is up and running, we can get access to the host and query all exposed functions with
global jvx_host_call_global
[a, entries] = jvx_host_call_global('external_call');
Our newly defined function is entries{2}. From displaying the parameters of the function call we can run a test call
[a, out] = jvx_host_call_global('external_call', 'CayfAuNStarter', 'ayf_starter_lib_process', randn(2,128));
to pass a 2 x 128 buffer to our function and process the data to output the result to variable 'out`,

In the final step, we can involve the core function in our Matlab processing environment. We do so by adding the lines from console in the function

With this modification, we can run the algorithm and verify every single frame.
Once the Matlab host is running, we can enable and disable the Matlab processing by using the checkbox with the label Enable Matlab. The C code is always executed as there is no option to deactivate it so far. In some case, it is desired to run only the Matlab code. For this purpose, we now add additional options to allow better control of the processing behavior.
We start our objective by adding a new file ayfstarter-develop.pcgf in folder sources/Components/AudioNodes/ayfAuNStarter/pcg" to define additional properties:

There is only one property skipInvolveCCode that can be used to decouple C processing from processing when the host processes data.
In the property definition file exports_node.pcg we need to add an include statement to involve the new pcgf-file,
include "ayfstarter-develop.pcgf";
However, the developer options shall only be available in the Matlab version of our node. Therefore, we engage the statement in an ifdef-clause,

Now, we need to add the options to the CMakeLists.txt file,

set(PCGPOSTFIX "m")
set(LOCAL_PCG_FILE_OPTIONS ${LOCAL_PCG_FILE_OPTIONS} -D JVX_COMPILE_FOR_DEVELOP)
The highlighted statement is used to realize the following behavior:
- Add an additional option
-D JVX_COMPILE_FOR_DEVELOPto run the property code generation tool with the ifdefJVX_COMPILE_FOR_DEVELOPenabled. - Output the generated header with the
PCGPOSTFIX. I our case, the generated header file is postfixed by am.
This modification is then considered in file CayfAuNStarter.h,

In the final step, the new property skipInvolveCCode must be linked to member variable config.skipInvolveCCode which is part of template class CjvxMexCallsProfileTpl,

Once these modifications are done, the system can be build. We use the start script to start Matlab and also the Matlab host. The new property can be accessed using the generated property functions
ayfAuNStarter.get_audio_node_develop_skipInvolveCCode
ayfAuNStarter.set_audio_node_develop_skipInvolveCCode

In the next step, we start creating test scripts for our new algorithm. For this purpose, we create a new folder sources/Libraries/ayfstarter-matlab-testscripts and store a script ayf_starter_run_test.m in that folder,

This new script can be run on the Matlab console and
- Starts the Matlab host with a specific configuration file,
- runs the processing to process an input signal into an output signal and
- stores the output signal in a wav file.

When running, the host opens, processes the full input source (0-100% on the progress scale), closes and stores the output file.

In the next step, we enable the Matlab profiling. This requires to wrap the profiling functionality given in the base class by a property. Again, we define the property involveProfiling in file ayfstarter-develop.pcgf,

and extend the association to the member variable config.matlab_profiling_enabled in template class CjvxMexCallsProfileTpl when activating class CayfAuNStarter,

Once build with these modifications, we can activate profiling in the Matlab test script,

By doing so, the profiling scripts in the subfolder matlab/m-files/profiling of our release-runtime folder will be involved:
- Script
jvxProfileConfigto be called before starting processing. Here, mostly processing parameters can be set. - Script
jvxProfileStartto be called in the processing loop right before the very first frame is processed. Here, we can request information about the length of the file and similar. - Script
jvxProfileStepto be called when processing of one frame has been completed. Here, data can be collected to be analyzed afterwards. - Script 'jvxProfileStop` to be called once processing is over. Here, we can analyze the data collectd during profiling.
Even though involved, at the moment, there is not too much to see that profiling is enabled.
The 4 profiling scripts shall now be used to forward profiling issues to local callbacks. For this purpose, we define 4 functions,

-
jvxLocalConfig: Forward call fromjvxProfileConfig -
jvxLocalStart: Forward call fromjvxProfileStart -
jvxLocalStep: Forward call fromjvxProfileStep -
jvxLocalStop: Forward call fromjvxProfileStop
These functions need to be assigned to the profiling substem as folows:

When running the script ayf_starter_run_test.m, the local profiler scripts are called. We see the output from the local profiler functions:

It has proven to be efficient and accurate to collect algorithm data at so-called testpoints. By defining testpoints, we define positions in the algorithm at which the algorithm data is collected. If we run an algorithm in C and Matlab, the implementation may be completely different but for a verification, data at the testpoints must match!
We add another property to our develop property set to define a single testpoint TP0,

In the next step, we need to start to collect algorithm data in Matlab. We need to activate the testpoint in the local profiler function jvxLocalConfig in file ayf_starter_run_test.m,

Then, we need to store the data that is available in the algorithm (file jvxProcessOffline_local.m in sources/Components/AudioNodes/ayfAuNStarter/matlab-local) in the profiling datastruct,

On the other side, we need to collect data also in the C implementation. Typically, the algorithm data is available in the C core library. Therefore, we need to extend this library as follows:
- We need to add C code for the collection, namely files
sources/Libraries/ayfstarterlib/include/ayfstarterlib_debug.handsources/Libraries/ayfstarterlib/include/ayfstarterlib_debug.cpp,

and

- We need to create two versions of our library, one for debugging and one for non-debugging mode. We do so by adding 4 lines in the file
sources/Libraries/ayfstarterlib/CMakeLists.txt,

- We need to add an additional debug struct handle to allow debugging data and the two member functions to allocate profiling structs to our class
CayfAuNStarterin file `sources/Components/AudioNodes/ayfAuNStarter/src/CayfAuNStarter.h,

- We need to add the profiling allocation functions for our class
CayfAuNStarterin filesources/Components/AudioNodes/ayfAuNStarter/src/CayfAuNStarter_mex.cpp,

- We need to add some properties in file
sources/Components/AudioNodes/ayfAuNStarter\pcg/ayfstarter-develop.pcgf, to activate the testpoints (testpoints) and to read back algithm data while in operation (bsizeSignal),

- Finally, in our C algorithm in file
sources/Libraries/ayfstarterlib/src/ayfstarterlib.c, we need to actually collect the data in the core processing function,

Once all the modifications have been added, it is now possible to run the profiling script in Matlab.
After running the start script, we review the test script ayf_starter_run_test.m.
a) In there, we modify the local profiler function jvxLocalConfig to enable the testpoint 0 using a property command and to disable the option to skip C code,

We will run Matlab and C version of the core algorithm and collect data from both versions and compare afterwards - to make sure the algorithm works in identical manners.
b) In the profiler function jvxLocalStart, we will prepare collecting data in the Matlab workspace. Mainly, we pre-allocate huge matrices that can hold the testpoint buffers for all frames,

In the function, we read hdl.nFramesto make sure we allocate space for all buffers in the current processing and hdl.hooks.data.M read from the property system of the host to find out the size of each buffer to be stored. We create one matrix hdl.hooks.data.matSignalC_cbr for the data collected in C code and one matrix hdl.hooks.data.matSignalMat_cbr for the data collected in Matlab code. The involved datatype jvx_dsp_base.cbr.jvxMatrixReference in that context is a huge matrix which, however, is treated as a handle (call-by-reference)and shall not be allocated when adding or reading data. In the function, we add one buffer originating from the C code (``)
c) In profiler function jvxLocalStep, we actually collect all buffers and attach the vectors to the previously allocated matrix,

d) In the profiler evaluation script jvxLocalStop the collected data is final evaluated,

Vectors are taken both collection matrices, hdl.hooks.data.matSignalC_cbr and hdl.hooks.data.matSignalMat_cbr, and compared to compute a squared log error measure. All log error measures are collected and are finally display in a linear plot,

In the shown example, the log error is -200 (dB) for all more than 12000 buffers since the two versions of the output buffers are completely identical!