Object Editing Implementation

• Related issues:
• Requested reviewers:

Description

This is a proposal for the object editing interface. It mainly adresses the point that the instance where a change of the object parameters need to happen is hidden somewhere in the GetSamples API function. In the proposal the API is restructured mainly by shifting all processing steps needed to arrive at this point to the FeedFrame API function to have access to the object parameters at the correct time in the processing chain w/o the need of a callback function.

Detailed Description

• changes in the low-level decoder functions
  • refactor function ivas_jbm_dec_feed_tc_to_renderer() into two functions
    • ivas_jbm_dec_feed_tc_to_renderer(): only transfer the TC buffers
    • ivas_jbm_dec_prepare_renderer(): prepare the rest of the rendering, especially the precalculation of ISM render gains.
      • this also lead to similar refactorings for the underlying digesting functions for ParamISM and ParamMC
    • add distinct members to the ISM_RENDERER_DATA struct for the edited parameters (this is definitely open for discussion if this is the best way, or maybe simply overwrite the ISM MD at the places where they are stored for the respective modes)
    • adapt the discrete ISM renderer and the TD binaural renderer to provide a working solution for at least one IVAS mode (again open for dicussion if there is a better way)
• changes on API level
  • move setup, TC and MD decoding, and feeding the TCs to the renderer from IVAS_DEC_GetSamples() to IVAS_DEC_FeedFrame(), after calling IVAS_DEC_FeedFrame() the editable paramters may be get or set.
    • add a new buffer for binaural rendering is needed to store any flushed samples that can be produced at a rate switch, this was before directly copied to the frontend buffer in IVAS_DEC_GetSamples(), obviously this is no longer possible.
  • introduce a new function IVAS_DEC_PrepareRenderer() that basically calls the ivas_jbm_dec_prepare_renderer() function (Might be hidden in IVAS_DEC_GetSamples(), the API is anyway envisioned to be restructured at some point)
  • add a new data structure IVAS_EDITABLE_PARAMETERS that shall hold all editable parameters, again open for discussion what should really be editable
  • add new IVAS_DEC_[Get|Set]EditableParameters() for the actual editing of object paramters.

As a result basically the call sequence with object editing (in non-JBM decoding) will look like:

IVAS_DEC_FeedFrame();
if ( object_editing )
{
    IVAS_DEC_GetEditableObjects();
    /* object parameter editing happening here */
    IVAS_DEC_SetEditableObjects();
}
IVAS_DEC_PrepareRenderer();
IVAS_DEC_GetSamples();

For JBM things are a little bit more complicated since "feeding" a frame to it means only enqueing it in the JBM packet buffer, and the point of editing is hidden within the IVAS_DEC_VoIP_GetSamples() function. I tried to have it exposed to the outside also here in one possible way.

Enabled functionality

• The possibility to edit the direction (azimuth and elevation angle) and gain of objects in audio formats containing objects (ISM, supported modes of OMASA: Param and DISC, and OSBA: DISC).
• The possibility to edit the gain of the MASA- or SBA-format component in the supported OMASA and OSBA modes.
• The parameters are edited at the application level after retrieving the editable parameters from the decoder library. An example edit function is enabled by the decoder argument -obj_edit.

Affected operating points

• Bitstream is BE
• Non-edited decoder output is BE
• Providing new output with edited (object and MASA or SBA) content for ISM format, OSBA format in DISC more, and OMASA format in Param or DISC mode