AudioFrame

Represents a frame of audio captured by the current client for its outgoing audio stream.

Audio frames are delivered to the application by the SFSAudioEvent.OUTGOING_FRAME event, if the ConfigData.AudioCfg.exposeOutgoingFrames setting is enabled.

The audio frame and its samples belong to the application, which can keep them as long as needed: they are never reused by the API. However, if multiple listeners are registered for the event, they all receive the same audio frame instance. A listener which needs to modify the samples (for example to apply a window function before calculating a FFT) must copy them first (for example using samples.slice()), not to alter what the other listeners receive; for the same reason, the samples' buffer must not be transferred to a Web Worker (the other listeners would receive an empty array).

See
Example

The following example shows how to receive the outgoing audio frames and send them to a Web Worker for analysis.

cfg.audio.exposeOutgoingFrames = true;
...

sfs.addEventListener(SFSAudioEvent.OUTGOING_FRAME, onOutgoingFrame, this);

function onOutgoingFrame(evtParams)
{
   const stream = evtParams[EventParam.MyAudioStream];
   const frame = evtParams[EventParam.AudioFrame];

   // Time offset of the frame from the beginning of the stream, in milliseconds
   const offsetMs = frame.sequenceNumber * stream.frameDurationMs;

   // The samples are copied by postMessage (don't transfer their buffer)
   analysisWorker.postMessage({ offsetMs: offsetMs, samples: frame.samples });
}

Members

(readonly) samples :Float32Array

The audio samples of the frame, as 32-bit floating point values (nominally in the -1 to 1 range); in case of multiple channels, samples are planar (all samples of the first channel, followed by all samples of the second channel, and so on).

The number of samples is sampleRate * frameDurationMs / 1000 * channels (see AudioStreamInfo).

The samples contain the captured audio signal after the audio processing performed by the browser, exactly as it is encoded and transmitted. Audio processing alters the signal in ways which are relevant if it is analyzed by the application:

  • echo cancellation (see enableAEC) removes the audio played back by the device (for example the voices of other users) from the captured signal, which prevents it from affecting the analysis;
  • noise suppression (see enableNS) applies a frequency-dependent gain to reduce background noise;
  • automatic gain control (see enableAGC) varies the signal level over time, also depending on whether speech has been detected;
  • when audio processing is enabled, the browser might also apply a high-pass filter, removing the lowest frequencies.

The actual effects of audio processing depend on the browser. If the signal level or spectrum is relevant to the analysis, consider disabling noise suppression and automatic gain control.

Type:
  • Float32Array

(readonly) sequenceNumber :number

Sequence number of the audio frame within the outgoing audio stream.

The first audio frame captured after the stream is started has sequence number 0; the value then increases by one for each captured audio frame, including those not delivered to the application because the stream is muted, or because no listeners were registered for the SFSAudioEvent.OUTGOING_FRAME event. The time offset of the audio frame from the beginning of the stream is sequenceNumber * frameDurationMs (see AudioStreamInfo#frameDurationMs).

Sequence numbers count captured audio frames, not elapsed time. If the browser doesn't capture audio for a while (for example because audio processing is suspended by the browser or the operating system), no audio frames are produced and no gap in sequence numbers results: the time offset calculated as described above can then be lower than the actual elapsed time.

Type:
  • number