feat: implement pull buffering

[Zekfad]

Description

My use case is audio book files which could span more than a few gigabytes data of MP3.

Using normal buffer will not work because you cant load 2GB MP3 on a low-end phone into RAM (and that's not even decoded samples, which requires way more memory), so I think releasing buffer is a great choice here, but you can't seek such a buffer.

Currently streaming in implemented in a push way: user pushes data as soon as it is available for engine to use.

I think that adding API for pull style buffering will solve this issue: audio engine can ask for data it's missing.
For example supplying data on demand using HTTP Range requests or reading from NAS.

There is also a case of using decryption on-the-fly, so this feature not limited to HTTP.

Requirements

• Allow seek in releasing buffer or make a releasing pull buffer with different semantic.
• Make an API for engine to request specific data range when buffering.
• Add configurable defaults when to ask for data e.g. 10 seconds before end of buffer while playing.
• Add configurable defaults for how much of data to ask in a single chunk.
• Copy supplied data, to allow reusing Uint8List on Dart side.

Additional Context

There is need for an additional logic during initial playback in case user ask to seek from the start, since you must first request file header with metadata and then you can seek to desired position. (Iff we're not supplying raw PCM)

I'm planning to cache played parts of file, but not download entire 2GB over a mobile network. For example first megabyte with metadata and then wherever user seeks to.

[DarthRainbows]

My app would benefit from this too. As is, I am polling the buffer size on a timer and pushing more data whenever the buffer gets too low. It works, but I dislike it.

[alnitak]

Sorry for my late reply. I had to think about this. And I didn't thinked enough till now :)

I think this could be a nice feature, but indeed a bit complex to achieve.

A solution could be a 3rd kind of BufferingType, something like BufferingType.onDemand with a completely new approach on C++ side and inter-communication between C and Dart.

[Zekfad]

Thanks for reply and consideration!

I also think it requires a completely different implementation with a direct access to soloud (meaning new c++ audio source and instance classes).

I believe the most difficult part is requesting that precious metadata with seek tables.

On a side note, what do you think about vbr mp3 audio in mp4 container (this takes advantage of mp4 moov to make precise seeking), is there any chance for support? But it also seems very complicated since moov can be at the end or at the beginning of the file?

[alnitak]

On a side note, what do you think about vbr mp3 audio in mp4 container

What can be done with audio is such a vast field that I can't even begin to imagine it. But I think everything (or almost everything, if you keep your feet on the ground) can be done! Also, for now, I'm alone¹ developing this plugin, which isn't small at all. So I don't think it will be possible in the short term.
That said, anyone who wants to participate is welcome :)

¹ even though Filip gave me a great help in the Dart part

[Zekfad]

It's this feature no longer desired? Surely there may be not enough resources to implement it right now, but I think it's still will be helpful to keep the issue in case someone might pick it up later

[alnitak]

Reading this before closing, I was thinking about implementing such a feature on native side. A thing that I don't think I have had the opportunity to implement. By this, I closed with "not planned". But after your latest message, I thought that maybe this should be done more easily on Dart side.

Maybe we could think about a class (in the utils or tools folder) that uses an Isolate to perform this kind of task. Anyway I don't know of a web server that provides me with the functionalities to test and start to write some code (if you have any, please tell me).

If you are willing to do something, I'll be glad to help you in any way.

[Zekfad]

I can procure some mp3 for testing, but it's possible to just generate long sine wave and it would work.
Ref: https://superuser.com/questions/724391/how-to-generate-a-sine-wave-with-ffmpeg

I'll look more into vbr, cbr and xing headers and prepare some funky files that would help testing.

Personally I don't think it's possible to cleanly implement from dart side currently, because of fundamental differences between pull and push buffering (with current push method you must have at least some headspace to push data, and you cant reliably switch data stream, for example during seek back you need to drop what's already pushed to buffer and start filling again)

As for web server I think it's not really needed, a good more generic API would only need a callback something like
Future<(Uint8List chunk, bool isEof)> requestBytes(int offset, [ int? length, ]) that should produce encoded bytes in range from offset to at most offset+length (meaning it can return 0 bytes, such EOF or network instabilities, in which case engine continue pulling until it gets EOF). With such API user can use any method to supply data - from network, file, encrypted stream.

For testing I'd recommend to start with just reading from disk by chunks, with generic API later on it would be easy to implement utilities to use network files.

Also could you point some docs and examples how to implement audio sources and audio source instances and relevant codec handling parts, I'm a little busy right now, but I could try to make some initial sketches on weekends.

If I'm reading docs (https://solhsa.com/soloud/newsoundsources.html) correctly, it should relatively easy to implement pull buffer, since audio source instance already modeled that way.

[alnitak]

Yes, the https://solhsa.com/soloud/newsoundsources.html is the starting point for this. But maybe I still don't have a clear idea of the workflow of a pulling buffer. Can you point me to some documentation or visual examples about that? Is the logic something like the YouTube buffering bar when you seek to another position?

My idea is:

1. Create the pull buffer using the stream header parameters (ie, the audio length)
2. New audio data is coming --> start decoding and playing it.
3. User seeks another position --> clear the buffer, ask for a new range of audio data --> go to 2
4. If the current playing position reaches a defined time before the end of the current buffer --> ask for new data --> go to 2

(with current push method you must have at least some headspace to push data, and you cant reliably switch data stream, for example during seek back you need to drop what's already pushed to buffer and start filling again)

regarding this, dropping data is supported by resetBufferStream. After calling that method, you can start buffering from another seek point.

Also could you point some docs and examples how to implement audio sources and audio source instances and relevant codec handling parts

I started from scratch by reading the doc you linked and the way the SoLoud author wrote soloud_wav.cpp and soloud_wav.h. I came up with the content of the audiobuffer folder.

As a side note, before starting to think of a new class inherited by the SoLoud AudioSource, we could think about extending the BufferStream because it has all the logic of a stream and can handle encoded audio. But this also depends on how much I understand all this 😄

I forgot to mention that for testing the BufferStream, I wrote websocketd, which acts as a websocket server. The websocket.dart example makes the connection to it and plays what is coming. Maybe this could be useful!

Sorry for the long message! There is much to think about.

[Zekfad]

I'm not sure as about related literature, but basically pull style API is how SoLoud Audio Source Instances already designed - engine asks for data when it's needed with minimal buffering ahead, quoting the docs:

The SoLoud::WavStream class represents a wave sound effect that is streamed off disk while it's playing.

Push API is how flutter_soloud buffer stream implemented: you push chunks via addAudioDataStream without prior knowledge if they are currently needed, buffering ahead.

With former method it's user responsibility to manage buffering ahead. It may seem as unnecessary at first, but it simplifies implementation for library and gives more freedom to users.
I'll give an example: transparently switching between reading cached chunks from filesystem and requesting new chunks when they are needed, in this example memory usage is minimized, because you're not filling buffer ahead more than engine needed and you're not keeping requested data, instead dumping it to disk.

With idea you're outlined I believe its already how AudioDataStream works: you add steam and call addAudioDataStream when data arrives which is you're pushing data to buffer asap, instead of waiting for engine to ask for it.

Also regarding duration it's not very reliably, after seeking VBR file using estimations you can end up with big difference. Just how currently chrome works - if you load big VBR Mp3 file and seek to end you'll have played time bigger than duration, because of accumulated seek error.

This issue can be eliminated by using efficient seek table (drmp3 using in library has support for arbitrary long seek tables, not just what's inside of headers, see drmp3_seek_to_pcm_frame__seek_table) which reduces long brute force decode-and-discard until seek time is reached.

My case includes targeting old phones (reducing memory usage by buffering to disk instead of memory) and saving data usage (reducing data requested by only doing so when needed).

[Zekfad]

Now that I think of it, it's like exposing Audio Source Instance API but hiding decoding under the hood, low level style API that gives enough freedom but doesn't concern user too much about how to interact with audio codecs. Tomorrow I'll look more closely to flutter_soloud internals and make draft API for further discussion, I think it will help to explain my proposal more cleanly.

[DarthRainbows]

With former method it's user responsibility to manage buffering ahead. It may seem as unnecessary at first, but it simplifies implementation for library and gives more freedom to users. I'll give an example: transparently switching between reading cached chunks from filesystem and requesting new chunks when they are needed, in this example memory usage is minimized, because you're not filling buffer ahead more than engine needed and you're not keeping requested data, instead dumping it to disk.

Push buffers are simpler in some instances and more complicated in others. The problem with push buffers is that they force the client to monitor the buffer and push only when needed to avoid buffer overflows. They certainly have their uses, and I would never suggest replacing them in the API with pull buffers exclusively. Both kinds are useful.

[Zekfad]

I would never suggest replacing them in the API

Sorry if any of my messages caused a misunderstanding. It was never an intention to replace anything, only adding new capabilities, not removing existing feature.

[Zekfad]

Here's very rough API I can think of and request your comments.
Since @DarthRainbows is also in need of this feature, I'd like to see if this would be satisfactory to your case.

api.dart

enum SeekMode {
  /// Precise seeking mode.
  ///
  /// Seek happens by decoding and discarding audio frames until desired time
  /// is achieved.
  ///
  /// It's highly recommended to have a good seek table which will allow to skip
  /// parts of stream for more efficient seeking.
  ///
  /// Works good for variable bitrate files.
  precise,
  /// Seek is done very quickly but can cause duration and playback time
  /// discrepancies when audio stream has variable bitrate.
  ///
  /// Seek table is recommend to reduce such discrepancies.
  ///
  /// Works good for constant bitrate files.
  estimated,
  ;
}

// List<int> is expected to be Uint8List most of the time, so implementation
// will have an `is` typecheck.
// More lax type will allow easier integration with HTTP streams.
typedef AudioChunk = (List<int> data, bool done);

// TODO: maybe add flac seek table later on?
sealed class SeekTable {}

// This is very low-level feature and implies that user decoded full file
// beforehand to produce precise seek points table.
//
// Modeled after drmp3_seek_point which is very precise and target exact PCM
// frames.
//
// See drmp3_seek_to_pcm_frame__seek_table
// See drmp3_calculate_seek_points
// See drmp3_bind_seek_table
typedef Mp3SeekPoint = (
  // Points to the first byte of an MP3 frame
  int byteOffset,
  // The index of the PCM frame this seek point targets.
  int pcmFrameIndex,
  // The number of whole MP3 frames to be discarded before pcmFramesToDiscard.
  int mp3FramesToDiscard,
  // The number of leading samples to read and discard.
  int pcmFramesToDiscard,
);

class Mp3SeekTable {
  const Mp3SeekTable(this.seekPoints);
  final List<Mp3SeekPoint> seekPoints;
}

abstract interface class RawAudioSource implements AudioSource {
  // TODO: constructor with callback fucntion that is usable via [getAudioChunk].
  // (and destructor or user should manage associated resources somewhere else?)

  /// Maximum size of chunk audio engine can request.
  int get chunkSize;

  /// Whether audio engine should align chunk requests.
  /// If alignment is enabled `offset` parameter of [getAudioChunk] will always
  /// be a multiple of [chunkSize].
  bool get alignChunkRequests;

  /// Minimum duration of decoded data that audio engine tries to buffer ahead.
  ///
  /// Do not set this value too big because buffer resets on seek backward and
  /// requires more RAM to store decoded PCM frames.
  Duration get bufferAheadDuration;

  /// Controls how audio engine preforms seeking.
  SeekMode get seekMode;

  /// When audio engine requires more data to decode [bufferAheadDuration] this
  /// method is used.
  ///
  /// You must return 2 values:
  /// - Audio chunk (you should not decode it, library hides codec detection
  ///   and container handling from you).
  /// - Whether audio stream has ended and no more data will be available.
  ///
  /// If you returned not full chunk when [alignChunkRequests] is enabled audio
  /// engine will request same chunk again.
  ///
  /// Returned buffer MUST NOT have length bigger than [maxSize].
  FutureOr<AudioChunk> getAudioChunk(int offset, int maxSize);

  /// Before loading audio engine will try to request custom seek table.
  /// If no such table is present it will try to load one from audio container
  /// headers, for example Mp3 XING header.
  ///
  /// This is mostly needed for Big VBR Mp3, since XING seek table is small and
  /// imprecise.
  FutureOr<SeekTable?> getSeekTable();
}

// Methods can return synchronously which allows synchronous file streaming
// iterations, such as read, process (e.g. decrypt) and playback on the fly.

I haven't looked at soloud audio source instance implementation that much but I think this API should be not very hard to implement. There are some questions left as how to manage internal buffer and multiple instances of same audio, such as what semantics has getAudioChunk when called multiple times in parallel (by different instances at different positions). Also it may be helpful to have callback similar to getSeekTable which will provide duration, or add duration to seek table.