proposal: audio package

[mattetti]

This proposal suggests a new standard audio package. The goal of this package is not to offer a library to consume audio data but instead to define a common interface that could be used by all audio libraries. This common interface would be audio.Buffer and concrete implementations would be provided.

According to the guidelines, I wrote a design document explaining things in details, it can be found here.
The proposal implementation in code is available here.
Note that this proposal is a follow up on an earlier proposal: #13432

I do realize that adding a new standard package is a big deal and that's why I kept the scope as small as possible and also use the implementation against non-trivial real code. The value added of such a package might not be immediately obvious to programmers not familiar with audio digital signal processing because end users would not see a direct benefit. However, this would be the corner stone of an entire ecosystem. Making audio libraries compatible with each other and offering a happy path means that Go developers would be able to chain a bunch of libraries and create their own audio processing chain without much effort. It's somewhat comparable of the benefits we all see in io.Reader.

Audio is everywhere and most available audio source code is written in C, but that doesn't have to be the case. With the rise of machine learning and language processing, developers wanting to play with audio data have the choice between hard to use C code and terribly inefficient Python code. Go is a great alternative and we have an opportunity to consolidate and optimize future efforts by defining a common interface.

My goal is to start the conversation now and in the case of acceptance, I would submit the code for review during the 1.9 cycle.

[nhooyr]

The first here link is dead.

[mattetti]

@nhooyr thanks, fixed.

[davecheney]

To paraphrase someone from the Python community

The standard library is where packages go to die.

Why does this need to be added to the state dad library, why can't this be something like

github.com/splice/audio

[mattetti]

@davecheney you are right and it is already here: https://github.com/go-audio/audio but here is the thing, if we want to unify audio libraries out there, we need to make the common interface standard and official. Note that this package is not a package providing codecs and audio tools. It's a package used by 3rd party package developers implementing a shared interface.

Of course all packages in the std lib could have been external and could still be moved outside but it greatly increases the friction and the chances of consolidation. If you go through the proposal, I hope it's clear that my goal is to provide common ground for all libraries. As you can imagine, if a new developer coming from company X want to implement time stretching in Go, she might start by looking at what's available in the standard library and go from there. If nothing is mentioned, or doesn't do the right google search, she might end up building a solution that wouldn't work well with others.

What I am trying to say is that the main value of having this package in the standard library is to have a shared entry point for all libraries. I fully agree that audio libraries implementing codecs, effects and other things should live on their own. That's also why I setup the go-audio organization :)

[davecheney]

But why does it need to live in the standard library? All interface declarations with the same method sets are equal and interchangeable.

[mattetti]

It's more than an interface, there are concrete implementations.
I'm not sure how to answer your question in a way that would satisfy you, can you maybe help me understand why the packages we currently have are part of the standard library? Since anything that doesn't define the langage itself can be external, there must be reasons that made us choose to have them as part of the standard library.

[davecheney]

As I understand it the majority of the standard library exists to support building Go and the go tool. @bradfitz would be the best person to answer this, he has done the archology. From my point of view, adding a package to the standard library is the kiss of death. They packages API is condemned to grow via accretion in unnatural ways, which we are seeing with the database/sql package. Note: I am specifically not taking aim at the work of those adding features to the database/sql package, simply noting that if that package lived independently from the stdlib it would have been a straight forward matter to bump the major revision of the package to add support for the context arguments added in 1.8.

…

On Tue, 3 Jan 2017, 18:38 Matt Aimonetti ***@***.***> wrote: It's more than an interface, there are concrete implementations. I'm not sure how to answer your question in a way that would satisfy you, can you maybe help me understand why the packages we currently have are part of the standard library? Since anything that doesn't define the langage itself can be external, there must be reasons that made us choose to have them as part of the standard library. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18497 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA3ikeXfN3AYl4cCDZbxk7TZJY22Hks5rOfrdgaJpZM4LZXk3> .

[egonelbre]

I'm worried about the Buffer implementation having to allocate. Maybe there's some way to wire multiple buffers together in a way that doesn't involve allocation, but I'm not sure how? Maybe you have some example of a real-time-audio-synth working on 1ms or smaller buffers?

The list of 6 conversion functions should probably work on slices rather than on single values.

[sbinet]

I am underqualified to assess whether this should live as "audio" or as "x/media/audio" (although I'd err towards the latter) and leave this aspect to more qualified people.

that said, the audio.Buffer interface has piqued my interest.
it seems to me an audio.Buffer is, conceptually, an io.Seeker + an io.Writer and/or io.Reader, so it could be mmap'ed from a file, or conveniently acquired from h/w source, or conveniently save to disk.
I suppose manipulation/indexing of []T slices is a somewhat easier API?

tangentially, do you think the "multi-dimensional slices" proposal could help reconcile the various []float64, []float32 and []int "interfaces"?
in python, audio operations are applied on "bytes-like" objects:

• https://docs.python.org/3/library/audioop.html
• https://docs.python.org/3/glossary.html#term-bytes-like-object
  ie: something that implements the buffer protocol, which IMHO is -- in Go -- something like a mix of the io.{Read|Writ}er interfaces w/ the multi-dim slice proposal.

having a "buffer protocol"-like interface would be more stdlib-worthy than audio.Buffer.

[mattetti]

@egonelbre good idea, I will post examples showing how to reuse the slice. The short version is that you have a slice and you don't have to reallocate. In what context are you concerned? In other words, how are you getting the input data that makes you think such an approach would require reallocation?

In regards to the conversion functions, can you say more about your suggestion?

[mattetti]

@sbinet yes matrices would improve things since frames would be nicer to write / read and more performant.

In regards to the bytes like buffer protocol I am not quite sure what that means concretely. When doing DSP, you have to convert the bytes into sample values, process them and convert them back into bytes at the end. But you'd want to keep the samples in a processable format until the end. The audio buffer approach is used in CoreAudio, the Cocoa framework for macOS/iOS.

The hard part here is to find the balance between performance and convenience. We want to avoid reallocations but we also want a nice API.

[egonelbre]

I mainly concerned about real-time-audio. A good example would be something like:

1. get input from keyboard/midi and generate sine-wave
2. get input from a looping and streaming PCM
3. mix 1. and 2.
4. have a gain node (adjustable)
5. have a pan node (adjustable)
6. output to device

Bonus points for:

1. when the looping/streaming PCM has different SampleRate/Format than the sine-wave generated

Basically, I'm missing the big picture how the Buffer should be used in practice.

[mattetti]

No problem, I have a generator package and effect libraries I developed against a slightly different version of the proposal, I will port them over shortly. I don't have an output to device tho, but I guess output to UI would do it.
The sample rate format change isn't an issue but if the streaming has a different format using a different data type, you do need 2 buffers to avoid reallocating.

[egonelbre]

In regards to the conversion functions, can you say more about your suggestion?

I mean when you are using IntToIeeeFloat, I assume, most of the time you are working on an []int rather than a single int. When people write the for loops manually for a particular conversion it makes more difficult to "upgrade" everyone to optimized versions of the conversions (e.g. asm optimized loops). (Alternatively, I misunderstood the purpose of the conversion functions.)