Processing audio in the browser!

The modern web is an amazing place. You can do almost anything in a browser that you can do in native code. Including processing audio in realtime. In javascript.

Nuts? Probably. Worth it? Definitely.

The Web Audio API lets you generate or modify audio on the fly. For example, Google Chrome Labs has an example that generates white noise by filling a buffer with random floats. MDN has an example that displays a simulated oscilloscope of the microphone input (and it’s extremely lightweight!). It’s really quite elegant.

The way that it works is that it gives us a way to programmatically modify the audio graph by inserting your own filters and processors. The specification includes this hilariously simply illustration:

modular-routing1.png

…but essentially:

  • The AudioContext interface represents the entire audio system (the “audio graph”)
  • The audio graph contains many AudioNodes, which connect the audio “source” (microphone, video, etc…) to the “output” (usually your speakers)
  • About 14 kinds of nodes (AnalyzerNode, AudioBufferSourceNode, AudioBufferDestinationNode, etc…) that we mostly don’t care about right now
  • Also contains AudioWorklet, AudioWorkletProcessor, interfaces and an AudioWorkletNode.

The last few are the ones I’m interested in. I want to build CinemaGazer(1)(2)(3 from ACM) for the web. This part of the API should, in theory, enable me to process the audio in realtime (using some Voice Activity Detection), and fast forward through the silent parts of a video.

It’s also not that had to set up. First, you write a little tiny processor class in a separate file (and register it):

// https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkletNode
// https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkletProcessor
class CinemaGazerProcessor extends AudioWorkletProcessor {
    process (inputs, outputs, parameters) {
        return true;
    }
}

registerProcessor('cinema-gazer-processor', CinemaGazerProcessor);

Then, you need to grab the AudioContext:

let audioContext = new AudioContext();
Load your module (with a promise):

​​​​audioContext.audioWorklet.addModule('cinema-gazer-processor.js')

Then all you need to do is to connect it up:
.then(function() {
        // debugger;
        const cinemaGazerNode = new AudioWorkletNode(audioContext, 'cinema-gazer-processor')
        const videoElement = document.getElementById("video-1");
        let audioSource = audioContext.createMediaElementSource(videoElement);
    
        // let audioFilter = audioContext.create
        cinemaGazerNode.connect(audioContext.destination);
})
eafe
You’re basically all set then! You can write your processing code in the process method.
In the future, I hope to elaborate on CinemaGazer.

~ by Alexander Riccio on July 18, 2019.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

 
Lucky's Notes

Notes on math, coding, and other stuff

AbandonedNYC

Abandoned places and history from the five boroughs and beyond.

Open Mind

Science, Politics, Life, the Universe, and Everything

I learned it. I share it.

A software engineering blog by György Balássy

Untapped Cities

Rediscover your city: Urban discovery and exploration in NYC and around the world

Bit9 + Carbon Black Blog

#ArmYourEndpoints

The Electric Chronicles: Power in Flux

If someone ever tells you that you don't need more power, walk away. You don't need that kind of negativity in your life.

Ted's Energy Tips

Practical tips for making your home more comfortable, efficient and safe

love n grace

feel happy, be happy

Recognition, Evaluation, Control

News and views from Diamond Environmental Ltd.

greg tinkers

Sharing the successes and disasters.

Sam Thursfield's Blog

I want music in my life not questions!

Cranraspberry Blog

Sharing the things I love

Biosingularity

Advances in biological systems.

The Embedded Code

Designing From Scratch

Sean Heelan's Blog

Program analysis, verification and security

EduResearcher

Connecting Research, Policy, and Practice in Education

Popehat

A Group Complaint about Law, Liberty, and Leisure

Warners' Stellian Appliance

Home & Kitchen Appliance Blog

Bad Science Debunked

Debunking dangerous junk science found on the Internet. Non-scientist friendly!

4 gravitons

The trials and tribulations of four gravitons and a postdoc

Strange Quark In London

A blog about physics, citylive and much procastination

The Lumber Room

"Consign them to dust and damp by way of preserving them"

In the Dark

A blog about the Universe, and all that surrounds it

andrea elizabeth

passionate - vibrant - ambitious

Probably Dance

I can program and like games

a totally unnecessary blog

paolo severini's waste of bandwidth

Musing Mortoray

Programming and Life

PJ Naughter's space

Musings on Native mode development on Windows using C++

  Bartosz Milewski's Programming Cafe

Category Theory, Haskell, Concurrency, C++

Brandon's Thoughts

Thoughts on programming

David Crocker's Verification Blog

Formal verification of C/C++ code for critical systems

10 Minute Astronomy

Stargazing for people who think they don't have time for stargazing.

One Dev Job

notes of an interactive developer

Enterprise Architect, IoT, Cloud, Mobile Apps, Technology Evangelist, Technical Pre-Sales, Business Evangelist, Speaker

Coder/Architect for IoT, Cloud Technologies and Mobile Apps, Azure Cloud, Amazon Cloud, Windows Phone 10 Apps, iPhone Apps, Scrum Master, Business Evangelist, Mobile apps developer in iOS and Windows 10 UWP, Azure IoT Hub, Machine Learning, Stream Analytics, Azure Mobile Service, APM Tools

The Angry Technician

No, the Internet is not broken.

Kenny Kerr

Author • Systems programmer • Creator of C++/WinRT • Engineer on the Windows team at Microsoft • Romans 1:16

IT affinity!

The Ultimate Question of Life, the Universe, and Everything is answered somwhere else. This is just about IT.

Eat/Play/Hate

The ramblings of a crazed mind

Molecular Musings

Development blog of the Molecule Engine

%d bloggers like this: