Processing audio in the browser!
The modern web is an amazing place. You can do almost anything in a browser that you can do in native code. Including processing audio in realtime. In javascript.
Nuts? Probably. Worth it? Definitely.
The Web Audio API lets you generate or modify audio on the fly. For example, Google Chrome Labs has an example that generates white noise by filling a buffer with random floats. MDN has an example that displays a simulated oscilloscope of the microphone input (and it’s extremely lightweight!). It’s really quite elegant.
The way that it works is that it gives us a way to programmatically modify the audio graph by inserting your own filters and processors. The specification includes this hilariously simply illustration:
…but essentially:
- The
AudioContextinterface represents the entire audio system (the “audio graph”) - The audio graph contains many
AudioNodes, which connect the audio “source” (microphone, video, etc…) to the “output” (usually your speakers) - About 14 kinds of nodes (
AnalyzerNode,AudioBufferSourceNode,AudioBufferDestinationNode, etc…) that we mostly don’t care about right now - Also contains
AudioWorklet,AudioWorkletProcessor, interfaces and anAudioWorkletNode.
The last few are the ones I’m interested in. I want to build CinemaGazer(1)(2)(3 from ACM) for the web. This part of the API should, in theory, enable me to process the audio in realtime (using some Voice Activity Detection), and fast forward through the silent parts of a video.
It’s also not that had to set up. First, you write a little tiny processor class in a separate file (and register it):
// https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkletNode
// https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkletProcessor
class CinemaGazerProcessor extends AudioWorkletProcessor {
process (inputs, outputs, parameters) {
return true;
}
}
registerProcessor('cinema-gazer-processor', CinemaGazerProcessor);
Then, you need to grab the AudioContext:
let audioContext = new AudioContext();
audioContext.audioWorklet.addModule('cinema-gazer-processor.js')
.then(function() {
// debugger;
const cinemaGazerNode = new AudioWorkletNode(audioContext, 'cinema-gazer-processor')
const videoElement = document.getElementById("video-1");
let audioSource = audioContext.createMediaElementSource(videoElement);
// let audioFilter = audioContext.create
cinemaGazerNode.connect(audioContext.destination);
})
process method.
