Sound Design Applications

Really proud to share something we’ve been working on for a while: Magenta RealTime 2 (MTR2), a live music model that is highly interactive (MIDI, audio, text, lots of parameters) and low-latency (~200ms end-to-end), and runs locally on a MacBook! We’re releasing the weights together with an OSS library (JAX/MLX), a C++ inference engine for efficient streaming generation, and a few app examples, including a plugin to run the model directly in your DAW 👇 Apps & plugins: https://lnkd.in/eGpxj9uC Blog post: https://lnkd.in/eShR5m5P  Code: https://lnkd.in/e4QPPRVy  Models: https://lnkd.in/e5Eyvrda Getting to such high-quality and controllable real-time audio synthesis has taken a lot of research and technical development. The blog has details on this (and there will be more to come), but in a nutshell here are some key elements. We get to high responsivity via frame-level AR with 40 ms frame size. This comes with challenges (e.g. growing context, length extrapolation, drifting) which we address with a combination of sliding window attention, sink embeddings, and careful design of information propagation through layers. As for the controls, we design the model to be robust to any combination of input signals via a masking scheme that becomes a creative tool at inference: different masking patterns result in different behaviour which we can compose and modulate into several “playing modes”. I have a bias for work that opens new possibilities instead of rehashing old ones, and MRT2 makes a lot of new things possible, while still having the level of control and interaction we expect from music-making devices. Most importantly, it’s really fun, malleable and hackable!

The filter comparison shows how different low-pass filters affect a signal as frequency increases. All four filters allow low-frequency signals to pass, but they reduce higher frequencies in different ways. The choice of filter changes signal smoothness, distortion, phase behavior, and noise rejection. Chebyshev filters remove unwanted frequencies very quickly, but the ripple in the passband can slightly distort signal amplitude. This makes the signal less smooth even before the cutoff region. Butterworth filters provide the flattest response in the passband. Signals remain smooth and natural with minimal amplitude distortion, which is why Butterworth filters are commonly used in audio and general-purpose analog circuits. Bessel filters focus on preserving waveform shape and timing. Their slower roll-off means they remove noise less aggressively, but pulses and transient signals stay cleaner with less phase distortion. This is important in communication and data systems. Elliptic filters provide the sharpest cutoff, blocking unwanted frequencies very efficiently. However, ripple appears in both the passband and stopband, which can introduce ringing and waveform distortion in sensitive signals. The graph demonstrates that sharper filtering usually comes with more signal distortion or ripple, while smoother signal preservation often requires a slower roll-off. #ElectronicsEducation #ElectronicsRD #LowPassFilter #SignalProcessing #AnalogFilter

Neural Audio Resynthesis in action: an original guitar sample (green) and three generated variations (white) fully synthesized from the extracted DNA of the original sound. And by DNA, we mean something very specific. Not just spectral character, but the room acoustics, the signal chain, the effects processing, and critically, the articulation and physical playing style of the musician who recorded it. *The full latent identity of a sound.* This is a problem the research community has been chasing for a long time. How do you disentangle the structural and compositional qualities of a sound from its tonal and environmental ones? How do you separate what an instrument is from how it was played and where it was recorded? Our ML R&D team continues to push the boundaries of what's possible with sound. We believe this introduces a new paradigm in the context of audio synthesis and editing technology. One that will bring a universe of creative possibilities in the hands of creators. The Splice catalog is becoming dynamic, adaptable, liquid.. Every sample carries not just audio data but an expressive latent identity that can be explored, varied, and steered. I like to call it semantic sound design. You can try Variations today via our Splice Plugin. We're just getting started. More to share soon! #machinelearning #audio #sound #musicproduction #musicindustry #artificialintelligence #ai

🚀 SupertonicTTS: Achieving Efficient and Scalable Text-to-Speech 🚀 Recent text-to-speech (TTS) systems have achieved remarkable feats, enabling natural voice synthesis for unseen speakers through techniques like zero-shot learning and offering functionalities within single models such as voice conversion and multilingual synthesis. Yet, must we always equate cutting-edge TTS with massive parameter counts, enormous datasets, and intricate pipelines involving grapheme-to-phoneme (G2P) modules and external aligners? The core innovation here lies in a novel system. The system consists of three modules: 👉 First, a speech autoencoder encodes input audio in the form of mel spectrograms into a continuous, low-dimensional latent space while preserving quality. This latent space, along with temporal compression, significantly reduces the computational cost of subsequent processing steps. 👉 Second, a text-to-latent module maps raw character-level text and speaker information to these temporally compressed latents. It utilizes a flow-matching algorithm to estimate a conditional probability path, generating latents from noise conditioned on text and reference speech. To enhance training efficiency and convergence, the system introduces context-sharing batch expansion, a technique that reuses conditional variables across multiple perturbed inputs. This method effectively mimics the benefits of a larger batch size while keeping computational costs low and improving text-speech alignment learning. 👉 Finally, instead of traditional phoneme-level duration models, SupertonicTTS employs an utterance-level duration predictor to estimate the total duration of synthesized speech. The system is designed to be lightweight and efficient, incorporating ConvNeXt blocks across all modules. Additionally, it simplifies the synthesis pipeline by using cross-attention mechanisms for text-speech alignment, operating directly on raw character-level text without requiring G2P modules or external aligners. The speech autoencoder was trained on 11k hours of audio, and the text-to-latent module and duration predictor were trained on 945 hours of English speech data using NVIDIA RTX 4090 GPUs. Notably, SupertonicTTS achieves competitive zero-shot TTS performance with only 44M parameters and exhibits exceptionally fast inference speeds (0.23 seconds for 10-second speech), significantly outperforming models with hundreds of millions or even over a billion parameters. This signifies a promising direction for future TTS research, suggesting that highly scalable and efficient systems with competitive performance can be developed with significantly reduced resources, potentially democratising access and enabling real-time applications more broadly. More Details: https://lnkd.in/gv8me4Va #TTS #TextToSpeech #AISpeech #SpeechSynthesis #ScalableTTS #EfficientAI #LatentRepresentation #FlowMatching #ZeroShot #SupertonicTTS

𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐣𝐮𝐬𝐭 𝐨𝐩𝐞𝐧-𝐬𝐨𝐮𝐫𝐜𝐞𝐝 𝐚 𝐓𝐓𝐒 𝐬𝐲𝐬𝐭𝐞𝐦 𝐭𝐡𝐚𝐭 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐞𝐬 𝟗𝟎-𝐦𝐢𝐧𝐮𝐭𝐞 𝐦𝐮𝐥𝐭𝐢-𝐬𝐩𝐞𝐚𝐤𝐞𝐫 𝐜𝐨𝐧𝐯𝐞𝐫𝐬𝐚𝐭𝐢𝐨𝐧𝐬. VibeVoice dropped this month with 10.2k GitHub stars. The technical approach is worth understanding. The core problem with long-form speech synthesis: context windows. Most TTS models struggle past a few minutes because audio tokens accumulate fast at typical frame rates (50-100 Hz). VibeVoice runs at 7.5 Hz. They built continuous speech tokenizers (acoustic and semantic) that preserve audio fidelity at this ultra-low rate. The architecture combines an LLM (Qwen2.5 1.5B) for dialogue understanding with a diffusion head for generating acoustic details—a next-token diffusion approach. Two model variants: The long-form model synthesizes conversations up to 90 minutes with 4 distinct speakers. Speaker consistency across that duration is the hard part most teams skip. The realtime model hits ~300ms time-to-first-audio with streaming text input. Useful for voice agents where latency compounds. What actually works: 🩸 English and Chinese 🩸Cross-lingual voice transfer 🩸Spontaneous singing (the demos are surprisingly good) Current limitations: 🩸No overlapping speech modeling 🩸No background audio/music 🩸English and Chinese only MIT licensed. Weights on HuggingFace. For anyone building voice interfaces or podcast automation: this is the best open-source baseline I've seen for multi-speaker long-form synthesis. The 7.5 Hz tokenizer design alone is worth reading the technical report. Link: https://lnkd.in/dzcr4-Bi

Researchers have created a new technology that can send sound only to a specific person, even in a crowded space. Imagine listening to music or having a private conversation without headphones and without disturbing anyone nearby. This breakthrough could change the way we experience sound in public spaces, from entertainment to communication. Normally, sound waves spread out as they travel, which makes it difficult to control where the sound goes. But researchers have found a way to focus sound using ultrasound, a type of sound that is too high-pitched for humans to hear. By using two silent ultrasound beams at different frequencies, they create a new sound wave in the spot where the beams meet. This new sound is the one that can be heard, but only in that particular spot. What makes this technology even more impressive is that the ultrasound beams can bend and follow a curved path, similar to how light can bend through a lens. This allows the sound to reach a specific location, even around obstacles. The sound is silent until it reaches the target area, making it possible to deliver sound only to one person without anyone else hearing it. While this technology isn’t ready for widespread use yet, it could have huge impacts on things like personalized audio in public places, creating quiet zones, and even improving privacy in conversations. However, challenges like sound quality and energy use still need to be worked out. Research Paper 📄 https://lnkd.in/ePCNqBig

The Fourier Transform is one of the fundamental mathematical tools in science and engineering, allowing us to analyze and understand complex signals by breaking them down into simpler components. At its core, it is a method for converting a function from the time domain (where signals are expressed as they evolve over time) into the frequency domain (where signals are represented as a sum of different frequency components). This transformation is essential because many natural and engineered processes, from sound waves to electrical signals and even quantum mechanics, can be better understood in terms of their frequency content rather than their raw time-based behavior. To grasp the essence of the Fourier Transform, one must first understand the idea that any continuous signal, no matter how complex, can be represented as a combination of sinusoidal functions—specifically, sine and cosine waves—each with a particular frequency, amplitude, and phase. This concept is rooted in the field of harmonic analysis and is a direct consequence of the superposition principle, which states that complex waveforms are merely the sum of simpler waveforms. The Fourier Transform identifies these individual frequency components and determines how much of each is present in the original signal. The transformation works using complex exponentials, a fundamental idea in mathematics where a signal can be expressed using Euler's formula, involving imaginary numbers and the exponential function. Mathematically, the Fourier Transform provides a function called the frequency spectrum, which tells us how the energy of a signal is distributed among different frequencies. The reverse operation, known as the Inverse Fourier Transform, allows us to reconstruct the original signal from its frequency representation, proving that no information is lost in this transformation. When working with digital signals—such as in computers, digital audio, or image processing—scientists use a discrete version of this method called the Discrete Fourier Transform (DFT). To make this computation more efficient, especially for large datasets, the Fast Fourier Transform (FFT) is employed, which significantly reduces computational time and is widely used in modern applications. The Fourier Transform is not just a theoretical tool; it has vast real-world applications. In signal processing, it helps filter unwanted noise, compress data (as seen in the JPEG format for images), and analyze audio and radio signals. In physics, it plays a crucial role in quantum mechanics, optics, and heat conduction, helping scientists solve complex differential equations that describe wave behavior and diffusion processes. In engineering, it is indispensable for analyzing electrical circuits, designing antennas, and optimizing communication systems. Even in machine learning and AI, Fourier analysis is used for extracting features from signals and improving pattern recognition techniques.

Imagine you have an analog signal from a sensor. After digitizing it with an ADC, you’re ready to analyze it on a microcontroller or DSP to track trends. However, the signal is full of unwanted high-frequency noise, making it difficult to see the data you need. To fix this without changing the hardware, you can use a digital low-pass filter to remove the high-frequency noise. One way to build this filter is with a windowed-sinc FIR filter: Set the Cutoff Frequency: Decide the highest frequency you want to keep. The filter will allow lower frequencies through and block higher ones. Create the Sinc Function: The sinc function forms the base of the filter and lets frequencies below the cutoff pass through. Apply a Window Function: To reduce side effects from cutting off the sinc function, multiply it by a Hamming window. This smooths the filter and gives better results. The resulting FIR filter can then be applied to the noisy signal, allowing you to focus on the main information while ignoring unwanted noise. This simple digital filter technique is effective in signal processing when hardware changes aren’t possible.

We have a new paper out, "SUNAC: Source-aware Unified Neural Audio Codec", led by our Mitsubishi Electric colleague Ryo AIHARA during his one-year visit at Mitsubishi Electric Research Laboratories. While most neural audio codecs encode mixtures of multiple sources in an entangled manner, we propose SUNAC, a source-aware codec that encodes individual sources directly from mixtures, conditioned on source type prompts. This enables user-driven selection of which source(s) to encode, including separately encoding multiple sources of the same type (e.g., multiple speech signals). SUNAC achieves competitive resynthesis and separation quality relative to a cascade of source separation followed by a conventional NAC, with lower computational cost. Joint work with Yoshiki Masuyama, Francesco Paissan, François Germain, and Gordon Wichern. https://lnkd.in/dXuVwZ9z

🚀 IEEE WASPAA 2025 paper alert: "Combolutional Neural Networks" 🎶🧠 Excited to share Cameron Churchwell's new work accepted to #WASPAA2025 — “Combolutional Neural Networks.” (It's not a typo!) Audio signals often have a harmonic structure, but common architectures like CNNs tend to miss this rich inductive bias. We introduce the combolutional layer, a learnable IIR comb filter designed to extract harmonic features directly from raw audio. 🔍 Why combolutional layers? ✅ Precisely capture harmonic patterns in the time domain ✅ 1,000× fewer MACs than convolutional layers ✅ Fully differentiable, interpretable, and efficient for on-device use ✅ We open-source GPU training kernels with Triton 📊 We test it on: 🎹 Piano transcription 🗣️ Speaker classification 🎼 Musical key estimation Across the board, our CombNet models achieve comparable or superior performance while drastically reducing parameter count and computational cost. Check out the code & paper below: 🔗 GitHub: https://lnkd.in/gZvqGxGT 📄 Paper: https://lnkd.in/gdefBVRj A big shoutout to Cameron for his first first-authored paper in his PhD studies!