How a Mel Spectrogram Works
A mel spectrogram begins with a standard spectrogram, which is computed by applying the Short-Time Fourier Transform (STFT) to an audio signal. The STFT breaks the signal into short, overlapping time frames and calculates the frequency content of each frame. The result is a two-dimensional representation with time on one axis and frequency on the other, where intensity indicates energy at each time-frequency point.
The key difference between a mel spectrogram and a standard spectrogram is the frequency axis. A regular spectrogram uses a linear frequency scale, while a mel spectrogram warps the frequency axis using the mel scale. The mel scale is a perceptual scale of pitch: it spaces frequencies so that equal distances on the scale correspond to equal perceived differences in pitch. In practice, this means lower frequencies are given finer resolution and higher frequencies are compressed together, reflecting how the human auditory system works.
The Mel Spectrogram Formula
To produce a mel spectrogram, a bank of overlapping triangular filters (called a mel filterbank) is applied to the power spectrum of each STFT frame. Each filter sums the energy within its frequency band. The classic mel scale conversion from frequency f in hertz is:
m = 2595 × log10(1 + f / 700)
The output is typically converted to a logarithmic (decibel) scale, producing what is often called a log mel spectrogram. The log compression better matches human loudness perception and improves the numerical stability of downstream models.
Mel Spectrogram vs. MFCC
Mel-frequency cepstral coefficients (MFCCs) are derived from mel spectrograms by applying a discrete cosine transform (DCT) to the log mel filterbank energies. MFCCs were long the default feature for speech recognition. Modern deep learning systems, however, often skip the DCT step and feed mel spectrograms directly into neural networks, because the networks can learn their own optimal transformations from the richer representation.
Applications in Speech and Voice AI
Automatic speech recognition (ASR): Mel spectrograms serve as the primary input to models like Whisper, Conformer, and other end-to-end architectures.
Text-to-speech (TTS): Systems such as Tacotron predict mel spectrograms from text, then convert them to audio waveforms using a vocoder.
Speaker verification and voice cloning: Mel spectrograms capture speaker-specific timbral characteristics used for identity modeling.
Audio classification: Environmental sound detection and music analysis also rely on mel spectrogram features.
Computing Mel Spectrograms in Python
Libraries like librosa make it straightforward to compute mel spectrograms in Python. The function librosa.feature.melspectrogram() accepts an audio signal and returns the mel spectrogram matrix, with parameters for the number of mel bands, FFT window size, and hop length. Visualization is typically done with librosa.display.specshow() alongside matplotlib.