Deep Learning Audio Denoising Model

20 Jun 2024          github repository

Quick demonstration of the model capabilities:

Caution: The noised audio files may be a little loud

Authors

• Piotr Wojsławski @piotrwojslawski

• Dominik Klisiewicz @DominikKlisiewicz

• Karolina Szachniewicz

Project Overview

The primary objective of this project is to develop a model capable of reducing noise in audio recordings, enabling clearer detection and understanding of human speech. The noise in this project is specifically rain and thunderstorm sounds. The original audio comprises human speech readings.

Data Sources

• Speech Corpus: The dataset used for human speech is from OpenSLR. For this project, only recordings by women, with durations between 3 to 5 seconds, were included.
• Noise Samples: The noise data, consisting of rain and thunderstorm sounds, is sourced from the ESC-50 dataset.

Data Preprocessing

1. Dataset Creation: A new dataset was generated by adding random noise samples to each speech recording. This resulted in a collection of audio files containing both the original speech and the superimposed noise. Only two types of noise sounds (aforementioned rain and thunderstorm sounds) were added due to computational restraints of authors - models were computed on private machines.
2. Feature Extraction: Each audio file was transformed into the Short-Time Fourier Transform (STFT) format using Fourier transformation to obtain a more informative representation of the data.
3. Data Splitting: The dataset was divided into training (80%), validation (10%), and test sets (10%).
4. Normalization: All datasets were normalized using the minimum and maximum values derived from the training set.

Model

1. Model Architecture: The model processes noisy speech audio after STFT transformation and normalization. The output is the denoised speech audio, which is subsequently de-normalized and converted back to the audio format. The loss function used is Mean Squared Error (MSE), which measures the difference between the denoised audio and the original clean audio.
2. Encoder-Decoder Structure: The model comprises an encoder and a decoder, each consisting of multiple convolutional layers.
   • Encoder: The encoder compresses the input audio representation into a lower-dimensional space. It consists of three convolutional layers, each followed by batch normalization, max pooling, and dropout for regularization. The final layer further encodes the data into an even more compact representation.
   • Decoder: The decoder reconstructs the audio from the encoded representation. It employs transposed convolutional layers to upsample the data back to the original dimensions. Each layer is followed by batch normalization and dropout to prevent overfitting. The final layer produces the denoised audio output.

Training Process

1. The autoencoder model was trained using an Adam optimizer, which dynamically adjusted the learning rate based on the batch size. The initial learning rate was set to adapt proportionally to the batch size relative to a base rate of 0.01. The model architecture comprised an input shape of (600, 264, 1), configured to reconstruct encoded representations through an encoder-decoder framework.
2. During training, model checkpoints were employed to save the best-performing model based on validation loss. Early stopping was implemented to cease training if the validation loss did not improve after 20 epochs, ensuring efficient model convergence. Additionally, a learning rate reduction strategy was applied, reducing the learning rate by a factor of 0.5 if no improvement in validation loss was observed after 3 epochs.

Robot-like outcomes were a big obstacle during training

Noised Audio

Denoised, Robotic Audio

3. The model underwent training over 100 epochs with a batch size of 16. The best-performing model was selected based on validation loss. Post-training, the achieved mean squared errors were as follows:
   • Training Set: 0.013817
   • Validation Set: 0.013727
   • Test Set: 0.014147

Final Results

This is the visualization of the audio corrupted by rain noise

This is the visualization of the same audio after undergoing a denoising process

This is the visualization of the original audio before any noise was added