LSTM-based Speaker Diarization System

This repository contains the implementation of an advanced LSTM-based speaker diarization system as described in the paper "Optimizing LSTM-based Speaker Diarization: Comparing Architectures and Clustering Methods" by Michael Reinegger.

Speaker diarization is the process of partitioning an audio stream into homogeneous segments according to the speaker's identity. This system uses Long Short-Term Memory (LSTM) neural networks to generate d-vectors from audio features, which are then clustered to identify unique speakers.

Key features of this implementation include:

• LSTM and BiLSTM architectures for d-vector generation
• Multiple clustering algorithms for speaker segmentation
• Comprehensive hyperparameter tuning
• Extensive experimental analysis

1. Feature Extraction: Mel-spectrograms are extracted from 30ms audio segments.
2. Voice Activity Detection (VAD): WebRTC VAD is used to filter out non-speech segments.
3. Neural Network: LSTM/BiLSTM network generates d-vectors from speech features.
4. Clustering: Various algorithms (Spectral, DBSCAN, K-Means, Agglomerative) group d-vectors into speaker clusters.
5. Post-processing: Short segments are merged and final diarization output is generated.

• Python 3.7+
• PyTorch 1.7+
• librosa 0.8+
• WebRTC VAD
• NumPy
• Matplotlib
• scikit-learn
• SciPy
• pyannotate

Install dependencies:

pip install -r requirements.txt

Prepare your audio datasets in the following structure:

data/
├── voxceleb1/
├── voxconverse/
└── callhome/

To train the model:

python train.py

You can modify training parameters in config.py.

To run inference on new audio:

python run.py path/to/model.pth path/to/audio.wav

Adjust hyperparameters in config.py:

# Neural Network Architecture
input_size = 40
hidden_size = 768
num_layers = 3
projection_size = 256

# Training Parameters
num_epochs = 100
batch_size = 64
learning_rate = 0.001

# Data Preprocessing
min_num_speakers = 2
max_num_speakers = 8
num_segments = 10

We conducted extensive experiments to optimize key hyperparameters:

Optimal parameters: σ = 7, percentile = 85%

Best configuration: hop-length = 40ms, 128 FFTs

We compared four clustering algorithms:

Spectral Clustering and Agglomerative Clustering showed the best overall performance.

Various LSTM and BiLSTM configurations were tested:

The 3-layer LSTM with 768 nodes per layer achieved the best balance of performance and computational efficiency.

The system achieves an average Diarization Error Rate (DER) of 20% on the Vox Converse dataset.

We welcome contributions to improve the system. Please follow these steps:

1. Fork the repository
2. Create a new branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

If you use this code in your research, please cite:

@techreport{reinegger2024optimizing,
  title={Optimizing LSTM-based Speaker Diarization: Comparing Architectures and Clustering Methods},
  author={Reinegger, Michael},
  year={2024},
  institution={Johannes Kepler University Linz},
  note={Available at: https://mljr.eu/papers/BA_Speaker_Diarization.pdf}
}

This project is licensed under the MIT License - see the LICENSE file for details.

• VoxCeleb and VoxConverse datasets
• WebRTC Project for the VAD implementation
• PyTorch team for the deep learning framework