Real-Time Multi-Speaker Voice Biometrics & Live Transcription Pipeline

Project Overview

Designed, engineered, and optimized an enterprise-grade, real-time voice biometric and multi-speaker speech-to-text pipeline hosted on an Oracle Linux 9 VM in Oracle Cloud Infrastructure (OCI). Connected via a secure Site-to-Site IPsec VPN, the system ingests raw 16kHz PCM audio streams directly from remote workstation microphones over WebSocket connections. The platform performs real-time audio chunking, contextual transcription, and dynamic speaker identification by comparing 192-dimensional deep learning voice embeddings in parallel.

System Architecture & Workflow

The solution replaces traditional file-upload audio pipelines with a low-latency, full-duplex WebSocket streaming model. Audio captured locally via client-side PyAudio streams is broken into discrete 3-second PCM byte buffers and transmitted continuously over port 8000. On the server side, an asynchronous FastAPI backend processes incoming frames in memory without disk I/O bottlenecks. Text generation is handled by faster-whisper configured for float32 CPU execution, while speaker identification executes concurrently using SpeechBrain’s spkrec-ecapa-voxceleb deep neural network architecture.

Key Technical Contributions

Asynchronous Streaming Pipeline: Built a high-throughput FastAPI WebSocket server handling continuous 16kHz 16-bit mono PCM audio buffers, converting raw bytes directly into normalized float32 NumPy arrays for zero-copy downstream inference.

Optimized Speech-to-Text Engine: Integrated faster-whisper utilizing the float32 compute type to deliver consistent, accurate text transcriptions with sub-second latency on standard CPU compute instances.

Dynamic Voice Biometric Diarization: Leveraged SpeechBrain’s ECAPA-TDNN model to extract normalized 192-dimensional speaker embeddings per audio chunk. Developed a cosine similarity matching engine that dynamically identifies new speakers on the fly, assigns randomized identities (e.g., Alex, Jordan, Taylor), and maintains persistent voice profiles across ongoing streams.

Rolling Vector Profile Refinement: Implemented a moving-average embedding update algorithm (0.8 \* profile + 0.2 \* new_sample) that continuously refines enrolled voice vectors as speakers continue talking, accounting for pitch shifts and vocal dynamics.

Persistent Biometric Enrollment API: Engineered an HTTP /enroll endpoint and JSON storage layer that allows pre-registering verified user voiceprints via WAV audio samples for automated identity verification during live streams.

Acoustic Tuning & Problem Solving

Overcame critical production challenges including WebSocket handshake failures under Uvicorn by configuring dedicated websockets backend protocols. To resolve "false speaker proliferation"—where natural pitch variation caused the system to spawn excessive new speaker identities—the cosine similarity threshold was empirically tuned from 0.25 to 0.52 alongside audio vector normalization, establishing high verification accuracy across real-world room acoustics.

Project Demonstration

Watch the full end-to-end demonstration of the real-time transcription and voice biometric pipeline in action on YouTube:

Watch Real-Time Multi-Speaker Voice Biometrics Demo on YouTube