AI with Python – Speech Recognition

Modern AI with Python – Speech Recognition

In this chapter, we’ll explore the fascinating world of speech recognition, a key component of Artificial Intelligence, using Python. We’ll cover how machines can understand human speech and the modern techniques involved.

Speech is fundamental to human communication. The primary goal of speech processing in AI is to enable seamless interaction between humans and machines. This involves several interconnected tasks:

• First, Automatic Speech Recognition (ASR), which allows machines to transcribe spoken words, phrases, and sentences into text.
• Second, Natural Language Processing (NLP), enabling machines to comprehend the meaning and intent behind the transcribed text.
• Third, Speech Synthesis (Text-to-Speech, TTS), allowing machines to generate human-like speech from text.

This chapter primarily focuses on Automatic Speech Recognition (ASR). We’ll delve into how raw audio signals, captured by a microphone or from a file, are processed and interpreted by a system to identify the spoken words.

Understanding and Building a Speech Recognizer

Automatic Speech Recognition (ASR) is a cornerstone technology for many AI applications, from voice assistants like Siri and Alexa to dictation software and in-car control systems. Building a robust ASR system is a complex endeavor due to the inherent variability in human speech.

Challenges in Developing Speech Recognition Systems

Creating a high-accuracy ASR system presents numerous challenges. These can be categorized along several dimensions:

• Vocabulary Size: A system designed for a small, fixed vocabulary (e.g., ‘yes’, ‘no’, ‘next’ in a voice menu) is simpler to build than one for a large vocabulary (e.g., general dictation with tens of thousands of words).
• Channel and Recording Quality: High-fidelity audio from a close-talk microphone is easier to process than noisy audio from a distant microphone or a low-bandwidth telephone call.
• Speaking Mode: Isolated words are easier to recognize than connected words (short phrases) or continuous, conversational speech, which often features disfluencies and coarticulation.
• Speaking Style: Read speech (e.g., audiobooks) is typically clearer and more formal than spontaneous, conversational speech, which is more challenging.
• Speaker Variability: Systems can be speaker-dependent (trained for a specific user), speaker-adaptive (adapts to a new user over time), or speaker-independent (aims to work for any user). Speaker-independent systems are the most complex but also the most versatile.
• Noise Levels: The Signal-to-Noise Ratio (SNR) significantly impacts performance. High SNR (>30dB) environments are ideal, while low SNR (<10dB) environments with significant background noise pose a major hurdle.
• Accents and Dialects: Regional accents, dialects, and individual speaking idiosyncrasies add layers of complexity.

Despite these challenges, significant advancements, particularly with deep learning, have led to highly effective ASR systems. The fundamental steps to build or understand a speech recognizer involve signal processing, feature extraction, acoustic modeling, and language modeling.

Visualizing and Understanding Audio Signals

The first step in any audio processing task is to load and understand the structure of an audio signal. Modern ASR systems typically work with digital audio.

Digital Audio Basics

When sound is recorded, it’s an analog wave. To be processed by a computer, it must be digitized. This involves:

• Sampling: The analog signal is sampled at regular intervals. The number of samples per second is the ‘sampling rate’ (e.g., 44100 Hz for CD quality, 16000 Hz common for speech). A higher sampling rate captures more detail.
• Quantization: Each sample’s amplitude is mapped to a discrete numerical value. The ‘bit depth’ (e.g., 16-bit, 24-bit) determines the resolution of these values.

We’ll use the librosa library, a powerful Python package for audio and music analysis. If you don’t have it installed, you can install it using pip:

pip install librosa matplotlib numpy

Example: Loading and Visualizing an Audio File

Let’s load an audio file (e.g., a WAV file) and visualize its waveform. Ensure you have an audio file named example_audio.wav in your working directory or provide the correct path.

First, import the necessary libraries:

import librosa
import librosa.display
import matplotlib.pyplot as plt
import numpy as np

Now, load the audio file. librosa.load() returns the audio time series (as a NumPy array) and the sampling rate.

# Replace 'example_audio.wav' with the path to your audio file
audio_file_path = 'example_audio.wav'

try:
    audio_signal, sampling_rate = librosa.load(audio_file_path, sr=None) # sr=None to preserve original sampling rate
except FileNotFoundError:
    print(f"Error: The file {audio_file_path} was not found.")
    print("Please ensure you have a valid audio file and path.")
    # As a fallback for the tutorial, create a dummy signal if file not found
    sampling_rate = 22050 # A common default
    duration = 3 # seconds
    audio_signal = np.sin(2 * np.pi * 440 * np.linspace(0, duration, int(sampling_rate * duration)))
    print("Using a dummy sine wave for demonstration.")

print(f'Signal Shape: {audio_signal.shape}')
print(f'Sampling Rate (Hz): {sampling_rate}')
print(f'Signal Datatype: {audio_signal.dtype}')
print(f'Signal Duration (s): {librosa.get_duration(y=audio_signal, sr=sampling_rate):.2f}')

librosa.load automatically normalizes the audio signal to be between -1 and 1. Let’s visualize the waveform:

plt.figure(figsize=(12, 4))
librosa.display.waveshow(audio_signal, sr=sampling_rate)
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
plt.title('Audio Waveform')
plt.tight_layout()
plt.show()

This plot shows the amplitude of the audio signal over time. You’ll see variations corresponding to the sound’s intensity. For speech, these variations represent spoken words and silences.

Expected output (textual, actual plot will be displayed by matplotlib):

Signal Shape: (some_number,)
Sampling Rate (Hz): your_audio_sampling_rate (e.g., 22050 or 44100)
Signal Datatype: float32
Signal Duration (s): duration_of_your_audio_file (e.g., 3.00 seconds)

The plot will be a waveform graph showing amplitude fluctuations over time.

Transforming to the Frequency Domain: Spectrograms

While the time-domain waveform is useful, much of the information in speech is better understood in the frequency domain. The Short-Time Fourier Transform (STFT) is commonly used to analyze how the frequency content of a signal changes over time. The result is often visualized as a spectrogram.

Example: Generating and Visualizing a Spectrogram

Using the audio_signal and sampling_rate from the previous example:

# Compute STFT
stft_signal = librosa.stft(audio_signal)

# Convert amplitude to decibels (dB) for better visualization
स्पेक्टrogram_db = librosa.amplitude_to_db(abs(stft_signal), ref=np.max)

# Plot the spectrogram
plt.figure(figsize=(12, 6))
librosa.display.specshow(स्पेक्टrogram_db, sr=sampling_rate, x_axis='time', y_axis='hz') # or y_axis='log' for logarithmic frequency
plt.colorbar(format='%+2.0f dB')
plt.title('Spectrogram (Hz scale)')
plt.tight_layout()
plt.show()

The spectrogram displays time on the x-axis, frequency on the y-axis, and the intensity of each frequency component (in dB) using color. Darker areas indicate higher energy at those frequencies. For speech, you can often see formants (resonant frequencies of the vocal tract) as horizontal bands.

A common variation is the Mel spectrogram, which uses the Mel scale for frequencies, mimicking human auditory perception. This is a crucial step for extracting Mel-Frequency Cepstral Coefficients (MFCCs).

# Compute Mel spectrogram
mel_spectrogram = librosa.feature.melspectrogram(y=audio_signal, sr=sampling_rate, n_mels=128, fmax=8000) # n_mels: number of Mel bands
mel_spectrogram_db = librosa.power_to_db(mel_spectrogram, ref=np.max)

# Plot the Mel spectrogram
plt.figure(figsize=(12, 6))
librosa.display.specshow(mel_spectrogram_db, sr=sampling_rate, x_axis='time', y_axis='mel', fmax=8000)
plt.colorbar(format='%+2.0f dB')
plt.title('Mel Spectrogram')
plt.tight_layout()
plt.show()

The spectrogram plots will show time-frequency representations of your audio. The first will have a linear or logarithmic frequency axis, while the Mel spectrogram will use the Mel frequency scale, which is often more informative for speech.

Generating a Simple Audio Signal

Understanding signal generation can help solidify concepts. Let’s create a simple sine wave (a pure tone) and save it as a WAV file. We’ll use NumPy for generation and soundfile library for saving (as scipy.io.wavfile.write can sometimes be tricky with float data or normalization).

Install soundfile if you haven’t already: pip install soundfile

import numpy as np
import matplotlib.pyplot as plt
import soundfile as sf # For saving audio files

# Parameters for the tone
duration_tone = 3  # seconds
sampling_rate_tone = 44100  # Hz
frequency_tone = 440  # Hz (A4 note)

# Generate time axis
time_axis = np.linspace(0, duration_tone, int(sampling_rate_tone * duration_tone), endpoint=False)

# Generate sine wave
tone_signal = 0.5 * np.sin(2 * np.pi * frequency_tone * time_axis) # 0.5 amplitude to avoid clipping

# Save the generated audio to a file
output_file = 'generated_tone.wav'
try:
    sf.write(output_file, tone_signal, sampling_rate_tone)
    print(f"Generated tone saved to {output_file}")
except Exception as e:
    print(f"Error saving audio file: {e}")

# Visualize a small portion of the generated signal
plt.figure(figsize=(10, 4))
plt.plot(time_axis[:500], tone_signal[:500]) # Plot first 500 samples
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
plt.title(f'Generated Sine Wave ({frequency_tone} Hz)')
plt.grid(True)
plt.show()

This code generates a 3-second audio signal of a 440 Hz pure tone and saves it. The plot will show a perfect sinusoidal wave, representing the generated tone.

Feature Extraction from Speech: MFCCs

Raw audio or even spectrograms are high-dimensional. For ASR, we need to extract compact and informative features. Mel-Frequency Cepstral Coefficients (MFCCs) are widely used traditional features that capture essential characteristics of the vocal tract and are somewhat robust to variations.

Example: Extracting MFCCs using Librosa

Continuing with the audio_signal and sampling_rate from our loaded audio file:

# Extract MFCCs
# Common parameters: n_mfcc (number of coefficients), hop_length, win_length
mfccs = librosa.feature.mfcc(y=audio_signal, sr=sampling_rate, n_mfcc=13)

print(f'MFCCs Shape: {mfccs.shape}') # (n_mfcc, number_of_frames)

# Visualize the MFCCs
plt.figure(figsize=(12, 4))
librosa.display.specshow(mfccs, sr=sampling_rate, x_axis='time')
plt.colorbar()
plt.title('MFCCs')
plt.tight_layout()
plt.show()

The mfccs array will have n_mfcc rows (coefficients) and a number of columns corresponding to time frames. The visualization shows how these coefficients change over time, representing the phonetic content of the speech.

Expected output (textual):

MFCCs Shape: (13, number_of_frames_in_your_audio)

The plot will display the MFCC features over time, with different colors indicating the values of the coefficients. This is a standard way to represent speech features for recognition models.

While MFCCs are foundational, modern ASR systems, especially deep learning models, often learn features directly from spectrograms or even raw audio, or use more advanced feature representations like Perceptual Linear Prediction (PLP) coefficients or filter bank energies.

Recognition of Spoken Words using SpeechRecognition Library

Now, let’s perform actual speech recognition. The SpeechRecognition library provides a convenient way to use various speech recognition engines, including online APIs (like Google Web Speech API, Wit.ai) and offline engines (like CMU Sphinx).

Install the library: pip install SpeechRecognition PyAudio (PyAudio is needed for microphone input). If you have issues installing PyAudio, consult its documentation for platform-specific instructions (e.g., on Windows you might need precompiled binaries, or on Linux, portaudio development libraries).

Example: Recognizing Speech from a Microphone

This example will use your microphone to capture audio and the Google Web Speech API to recognize it. An internet connection is required.

import speech_recognition as sr

# Initialize the recognizer
recognizer = sr.Recognizer()

# Use the default microphone as the audio source
with sr.Microphone() as source:
    print("Calibrating microphone for ambient noise... Please wait.")
    # Adjust for ambient noise for 1-2 seconds for better accuracy
    recognizer.adjust_for_ambient_noise(source, duration=1)
    print("Microphone calibrated. Please say something clearly!")

    try:
        audio_data = recognizer.listen(source, timeout=5, phrase_time_limit=10) # Listen for up to 5s, max 10s phrase
        print("Recognizing your speech...")

        # Recognize speech using Google Web Speech API
        text = recognizer.recognize_google(audio_data)
        print(f"You said: {text}")

    except sr.WaitTimeoutError:
        print("No speech detected within the timeout period.")
    except sr.UnknownValueError:
        print("Google Web Speech API could not understand the audio.")
    except sr.RequestError as e:
        print(f"Could not request results from Google Web Speech API; {e}")
    except Exception as e:
        print(f"An unexpected error occurred: {e}")

When you run this, it will listen for your voice. After you speak, it will attempt to transcribe what you said.

Example interaction (textual):

Calibrating microphone for ambient noise... Please wait.
Microphone calibrated. Please say something clearly!
(You speak: "Hello world AI")
Recognizing your speech...
You said: hello world AI

Recognizing Speech from an Audio File

You can also recognize speech from an existing audio file (e.g., WAV format). Ensure the file is clear.

# Assuming 'example_audio.wav' is a clear recording of speech
audio_file_for_recognition = 'example_audio.wav'

with sr.AudioFile(audio_file_for_recognition) as source:
    try:
        audio_data_file = recognizer.record(source)  # Read the entire audio file
        print(f"Recognizing speech from {audio_file_for_recognition}...")
        text_from_file = recognizer.recognize_google(audio_data_file)
        print(f"The audio file contains: {text_from_file}")
    except FileNotFoundError:
        print(f"Error: The file {audio_file_for_recognition} was not found.")
    except sr.UnknownValueError:
        print("Google Web Speech API could not understand the audio from the file.")
    except sr.RequestError as e:
        print(f"Could not request results from Google Web Speech API; {e}")
    except Exception as e:
        print(f"An unexpected error occurred: {e}")

The SpeechRecognition library supports other engines too. For instance, recognizer.recognize_sphinx(audio_data) can be used for offline recognition with CMU Sphinx (requires PocketSphinx to be installed: pip install pocketsphinx). This is useful for applications without internet access or for privacy concerns.

For more advanced or custom ASR solutions, consider exploring toolkits like Kaldi, or pre-trained deep learning models from Hugging Face Transformers (e.g., Wav2Vec2, Whisper), TensorFlow, or PyTorch. These often provide higher accuracy but require more setup and computational resources.

Additional Resources:

• Librosa Documentation: https://librosa.org/doc/latest/index.html
• SpeechRecognition Library Documentation: https://pypi.org/project/SpeechRecognition/
• A beginner’s guide to STFT: https://www.gaussianwaves.com/2015/11/interpreting-fft-results-obtaining-magnitude-and-phase-information/
• Introduction to MFCC: https://wiki.aalto.fi/display/ITSP/Mel-frequency+cepstral+coefficient+(MFCC)