• Speech Recognition
  • feature extraction

Speech Recognition

broader speech processing

• wake-up word detection: binary classification
  • small window → low latency
  • multiple models, threshold & size
• echo cancellation: subtract (complex) echo of sound from speaker
• sound source localization: microphone array, delay

online/streaming mode: continuous conversion

omnidirectional/ cardioid/ hyper cardioid microphone

sound sample rate

• need at lease 2x of highest frequency wanted, else aliasing (Nyquist theorem)
• 44.1kHz for CD
• 16kHz good enough for speech

$\mu$-curve

audio format: use PCM .wav

online endpointing format: push to talk, hit to talk, continuous listening

feature extraction

spectrogram: energy distribution over frequency vs time

1. preemphasize speech signal: boost high frequency

   $$\begin{gathered} s_{preemp}(n)=s(n)-\alpha s(n-1) \\ \alpha =0.95 \end{gathered}$$

2. spectrogram: from time series to frequency domain

   • discrete Fourier transform (DFT)
   • symmetric, only first half + midpoint meaningful (by Nyquist theorem)
   • magnitude: $0\sim \frac{S_R}{2}$Hz, frequency $\frac{i}{M}{S}_R$ at point $i$
     • $S_R$: sample rate, $M$: number of sample
     • power: magnitude squared
   • flat noise from jump in sample windowing
     • solution: multiple input by bell-shape windowing function and half-overlap windows to avoid losing information
   • before using fast Fourier transform (FFT): zero padding
     • cause fake interpolated detail

3. auditory perception: from frequency to Bark

   • mimic human ear, which distinguish low frequency better
   • frequency warping with Mel curve
   • filter bank: triangular filter on Mel curve to be multiplied
     • evenly-spaced, half-overlap, 40 enough, 80 good
     • translate back to equal-area triangles on frequency axis
       • not directly on Mel curve for simplicity

4. log Mel spectrum: log of integration of each filter bank

5. Mel cepstrum discrete cosine transform (DCT) compression

   • reduce redundancy from filter bank overlap
   • subtract the mean to remove microphone/noise difference
     • microphone response is speech with convolution
     • convolution -DFT → multiplication -log → summation

longest common subsequence: dynamic programming, dummy char padding, streaming, search trellis, sub-trellis, lexical tree

• pruning: hard threshold vs beam search

DTW: dynamic time warping: disallow skipping (vertical on trellis), non-linear mapping (super diagonal)

• $P_{i,j}$: best path cost from origin to $(i,j)$
• $C_{i,j}$: local node cost (vector distance)
• global beam across templates for beam search
• combine multiple template
  • template averaging: first align templates by DTW
  • template segmentation: compress chunks of same phoneme
  • actual method: iterate from uniform segments to stabilize variance

Mahalanobis distance

$$d(x,m_j)=(x-m_j{)}^T{C}_j^{-1}(x-m_j)$$

• can be estimated with negative Gaussian log likelihood

$$d(x,m_j)=\frac{1}{2}\log \left((2\pi {)}^D|C_j|\right)+(x-m_j{)}^T{C}_j^{-1}(x-m_j)$$

covariance

$$\Sigma =\left(\begin{array}{cccc}{\sigma }_{11}& {\sigma }_{12}& \cdots & {\sigma }_{1d}\\ {\sigma }_{21}& {\sigma }_{22}& \cdots & {\sigma }_{2d}\\ ⋮& ⋮& \ddots & ⋮\\ {\sigma }_{d1}& {\sigma }_{d2}& \cdots & {\sigma }_{dd}\end{array}\right)$$

• estimated with diagonal covariance matrix when elements largely uncorrelated

self-transition penalty: model phoneme duration

MFCC (Mel frequency cepstrum coefficient)

dynamic time warping (DTW)

• align multiple training sample
  1. segment all model sample uniformly by #phoneme, and average
  2. align each sample against model sample to get new segment and new average
  3. iterate until convergence
• transition probability: #segment switch over #frame in segment

hidden Markov model for DTW

• use log probability so total score is log total probability
• simulation by transition probability matrix $T$ (each row sum to 1)
• initial probability $\pi$

expectation-maximization (EM) algorithm

1. initialize with k-means clustering

2. auxiliary function: conditional expectation of the complete data log likelihood

   $$Q(\Theta ,{\Theta }^{(t)})=\sum _{y}p(y|X,{\Theta }^t)\log (p(X,y|\Theta ))$$

3. evaluate ${\Theta }^{t+1}$

   $${\Theta }^{t+1}=\underset{\Theta }{\mathrm{argmax}}Q(\Theta ,{\Theta }^t)$$

4. iterate until convergence

forward algorithm

1. initialize $\alpha (s,1)={\pi }_{s}P(o_t|s)$
2. iterate $\alpha (s,t+1)={\sum }_{s^{\prime }}\alpha (s^{\prime },1)P(s|s^{\prime })P(o_{t+1}|s)$

Baum Welch: soft state alignment

log-domain math

$$\begin{gathered} \log (x^y)=\log (x)2^{\log (y)} \\ \log (x+y)=\log (x)+\log (1+2^{\log (y)-\log (x)}) \end{gathered}$$

continuous text recognition: small-scale problem, e.g. voice command

• wrap back from end of template to start dummy variable
  • high wrap back cost → discourage space
• lextree
• non-emitting state/ null state: only for connecting, no self-transition
  • no transition time
• prior probability for word: add to the start of its HMM
• word transition probability for edge cost between word HMM
• approximate sentence probability with best path

grammar: only focus on syntax not semantics

• finite-state grammar (FSG)
• context-free grammar (CFG)
• backpointer: only word-level, additional script on word transition

training with continuous speech: bootstrap & iterate

• silence: silence model, $\epsilon$ bypass arc, self-loop non-emitting state
  • loop need to go through emitting state, else infinite

N-gram

• N-gram assumption: $P(w_k|w_1,\dots ,w_{k-1})=P(w_k|w_{k-(n-1)},\dots ,w_{k-1})$
• start/end of sentence: <s>, </s>
• unigram, bigram. 5-gram good enough for commercial use
• N-gram with D words need ${\sum }_{i=1}^{N-1}{D}^i$ transition node
• good Turing
  • Zipf’s law
• backoff

state-of-the-art system

• unit of sound by clustering large unlabelled dataset
• transfer learning with small labeled dataset

phoneme: 39 English, Mandarin with tone

• few rare phoneme, much more common than rare word → beat Zipf’s law
• defined by linguistic, or learned by clustering
• mono-phone/tri-phone: context-independent/dependent (for neighbor)
• absorbing/generating state → non-emitting
• locus: stable centers in spectrum

multiple pronunciation: multiple internal model + probability

mono-phone/ context-independent (CI) model

di-phone: model previous and current phoneme. problem: cross-word effect

tri-phone: model multiple current phoneme based on previous and next phoneme

• many cases not seen, back off to mono-phone
• share Gaussian with mono-phone, different weight
  • decision tree

inexact search: run n-gram on (n-1)-gram model by applying word-transition prob