Download Entropy(Outlook)

BACK END PROCESSING

Acoustic-phonetic Based: Using acoustic properties of a
fixed set of language phonemes to perform speech recognition.

Pattern recognition based: Templates of words, phrases,
etc. are stored and learned by a training procedure

Artificial intelligence based: Combine the above two
approaches and adapt the parameters over time.
ACOUSTIC-PHONETIC APPROACH
Implementation Steps
1.
Segment the speech into phonetic boundaries



2.
Phonetically label each segment based on
acoustic characteristics of the various phonetic
units

3.
Peer reviewed papers describe segmentation algorithms
Supervised algorithms: 90% accuracy
Unsupervised algorithms: 70-80% accuracy
Clustering algorithms learn the phonetic characteristics
Match most likely word sequence to labeled data


Language grammatical models assist with this step
Database dictionary contains recognition possibilities
PATTERN RECOGNITION APPROACH
Implementation Steps
1.
2.
Training phase to “learn” the patterns of the lexical units to
recognize and storing acoustic templates into a pattern
classification set.
Recognition phase to compare the speech with a list of
speech patterns
Advantages
1.
2.
3.
Widely used, easy to understand
Robust across languages, vocabularies, background
conditions
Proven high performance
ARTIFICIAL INTELLIGENCE APPROACH
An Expert System introduced
Helps with the labeling process: uses more
than simply the acoustic characteristics
Contains phonemic, lexical, syntactic,
semantic, and pragmatic knowledge
 Adjust the dynamic components of the
system over time (i.e. threshold values)

TRAINING PROCEDURES

Supervised training:



An algorithm “learns” the parameters using a training set of
labeled data. The “trained” algorithm then is ready to run in
an actual environment.
Sometimes adequate labeled training data is unavailable
Unsupervised training:


An algorithm trains itself by computing categories from
unlabeled training data.
Generally less reliable than supervised training. However,
useful if labeled training data is not available
CLUSTER
Definition: A group of audio frames having similar
feature values
Goal: Devise an algorithm that partitions frames of an
utterance into groups of clusters
Issues:





How is the algorithm initialized?
Which audio features are relevant?
What distortion measure represents similarity?
How many clusters should we use?
What is an appropriate training algorithm?
VECTOR QUANTIZATION
Partition the data into
cells
 Compute cell centroids
(red circles in the
diagram)
 Compute the distance
between received data
and centroids
 Assigned the data to
the nearest centroid

K- MEANS ALGORITHM

Input:



Output:



F = {f1, …, fk} is a list of feature vectors
N = desired number of categories (ex: phoneme types)
C = {c1, …, cN) centroid for each category
m: F->C Maps feature vector to one of the categories
Pseudocode
Randomly put members of F into an initial C
Compute the center of each member of C
DO
FOR EACH fj ∈ F assign fj to the closest ck
Re-compute the center of each member of C
WHILE iterations < THRESHOLD and at least one reassignment

Note: A poor initial selection will lead to incorrect results or poor performance.
K-Means is a greedy algorithm which converges to a local minima
GAUSSIAN DISTRIBUTION



When we analyze probability involving many random processes,
the distribution is almost always Gaussian.
Central Limit Theorem: As the sample size of random variables
approach ∞, the distribution approaches Gaussian
Probability distribution:
f(x | µ,σ2)
= 1/(2 πσ)½ * ez
where
z = -(x-µ)2 / (2 σ2)
MULTIVARIATE MIXTURE GAUSSIAN


Multiple independent random variables
Each variable can have its own mean and variance
Two independent
random variables
X and Y
MULTIVARIATE NORMAL DISTRIBUTION

(
EM ALGORITHM
EM = Expectation-Maximization
1.
2.
3.
4.
Perform an initial Maximum Likelihood (MLI) estimation
Expectation Step: Compute the expected value of the
Maximum likelihood function with respect to the observed
distribution
Maximization Step: Use the adjusted values computed in
step 2 to refine the expectation estimation
Repeat step 2 until algorithm converges
Examples: K-Means, EM Clustering, and the Baum-Welsh Hidden
Markov Algorithms, which we will discuss later, are special cases of
the EM Algorithm
Note: These are “greedy” algorithms that converge to local minima.
The initial MLE estimation is a critical component.
K MEANS WITH CLUSTER DISCOVERY
1.
2.
3.
Let M= 1 to form a single partition
Find centroid of all training data ( 1/T ∑i=0,Txi )
While (M < desired number of partitions)
For each M
Compute centroid position
ii.
Replace old centroid with new one
iii.
Partition the partition in half
iv.
Estimate centroid in each half
v.
Use the k-means algorithm to optimize centroid position
vi.
M = 2*M
Refinements to:
• Choose which clusters to split
o Akaike Information Criterion (AICC) or Bayesian Information criterion (BIC)
o Cluster with the widest variance
o Clusters with containing a minimum number of entries
• Decide how to reassign data to the new clusters
i.
EM CLUSTERING ALGORITHM
Assumption: Clusters conform to a multivariate Normal Distribution

Probability of frame, f, belonging to cluster, c (d=feature vector for frame f, ∑
= covariance matrix, ∑-1 = inverse of covariance matrix, |∑|= determinant of
covariance matrix, u = mean)
𝑇
𝑝𝑑𝑓𝑐, 𝑓 =
𝐹
−2
(2𝜋)
1
1
−2 −2 𝑓𝑟𝑎𝑚𝑒𝑠𝑓 𝑡−𝑢𝑐
|∑𝑐| 𝑒
,
∑−1
𝑐
(𝑓𝑟𝑎𝑚𝑒𝑠𝑓 𝑡−𝑢𝑐)
,
𝑡=0

Mean step value for a frame f of component c (frc is fraction of frames
belonging to this component)
stepc,f = frcc*pdfc,f / (∑𝐶−1
𝑖=0 𝑝𝑑𝑓𝑖 , 𝑓 )

Total step value for component c: totalStepc = ∑𝐹−1
𝑓=0 𝑠𝑡𝑒𝑝𝑐,𝑓

𝑐𝑓
,
New mean (feature t of cluster c): newMeanc,f = ∑𝑇𝑡=0 𝑓𝑟𝑎𝑚𝑒𝑠f,t * 𝑡𝑜𝑡𝑎𝑙𝑆𝑡𝑒𝑝

𝑠𝑡𝑒𝑝
New fraction (frames belonging to cluster c): newFrcC = totalStepc / F
𝑐
EM CLUSTERING ALGORITHM
1.
2.
3.
4.
5.
Seed the clusters (C) with initial mean values and initial
fraction belonging to each cluster (1/C)
Save current mean values (uprevc) for each component
Compute pdfc,f , stepc,f, and totalStepc arrays for all
components and frames
Compute newMeanc,f and newFrcc for all clusters and
features
Compute change in mean value
Δ𝑠𝑡𝑒𝑝 =
6.
∑𝐶−1
𝑐=0 𝑢𝑝𝑟𝑒𝑣𝑐 − 𝑢𝑐
2
∑𝐶𝑐=0 𝑢𝑝𝑟𝑒𝑣𝑐 ∗ 𝑢𝑝𝑟𝑒𝑣𝑐
If number of iterations less than max or Δstep >
THRESHOLD, then repeat step 3
MEAN, VARIANCE, STANDARD DEVIATION

The mean or expected value




µ = E(x)
1
=
𝑁
∑ xi over all x values
Variance
1
𝑁−1
(∑x2 – (∑x)2 /N) =
1
𝑁−1

σ2 =
(∑ (x – ux)2

The first form leads to rounding errors in practice
Standard Deviation: σ = square root of variance
Intuition





Mean: center of the distribution (1st moment)
Variance: spread of the distribution (2nd moment)
standard deviation: percent within a distance from the mean
Skew: asymmetry of the distribution (3rd moment)
Kurtosis: how peaked is the distribution (4th moment)
Note: Same mean, different variances
COMPUTE MEANS, VARIANCE, AND STD
public double[][] computeAverages(int[][] frames, int F)
{
double[] mean = new double[F], variance = new double[F], std = new double[F];
for (int frame=0; frame < frames.length; frame++)
for (int f=0; f<F; f++) mean[f] += frames[frame][f];
for (int f = 0; f<F; f++) mean[f]/= frames.length;
for (int frame=0; frame < frames.length; frame++)
for (int f=0; f<F; f++) variance[f] += pow(frames[frame][f] - mean[f], 2);
for (int f=0; f<F; f++)
{
variance[f] /= N - 1;
std[f] = Math.sqrt(variance[f]);
}
return new double[]{mean, variance, std};
}
Note: F = number of features
NORMALIZE MEAN AND VARIANCE
public void meanNormalization
(double[][] frames, double[] mean, int F)
{ for (int frame=0; frame<frames.length; frame++)
for (int f=0;f<F;f++) frames[frame][f]-=mean[f];
double[] variance = new double[F];
for (int frame=0; frame<frames.length; row++)
for (int f=0; f<F; f++) variance[f] += frames[frame][f] * frames[frame][f];
for (int f = 0; f<features; f++) variance[f] /= (frames.length – 1);
for (int frame=0; frame<frames.length; row++)
for (int f = 0; f<features; f++)
if (variance[f] != 0) frames[frame][f] /= Math.sqrt(variance[f]);
}
Note: After this algorithm, the mean is 0 and the variance is 1
COVARIANCE
Covariance determines how two random variables relate

A positive covariance occurs if two random variables tend to
both be above or below their means together

A negative covariance occurs when the random variables
tend to be on opposite sides of the mean

If no correlation, the covariance will be close to zero

Covariance formula

Discrete: Cov(X,Y) = ∑x (xi-µx) * (yi-µy)/(N-1)

Correlation Coefficients: Corr(X,Y) =
𝐶𝑜𝑣(𝑋,𝑌)
𝑠𝑡𝑑 𝑋 𝑠𝑡𝑑(𝑌)
COVARIANCE MATRIX


The code below assumes the each feature in the frame
array is normalized to mean=0 and variance=1
It produces a feature x feature two dimensional array
for use to compute Mahalanobix distance
frames += frame.length;
for (int r=1; r<=FEATURES; r++)
for (int c=1; c<=FEATURES; c++)
for (int t=0; t<frames; t++)
covariance[r-1][c-1] +=
frame[t][r]*frame[t][c];
GENERAL COVARIANCE MATRIX CREATION
private double[][] createCovariance throws exception
(ArrayList<Integer> cluster, double[][] dataSet, double[] mean, double[] std)
{ int frames = cluster.size(), len = mean.length, frameEntry;
double[ cov[][] = new double[len][len], features[], variance;
for (int frameEntry: cluster)
{ features = dataSet[frameEntry];
for (int i=0; i<features.length; i++)
for (int j=i; j<features.length; j++)
{ variance = (features[i]-mean[i])/std[i] *(features[j]-mean[j])/std[j];
cov[i][j] += variance;
} }
for (int i=0; i<len; i++)
for (int j=i; j<len; j++) { cov[i][j] /= (frames-1); if (i!=j) cov[j][i] = cov[i][j]; }
return cov;
}
DISTORTION METRIC REQUIREMENTS
Definition: Measure similarity of two frames of speech.
The vector xt , yt contain the features from frames of two signals
A distance measure should have the following properties:
0  d(xt,yt)  
(positive definiteness)
0 = d(xt,yt) iff xt = yt
d(xt,yt) = d(xt,yt)
(symmetry)
d(xt,yt)  d(xt,zt) + d(zt,yt)
(triangle inequality)
A speech distance metric should correlate with perceived distance.
Perceptually-warped spectral features work well in practice
23
DISTORTION MEASURES
After partitioning a speech signal into windowed frames, we have
an array, frame[t][f]


t is time (typically in 12.5 MS chunks)
Each frame represents features (f) (typically for 25 MS of speech)

L1 Norm ∑𝐹−1
𝑓=0 |𝑓𝑟𝑎𝑚𝑒 𝑡 [𝑓] − 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒[𝑓]|

Euclidean

Mahalanobis

2
∑𝐹−1
𝑓=0(𝑓𝑟𝑎𝑚𝑒 𝑡 𝑓 − 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒[𝑓])
(𝑓𝑟𝑎𝑚𝑒 𝑡 − 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒)𝑇 ∑ − 1 (𝑓𝑟𝑎𝑚𝑒 𝑡 − 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒)
Normalized Dot Product
𝑓𝑟𝑎𝑚𝑒 𝑡
∑𝐹
𝑓=0 𝑓𝑟𝑎𝑚𝑒 𝑡 [𝑓]
𝑇
2
.𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒
∑𝐹
𝑓=0 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒[𝑓]
2
Notes:
• frame[t], template are vectors
• The superscript T indicates a transpose vector
• Mahalanobis and Normalized Dot Products involve a matrix multiplication
• ∑-1 is the inverse of the covariance matrix corresponding to the features
DECISION TREES
Partition a series of questions, each with a discrete set of answers
x x
x
x
x
x
x
x
x x
x
x
x
x
x x
x
x
x
x
x x
x
x
x
x
x
x
x
Reasonably Good Partition
x
x x
x
x
x
x
x
x
x
x x
x
x
Poor Partition
x
CART ALGORITHM
Classification and regression trees
1.
Create a set of questions that can distinguish between
the measured variables
a.
b.
2.
3.
4.
5.
6.
7.
Singleton Questions: Boolean (yes/no or true/false) answers
Complex Questions: many possible answers
Initialize the tree with one root node
Compute the entropy for a node to be split
Pick the question that with the greatest entropy gain
Split the tree based on step 4
Return to step 3 as long as nodes remain to split
Prune the tree to the optimal size by removing leaf nodes
with minimal improvement
Note: We build the tree from top down. We prune the tree from bottom up.
COMPUTING ENTROPY


Entropy: Bits needed to store possible question answers
Formula: Computing the entropy for a question:
Entropy(p1, p2, …, pn) = - p1lg p1 – p lg2p2 … - pn lg pn

Where
pi is the probability of the ith answer to a question
lgx is logarithm base 2 of x

Examples:


A coin toss requires one bit (head=1, tail=0)
A question with 30 equally likely answers requires
∑i=1,30-(1/30)lg(1/30) = - lg(1/30) = 4.907
EXAMPLE: PLAY OR NOT PLAY?
Outlook
Temperature
Humidity
Windy
Play?
sunny
hot
high
false
No
sunny
hot
high
true
No
overcast
hot
high
false
Yes
rain
mild
high
false
Yes
rain
cool
normal
false
Yes
rain
cool
normal
true
No
overcast
cool
normal
true
Yes
sunny
mild
high
false
No
sunny
cool
normal
false
Yes
rain
mild
normal
false
Yes
sunny
mild
normal
true
Yes
overcast
mild
high
true
Yes
overcast
hot
normal
false
Yes
rain
mild
high
true
No
Questions
1) What is the outlook?
2) What is the temperature?
3) What is the humidity?
4) Is it Windy?
Goal: Order the questions in
the most efficient way
EXAMPLE TREE FOR “DO WE PLAY?”
Goal: Find the optimal tree
Outlook
sunny
overcast
Humidity
Yes
rain
Windy
high
normal
true
false
No
Yes
No
Yes
WHICH QUESTION TO SELECT?
witten&eibe
EXAMPLE: QUESTION “OUTLOOK”
Compute the entropy for the question: What is the outlook?
Original Entropy: -9/14lg(9/14)-5/14lg(5/14) = 0.94028595867
Entropy(Outlook = Sunny) =Entropy(0.4, 0.6)=-0.4 log2(0.4)-0.6 log2(0.6)=0.971
Five outcomes, 2 for play for P = 0.4, 3 for not play for P=0.6
Entropy(Outlook = Overcast) = Entropy(1.0, 0.0)= -1 log2(1.0) - 0 log2(0.0) = 0.0
Four outcomes, all for play. P = 1.0 for play and P = 0.0 for no play.
Entropy(Outlook = Rainy)= Entropy(0.6,0.4)= -0.6 log2(0.6) - 0.4 log2(0.4)= 0.971
Five Outcomes, 3 for play for P=0.6, 2 for not play for P=0.4
Entropy(Outlook) = Entropy(Sunny, Overcast, Rainy)
= 5/14*0.971+4/14*0+5/14*0.971 = 0.693
The Entropy gain if we choose this question to be the root of the tree
Entropy gain(Outlook) = 0.940 - 0.693 = 0.247
Goal is to choose the question that minimizes the remaining entropy
COMPUTING THE ENTROPY GAIN

Original Entropy (14 outcomes, 9 for Play P = 9/14, 5 for not play P=5/14)
Entropy(Play) = Entropy(9/14,5/14)=-9/14log2(9/14) - 5/14 log2(5/14)=0.940

Information gain equals (information before) – (information after)
gain(Outlook) = 0.940 – 0.693 = 0.247

Entropy for the other questions
 Entropy(Humidity) = 7/14(-3/7lg(3/7)-4/7lg(4/7))
+7/14(-6/7lg(6/7)-1/7lg(1/7)) = 0.788 {gain = 0.152}
 Entropy(Windy) = 8/14(-.75lg(.75)-.25lg(.25)) + 6/14 (-.5lg(.5) - .5lg(.5))
= 0.892 {gain = 0.048}
 Entropy(Temperature) = 4/14(-.5lg(.5)-.5lg(.5))
+ 6/14(-2/3lg(2/3)-1/3lg(1/3)) + 4/14(-.75lg(.75)-.25lg(.25)) =0.911
 gain(Humidity) = 0.152, gain(Windy) = 0.048, gain(Temperature) = 0.029

Conclusion: Ask, “What is the Outlook?” first
CONTINUING TO SPLIT
yes
no
no
gain(" Temperatur e" )  0.571 bits
gain(" Humidity" )  0.971 bits
gain(" Windy" )  0.020 bits
For each child question, do the same thing to form the complete decision tree
Example: After the outlook sunny node, we still can ask about temperature,
humidity, and windiness. The humidity question has the best gain in entropy.
THE FINAL DECISION TREE
Note: The splitting stops when further splits don't reduce
entropy more than some threshold value