Download Project Progress Presentation (Fall)

DJ Spatial Tracking and Gesture
Recognition for Audio Effects and Mixing
Project Progress
Andrew Hamblin, Evan Leong, and Theo Wiersema
Dr. Jose Sanchez
Bradley University ECE
November 20, 2015
Objective
• Glove for disc jockeys
(DJ)
• Gestural control of
music effects
Fig. 1. System diagram
2
Goals
• Glove with tri-color light-emitting diodes (LED)
• Acquire and recognize gesture
• Seamless communication
• Real-time dynamic effects
3
Motivation
• Passion for music
• Unique product idea
4
Significance
• Disconnect for DJ
• Complexity of DJ equipment
• Lack of natural connection
Fig. 2. DJ board
5
System Block Diagram
Fig. 3. System block diagram
6
Top-Level State Diagram
Fig. 4. Top-level state diagram
7
Glove State Diagram
Fig. 5. Glove state diagram
8
Responsibilities Chart
TABLE I. HIGH-LEVEL RESPONSIBILITIES
Andrew Hamblin
Evan Leong
Theo Wiersema
Mixxx plugin
Glove development
Hidden Markov
model (HMM)
Raspberry
Pi-computer
communication
Pixy-Raspberry Pi
communication
Trinket code
9
Gantt Sub-Chart: Andrew Hamblin
Fig. 6. Andrew Hamblin Gantt chart
10
Pixy Object Detection: Progress
Fig. 7. Andrew Hamblin Gantt chart (1)
11
Pixy Object Detection: Goals
• Identify LED color
• Track LEDs
• Perform under
various ambient
light conditions
Fig. 8. System visual diagram
12
Pixy Object Detection: Data
• Sends data on every frame
• (x,y) position of top left corner
• Height and width of virtual object box
• Data used for gesture acquisition
13
Pixy Object Detection: Example
Blue
Green
Red
Blue
Green
Fig. 9. Pixy LED detection
14
Pixy Object Detection: Reliability
• LED detection
• White light perception
• False detection
• LED tracking
• Grouping
• Diffusion
Fig. 10. Red LED line object detection
Fig. 11. Red LED line
15
Pixy Object Detection: Blue
Fig. 12. Blue LED detection
16
Pixy Object Detection: Green
Fig. 13. Green LED detection
17
Pixy Object Detection: Red
Fig. 14. Red LED detection
18
Pixy Object Detection: Next Step
• Continue testing with LEDs
• Determine LED configuration
• Establish serial peripheral interface (SPI)
communication with Raspberry Pi
19
MIDI Mapping: Progress
Fig. 15. Andrew Hamblin Gantt chart (2)
20
MIDI Mapping: Goal
• Map signals to
Mixxx DJ effects
Fig. 16. System visual diagram
21
MIDI Mapping: Flow Chart
Fig. 17. MIDI mapping flow chart
22
MIDI Mapping: MIDI Scripting
• MIDI signals [1]
• 3 bytes
• Describes command
• Status byte
• Op-code
• Channel
• Data bytes
• Command descriptor
Fig. 18. MIDI signal breakdown
23
MIDI Mapping: Extensible Markup Language (XML)
• MIDI Learning Wizard [2]
• Prompts basic effects
• Designer transmits desired command
• Generates XML file
• XML file [2]
• Describes data for JavaScript use
24
MIDI Learning Wizard
Fig. 19. Learning Wizard window [3]
25
MIDI Mapping: Next Step
• Research
• XML data description
• JavaScript language
• Generate code
• XML file
• JavaScript plug-in
• Test/debug
• MIDI Monitor [4]
26
Andrew Hamblin: Upcoming Work
Fig. 20. Andrew Hamblin upcoming work Gantt chart
27
Gantt Sub-Chart: Evan Leong
Fig. 21. Evan Leong Gantt chart
28
LED Circuit: Progress
Fig. 22. Evan Leong Gantt chart (1)
29
LED Circuit: Goals
• LED brightness for object detection by Pixy
• Switch colors with button
30
LED Circuit: Diode Characteristics
• Common cathode design
• Red, green, and blue leads
• Turn-on voltages
• Red: 1.30 V
• Green: 2.07 V
• Blue: 2.26 V
Fig. 23. Tri-color LED
31
Transistor Circuit: Progress
Fig. 24. Evan Leong Gantt chart (2)
32
Transistor Circuit: Goal
• Low power dissipation for safe use
33
Transistor Circuit: 1st Design
• Three transistors
• Saturation and off
mode
• Current draw in both
modes
Fig. 25. Transistor circuit of one color lead
34
Transistor Circuit: 1st Design Power Dissipation
Collector current and collector-emitter (CE) voltage
• Saturation mode
• Red: 0.1602 A and 2.64 V
• Green: 0.1566 A and 2.77 V
• Blue: 0.1582 A and 2.83 V
• Off mode
• Red: 0.1023 A and 0.55 V
• Green: 0.1015 A and 0.43 V
• Blue: 0.0998 A and 0.49 V
35
Transistor Circuit: 1st Design Power Dissipation
Power dissipation
• Saturation mode
• Red: 0.423 W
• Green: 0.434 W
• Blue: 0.448 W
• Off mode
• Red: 0.0563 W
• Green: 0.0436 W
• Blue: 0.0489 W
• Power dissipation is too much
36
Transistor Circuit: 2nd Design
• One transistor
• Saturation mode
• Always on
Fig. 26. Common emitter configuration
37
Transistor Circuit: 2nd Design Power Dissipation
• Collector current and CE voltage
• Red: 27.2 mA and 43.0 mV
• Green: 11.2 mA and 24.0 mV
• Blue: 10.7 mA and 23.0 mV
• Modes are not simultaneous
38
Transistor Circuit: 2nd Design Power Dissipation
• Transistor power:
• Red: 1.17 mW
• Green: 0.269 mW
• Blue: 0.246 mW
• Battery life:
• 4 AAA - 4000 mAH [5]
• Min. life - 4000 mAH / 44 mA (red mode) = 90.9 hrs
39
Circuit: Next Step
• Test 5V Trinket Pro
• Finalize circuit
40
Evan Leong: Upcoming Work
Fig. 27. Evan Leong next step Gantt chart
41
Gantt Sub-Chart: Theo Wiersema
Fig. 28. Theo Wiersema Gantt chart
42
Programming the Trinket
Fig. 29. Theo Wiersema Gantt chart (1)
43
Programming the Trinket: Data
• Arduino software
• State machine
• Cyclic
Fig. 30. Trinket button circuit
44
Programming the Trinket: Results
Fig. 31. Changing LED colors
45
HMM: Forward-Backward Algorithm [6]
Fig. 32. Theo Wiersema Gantt chart (2)
46
Forward-Backward: Goals
• Find unknown parameters of HMM
• Adjusts transition and emission matrices based
on observations
47
Forward-Backward: Status
• Completed
• Initialization
• Recursive (forward and backward) steps
• Results
• α and β matrix: Probability of observation given a state
48
Forward-Backward: Test [7]
• Initial conditions (𝜋)
• s is 0.85
• t is 0.15
• States: s,t
• Observations: A,B
Fig. 33. Transition and emission probabilities
49
Forward: Test Results
• Observation sequence: ABBA
• Forward algorithm calculates correct values
TABLE II. GIVEN RESULTS
TABLE III. SIMULATION RESULTS
Iteration
s
t
Iteration
s
t
1
0.3400
0.08000
1
0.3400
0.0750
2
0.0660
0.1550
2
0.0657
0.1528
3
0.0212
0.0929
3
0.0210
0.0917
4
0.0063
0.0492
4
0.0062
0.0486
Total probability = 0.00625 + 0.04919 = 0.05544
Total probability = 0.0062 + 0.0486 = 0.0548
50
Backward: Test Results
• Observation sequence: ABBA
• Backward algorithm calculates correct values
TABLE IV. GIVEN RESULTS
TABLE V. SIMULATION RESULTS
Iteration
s
t
Iteration
s
t
4
0.1332
0.1273
4
0.1331
0.1273
3
0.2561
0.2487
3
0.2561
0.2487
2
0.4700
0.4900
2
0.4700
0.4900
1
1.0000
1.0000
1
1.0000
1.0000
Total probability = 0.13315 + 0.12729 = 0.26044
Total probability = 0.1331 + 0.1273 = 0.2604
51
HMM: Viterbi Algorithm [8]
Fig. 34. Theo Wiersema Gantt chart (3)
52
Viterbi: Goal
• Identify most probable state at each time interval
53
Viterbi: Status
• Completed
• Initialization
• Recursive computations
• Results
• 𝛿 matrix
54
Viterbi: Test [9]
• Initial conditions (𝜋)
• s is 0.6
• t is 0.4
• States: s,t
• Observations: A,B,C
Fig. 35. Transition and emission probabilities
55
Viterbi: Test Results
• Observation sequence: ABC
• Viterbi algorithm calculates correct values
TABLE VI. GIVEN RESULTS
TABLE VII. SIMULATION RESULTS
Iteration
s
t
Iteration
s
t
1
0.3
0.004
1
0.3
0.004
2
0.084
0.027
2
0.084
0.027
3
0.00588
0.01512
3
0.00588
0.01512
Most probable sequence of states: s,s,t
Most probable sequence of states: s,s,t
56
HMM: Next Step
• Integration of all algorithms
• Initialization
57
Trajectory Extraction
Fig. 36. Theo Wiersema Gantt chart (4)
58
Trajectory Extraction: Goals
• Calculate center of
LEDs
• Calculate angle of
hand’s motion
Fig. 37. System visual diagram
59
Trajectory Extraction: Simulation
Fig. 38. Trajectory extraction (1)
60
Trajectory Extraction: Simulation (cont.)
Fig. 39. Trajectory extraction (2)
61
Trajectory Extraction: Simulation (cont.)
Fig. 40. Trajectory extraction (3)
62
Trajectory Extraction: Next Step
• Assign weights based on size
• Test/debug with real motion data
63
Theo Wiersema: Upcoming Work
Fig. 41. Theo Wiersema Gantt chart
64
Summary: Pixy Object Detection
• Identify LED color
• Track LEDs
• Perform under various
ambient light
conditions
• Reliability
Fig. 42. System visual diagram
65
Summary: Mixxx MIDI Mapping
• Map signals to Mixxx
DJ effects
Fig. 43. System visual diagram
66
Summary: Circuit Design
• LED brightness for
object detection by Pixy
• Switch colors with button
• Low power dissipation
for safe use
Fig. 44. System visual diagram
67
Summary: Forward-Backward Algorithm
• Find unknown
parameters of HMM
• Adjust transition and
emission matrices
Fig. 45. System visual diagram
68
Summary: Viterbi Algorithm
• Identify most probable
state at each time
interval
Fig. 46. System visual diagram
69
Summary: Trajectory Extraction
• Find center point of hand
• Find angle between
frames
Fig. 47. System visual diagram
70
DJ Spatial Tracking and Gesture
Recognition for Audio Effects and Mixing
Project Progress
Andrew Hamblin, Evan Leong, and Theo Wiersema
Dr. Jose Sanchez
Bradley University ECE
November 20, 2015
Fig. 48. Gantt Chart
72
MIDI Mapping: JavaScript
• Defines audio effect functions
• Reads data from XML file
• Relays command to Mixxx DJ software
73
Op-Amp and Loading Effect
• Loading Effect
• RLOAD < RIN
• Large voltage drop
across RIN
• Small voltage drop
across RLOAD
• Less load power
Fig. 49. Loading effect example
• Solution
• Unity gain buffer op-amp
Fig. 50. Unity gain follower
74
Transistor Circuit: Design Considerations
• Set base current
value
• Calculate collector
current
• Calculate resistor
values
Fig. 51. Blue lead transistor circuit
75
Transistor Circuitry: Calculations
•
Base
• IB = 5.00 mA
• VBatt = 6.0 V
• RB = 1060 Ω
•
𝛽=5
•
Collector
• IC = 25.0 mA
• VPWM = 3.3 V
• RB = 29.6 Ω
Fig. 52. Blue LED example with values
76
Glove Design
• Conceptual design
• Placement of LEDs
• Intuitive button location
• Sheaths to direct light
• Battery pack placement
77
Angle Quantization
• Divided among “bins”
[10]
• Angles are rounded to
the nearest bin
Fig. 53. Quantized angle bins
78
Training with Test Gestures
Fig. 54. Test gestures
79
Angles of Test Gesture
Fig. 55. Gesture angles
80
Quantized Gesture Angles
Fig. 56. Quantized gesture angles
81
Forward Algorithm Equations
• Initialize
(1)
• Run through gesture forwards
(2)
82
Backward Algorithm Equations
• Initialize
(3)
• Run through gesture backwards
(4)
83
Viterbi Algorithm Equations
• Initialize delta
(5)
• Recursive delta
(6)
84
Introduction to HMM
Dwight
Fig. 57. Gumball example
85
Introduction to HMM (Cont.)
Fig. 58. Gumball machine emission probabilities
86
Introduction to HMM Cont.
Fig. 59. Gumball state diagram
87
Introduction to HMM (Cont.)
Fig. 60. Gumball example
88
Introduction to HMM Cont.
Fig. 61. Possible outcomes [9][10][11]
89
How do Gumballs Relate to HMM?
• Gumballs → observations
• Gumballs on conveyor belt → observation
sequence
• Gumball machines → states
• Succession of machines dropping gumballs →
sequence of states
• Food → result of observation
90
How HMM Relates to Gesture Recognition
• Observations → angles
• Sequence of observations → trajectory of glove
• States → hidden, abstract representation of
angles
• Sequence of states → abstract representation of
gesture
• Result of observation → audio effect applied
91
References
[1] http://www.mixxx.org/wiki/doku.php/midi_crash_course
[2] http://www.mixxx.org/wiki/doku.php/midi_controller_mapping_file_format
[3] http://static1.squarespace.com/static/50707813e4b00907bc18522b/t/51d1ba1fe4b025acba75a0c3/1372699169773/mixxxmidi-learn.jpg
[4] http://www.snoize.com/MIDIMonitor
[5] http://data.energizer.com/PDFs/E92.pdf
[6] https://en.wikipedia.org/wiki/Forward%E2%80%93backward_algorithm
[7] http://www.indiana.edu/~iulg/moss/hmmcalculations.pdf
[8] https://en.wikipedia.org/wiki/Viterbi_algorithm
[9] http://mas115.group.shef.ac.uk/projects/project2a/group23/typesofmarkov.html
[10] http://www.iis.ee.ic.ac.uk/ComputerVision/docs/pubs/John_Report_2014.pdf
[11] http://www.clipartbest.com/hamburger-black-and-white-clipart
[12] http://www.clipartpanda.com/clipart_images/pizza-coloring-pages-1-33796411
[13] http://www.clipartbest.com/clipart-RTA6qk8nc
92