Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
KnoteBox Joe Kramer, Leo Ovanesyan, Jimmy Thompon Project Objectives KnoteBox MIDI Project Objectives • Recognize musical notes from an analog sound input in real-time and output MIDI data to a computer • Why use the MIDI protocol? – It contains the information needed to display notes – It is standard in the music industry – Software already exists to manipulate MIDI data Project Motivation • Learning a new song or instrument can be irritating and discouraging • Music-based video games can be used for learning music but have simplified controllers • A music-based video game using real instruments would be enjoyable and instructional Project Motivation • There are systems which detect notes in audio signals • LittleBigStar – attempts to recognize chords but does not have much accuracy • The KnoteBox is different and better because it will attempt to detect multiple notes at the same time, do it in real time, and with more accuracy KnoteBox > Current Technology Applications • Guitar Hero / Rockband clone • Use MIDI input to create a track that must be played as notes scroll by • Learning / Teaching capabilities • Entertainment Game Applications • Sheet music generator • Play into the system in order to get sheet music for the song played • Can also generate a MIDI file Applications • Tuner • Ability to show frequency information or MIDI data to tune a specific instument • Instrument / Song learning tool • Ability to show what note is actually being played Overview • Analog audio signal in – Single instrument – Polyphonic – Melodic Audio in Audio processing I/O Note matching MIDI to PC over USB • MIDI over USB out – Buy MIDI to USB chip – MIDI note on->note off • Box I/O – Depends on algorithm MIDI Protocol • Industry standard protocol defined in 1982 • Defines interface between electronic instruments and file format MIDI Note on Pitch, velocity • MIDI events – Note on with velocity – Note off – Aftertouch, pitch bend MIDI Note off Pitch • Pitch ranges from 0-127 (~12 octaves) Music Theory • Pitch – Fundamental frequency determines note being played – Musical scale is logarithmically spaced (equal temperament) – Linearly spaced overtones • Envelope – Time based attack, sustain, decay, release • Timbre – Differences in sound of voices or instruments – Determined by differences in overtones and envelope Frequency Analysis • Constraints – Delay – Resolution – Computation • Time-frequency Algorithms – – – – – Resonators Short form Fourier Transform Wavelet analysis Bilinear frequency analysis Q-constant and Tonal Centroid Harmonic Peak Analysis • Pick peaks from timefrequency image – Thresholding, noise canceling – Decreasing power at high notes • Form tracks of peaks over time • Match peaks to a combination of notes • Trigger note on when peaks rise together • Keep state of notes being played Software Analog waveform Time frequency data DSP / FPGA Thresholding/ De-noising Note recognition MCU MIDI formatting Hardware Power Audio jack ADC/ filters DSP/ FPGA MIDI driver chip MCU JTAG Power Power supply Mini USB out Hardware – front end • Audio jack – 6.5mm connector • Typical for electrical guitars and professional audio Hardware – pre-digital processing • ADC chip – Must have sampling rate of about 44.1KHz • Standard audio CD quality • Filter chip or RC circuit – Anti-aliasing low-pass filter • Amplifiers – Op-amp: BW up to 44.1KHz Hardware – for heavy processing • DSP – Hardware works, just need software – Difficult to parallelize solution – Limited to instruction set • FPGA – Solution can be parallelized in hardware – Hardware needs to be verified Hardware – for heavy processing • DSP – TI DaVinci video processor • DSP and MCU • Acceleration hardware and lots of RAM • FPGA – Xilinx Virtex • High performance FPGAs Hardware – for post-processing • Microcontroller – MSP430 • Cheap and popular (lots of example code) • Low power (not an issue in the application) – ARM • Used in cell phones and other demanding applications • 100MHz-1GHz range Hardware – output • Ploytech GM5 chip – Provides a MIDI driver through a USB interface – Computer sees a MIDI device • Hirose mini USB connector Budget – for development Item Approximate cost Two PCBs $130 DSP / FPGA $200 (x2?) Microcontroller <$50 (x2?) Amplifiers, filters, ADCs <$50 USB to MIDI chip $10 Miscellaneous $100 Total $540-$790 Budget – for production Item Approximate cost One PCB $50 DSP / FPGA $150 Microcontroller <$50 Amplifiers, filters, ADCs <$50 USB to MIDI chip $5 Casing $30 Total $335 Schedule Division of Labor Task Joe Leo Jimmy Feature Extraction x x x Pattern Recognition x x x X x PCB Design X FPGA Prototyping x Microcontroller X x PC Software x x x Risks and Mitigations • Inaccuracy with picking out notes and peaks – Perform Matlab testing at the beginning to find and verify the best method – Perform FPGA testing on a test board before building our own hardware • Speed (Speed vs. Accuracy) – Consider parallelizing the system – Store information instead of having the system be realtime Risks and Mitigations • Noisy signal (especially at Expo) – Have filters in place to remove the noise – Consider having several microphones to create a noise array for noise cancellation • Different instruments have different sound signatures – Have different algorithms in place that can be used depending on the instrument – Have a switch on the box to select what sound type the signal is Risks and Mitigations • Time Constraint – Plan time accordingly – Stick to the schedule • Don’t get funding / Need more funds – – – – Do the URAP application right the first time Find a local sponsor Get student discounts Look for other research grants Risks and Mitigations • Problems or errors with PCB design and population – Fix mistakes and order new iteration – Make room for an error in our timeline – Do it right the first time • Shipping delay / error – Order parts early – Ensure the part is available Risks and Mitigations • Leo gets hits by a bus – Ol’ well, we don’t need him – Recruit from another group • Jimmy or Joe die / have an emergency – Triangle of stability – Have at least two people working on all the different parts Questions / Comments