Download Final Project Report, Mechatronics

Polytechnic Institute of New York University
Department of Mechanical Engineering
Final Project Report, Mechatronics - ME5643
Air Piano
Professor
Sang-Hoon Lee
Students
Lucas Gregolin Dias - 0513304
Fabio Mantelli - 0513305
1
Table of Contents
Introduction ............................................................................................................................................................................... 3
Methodology - Hardware ..................................................................................................................................................... 4
Microcontroller Module ................................................................................................................................................... 4
Keyboard Module ............................................................................................................................................................... 5
Control Module .................................................................................................................................................................... 6
Prototype Costs ................................................................................................................................................................... 8
Methodology - Software........................................................................................................................................................ 9
MIDI Protocol ....................................................................................................................................................................... 9
MIDI Messages ..................................................................................................................................................................... 9
MIDI Digital Audio Workstation (DAW).................................................................................................................. 11
Code ........................................................................................................................................................................................ 11
Results ........................................................................................................................................................................................ 15
Conclusion ................................................................................................................................................................................ 16
Bibliography ............................................................................................................................................................................ 17
2
Introduction
Since it was developed, the MIDI (Musical Instrument Digital Interface [1]) protocol
has revolutionized the concept of making music, quickly becoming one of the main pillars of
music production and recording nowadays. The characteristics of this protocol, allied with
new and more accessible technologies, also changed the paradigm of digital instruments.
Previously, they were used exclusively to emulate analogic instruments (such as pianos and
drum kits). Now they are acquiring their own characteristics, in different and more creative
ways.
Another technology that has been getting a lot of attention lately is gesture
recognition. Different applications have been using the technology as a way of interacting
more naturally with the end-user.
Immersed in this context we find the “Air Piano” project. It is a musical instrument
that can sense finger gestures on top of an array of sensors to trigger notes and send them
to MIDI devices. It also has a control board that changes the volume and effects of the sent
notes depending on its distance to the ground and orientation.
This device was inspired by some unusual musical instruments which use sensor
measurements to produce tones, like the Theremin [2], the Martenot [3], and other more
recent ones [8]. Its gesture sensing is inspired on the recent advent of hands-free interfaces
like the Kinect [4], Leap [5] and Oculus Rift [6].
3
Methodology - Hardware
The device built is composed by a microcontroller, photoresistors, an accelerometer
and an ultrasonic distance sensor. For the structure, we used cardboard and breadboards.
Microcontroller Module
Figure 1 - Microcontroller Module
For the prototype’s microcontroller module a breadboard was used as the mechanical
structure. The microcontroller used was the Arduino Micro [7]. We chose this over the
suggested BASIC Stamp 2 microcontroller due to the necessity of 12 analog ports. The
specifications of this microcontroller are as indicated in Table 1.
Figure 2 - Arduino Micro
4
Microcontroller
ATmega32u4
Operating Voltage
5V
Input Voltage (recommended)
7-12V
Input Voltage (limits)
6-20V
Digital I/O Pins
20
PWM Channels
7
Analog Input Channels
12
DC Current per I/O Pin
40 mA
DC Current for 3.3V Pin
50 mA
Flash Memory
32 KB (ATmega32u4), 4 KB used by bootloader
SRAM
2.5 KB (ATmega32u4)
EEPROM
1 KB (ATmega32u4)
Clock Speed
16 MHz
Table 1
Keyboard Module
Figure 3 - keyboard
The prototype keyboard was built as a cardboard box-like structure with 11 dividers,
resulting in 12 holes corresponding to the 12 notes of one octave of the musical scale (C to
B), displayed on the subtitles. Plastic tape was used to stick the parts together, and the wires
were soldered together. Each hole contains a photoresistor, which was connected to the
microcontroller as portrayed on Figure 4.
5
Figure 4 - photoresistor circuit
Figure 5 - photoresistor
A photoresistor resistance decreases with increasing incident light intensity. So with
this circuit, the analog pin measurement will be inversely proportional to the amount of light
shed on it. The user will use his/her fingers to shade the photoresistor, changing the state on
the analog port and triggering the notes.
Control Module
Figure 6 - control module
6
The control module prototype also has a breadboard structure, and it is basically
composed by an ultrasonic distance sensor, which is used to detect its distance to the ground,
and an accelerometer, which measures the board spatial orientation. The accelerometer
used 2 microcontroller ports, one for each axis, and the ultrasonic sensor used only one port.
The other wires seen in Figure 6 are power supply and ground wires for the digital sensors.
The accelerometer used is the MX2125 by parallax (Figure 7). It measures on a range
of ±3g on each axis and sends a PWM signal through its output ports with the width
modulated by the acceleration readings. The components of the g force in each direction
allow us to measure tilt in relation of two orthogonal axes. The electrical and mechanical
specifications of the MX2125 design met our requirements.
Figure 7 - accelerometer
The ultrasonic sensor used in this prototype was the PING by parallax. It is a simple
pulse in/pulse out device that measures distance on a range of 2 cm to 3 m, without contact.
Its convenience also resides in the fact that it doesn’t require special light conditions and that
it uses only one I/O digital pin port on the microprocessor. Attached on the control module,
the sensor can be used by pointing it to a flat surface and bringing the module nearer or
farther away from it.
7
Figure 8 - ultrasonic sensor
Prototype Costs
We estimated the components’ cost as shown in Table 2.
Component
Quantity
Unit Price
Arduino Micro
1
$22.95
MX2125 Accelerometer
1
$29.00
PING Ultrasonic Sensor
1
$25.00
Photoresistor
12
$1.25
1 kΩ resistor
12
$0.80
Breadboard
2
$5.00
Assorted wires
~35
$0.30
Cardboard
-
$0.00
Total cost
$122.05
Table 2
These costs can be reduced by mass production, by embedding the circuits in printed
circuit boards, cutting the breadboard and wires’ cost, and by using cheaper accelerometer
and ultrasonic sensors.
8
Methodology - Software
MIDI Protocol
Before the creation of MIDI protocol, many other proprietary standards were used to
interface between different digital instruments. They allowed the communication and the
exchange of messages between two instruments of the same brand, but failed to do that
when is dealing with devices that used different protocols.
The MIDI protocol was created in 1983 to standardize the interface between digital
instruments. It created requirements for the hardware and communication. The relevant
aspects of the protocol to the project are the message standard and communication standard.
MIDI messages don’t carry information about the sound being played, but they tell which,
when and for how long notes are being played, analog to music notation. Some of the
advantages of MIDI protocol over the previous standards are the fact that it is open-source,
simple, fast and lightweight (small messages). [1]
MIDI Messages
MIDI messages are delivered through serial port, and are 8 bit (1 byte) long. There
are two types of message: status messages and data messages.
The MSB (most significant bit) of the status messages is 1. The next 3 bits are the
command and the last 4 is the channel the message is being sent to. Depending on the status
message, 0 to 3 data bytes are expected to follow it.
The MSB of the data messages is 0. The next 7 bits are the data value for the message.
You can send 128 different values using one data message.
Figure 9
9
The two status messages used by the system are the “note on” and the “control
change” message. They are sent on the channel 1 (for example) and you can see their
structure in Table 3.
Status
D7----D0
Data Byte(s)
D7----D0
Description
1001nnnn
0kkkkkkk
0vvvvvvv
Note On event. This message is sent when a note is depressed
(start). (kkkkkkk) is the key (note) number. (vvvvvvv) is the
velocity.
1011nnnn
0ccccccc
0vvvvvvv
Control Change. This message is sent when a controller value
changes. Controllers include devices such as pedals and levers.
Controller numbers 120-127 are reserved as "Channel Mode
Messages" (below). (ccccccc) is the controller number (0-119).
(vvvvvvv) is the controller value (0-127).
Table 3
10
MIDI Digital Audio Workstation (DAW)
Since the MIDI messages don’t carry any sound information, in order to test and play
music, a MIDI instrument is needed. For this project, the Hairless MIDI to Serial Bridge [10]
computer driver was used to interpret the MIDI messages received through the serial port
and send it to Reason [11], a DAW, for the messages to be played in one of its MIDI software
instruments.
Code
The Arduino Micro is programmed using the C language.
The photoresistor ports are the 12 analog ports of the microcontroller. The initial values for
the sensor measurements are zero.
// Photoresistor ports
int light[] = {A6, A7, A8, A9, A10, A11, A5, A4, A3, A2, A1, A0};
int lightValue[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
The MIDI protocol uses natural numbers from 0 to 127 to represent each note, so that one
unit stands for one semitone. The central piano C note number is 60. The array noteValue
stores the current state of each note, pressed (1) or not pressed (0).
// MIDI notes from C to B
int note[] = {60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71};
int noteValue[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
The array noteThresholdOn will store the minimum sensor measurement necessary to trigger
the note on. Likewise, the array noteThresholdOff will store the maximum value which will
trigger the note off. This acts like a digital Schmidt trigger, necessary to reduce note triggering
caused by noise in the sensor readings. This arrays are populated with the calibrate function.
// Ambient light adjustment values
int noteThresholdOn[12], noteThresholdOff[12];
Sensor ports, initial values and auxiliary variables.
// Ultrasonic sensor
int ultra = 7, ultraValue = 0x45, duration, velocity;
// Accelerometer
int accelX = 5, accelY = 3, valueX = 0, valueY = 0, pulseX, pulseY;
11
We read the accelerometer and ultrasonic sensors once every 100 iterations to decrease lag
when transmitting the signals though the serial port.
// Sensing interval counter
int i = 0;
// Photoresistor calibration
int calibrated = 0;
void setup() {
Serial.begin(31250); // MIDI protocol baud rate
}
void loop() {
if (calibrated == 0) calibrate();
// Read sensors every 100 iterations
if (i == 100) {
readAccel();
readUltrasonic(); // Read ultrasonic sensor and update ultraValue
sendControl(0x03, valueY); // Send effect to MIDI channel 3
sendControl(0x01, valueX); // Send effect to MIDI channel 1
i = 0;
}
i++;
We read the values on the 12 photoresistors and decide if the corresponding keys were
triggered.
for (int i = 0; i < 12; ++i) {
lightValue[i] = analogRead(light[i]);
if (lightValue[i] >= noteThresholdOn[i]) { // The key was "pressed"?
if (noteValue[i] == 0) {
noteValue[i] = 1; // Change state
sendNote(0x90, note[i], ultraValue); // Turn on note
}
}
else if (lightValue[i] < noteThresholdOff[i]) { // The key was "released"?
if (noteValue[i] == 1) {
noteValue[i] = 0; // Change state
sendNote(0x90, note[i], 0x00); // Turn off note
}
}
}
}
The functions sendNote and sendControl implement the MIDI messages and the serial
communication to turn on or off a note and send a control message (which can be used for
several purposes, in particular we will use it to produce sound effects).
// Send a MIDI note
void sendNote(int cmd, int pitch, int velocity) {
Serial.write(cmd);
Serial.write(pitch);
Serial.write(velocity); // Velocity = intensity or volume
}
12
// Send a MIDI control
void sendControl(int control, int value) {
Serial.write(0xB0);
Serial.write(control);
Serial.write(value);
}
The calibration routine populates noteThresholdOn with sensor measurements when all holes
are open and noteThresholdOff when all holes are closed. This is critical since ambient light
drastically changes the photoresistor behavior.
// Photoresistors setup
void calibrate() {
while (!Serial.available());
int incoming = Serial.read();
Serial.println("Calibration starting. Leave all holes open and press any key.");
while (!Serial.available());
incoming = Serial.read();
Serial.println("Values for threshold off will be: ");
for (int i = 0; i < 12; ++i) {
noteThresholdOff[i] = analogRead(light[i]) + 30;
Serial.print(noteThresholdOff[i]);
Serial.print(" ");
}
Serial.println(" ");
Serial.println("Close all the holes and press any key.");
while (!Serial.available());
incoming = Serial.read();
Serial.println("Values for threshold on will be: ");
for (int i = 0; i < 12; ++i) {
noteThresholdOn[i] = analogRead(light[i]) - 20;
Serial.print(noteThresholdOn[i]);
Serial.print(" ");
}
Serial.println(" ");
calibrated = 1;
}
The ultrasonic distance sensor works by sending one input pulse and measuring the time for
the echo pulse to arrive. The function microsecondsToVelocity is used to convert this waiting
time to a number from 0 to 127, which will be the MIDI velocity (volume).
// The ultrasonic sensor is used for measuring the velocity (volume) of the notes
played.
void readUltrasonic() {
// The PING sensor is triggered by a HIGH pulse of 2 or more microseconds.
// We give a short LOW pulse beforehand to ensure a clean HIGH pulse.
pinMode(ultra, OUTPUT);
digitalWrite(ultra, LOW);
delayMicroseconds(2);
digitalWrite(ultra, HIGH);
delayMicroseconds(5);
digitalWrite(ultra, LOW);
// The same pin is used to read the signal from the PING: a HIGH
13
// pulse whose duration is the time (in microseconds) from the sending
// of the ping to the reception of its echo off of an object.
pinMode(ultra, INPUT);
duration = pulseIn(ultra, HIGH);
if (duration != 0) ultraValue = microsecondsToVelocity(duration);
}
int microsecondsToVelocity(int microseconds) {
// The speed of sound is 340 m/s or 29 microseconds per centimeter.
// The ping travels out and back, so to find the distance of the
// object we take half of the distance travelled.
int cm = microseconds / 29 / 2;
// Velocity (intensity or volume) in MIDI goes from 0 to 127, and we measured the
// minimum distance being 30cm and the maximum being 145cm
if (cm >= 60) return 127;
else if (cm <= 3) return 0;
else return (127 * cm - 3 * 127) / 57;
}
The accelerometer is read in this function. Likewise, there is a conversion from its
measurements to a 0-127 scale understandable by the MIDI protocol. In particular, the X and
Y accelerations are used to produce 2 MIDI sound effects.
// The accelerometer is used for 2 MIDI effect controls
void readAccel() {
pinMode(accelX, INPUT);
pinMode(accelY, INPUT);
// read pulse from x- and y-axes:
pulseY = pulseIn(accelY, HIGH);
pulseX = pulseIn(accelX, HIGH);
// convert the pulse width into acceleration
// accelerationX and accelerationY are in milli-g's:
// earth's gravity is 1000 milli-g's, or 1g.
valueX = (((pulseX / 10) - 500) * 8);
valueY = (((pulseY / 10) - 500) * 8);
valueX = abs(valueX);
valueY = abs(valueY);
if (valueX >= 1072) valueX = 127;
else valueX = (valueX / 1072.0) * 127.0;
if (valueY >= 1072) valueY = 127;
else valueY = (valueY / 1072.0) * 127.0;
}
14
Results
After some debugging and adjustment, the prototype reliability was acceptable. The
calibration routine was essential to make it work, since ambient light turned to be a great
influence. Also, the introduction of two lighting thresholds (similar to a Schmitt trigger) was
important to reduce sensor jittering.
The advantages of using breadboards were the ease to assemble the prototype and
the possibility of quickly change the circuitry. However, faulty connections were a common
problem that could be solved by using printed circuit boards.
15
Conclusion
The device built shows that motion sensing does not necessarily need expensive
sensors and complicated architectures. It is a perfect demonstration of the creative potential
of the technology on music, and it brings a new way to create and perform music on live or
on studio. An improved version could be sold as a DJ equipment, toy or video game. The
involved gesture sensing technologies could also be used for other applications, for example:
socially assistive robots, remote controllers and alternative computer interfaces.
Possible improvements on the prototype for the second version include replacing the
mechanical parts of it by more comfortable and ergonomic ones, specifically designed for it.
There is also room for more features, such as control knobs and faders on the control module,
for a more practical and complete control of other parameters of the music. Giving the option
of the user to change MIDI parameters of the system such as controller numbers or channel
numbers is another functionality that could be added on a future version of it. This could be
made coupling a LCD display and buttons to the system or through serial communication to
a computer or smartphone. Other sensors could be added to provide even more ways to
change the music through gesture, such as a 3 axis accelerometer and gyroscopes.
16
Bibliography
[1] MIDI Manufacturers Association. “An Introduction to MIDI”.
http://www.midi.org/aboutmidi/intromidi.pdf
[2] “What’s a Theremin?”. http://www.thereminworld.com/Article/14232/what-s-a-theremin[3] “Ondes Martenot”. http://www.soniccouture.com/en/product/g27-ondes/
[4] Microsoft Corporation. “Kinect Sensor”. http://www.xbox.com/en-US/kinect
[5] Leap Motion Inc. “Leap Motion 3D Controller”. https://www.leapmotion.com/
[6] Oculus VR Inc. “Oculus Rift Virtual Reality Headset”. http://www.oculusvr.com/
[7] Arduino. “Arduino Micro”. http://arduino.cc/en/Main/arduinoBoardMicro
[8] MIDI Manufacturers Association. “MIDI Products”.
http://www.midi.org/aboutmidi/products.php
[9] “Understanding Schmitt Triggers”. http://www.ti.com/lit/an/scea046/scea046.pdf
[10] “Hairless MIDI to Serial Bridge. http://projectgus.github.io/hairless-midiserial/
[11] “Reason Digital Audio Workstation”. http://www.propellerheads.se/products/reason/
17