MidiFly® is a gesture based MIDI controller. Instead of pressing physical buttons, the user generates MIDI messages by moving his or her hands in the air, intercepting a virtual grid of pads. By default, every virtual pad generates a note message, unless it's disabled. A disabled pad can either do nothing at all, or send a dynamics related message.
MidiFly® has 3 main screens: the Performance screen the Note Editor screen and the the Grid Editor screen.
This Performance Screen is where the MIDI instrument is actually played. It is composed of: a 3D view of the interaction grid a 2D view of the interaction grid a debug view of the depth camera output a MIDI setup menu, and an interaction menu.
a. MIDI menu
port name is the virtual MIDI port opened by MidiFly®: in most host software (synths, DAWs, etc.) you just need to look for a MIDI port with the same name.
channel is the MIDI channel used (if you're not sure, leave the default value of 1, or check the requirements of the host software you want to use).
program is the MIDI program used (if you're not sure, leave the default value of 0, or check the requirements of the host software you want to use).
save saves any changes to the MIDI configuration to file.
load loads the last saved changes to the MIDI configuration from file.
b. Interaction menu
The mirror toggle will mirror the grid.
The notes toggle, when disabled, will prevent note from being sent, even if the user is interacting with the grid.
The bend widget allows you to setup the dynamic bend control, using the standard MIDI bend message: in most synth software this automatically map to the bend control. Clicking on the widget iterates between 3 states.
The after widget allows you to setup the aftertouch control, using the standard MIDI aftertouch message: in most synth software this automatically maps to the aftertouch control. Clicking on the widget iterates between 3 states.
The CC1 and CC2 widgets allow you to send custom CC MIDI messages: in most synth software these messages can be mapped to custom controls. Refer to the manual of your chosen host software to map it correctly.
The velocity slider sets the amount of velocity applied to each pad interaction. On a regular keyboard, velocity is normally mapped to the strength used by the musician when pressing a given key; since there's not such a thing as strength in MidiFly®'s interaction paradigm, velocity here is a function of how quick you get from near the grid to into the grid. When your hand is near the grid a yellow block will appear in the 3D view, to give you a visual feedback: if you keep your hand at this distance for a moment before pushing through the grid, activating a pad, your velocity value will be lower than if you stab the pad quickly.
edit notes brings you to the Note Editor screen.
edit grid brings you to the Grid Editor screen.
c. Dynamics/Expressive messages
By default the virtual grid sends out notes. When you disable the note message for a specific pad, you have the possibility to use that pad to send dynamics messages instead, using the bend, after, CC1 or CC2 widgets.
Clicking on these widgets iterates between 3 states: disabled, meaning the message is not active at all. X, meaning the feature is mapped over the horizontal axis of the grid. Or Y, meaning the feature is mapped over the vertical axis of the grid.
As an example, let's imagine that you want to use the top row of your grid to control the bend and the last column on the right to control aftertouch, while still using all the other pads to send notes. First you would setup your notes, as described in 1.2. Then you would disable all the pads in the top row and the last column on the right. Then you would click on bend until it's mapped to X. Finally you would click on after until it's mapped to Y. With this setup, whenever your hand is activating any pad not belonging to the top row or the right column, you would be sending the notes you mapped. If you push your hand in the position of the first pad on the left of the top row, you would send a message setting the bend to the lowest value; if you push your hand in the position of the last pad on the right of the top row, you would send a message setting the bend to the highest value. If you move your hand along the pads on the top row, you will be moving the bend value accordingly. Similarly, if you move your hand along the right column, you will be moving the aftertouch accordingly.
The CC1 and CC2 settings also have a numeric parameter: this allow you to specify the CC number you want. Normally this is an arbitrary parameter you use to map a specifc CC to a chosen parameter of a host synth (please refer to MIDI specs and your specific software manual, in case of doubts).
Generally speaking, these messages are mapped on to an axis similarly to the way a fader operates, so mapping them on a row or column will make that row or column act as a touchless fader. If all notes are disabled and dynamics messages are mapped on to the whole grid, the grid will become similar to a touchless KaossPad. Finally, the user is free to experiment with any possible combination of notes and dynamics messages.
Note Editor screen
The Note Editor screen is where you configure the notes associated with each pad of your grid.
a. Setting notes individually
click on a pad to shift the note 1 semitone up
shift+Click on a pad to shift the note 1 semitone down
X+Click on a pad to disable or enable the note
b. Setting note for the whole grid
oct up and oct down buttons will shift all notes on the grid one octave up or down
The scale widget allows you to apply a scale to the grid. The first parameter is the scale itself; use the arrow buttons on the sides to explore the available scales. The second parameter is the key note; use the arrow buttons on the sides to move between notes. After selecting the 2 parameters, click on the apply button to apply the selected scale to the grid.
c. Other widgets
reload notes reloads the last notes saved to file
save notes writes the current note configuration to file
back brings you back to the Performance screen
Grid Editor screen
The Grid Editor screen is where you configure the size and composition of your grid. Every change in this screen is previewed in the 3D view and applied to the real world grid, so that you can test your settings before saving.
sensor distance is the distance in mm between the lens of your sensor and the centre of your grid. (i.e.: a value of 1000 will set the centre of your grid at 1m from your camera).
pads is the size in mm of the side of each pad (i.e.: a value of 100 will get you 10cm pads).
grid size is the number of pads on each side of your grid (i.e.: a value of 6 will get you a 6x6 grid).
save grid saves any changes to file.
reload grid will load the last values you saved to file.
back brings you back to the Performance screen.
As an example, let's assume that you have your sensor on the floor, with the lens pointing up. Let's also assume that you want an 8x8 grid composed of 20cm pads: you want to set grid size to 8 and pads to 200. After doing this you test your new setup moving your hand over the sensor. Let's imagine that you want to play MidiFly® while standing and that, during your test, you discover the grid is a bit too low to be comfortable for you: all you have to do is adjust the "sensor distance slider until you like the final position. When you're happy with your settings, you press the save grid button.
MidiFly® has been developed and tested on OSX, Windows, Linux.
Developed and tested with a MS Kinect,IntelRealsense, Kinekt for Azure, Kinect 2, Asus Xtion, Orbbec Astra, Mynt Eye and others.
Before starting,MidiFly® will look for and try to validate a file called MidiFly.secret, located in ~/[userName]/MidiFly/. This file links a specific user to a specific sensor, controlling a one way hash of the sensor's serial number.