Discover how Virtual Drums works.
How, from the sticks' motion, can we reproduce realistic drum sounds depending on your positions, speed? How can we handle differentlighting , background behind the player, different distance between the player and the cameras? Discover the 7 main steps:
Video acquisition of the 2 cameras simultaneously, at a frequency of 30Hz. At this step, image data are huge: 640 * 480 * 3 bytes * 2 cameras * 30Hz = 55Mb/s transferred on the firewire bus!
Step 2Setup the camera parameters: gain, saturation, white balance, shutter speed. The resulting image resulting is then easier to analyse.
Step 3Algorithms designed to process the colors and detect shapes are processing the images. At this step, we have to ignore the noise and all other colors but the sticks. We finally have the 2d positions of the sticks on the screen.
Step 4Because we have the 2d positions of the sticks on the screen, and because we know the position of the cameras in 3 dimensions, we can determine via geometrical algorithms the 3d positions of the sticks. We also handle the distortion of the lenses.
Step 5Once the sticks are detected, we process their speed. We also look for other pertinent information such as apparition frequency, percentage of theoretical object similarities, etc.
Step 6Using a positioning algorithm, we determine if the stick is hitting a drum element. We then compute the sound localization and the volume, depending on the speed.
Step 7The 3d Interface displays in real time the sticks and the whole drum kit. Drums are highlighted each time the player hits it. The fun is maximum!
You can look at the presentation of the Virtual Drums project for the Shaastra festival at the IIT Chennai, in India. Download the presentation PDF