Percussive Sound Recognition

Description

control games or applications by making real-world sounds

SampleSumo's Percussive Sound Recognition (PSR) engine is a machine learning-based middleware package that analyzes incoming audio (for example from a microphone input) and listens for occurrences of pre-defined percussive sounds.

choose actions that produce different sounds: hitting a glass with a spoon, clapping your hands, knocking on the table, ...
tell the PSR engine about these sounds by recording some examples
let the engine train a sound recognition model from these examples
now receive trigger events in your game or application whenever you make one of these sounds

These trigger events can be used to control various gameplay elements: the sound of tapping on the table could trigger a jump action, the sound of hitting a glass could cause a protection shield to pop up, the sound of hitting a box could cause a canon to fire, etc...

Licensing

The PSR engine is available as a C/C++ library with documentation, development guide and example code. We currently support Windows, macOS and iOS (let us know if you would like to use it on other platforms).

Evaluation

If you would like to evaluate the technology yourself, we offer the following options:

Windows and macOS
simply download our technical demo application

iPad
contact us with your iPad's Unique Device ID and we will give you access to the technical demo application via TestFlight

GDC 2012 video

Brian Crecente from The Verge interviewed us at GDC 2012 in San Francisco about our middleware and the demo game we demonstrated there. This gives a nice introduction to the technology and what it can do:

Applications and use cases

Games

make a bomb explode by clapping your hands
activate a protection shield by hitting a glass
control a game by hitting real-life objects (the table, a box, a cup, …)
imitate a certain sequence of sounds in the right order to unlock items

Interactive toys

teach your toy pet to make different moves for different sounds
make real drum sounds by imitating drum sounds into a microphone

Advertising

supply two different objects with a cornflakes bag, let your customers surf to a web site and have them perform a certain sequence of sounds with these objects to win a prize

Domotics and security

turn out the lights by clapping your hands once, turn them on by clapping twice
detect breaking of glass, dog barks, shouting, …

Demo game

One of the first games using this technology, was a tech demo game called "Cows'n'order" developed by a team of game developement students from mADE/NTHV in Breda (NL). It's a fun and collaborative game, in which 4 cowboys need to guide their cows over a track filled with obstacles. Each time an obstacle can not be avoided, a cow dies... Each cowboy can remove certain obstacles, but only if he is at the front of the group at the right moment. The fun thing is that you can make your cowboy move to the front by hitting the right real-world object: a glass, a box, a table, two spoons, etc... This is the trailer they made for the game:

Game credits: Karlijn Koning (team lead, artist) Dieter de Baets (tech lead, programmer) Rick de Water & Carsten Frentz (programmers) Bart Geels, Monica Pranoto & Janice Tirantautama (artists) Nils Desle, Robbie Grigg & Ronny Franken (supervisors).