To run this sketch 1) click the Carrier button, then 2) click the Modulator button to start the sound. Left wrist: UP/DOWN affects carrier note. Right wrist: UP/DOWN affects modulator amplitude, LEFT/RIGHT affects modulator frequency. Note: this sketch uses your computer or phone's camera. If prompted, plase click "allow" for it to work. (No images are captured or uploaded.)
I'm utilizing ML5.js and the PoseNet model tracks your right wrists x and y positions (left/right and up/down). It tracks your left wrists y position (up/down). This data then feeds into an FM Synthesizer and modulates a carrier frequency. Move your left wrist up/down to affect the base frequency. Move your right wrist, (arm & hand) up or down to increase or decrease the depth of modulator (from -150 to +150). Move it from left to right to increase the frequency (far left is 0, far right is 75) of modulator. Hint: more extreme effects are heard near the corners.
In this sketch the carrier is a sine wave and the modulator is a sawtooth wave.