Mediapipe Web Integration

Goal

I first explored MediaPipe on a Raspberry Pi while studying robotics, then later used it in TouchDesigner for interective art. My goal was to bring this interaction into a web app that anyone with internet access and a camera could open, removing the difficult setup of installing TouchDesigner and finding a compatible MediaPipe integration.

I developed a custom Framer code component that connects each CMS item’s image, title, description, and link to the 3D environment. The cards cycle through projects, move laterally across the sphere, and display more information when hovered over or focused.

Controls

• Right middle finger and thumb pinch: Zoom in or out

• Right thumb crosshair: Hover over and select projects

• Left hand, closed to open: Change gallery modes

• Left hand, closed to open over a card: Focus or unfocus the project

Mouse Controls

• Left mouse drag: Move around

• Mouse wheel: Zoom

• Right mouse drag: Rotate the view

• Mouse click: Select or focus a project

What I learned

I learned how to combine MediaPipe hand tracking, Framer CMS, and 3D interactions to create a responsive web experience. I also gained a better understanding of designing intuitive gesture controls while providing reliable mouse controls as a fallback.

Future

In the future, hand-tracking signals from the web app could be sent to connected devices such as lights, projectors, motors, or haptic systems. This would extend the 3D interaction beyond the screen and create a more immersive physical experience.

I do not own any images used as a place holder, images used only for the purpose of demonstrating.