Case study · Personal project · Sep 2026 – Present
Motion Sense Game
A hands-free third-person action game — your webcam is the controller.
Motion Sense Game maps real-time hand gestures to full gameplay: walk, jump, slash, punch, blast, inventory, and a God of War–style orbit camera. A Python + MediaPipe vision stack classifies static poses and dynamic trajectories, then streams JSON events over a local WebSocket into Unity 6 — so computer vision and game systems stay fully decoupled and production-ready.
The hook
Controllers are great — until you want to play standing up with nothing in your hands. Motion Sense Game treats the webcam as a full input device: left hand owns locomotion and combat, right hand owns look and UI, and both work at the same time.
- No wearables — laptop camera in, playable character out.
- Game-quality feel — orbit camera, grounded jump, attack feedback, not a demo overlay.
- Clean architecture — CV process and Unity never share memory; they speak JSON events.
Control scheme
| Hand | Gesture | Gameplay |
|---|---|---|
| Left | Two fingers down + motion | Walk (camera-relative) |
| Left | Two fingers curl up | Jump (ground-gated) |
| Left | Open palm, fingers spread | Blast |
| Left | Fingers together + swipe | Slash |
| Left | Fist + forward thrust | Punch |
| Right | Fist + move | Orbit camera look |
| Right | Back of hand + finger count | Inventory open / select |
| Right | Index pointer / thumbs | Aim cursor · accept / deny |
Vision → game pipeline
- Capture — OpenCV webcam frames → MediaPipe Hand Landmarker (21 landmarks × up to 2 hands).
- Classify — pure functions for static poses; rolling buffers for walk / slash / punch / look deltas.
- Dispatch — priority state machine (inventory / combat / locomotion) emits small JSON events.
- Bridge — Python WebSocket server on
ws://127.0.0.1:8765. - Play — Unity client marshals events to the main thread and drives Rigidbody + orbit camera.
Same pattern as a hardware controller driver — gestures are just another input device.
Static vs dynamic gestures
The codebase keeps a hard split from day one: single-frame landmark shapes vs sequences over time. That separation is what keeps blast from stealing slash, and inventory from stealing walk when lighting or orientation is noisy.
- Static — open palm blast, thumbs, finger-count inventory, pointer tip.
- Dynamic — walk cadence, jump pose edges, lateral slash, punch thrust, fist look deltas.
- Orientation scoring — palm vs back using depth + 3D normals so inventory never false-fires on an open palm.
Third-person camera
Right-hand fist look is not freelook FPS. The camera orbits the player at a fixed distance with shoulder bias, smooth follow, and pitch clamps — the same “stuck behind you” feel as God of War–style action games. Walking always moves along camera planar forward so look and locomotion stay aligned.
Hard problems solved
Black / feedback webcam frames
Forced 1280×720 and screen-in-camera loops killed the feed. Native resolution probing, buffer flush, and a small preview window fixed capture reliability on Windows.
Gesture collisions
Open palm was firing walk/slash. Split blast (spread fingers) vs slash (fingers together + swipe), locked walk to downward two-finger motion, and assigned left/right hand roles.
Unity not receiving look events
NumPy scalars crashed JSON encode; WebSocket callbacks ran off the main thread. Added JSON sanitization and a concurrent queue drained in Update.
Camera feel
First look implementation rotated like FPS. Rebuilt as a world-space orbit with look-at focus, shoulder offset, and camera-relative walk.
Stack & what I owned
Solo end-to-end: gesture research, MediaPipe pipeline, state machine, WebSocket protocol, Unity input router, third-person camera, and player feel systems.
Status
Full control loop shipped and playable: dual-hand gestures, combat events, inventory hooks, and third-person camera. Source on GitHub; next iteration is combat enemies, inventory loadouts, and a polished arena level.