LumaSynth
Light-to-Sound Synthesizer · Android Application
/ Overview
Turn light into music — LumaSynth is a real-time light-to-sound synthesizer for Android that uses the device camera as a living, multi-channel instrument. Six independent sensors read luminance values from the live camera feed and map them directly to pitch, amplitude and timbre across 24 built-in synthesizer voices. Point the camera at anything — a candle, a sunset, a monitor — and the world becomes your instrument.
See it live/ Project Details
Name: LumaSynth – Light-to-Sound Synthesizer Android App
Category: Music Technology / Creative Instrument
Platform: Android (Jetpack Compose + Native C++)
Stack: Kotlin · C++ (Oboe) · JNI · CameraX · Jetpack Compose
Date: November 2025
/ Concept
The idea behind the project was to transform a smartphone camera into a playable audiovisual instrument. By continuously capturing live visual data and translating luminance values into musical parameters, the app creates a direct relationship between light, motion, and sound. Multiple virtual sensor zones interpret different areas of the camera feed simultaneously, allowing to shape melodies, textures, and rhythms.
/ Features
Sensor Engine
Six on-screen light sensors are placed evenly across the camera preview. Each sensor independently samples the camera feed at ~20 FPS, converting luminance values to voltage-like levels (1V–5V) that directly control pitch and amplitude. Sensors are individually assignable to any of 24 synthesizer voices.
S-Mode (GATE)
A dedicated S mode quantizes each sensor's luminance reading to a five-note pentatonic scale — C4, E4, G4, A4, C5. The 1V–5V voltage buttons become live real-time indicators showing the active pitch step, turning any sensor into a touch-free melodic instrument that responds to light intensity.
24-Voice Synthesizer
The instrument bank spans pads, organs, basses, acid leads, bells, glass tones, choirs, strings, noise sweeps and more. Each voice is rendered by a C++ Oboe audio engine with per-voice oscillators, envelopes and timbral shaping — all running in a dedicated audio thread.
/ System
LumaSynth uses a two-layer system architecture combining a Kotlin/Compose interface with a native C++ audio engine powered by Oboe and JNI. The Compose layer manages camera preview, sensor interaction, and visual feedback, while the native engine handles real-time synthesis, mixing, and recording. A JNI bridge continuously transfers luminance data from the camera pipeline to the audio engine, enabling responsive low-latency sound generation while keeping the UI and audio processing independent for smooth performance.
/ Tech Stack
Kotlin & Jetpack Compose
The entire UI is built with Jetpack Compose — sensor overlays, toggle buttons, live waveform graphs, mixer screen and instrument gallery are all declarative composables. Canvas-based rendering drives the luma wave graph, audio waveform visualizer and sensor divider lines in real time.
C++ DSP Engine
The synthesis core is a custom C++ engine using Google's Oboe library for low-latency Android audio. Per-voice oscillators, amplitude envelopes, a stereo widener, reverb, delay, filter and drive chain all run in the native audio callback at the device's preferred buffer size and sample rate.
CameraX and Image Analysis
Google's CameraX library powers the live camera pipeline. An ImageAnalysis use case feeds YUV frames to a background analyzer at up to 20 FPS. A pixel sampler reads the sensor regions and converts luminance to normalized float values used by both the audio engine and the UI. synthesis.
WAV Recording System
Stereo WAV recording captures the mixed output directly from the audio engine buffer. Recordings are auto-numbered sequentially and accessible through the built-in recording library with playback and file management.
/ About Lumasynth
LumaSynth began as an exploration of whether ambient light could become a musical controller. The result is a fully playable, studio-quality synthesizer that runs entirely on an Android phone — no MIDI controllers or external hardware required, just a camera, light, and sound.
