System

FanRows is a browser-based platform for creating and experiencing responsive embodied environments.

A standard webcam captures body movement locally in the browser. From that movement, FanRows derives continuous signals such as posture, angles, velocity, openness, direction and stillness. These signals can influence sound, visual behaviour and the evolving state of an environment.

The system consists of two closely connected parts: the Studio, where responsive environments are created and refined, and the Runtime, where they are experienced through movement.

Both operate on the same underlying room and session definitions.

From Body to Environment

A FanRows interaction begins with the body, but it does not end with movement recognition.

The current processing path can be described as:

Webcam → Pose Tracking → Movement Signals → Responses → Room State → Sound and Visual Environment

A movement signal might describe the opening of an arm, an elbow angle, torso lean, overall movement velocity or a period of relative stillness.

These signals can be mapped continuously to properties of the environment. An arm movement may gradually open a filter, while velocity changes the intensity of another sound layer and torso movement affects spatial balance.

Several relationships can operate at the same time. The resulting response is therefore shaped by the current configuration of the body rather than by a single isolated command.

The environment that results from this process is again perceived by the participant and can influence the movement that follows.

Full-Body Interaction in the Browser

FanRows currently uses MediaPipe-based pose tracking to derive full-body movement information from a standard webcam.

No dedicated motion-capture system or wearable sensor is required for the basic experience. A participant can open a FanRows room in a modern browser, allow camera access and begin moving.

The current system can work with signals derived from areas such as:

• shoulder and arm configuration
• elbow angles
• torso orientation and lean
• body openness
• movement velocity and activity
• direction and change
• stillness and sustained posture

These are not treated as a fixed vocabulary of gestures.

Instead, they provide continuously changing dimensions that can be combined by a creator to shape the behaviour of an environment.

Studio and Runtime

FanRows separates the creation of an environment from the experience of it without separating their underlying definition.

Studio

The Studio is the creator-facing part of FanRows.

It provides tools for configuring movement signals, sound material, mappings, ranges, smoothing, effects, scenes, visual behaviour and other properties of a responsive environment.

A creator can change a relationship, immediately experience it with the body, return to the Studio and refine it.

This creates a development loop that is central to FanRows:

Build → Move → Listen → Observe → Adjust → Test again

Not every technically correct interaction feels convincing when experienced physically. A mapping can respond exactly as configured and still feel too abrupt, tiring, insignificant or predictable.

For this reason, embodied testing is treated as part of the design process rather than as a final validation step.

Runtime

The Runtime is the participant-facing environment.

It removes the authoring interface and presents the resulting room as an experience that can be explored through movement.

A participant does not need to know which technical signals or mappings are active. The aim is to allow relationships to be discovered through movement, listening and repeated interaction.

Continuous Responses

Many interactive systems are based on discrete events: a gesture is recognized and an action is triggered.

FanRows can use discrete behaviour where appropriate, but its current design emphasizes continuous response.

A shoulder angle can gradually change a sound rather than switch it on. Movement velocity can influence several properties at once. Different ranges of the same signal can affect different sound layers or effects.

This allows interaction to remain active between recognizable poses or events.

A movement does not simply produce an action. It can move the participant through a continuously changing region of the environment.

Scenes and Changing Conditions

A FanRows room can contain several scenes.

Scenes allow sound material, movement relationships, visual conditions and responsiveness to change during an experience without requiring the entire environment to be replaced.

Transitions can be gradual, allowing one condition to develop into another while preserving continuity.

A scene can therefore represent more than a conventional musical section. It can define a different condition within the same responsive environment.

This makes it possible for a room to evolve over time while still remaining open to movement and exploration.

Sound and Visual Response

Sound is currently the most developed output modality in FanRows.

The audio system is based on Web Audio and Tone.js and can combine layered sound material with effects and continuously changing parameters.

Movement can influence properties such as:

• gain
• filtering
• spatial and ambience effects
• layer balance
• density
• timing-related behaviour

Visuals can respond alongside sound and contribute to the perceived character of a room.

FanRows does not require sound and visuals to represent movement literally. Their purpose can instead be to change the atmosphere, spatial impression or responsiveness of the environment.

Higher-Level Interaction Design

Direct parameter mapping is useful, but complex environments quickly become difficult to design if every relationship is treated only as a low-level numerical connection.

FanRows therefore also explores higher-level interaction concepts.

Signal Presets describe movement qualities such as openness, movement energy or stillness.

Response Recipes can represent more experiential intentions such as gradual opening, spatial widening, increasing rhythmic density or atmospheric depth.

These concepts are intended to help creators think about the behaviour of a room rather than only about individual technical parameters.

They remain an evolving part of the Studio.

Room Life

Most digital interaction responds primarily to the current input value.

FanRows is also exploring what happens when an environment retains aspects of its recent interaction history.

A participant may remain still for a period, repeatedly explore a movement pattern or gradually increase movement intensity. Instead of treating each moment independently, Room Life can allow these developments to influence the longer-term state of the environment.

This creates different temporal layers of response.

Some changes can happen immediately with movement, while others may develop over several seconds or over a larger part of a session.

Room Life is currently experimental. Its purpose is not to simulate intelligence or emotion, but to explore how a responsive environment can develop a form of temporal continuity.

Interaction Analysis

FanRows includes experimental tools for observing interaction over time.

These can make selected movement signals, repeated configurations and system responses visible after or during an embodied test.

The purpose of this analysis is not to determine what a participant thinks or feels.

A graph may show that movement slowed down, that a posture recurred or that activity changed after a system response. It cannot by itself determine why this happened or what the experience meant to the participant.

Analysis therefore complements embodied observation rather than replacing it.

It can help a creator or researcher identify moments worth examining more closely and compare different interaction configurations.

Local-First Processing

Camera-based pose tracking is performed locally in the participant's browser.

FanRows does not require webcam images to be uploaded to a server in order to interpret movement.

This local-first approach is important both for privacy and for accessibility. A responsive environment can be experienced using equipment that many people already have:

computer + browser + webcam + headphones or speakers

The project is deliberately not designed around facial emotion recognition, identity recognition or psychological profiling.

Creating with FanRows

A typical creation process begins with an environment rather than with a predefined performance.

A creator selects sound and visual material, chooses movement signals and defines how different aspects of the body influence the resulting space.

Scenes can then introduce changing conditions, while embodied testing reveals which relationships remain interesting when they are actually experienced.

The process may therefore look like:

Create → Configure → Experience → Revise → Experience again

One of the next important steps for FanRows is to open this process to creators other than its developer.

Artists, designers, students and researchers should be able to approach the same system with different intentions and discover uses that were not predetermined by FanRows itself.

This will help reveal whether FanRows functions as a genuine creative platform rather than only as a tool for producing its existing Resonance Rooms.

Experiencing FanRows

For the participant, the same system should remain much simpler.

A typical interaction requires only:

Open → Allow camera → Start → Move → Explore

The participant does not need to understand MediaPipe, Web Audio, mappings or scene configuration.

Technical complexity remains behind the responsive environment.

The aim is for attention to move progressively away from operating an interface and toward movement, listening and the changing character of the space.

Current Capabilities

The current FanRows platform already supports browser-based full-body tracking, continuous movement signals, configurable mappings, layered responsive audio, visual environments, multiple scenes and gradual transitions.

The Studio provides editing and preview tools for these relationships, while the Runtime presents the resulting environments independently of the authoring interface.

Interaction analysis, higher-level response abstractions and Room Life are still evolving.

A small-group version for approximately three to five participants is planned as a later Phase 2 prototype.

Current Limitations

FanRows is an active independent development project rather than a finished production platform.

Webcam pose estimation is not equivalent to high-precision motion capture. Tracking quality can depend on lighting, camera position, available space and computer performance.

The current Runtime is primarily designed around a single participant, and creator usability has so far been tested mostly during the internal development of the project.

Several parts of the system — particularly Room Life, interaction analysis and higher-level authoring concepts — remain experimental.

The sound language of the first Resonance Rooms is also still being developed.

These limitations are not hidden behind the platform concept. They define much of the work that remains to be done.

Current Focus

The main Phase 2 focus is Nebula / Resonance Room 01.

The objective is not to continuously add more features, but to make one responsive environment stable, convincing and worth spending time inside.

At the same time, FanRows is preparing for a new stage of testing in which people other than its developer create environments with the Studio.

This will make it possible to observe not only whether people enjoy an existing FanRows room, but also whether the underlying system enables them to develop their own ideas.

A first small-group prototype will follow once the single-participant experience and creator workflow are sufficiently stable.

An Open Platform Direction

FanRows currently begins with movement, sound and visuals, but the architecture is not intended to remain limited to one tracking technology or output modality.

Future versions may incorporate other sensors, spatial audio, lighting, haptics, AI-assisted interaction or other forms of environmental response.

The underlying idea remains the same:

a body acts within an environment, the environment responds, and the resulting experience becomes part of what happens next.

FanRows is being developed as a creative system for constructing and exploring these relationships.

The experiential principles behind the system are described in Concept →.

Its possible research uses and emerging questions are discussed in Research →.

Further information about the technical implementation can be found in Architecture →.

FanRows — Building responsive environments for embodied exploration.