Building FanRows
A longitudinal case study of an experimental movement-and-sound system
Contents
- When movement was still a control
- From prototypes to sound spaces
- Building the system
- Studio and the creator workflow
- Sound spaces as experiments
- Failure, revision and embodied testing
- Testing with other people
- Eighteen months of working with AI
- What changed in the development process
- From individual interaction to collective systems
- Open questions
FanRows has been under continuous development for roughly eighteen months. During that time it has changed enough that describing only the current system would leave out much of what has become interesting about the project.
It did not begin as a research project about artificial intelligence, and there was no finished model of an embodied sound environment waiting to be implemented. The early problem was more concrete: could movement in front of an ordinary camera become a meaningful way of interacting with sound?
A first answer to that question is relatively easy to build. A camera can detect a body. A pose can be recognized. A gesture can trigger a sound. An arm can control a parameter.
The more difficult questions appeared after these mechanisms began to work.
What happens after someone has discovered the gesture? Does the interaction still have anything to offer? Does a person begin moving differently because of what is heard? Can several parts of the body influence a sound environment at the same time without turning the experience into a collection of invisible sliders? How clearly should cause and effect be exposed? At what point does an understandable interface become predictable, and at what point does an open system become arbitrary?
These were not questions formulated at the beginning and then tested against a finished prototype. They emerged while building and repeatedly entering the system.
The project therefore developed through an unusually tight relationship between implementation and bodily experience. Code produced an interaction. The interaction was tried in front of the camera. Something that appeared sensible in the data or interface could feel rigid when performed. A sound that seemed promising while auditioning it could become tiresome after several minutes of repetition. A mapping could be technically smooth and nevertheless lead the body toward an uninteresting movement. Sometimes a small change in timing or range altered the character of an entire session.
FanRows gradually became less concerned with recognizing gestures and more concerned with what happens inside this continuing exchange.
A person moves. The system changes. What is heard affects the next movement. That movement changes the system again.
The interaction is not contained in either side alone.
This case study follows how that shift happened, how it affected the architecture and the creative tools around FanRows, and how the development process itself changed over time. It also documents another development that was not planned when the project began: artificial intelligence became increasingly involved in the work, first as an aid for individual technical problems and eventually across software architecture, debugging, interface design, experimental planning, writing and conceptual reflection.
That part of the story is important, but it is not separate from the project.
It can only be understood by looking at what was actually being built.
1. When movement was still a control
The most obvious way to connect a body to software is to treat movement as input. That logic is familiar from interactive installations, games, motion controllers and musical interfaces: the system detects something meaningful and assigns a response to it. A hand goes up and a sound starts, the body turns and a parameter changes, a particular pose becomes equivalent to pressing a button. This approach was useful in the early FanRows experiments as well. Discrete actions are easy to test, easy to explain and technically convenient when an interaction needs a clear trigger.
The limitation only became apparent after the interaction had been used for longer than a short demonstration. Once a gesture has been understood, much of the exploration is over. The participant may repeat it, combine it with another gesture or perform it with more energy, but the underlying relation is already known. In most software that is desirable; an interface should be predictable. A sound environment posed a different problem. If someone was expected to remain inside the interaction for several minutes, legibility alone was not enough. The relation between body and sound needed to remain stable enough to be perceived, yet open enough that the experience did not collapse into operating a set of invisible controls.
This gradually shifted FanRows away from discrete gestures toward continuous movement properties. MediaPipe provided body landmarks in the browser, but the landmarks themselves were only raw material. FanRows began deriving values such as joint angles, torso lean, shoulder tilt, movement velocity, directional changes, broader body activity and periods of relative stillness. These signals did not have to wait for a pose to become true or false. They could vary continuously as the body moved and could influence several parts of the sound environment at the same time. An arm no longer had to reach a predefined position before anything happened. Moving through a range could gradually reveal a sound layer. Velocity could influence another parameter while a change in posture affected filtering, spatial behaviour or the balance between several sounds. A participant no longer had to perform a small vocabulary that the system had defined in advance. Movement could begin to act across a field of relationships.
The technical difference between these approaches is relatively simple. The experiential difference is larger. A gesture that starts a sound makes causality obvious. A continuous signal can establish a relationship that remains active and changes gradually. This opened a design problem that would remain central throughout the project: how obvious should the connection between movement and sound be? During technical testing, the temptation is to make every response unmistakable. Raising an arm might produce a large gain change; moving faster might strongly increase an effect. Such mappings are useful because they confirm that tracking and audio response are working. Yet when every physical change produces an equally conspicuous sonic change, participants quickly begin to manipulate the system deliberately. Attention shifts from the sound environment to the search for controllable effects: what do I have to do to make that sound happen?
FanRows did not need to remove causality. Without a perceptible relationship, movement would cease to matter. The question was how causality could be distributed over time and across several simultaneous mappings. Some relationships could remain immediate, while others could emerge slowly. A sound might already be present and become more exposed through movement. Several small influences could combine into a change that became perceptible only after a while. A response could depend not only on posture but on how movement had developed during the previous seconds. The most interesting sessions increasingly seemed to sit between complete transparency and complete uncertainty. A participant could begin to form expectations about the system without reducing the experience to a memorized control scheme. Different sound spaces could occupy different positions within this range, and there was no reason to assume that one level of explicitness would be correct for all of them.
The audio material introduced another constraint. Traditional music production generally assumes some form of timeline. Even when a composition contains improvisation or generative processes, a producer can usually determine when material enters, how long it remains and what follows it. FanRows cannot depend on that sequence. One participant may move immediately while another remains almost still. Someone may repeatedly explore a narrow movement range while another uses the whole body. Some relationships may be discovered quickly and others not at all. The sound therefore has to survive combinations and durations that were never explicitly composed.
This became apparent very early when selecting loops and textures. A sound that was attractive in isolation could become tiring after several minutes. Two individually convincing layers could occupy too much of the same sonic space when movement caused them to overlap. A strong transient might be effective once and intrusive when a participant repeatedly crossed the same movement range. The problem was not simply choosing better sounds. The material had to tolerate the behaviour of the interactive system.
The term sound space became increasingly useful because it described this condition more accurately than a linear composition. A FanRows session establishes a bounded sonic environment in which drones, textures, loops, rhythmic elements and effects can coexist in changing proportions. Movement alters their presence or character, while scenes can change the larger condition of the environment without turning the session into a playlist. The aim is not unlimited variation. A room still needs an identity. Its material has to belong together strongly enough that unplanned combinations remain plausible and that a participant can develop familiarity with the space over time.
As this structure became more complex, configuration itself became part of the creative process. A session could contain sound sources, motion signals, ranges, mappings, effects and scenes. Adjusting one relationship often required hearing it immediately in the context of several others. The work began to resemble composition, but the object being composed was not a sequence of sounds on a timeline. It was behaviour.
FanRows Studio grew out of this pressure. It provides access to the structures that Runtime later executes: sound material, signals, mappings, scenes and responses. The distinction between Studio and Runtime eventually became an important architectural boundary. Studio exposes the machinery because a creator needs to shape it; Runtime hides most of it because a participant should experience the sound space rather than the configuration behind it. This separation now appears natural, but it was not part of the original idea. It emerged because the project had accumulated enough interacting relationships that editing code and configuration files was no longer a practical way to develop the experience.
Many software problems can be investigated while sitting in front of a screen. FanRows repeatedly required standing up. A mapping could look smooth in a graph and still feel awkward when performed. A threshold could work reliably while encouraging an uncomfortable posture. Holding both arms in a certain position might be trivial from the perspective of pose recognition and physically tiring after thirty seconds. A transition could be technically correct and still interrupt the continuity of a session. Repeated use exposed problems that short functional tests did not.
The body therefore became part of the development process in a literal sense. It was necessary to enter the system repeatedly in order to understand whether it worked. Some problems became obvious within seconds, while others appeared only after several minutes: repetitive movement, tiring poses, dominant sound layers, transitions that felt too abrupt, or mappings that seemed insignificant at first but became interesting once the participant had begun to anticipate them.
By this stage, the original question had already changed. FanRows was no longer primarily asking whether body movement could control sound. That had been demonstrated. The more interesting issue was what happens once sound responds continuously and the person begins adapting movement in return. The system observes measurable properties of the body but has no access to intention or subjective experience. The participant, however, hears the consequences of those measurements and changes subsequent movement partly in response to what is heard. Movement changes sound; sound alters perception; perception influences the next movement.
This closed loop was not the premise from which FanRows had been designed. It emerged from building and using the system, and it would gradually become one of the central ideas behind the project.
2. From prototypes to sound spaces
The early FanRows prototypes were much simpler than the present system. The essential loop was already recognizable: a webcam observed the body, movement was translated into measurable values and those values affected sound in the browser. At first, the practical questions dominated. Could ordinary consumer hardware provide tracking that was stable enough for musical interaction? Could the response remain immediate enough that movement and sound felt connected rather than delayed? Could several relationships operate at the same time without making the system unreliable?
Once workable answers existed, the limitations of the first approach became more instructive than its success. A shoulder angle could alter brightness, movement velocity could affect an effect parameter, or a held pose could introduce another layer. Each mapping worked in isolation, yet several minutes of use revealed that a collection of successful mappings did not automatically become a convincing environment. The body could control several parameters while the experience still felt like operating a machine.
This led to a more explicit layer between pose tracking and audio behaviour. MediaPipe continued to provide landmarks, but FanRows increasingly derived movement features from them rather than mapping landmarks directly. Joint angles, relative positions, velocity, periods of stability and persistence across time became part of the signal model. A shoulder angle could be interpreted as a continuous quantity rather than a binary condition such as “arm raised”. Body velocity could be measured over time instead of reduced to whether movement had occurred at all. Stillness could become meaningful because the system could distinguish a brief pause from a more sustained reduction of movement.
An early family of experiments combined these different temporal relations. A shoulder angle might continuously influence tonal brightness, lower body velocity might increase global reverberation, and a separate posture held for a defined period could gradually introduce another texture. The mechanisms were individually simple, but together they changed the character of the interaction because the system no longer responded only to events. It could remain sensitive to changing states and to the duration of those states.
The need to stabilize this behaviour introduced what became a regulation layer. Raw motion data fluctuates, thresholds can be crossed repeatedly within a short period and a participant may pass through a position that should only matter if it is maintained. Persistence, hold-times and state transitions became practical means of distinguishing a brief measurement from a meaningful condition. At first this was primarily an engineering concern: without regulation, sound could become unstable or twitchy. Over time it also changed the conceptual model. FanRows could respond to tendencies, duration and evolving conditions rather than demanding precise commands from the body.
The audio engine developed in parallel. Hard starts and stops made the mechanics of the interface too audible, so layered material increasingly changed through gain ramps, filtering and effects. Sounds could emerge, recede or remain near the edge of perception instead of being switched on and off. A loop could stay present for a longer period if its role changed gradually, moving between foreground and background as the body changed. Reverb, filtering, pitch-related processing and other effects could alter the character of existing material without requiring a new sound to be triggered.
This made repetition one of the central design problems. Conventional sample libraries are usually created for linear production, where an arranger controls when a loop enters and when it disappears. In FanRows, part of that timing is determined indirectly by the participant. A loop may remain available far longer than expected, return frequently because the same movement range is crossed repeatedly, or overlap with other material in combinations that were never intended by its original producer. Some sounds that seemed immediately attractive became irritating after five minutes. Strongly characteristic material could dominate an entire room, while restrained drones or textures sometimes proved more durable precisely because they left space for other layers and for the participant's movement.
Movement itself could become repetitive in the same way. If an attractive response was linked to an easily discovered gesture, a participant might find the effect, repeat the movement, hear the reward and repeat it again. This could be engaging for a short period, but it did not necessarily produce exploration. Repetition therefore became something to observe rather than simply eliminate. It could indicate that a mapping was compelling, but it could also reveal that the system had become too obvious. The distinction depended on whether repetition developed into a richer rhythm of interaction or collapsed into a single learned action.
As experiments accumulated, they also had to become reproducible. This was the point at which the concept of a Session became increasingly important. A session could contain the configuration required to reconstruct a particular environment: sound material, movement signals, mappings and the larger conditions under which they operated. Separating the session from the application made it possible to create new experiments without rewriting the core system, and it also made comparison possible. A configuration could be repeated, changed and tested again rather than existing only as a temporary state in the code.
Scenes introduced another level of structure, although their role gradually moved away from the conventional idea of a sequence on a timeline. Within FanRows, a scene became better understood as a bounded condition inside the same larger sound environment. It could contain different material, mappings or degrees of responsiveness without behaving like a separate track. Moving from one scene to another could make the environment denser, more open, more rhythmic or less explicit while preserving enough continuity that the participant remained inside the same session.
Transitions therefore became part of the interaction architecture rather than simple playback changes. A hard switch could make the participant feel that one piece had ended and another had begun. Gradual changes allowed another interpretation: the participant had entered a different region of the same space. This was particularly important in sessions with several scenes, where development over time was useful but a fixed musical timeline would have contradicted the openness of the movement interaction.
Repeated use also suggested that variation should not be confused with novelty. If a sound space changed unpredictably every time it was entered, there was little opportunity for familiarity to develop. FanRows increasingly valued recognizability across repeated sessions. A participant could return to the same environment, encounter relationships that were already partly known and still discover aspects that had not been noticed before. The comparison with a physical place became useful: returning to the same landscape does not reproduce the same experience, but familiarity provides a background against which smaller differences and new routes become perceptible.
By then, direct editing of session data had become an obstacle to experimentation. Signals had useful ranges, mappings had parameters, layers needed gain and effect settings, and scenes needed transitions. A configuration that appeared sensible in JSON often needed several adjustments once it was experienced through the body. FanRows Studio emerged because the project required an interface for shaping these relationships while remaining close to the runtime behaviour.
Studio was not conceived as a conventional digital audio workstation. There was no timeline to arrange in the usual sense. Its purpose was to expose the relationships inside a sound space: movement signals, sound material, mappings, responses, scenes and the parameters that regulate them. As this creator environment developed, the distinction between Studio and Runtime became clearer. Runtime is where a participant enters a room. Studio is where the conditions of that room are assembled and calibrated.
This was a significant change in the nature of the project. FanRows had initially been an application containing a series of experiments. With Studio, it began to become a system for creating experiments.
By early 2026, a recognizable architecture had emerged around this work. Webcam tracking supplied body landmarks; feature extraction turned them into useful movement signals; regulation stabilized and interpreted those signals over time; mappings connected them with an audio engine built around layered sound and real-time processing. Sessions provided the configuration surrounding this chain, scenes structured changing conditions within a session, and Studio made the relationships editable.
Seen retrospectively, this architecture appears coherent. During development, however, its parts did not arrive as a single design. Feature extraction appeared because raw landmarks were not expressive enough. Regulation appeared because direct values were unstable or too immediate. Layered audio developed because simple triggering did not sustain an environment. Sessions became necessary because experiments needed to be reproducible. Scenes emerged because one environment needed to change without becoming a conventional timeline, and Studio appeared because the growing number of relationships could no longer be shaped effectively in code alone.
The architecture therefore records part of the history of FanRows. Each layer reflects a problem that became visible only after the previous one had begun to work. By the time this structure was reasonably stable, the main difficulty had shifted again. The project was no longer only concerned with whether a sound space could be built. The harder question was how to make one worth entering repeatedly.
3. Building the system
By the time FanRows had moved beyond isolated motion-to-sound experiments, the software had acquired a different kind of problem. It was no longer sufficient to make tracking, mappings and audio processing work independently. They had to remain responsive while running together, and they had to be structured in a way that allowed the interaction model to keep changing. The architecture therefore developed around a relatively simple path from body to response, but with increasingly clear boundaries between the stages: webcam input is processed into body landmarks, landmarks are transformed into movement signals, those signals pass through mapping and regulation logic, and the resulting values affect audio and visual behaviour.
The system runs primarily in the browser. This was initially a practical choice but became an important characteristic of FanRows. A participant does not need a dedicated motion-capture system, external tracking server or specialized audio workstation to enter a session. An ordinary webcam provides the visual input, MediaPipe estimates the body pose, and movement analysis takes place on the same device. Audio synthesis and processing also run locally through browser audio technologies. Video frames and derived pose information therefore do not have to be transmitted to a remote service for the core interaction to function. Keeping this loop local reduces latency and also establishes a clear privacy boundary: the camera is used to derive movement geometry, not to identify the person or infer an internal state.
MediaPipe solves only the first part of the problem. Its landmarks describe where body points are located, but a coordinate by itself has little musical meaning. FanRows therefore maintains a feature layer that turns successive pose estimates into signals that are more useful for interaction. Joint angles can describe articulation; relative positions can describe posture; changes between frames can provide velocity or directional information; combinations of measurements can represent broader movement activity or stillness. This feature layer also creates some independence from the tracking technology itself. The audio system does not need to know how a shoulder coordinate was obtained. It consumes a movement signal with a defined range and behaviour.
The distinction became important as the number of signals increased. Torso lean, shoulder tilt, left and right movement velocity, body velocity and other continuous values can coexist with more explicit pose conditions. A room may use several of them at the same time. One signal can influence gain while another changes filtering or spatial processing, and the same signal can affect more than one sonic parameter. The mapping layer therefore does not describe a one-to-one correspondence between gesture and sound. It describes a network of relationships whose effects overlap during a session.
Raw values rarely behave well enough to be sent directly to audio parameters. Camera tracking contains small fluctuations, human movement is irregular, and a threshold that appears precise numerically can become unstable when the body moves around its boundary. FanRows therefore applies temporal and regulatory behaviour between detection and response. Values can be smoothed, ranges can be constrained, conditions can require persistence, and a posture can remain relevant only after it has been held for a specified period. This makes it possible to distinguish passing through a configuration from settling into one. In practical terms, the regulation layer prevents the audio system from reacting to every small fluctuation; conceptually, it gives duration and stability a role in the interaction.
The audio engine follows the same preference for continuous state over discrete events. Sound layers are generally kept synchronized and regulated through gain and parameter changes instead of repeatedly starting and stopping playback. This allows a layer to remain available while becoming almost inaudible, then return gradually as movement changes. Several layers can coexist and move between foreground and background without losing their temporal relationship. Effects can be applied at global or per-loop level, providing a baseline state that movement mappings modify rather than reconstructing the processing chain for every response.
This distinction between a baseline and modulation became useful as rooms grew more complex. A sound can have a stable underlying level, filter state or spatial character while movement adds controlled deviation around that state. The same principle applies to scene-wide processing. A scene can establish a particular sonic condition and movement can continuously alter it without destroying its identity. Parameters such as filtering, reverberation, modulation or pitch-related processing can therefore participate in the interaction alongside gain. The body is not merely deciding which loops are audible; it can change the character of material that is already present.
The current FanRows architecture separates motion tracking, movement signals, configuration and real-time execution while keeping them connected through the same room model.
Sessions became the unit that binds these relationships together. Their configuration is stored as structured data rather than embedded directly in application code. A session can define its sound material, scenes, mappings, regulatory parameters and other behaviour in JSON, allowing the same runtime to execute very different sound spaces. This separation was important for experimentation because changing a room no longer meant changing the application itself. It also introduced a practical requirement that would later become increasingly significant: configuration formats had to remain understandable and maintainable as the software evolved.
Scenes sit inside this session model. They define bounded conditions rather than positions on a conventional musical timeline. A scene can determine which sound layers and mappings are active, establish particular regulatory values and influence how the audio environment behaves. Transitioning between scenes changes these conditions while the audio system continues running. This allows a session to develop without requiring the participant to follow a predetermined sequence of musical actions.
Not all variation has to come directly from movement. The Cue Engine was introduced to create controlled temporal changes within a scene. It can alter selected parameters through gradual curves and probabilistic activation without interrupting playback. This provides another source of change alongside the body. The participant is therefore not interacting with a completely static mapping surface: aspects of the environment can evolve while movement continues to influence them. This raised a design issue that became important later. Variation generated by the system must remain subtle enough that participants can still perceive their own relationship with the room. If autonomous changes become too dominant, the connection between movement and consequence becomes difficult to read.
As configuration became richer, reusable structures began to appear in Studio. Signal Presets provide starting points for recurring kinds of bodily input such as movement energy, openness, angular change or stillness. Response Recipes perform a similar role on the sonic side by describing useful response characteristics that can then be calibrated for a particular room. These are not intended to turn interaction design into a menu of fixed behaviours. Their practical purpose is to avoid rebuilding common structures while keeping the underlying ranges and mappings editable.
The sound catalog addressed another scaling problem. Early prototypes could refer directly to a small number of audio files, but this approach became increasingly impractical as the material grew. The catalog adds searchable information about sounds, including their role, source, musical or structural characteristics and manually refined annotations. This matters because filenames and conventional genre labels reveal little about how a sound behaves in an interactive environment. A restrained drone, a pulse, a fragile texture and a sharply defined rhythmic loop may require very different treatment even if they originate from the same sample collection.
Visual feedback developed as a parallel output rather than as the primary representation of the interaction. The tracked body can be displayed directly, but FanRows has also experimented with more abstract visual material. The architectural intention is that visuals can receive movement and system signals through a boundary similar to the audio system rather than becoming inseparable from pose tracking itself. This leaves room for different visual approaches without changing the underlying body–sound relationship and also permits sessions in which sound remains the dominant feedback channel.
As these parts accumulated, architectural separation became more valuable. Motion input, feature extraction, configuration, audio rendering, visual feedback and storage increasingly needed to evolve at different speeds. FanRows therefore moved toward explicit domain concepts and clearer boundaries rather than allowing Studio components, tracking code and audio behaviour to depend directly on one another. Session, Scene, Mapping, Loop, Catalog, Studio, Runtime and Audio Engine became meaningful concepts in the software itself, not merely names used in the interface. The architecture has also been influenced by ports-and-adapters thinking: technologies at the edges of the system can change without requiring the central interaction model to be redesigned.
The distinction between Studio and Runtime is one consequence of this separation. Studio operates on an editable representation of a room. Parameters may be incomplete, changed repeatedly or inspected while the creator experiments. Runtime has a different responsibility: it projects a defined session into a live interaction and should not expose the complexity required to construct it. The separation is loosely CQRS-like in the sense that editing the model and executing it are treated as different concerns. It is not an attempt to impose an enterprise architecture pattern on an artistic system; it developed because mixing configuration state with live interaction repeatedly made both sides harder to reason about.
This architecture also made failures easier to locate. If motion tracking is unstable, it can be distinguished from a poorly chosen mapping. If the mapping behaves correctly but the sound is unconvincing, the problem may lie in the material or audio response rather than in tracking. If a session works in Runtime but is difficult to modify, the problem belongs to the creator workflow. This separation does not remove the interdependence of the experience — a participant ultimately encounters all of these layers at once — but it gives development a way to examine them individually.
The system that resulted is more elaborate than the first prototypes, but most of its complexity did not come from an ambition to build a large software platform. It accumulated because maintaining a continuous body–sound relationship proved to require distinctions that were easy to ignore in a short demonstration. Tracking had to be separated from interpretation, interpretation from regulation, regulation from sonic response, and creation from execution. Sound material needed its own organization, scenes needed continuity, configuration needed to survive experimentation, and local execution needed to remain fast enough that the loop still felt immediate.
The architecture is therefore not only infrastructure beneath FanRows. It contains assumptions about the kind of interaction the project is trying to make possible. Continuous signals are preferred where a binary gesture would remove too much variation. Audio layers remain alive because abrupt triggering tends to expose the machinery. Sessions preserve conditions because an experiment needs to be repeatable. Studio and Runtime are separated because designing an interaction and inhabiting it are different activities.
By this stage FanRows had developed enough technical structure to create substantially different rooms without rebuilding the underlying system. That solved one class of problems and exposed another. The available architecture could produce many combinations of movement and sound, but technical possibility provided almost no guidance about which combinations were worth using. The next phase therefore moved increasingly toward the creator workflow itself: how a room is assembled, heard, revised and gradually given a coherent character.