Research
FanRows explores embodied interaction in responsive sound environments. Its starting point is not the idea of controlling music with gestures, but the possibility that movement, perception and sound may gradually form a shared situation.
When a person raises an arm and a texture opens, the sound is not merely the result of that action. It changes the environment in which the next movement takes place. The altered sound may encourage the arm to continue, slow down, change direction or return to stillness. Over time, a relationship can emerge that was neither fully composed in advance nor produced by the participant alone. It develops through the continuing exchange between body, sound and space.
This is where FanRows connects to embodied and enactive approaches to cognition. These approaches understand perception and action as mutually dependent processes rather than separate stages. FanRows does not claim to measure cognition directly. It creates an artistic and technical environment in which aspects of perception–action coupling can be experienced, observed and refined.
The interaction is experienced first. Analysis makes selected aspects of its development visible.
From Experience to Observation
The research begins inside the interaction. A movement may unexpectedly feel connected to a sound layer. A repeated gesture may acquire a recognisable sonic contour. A period of stillness may allow the room to settle into a state that could not be reached through constant activity. Such moments are not discovered by reading values on a screen. They are noticed through moving and listening.
The analysis environment was developed to return to these moments more carefully. During a session, it records temporary traces of movement activity, repetition, coordination, recurring body configurations and changes requested from the audio engine. These traces can show that a response arrived later than expected, that several audio parameters changed at once or that a movement pattern returned after a period of variation.
They do not explain what the relationship meant to the participant. A technically clear correspondence may still feel empty, while an important bodily experience may leave only a modest numerical trace. The graphs therefore remain subordinate to embodied judgement. Their role is to make aspects of the interaction visible that may help explain why a resonance room feels coherent, delayed, overloaded or unexpectedly compelling.
This distinction is fundamental to FanRows. The system does not attempt to translate the participant into data. It observes selected properties of an interaction while leaving its artistic, personal and experiential meaning open.
Embodied Testing
The practical development of a resonance room moves repeatedly between exploration and inspection. A participant enters the room without having to perform a fixed sequence. They try different movements, listen to the response and notice whether a relationship begins to stand out. A slow opening of the arms may widen the sound field. A circular movement may produce a recurring shift in texture. A pause may reveal a layer that remained hidden during more active movement.
When such a relationship feels promising, it can be repeated. The participant listens for whether it remains perceptible across several attempts and whether it invites variation rather than becoming a rigid command. The analysis views can then help identify what changed technically: which body signals were involved, how the sound controls responded and whether the timing remained stable. The room can be adjusted and entered again.
FanRows refers to this evolving practice as embodied testing. It is not a conventional test in which a movement either passes or fails, and it does not ask whether the participant performed correctly. It asks whether a relationship between body and sound becomes clear enough to inhabit, repeat and develop without losing its openness.
At present, embodied testing is an artistic and technical working method rather than a scientifically validated protocol. Its purpose is to keep technical decisions connected to bodily experience. A mapping is not considered successful simply because its values are stable. It must also make sense while moving and listening.
Move → Listen → Notice → Repeat → Inspect → Reshape
Reading the Interaction
The current analysis environment offers several views of the same local session. They are not separate judgements about the participant, but partial perspectives on an evolving body–sound relationship. Some focus on timing between movement and audio response, others on repetition, continuity, coordination or the return of similar movement states.
Body–Sound Coupling
Movement activity and changes in live sound controls over the same period.
The Body–Sound Coupling view compares changes in bodily activity with changes requested from the audio engine. It shows whether movement and sonic modulation developed during similar periods, how many controls were involved and whether a measurable delay appeared between them.
This is useful when an interaction feels disconnected. A response may arrive too late, change too abruptly or involve so many parameters that the relation becomes difficult to perceive. The graph can help locate such problems, but it cannot decide whether the resulting experience is artistically convincing.
The coupling value describes a technical relationship between changing signals. It is not a measure of emotional resonance, attention, intention or artistic quality. A high value is not automatically desirable, and a meaningful experience does not necessarily produce a strong numerical correlation.
The question behind this view is therefore not whether more coupling is better. It is how much temporal correspondence is needed before movement and sound are experienced as parts of the same evolving situation.
Rhythm, Repetition and Flow
Approximate movement cycles, frequency and regularity.
The Rhythm and Repetition view searches for approximate cycles in selected regions of the body. It estimates whether a movement appears to recur, how long a cycle lasts and how regular the pattern becomes. A detected cycle may result from deliberate rhythm, habit, tracking variation or a short accidental movement. Its relevance begins when the participant notices it and chooses to return to it.
This possibility is central to FanRows. A movement does not need to be assigned a fixed function before the session starts. It may first emerge through exploration and gradually become part of a personal movement–sound vocabulary. The participant learns what the movement does by hearing, repeating and varying it rather than by memorising a command.
Continuity, interruption and stillness in tracked movement.
The Movement Flow view approaches the same session from another angle. It shows whether movement unfolds continuously, breaks into shorter passages or settles into relative stillness. Flow is treated as a temporal characteristic rather than as a measure of quality. Continuous motion is not inherently better than interruption, and stillness is not the absence of interaction. In some resonance rooms, stillness is the condition in which the sound gains clarity or maintains a temporary balance.
Together, these views help examine how a movement pattern develops over time. They make it possible to compare a fluid sequence with a hesitant one, or to observe how repetition changes as the participant becomes more familiar with the room.
Coordination and Recurrence
Temporal relationships between movement on both sides of the body.
The Left–Right Coordination view compares activity on both sides of the body. It can show whether the sides move together, independently, in opposition or with a temporal offset. The relation is descriptive rather than normative. Symmetry is not treated as better than asymmetry, and delayed or opposing movements may be entirely appropriate for a particular sonic situation.
This matters because bodily organisation is rarely limited to a single joint. A resonance room may feel different when both arms open together, when one side follows the other or when the torso remains stable while the hands move independently. The graph offers one way of observing these differences without classifying them as correct or incorrect.
Similarity between movement states at different moments in time.
The Recurrence Map compares movement states across the selected time window. Bright regions indicate that two moments were technically similar, while darker regions indicate greater difference. Because the map does not show the session only as a line moving from past to present, it can reveal returning trajectories, sustained states and repeated structures that may remain hidden in an ordinary graph.
Its interpretation remains deliberately cautious. Similar body configurations do not imply identical intention or experience. The same posture can feel different depending on the movement that preceded it, the current sound and the participant’s expectation. The map therefore identifies moments worth comparing; it does not claim that the same experience has returned.
Spatial summary of tracked activity across the upper body.
The Body Activity Map adds a compact spatial view. It shows where visible movement was concentrated during the observed interval and places the temporal graphs in relation to shoulders, elbows, hands and torso. It does not measure muscular effort or expressive intensity. Its role is simply to show how different bodily strategies may contribute to similar or different sonic developments.
Artistic Research Through Resonance Rooms
FanRows treats every resonance room as both an artistic environment and an experimental configuration. Sound material, motion-derived signals, mapping relationships, temporal behaviour, scene structure and visual atmosphere are composed together and then examined through bodily interaction.
The artistic and research aims cannot be separated cleanly. A room must first become experientially convincing before its interaction dynamics can be investigated meaningfully. At the same time, an evocative atmosphere is difficult to refine when the relation between movement and sound remains unstable or opaque.
The work therefore moves repeatedly between intuition and observation. An initial room is composed from an artistic intention. Bodily exploration reveals what feels coherent, confusing or unexpectedly strong. Analysis may suggest a technical reason for that impression. The room is then reshaped and experienced again.
This is why FanRows is not simply an analysis tool attached to an interactive sound system. The room, the method and the research questions develop together. The graphs do not stand outside the artistic process; they become useful only when they help to create a more convincing embodied experience.
The central theoretical perspective is embodied cognition, but the current object of research remains embodied interaction. FanRows does not yet claim to demonstrate how cognition changes. It investigates the conditions under which sound may become part of a sensorimotor environment: a space whose behaviour is gradually learned through movement, listening and repetition.
The decisive question is not merely whether the system responds to the body. It is whether the participant begins to perceive the room differently because of the relationship that has developed. Does a new movement arise because of what has just been heard? Does an accidental gesture become recognisable and repeatable? Does the room begin to feel less like a collection of effects and more like an environment that can be inhabited?
Research in Progress
FanRows is still in active development. The current analysis views are exploratory instruments whose usefulness must be tested through actual resonance-room sessions.
Some measurements may prove valuable for identifying unstable mappings, comparing repeated trials or documenting how an interaction changes over time. Others may remain descriptive without improving either the room or the research. Part of the work is therefore to determine which traces genuinely clarify the interaction and which merely add visual complexity.
The analysis is also affected by practical limits. Camera position, lighting, partial body visibility, tracking quality, frame rate and the selected observation window can alter the result. Repetition does not establish intention, technical coupling does not establish causation, and similarity between movement states does not prove that two moments were experienced in the same way.
These limits define what the current graphs can and cannot support. FanRows does not infer identity, emotion, attention, personality, health status or artistic ability, and it does not evaluate movement as good or bad. The system analyses movement geometry and technical interaction dynamics, not the inner state of the participant.
The current implementation runs locally in the browser. Webcam frames remain on the participant’s device, while pose processing, audio synthesis and live analysis take place locally. Persistent recording is not required for the interaction.
The purpose is not to produce more data. It is to discover which observations help create stronger embodied sound relationships.
Next Steps
The immediate research task is closely tied to the creation of the resonance rooms themselves. Before broader claims can be made, several rooms must become stable and experientially convincing. Their movement–sound relationships need to be explored, repeated and documented so that the analysis views can be judged against actual embodied experience.
A central question will be how intended mappings differ from relationships that participants discover on their own. A room may be designed around opening, balance or stillness, yet a participant may find a different movement that makes the space meaningful. These discoveries may reveal more about the potential of the system than the original mapping intention.
The planned small-group prototype will extend the research beyond the individual body. It will examine how several participants can influence a common sound environment while retaining local agency. The challenge is not merely to combine their movement data, but to create conditions in which mutual adaptation and shared regulation can emerge.
Current visual experiments add another dimension. Tracked body regions may locally influence fog, fluid or other visual material, while sonic parameters shape persistence, diffusion, turbulence or brightness. The aim is not to attach decorative graphics to the sound, but to investigate whether movement, sound and visual material can appear as aspects of one responsive environment.
FanRows is seeking critical exchange with people working in embodied interaction, embodied and enactive cognition, sound and music computing, movement research, artistic research, HCI and real-time graphics. The project is particularly interested in small, concrete forms of collaboration: examining a resonance room, discussing the analysis methods, comparing movement–sound mappings or helping to formulate an initial exploratory study.
The research questions remain open, and the methods are still evolving. The strongest evidence must come from the rooms themselves.
FanRows — Embodied Interaction in Responsive Sound Environments