2001
Communication between Network Visualization and Music
- / -
Background
Many of the databases offered via the web by the Union of International Associations enable users to represent the relationships between profiles in which they are interested as a dynamic network map of hyperlinks by clicking on the map logo. The user is then free to manipulate the network, increasing and decreasing its complexity, and configuring it in ways that are more meaningful. Progress in developing these facilities is described in a separate note: Interactive Hyperlink Map: Auto-generated, Self-organizing Link Visualization.
This map explored the possibility of attaching simple sound files to each node, allowing the user to trigger them individually by mouse operations. This was seen as the basis for developing an acoustic mnemonic code for structures. Ways of packing sequences of notes into each nodal file were envisaged.
This work was initiated in connection with a 1998-9 contract of the UIA with the European Commission (DG-XIII) on the Information Context of Biodiversity Conservation that has a specific emphasis on innovative uses of multi-media to enhance access to information (explanation)
An extensive bibliography (annotated) of items providing the rationale for this sonification approach is provided by the International Community for Auditory Display . Selected items have been incorporated into the references study on Knowledge Gardening through Music: patterns of coherence for future African management as an alternative to Project Logic.
Sound experiments
A more structured approach to the use of sound to enhance comprehension of complex patterns of information emerged as a result of exploration of use of the Koan software provided freely over the web as a browser plugin by SSEYO (UK). This focuses on the use of generative music seeded by particular (data) patterns and controlled by an extensive array of parameters familiar to musicians. Generative music has the additional advantage of avoiding some of the obvious copyright issues associated with supplying sound over the web. Of special interest is the fact that the amount of data transferred as a pattern to the plugin is normally less than 30k, since the music is generated by the plugin on the user's computer rather than having to be downloaded in its entirety.
From November 1999, ways of integrating sound into the visualization of complex networks is being explored. In order of increasing challenge and significance, these possibilities can be tentatively presented as:
- Music unlinked to data or map movement
In this case the music is basically generated in parallel with the network as a form of accompaniment (like in the silent movies). The issue here is whether any such accompaniment can usefully enhance comprehension of the map by judicious aesthetic choices, or whether it is purely decorative and "for effect" -- especially if the user is free to modify the music at will. Rersearch elsewhere has shown that background music can assist in the absorption/retention of information. i.e. even audio at this simple level, can be of real benefit, assuming it is suitably chosen.
- Music driven by data (but unlinked to any map
movement)
In this mode, data from which the map is generated is also used to affect or determine the "music". The music is therefore generated from the data and thus to some degree encodes the complexity represented visually in the accompanying map. The challenge here is to determine useful ways to translate the map coding into the patterns through which the music is generated. The question is the degree to which this musical encoding is meaningful in new ways, especially since it does not affect visually the dynamics of the map to which the user is exposed. In its more challenging forms, this mode would give rise to "music" unique to each map. How musical or meaningful it would be could be a challenge for the software facility and the user's mastery of its features. Particular patterns of sounds, or musical sequences, could be associated with the nodes of the map to be triggered by mouse operations.
- Map movement driven by music (through common
data)
In this mode, the data from which the map is generated is also used to trigger dynamics in the map, namely movements of particular nodes in response to particular musical notes and/or instruments. The map therefore moves in rhythm to the music. Again the question arises as to the degree to which this is simply an intriguing effect, as opposed to enhancing any form of comprehension of what the map (and the music) then represent.
- Music driven by map movement (through common
data)
In this mode, the user's manipulation of the map has effects on the pattern of sound. This could be merely a trivial effect to accompany normal mouse manipulation of the map. However it is possible that a user might benefit from this effect in unsuspected ways.
- Map movement driven interactively by music
(AND vice versa)
This mode essentially combines the two previous modes. The key here is the centre of gravity of control. Does user movement of the map take greater or lesser precedence over the music driving that movement on the basis of the data.
- Map movement driven via user (micro) input
This mode explores the use of external user sound input as a means of reconfiguring the map interactively, in effect by playing to it -- "taming the beast". Emphasis in this case is on the musical ability of the user in playing sounds that entrain the map into new patterns or sequences of patterns. Again the question, arises as to the extent this is more than an intriguing toy, and whether it offers new opportunities for comprehending and working with complex patterns.
- Strategic coordination (possibilities to be
explored)
It is at this stage that it becomes possible to explore the use of insights from music to effectively "harmonize" complex patterns of relationships.
As of November 1999, UIA on-line experiments are at Stage 1, verging into Stage 2 in the above schema.
Much of the challenge for the networks of strategies, developed by networks of organizations, in response to networks of problems, based on networks of values, lies in how these are to be coordinated or "harmonized" in some way. The well-explored conventional approach, based on some simplistic consensus, has a relatively poor track record and few prospects for greater efficacity. The dimensions explored by music redefine "consensus" in richer musical terms that offer many more ways to explore relationships between seemingly disparate elements, using both consonant and dissonant features to advantage.
The use of sound is therefore seen as a way of benefitting from insights into harmony that are widely and intuitively understood. Hopefully it will also help to reframe strategic responses to complex issues in ways to which younger generations can resonate more optimistically.
