This article was written by Dr. Michael Haverkamp and published in Bronner, K., & Hirt, R. (2009). Audio Branding: Brands, Sound and Communication. https://www.nomos-shop.de/en/p/audio-branding-gr-978-3-8329-4352-3
Introduction: Why Synesthetic Brand Design?
During the last decades it has been realized that corporate design gains crucial value if it provides activation of various senses. Synesthetic brand design means an optimization of a brand image and its symbols by using various references to multi-sensory perception. This design process implies synesthetic principles which are based on common properties of human perception (Haverkamp, 2008). The wording follows an extensive definition of synesthetics (Filk, 2004).
All types of processes of cross-sensory coupling within the perceptual system are covered by this approach. Beside common ways of interaction between the senses, the generic concept additionally refers to individual phenomena like genuine synesthesia, which are currently subject of various scientific studies. Synesthetic design can also be applied to product development and refinement of living environments.
The human perceptual system is designed to process multi-sensory data. We know from daily experience that we can use each sensory space separately: we can listen, and only listen, see, and only see, smell, taste etc. This experience makes us believe that sensory processing is split up into different spaces, strictly separated between each other. But in fact the phenomenon of separated senses is a result of a focus of attention on a specific sensory space. In contrary, when we are using sensory information intuitively during daily life, there is no separation of the senses, while nearly all stimuli are referred to multi-sensory objects.
First, on a physiological basis, our senses are specific. We always can clearly distinguish audible sensations from visual sensations from tactile sensations from smell … etc. Stimuli are processed via very different sense organs and through specific neuronal networks, finally projected upon specific areas of the brain (primary fields). Secondly, within brain structures of higher level, perception tends to match the data to provide a clear description of objects we are dealing with.
For daily live activities and behaviours it does not make sense to handle data of different sensory spaces separately, but to identify objects with multi-sensory properties. A bell is not defined by only its shape, but by a variety of visual, tactile, auditory, olfactory and other features. Only the combination of all is what we describe as bell. While not all sensory data are available at one time – the bell may only be heard from distance – memory helps to add the missing features from previous experience. Only the combination of multi-sensory features allows us to identify the object.
Therefore the perceptual system always tries to relate even single stimuli to a multi-sensory model. Thus, with view on design tasks it can be concluded that multi-sensory references must be taken into account even if design is limited to one sensory space. It must clearly be understood that even a purely graphical logo can have multi-sensory references which strongly can influence its perception and subjective assessment, although it only stimulates vision as a single sensory channel.
Perceptual Objects
In general, the perceptual system tends to provide multi-sensory models of physical objects. These models are needed by the individual to interpret his environment and to coordinate his actions, while it is surrounded with objects which physical nature cannot be accessed directly. As a result of perception and cross-sensory integration, an image occurs in consciousness. This image can show aspects of vision, audition or of any other sensory channel.
In contrary to the physical object, which provides the stimuli, these subjective representations are here named perceptual objects. A sensory channel will further on be described as modality. Within this context, image means a multi-sensory, not only visual representation. Here, a physical object is anything that provides specific stimuli. It is not necessarily a solid structure, but can be a liquid, a gas, a human being, an animal – or can be characterized by lack of material, like a hole.
A subjective representation of a physical object is based on a sensory hypothesis, generating a perceptual object that in some ways correlates to its physical source. While physical objects always provide various stimuli, perceptual objects usually appear to be multi-sensory, i. e. they contain auditory, visual, tactile and other data. Only if stimuli of a single sensory channel are presented which are unknown to the individual, a first approach is made by generating perceptual objects of this single modality, e.g. auditory or visual. It therefore can be stated that – in case of known stimuli – perceived auditory, visual or tactile data are always related to multi-sensory perceptual objects.
Sensory information which is not perceived at a moment is supplemented by data stored in memory, which is a result of previous experience (figure 1). If no mnemonic data has been stored, the whole features must be taken from instantaneous perception. In this case more time is needed for a multi-sensory observation of the object. The combination of actual stimuli and data from memory is an effective way of generating multi-sensory perceptual objects.
Figure 1. A multi-sensory perceptual object is a model of a physical object. It is generated from instantaneous sensory input and data from memory.
The quality of a perceptual object is only experienced by the perceiving subject itself. The various qualitative aspects inherent to a specific perceptional object are named qualia. The function of the perceptual system is aimed at identification of physical objects by generating multi-sensory perceptual objects (figure 2).
Figure 2. Contributions of various modalities shape a multi-sensory perceptual object.
Those perceptual objects cumulate the various sensory spaces, each including a specific set of features, like color, pitch, loudness, hardness and many others. If this object is known and represented in memory, its cross-sensory features can be recalled by stimulation of only one single modality. Learning to handle physical objects of daily life initially requires testing of all sensory properties, like vision, audition, smell, taste, surface structure, hardness and many others.
The learning process can easily be observed while a baby is handling a toy. It looks at the object, immediately grasps it, hits it towards another object to experience its sound, puts it into its mouth etc. After a while, it has learned that a specific sound indicates a hard or a soft surface, a heavy or lightweight object etc. It then is enabled to remind the perceptual object of a toy by hearing the sound, and this recalls all cross-sensory properties which have been experienced before.
From these observations it must be concluded that design of brand symbols as well as of product design must be based on analysis of possible references to multi-sensory perceptual objects, even if only stimuli in single sensory spaces are presented. Parallel processing is a basic feature of the brain (see e.g. Campenhausen, 1993). It can be found on all levels of neuronal activity, from the interaction of single neurons with its inhibitions and amplifications up to the binding activity of complete cortical domains.
Coupling Strategies
A multi-sensory perceptual object can be understood as a cluster of perceptual objects of single modalities. The data of different modalities can be coupled by various strategies (Haverkamp 2004, 2007a). Those strategies are methods which are applied by the human perceptual and cognitive system to couple different sensory spaces. On the other hand, these strategies can be understood as basis of design procedures, which closely refer to the cognitive and intuitive capabilities of the customer.
Figure 3 provides an overview. The main intuitive strategies are:
analogies = correlations of single features/attributes
iconic coupling (concrete association) = identification of sources of stimuli
symbolic connections = semantic correlations by analysis of meaning
Figure 3. Model of cross sensory strategies. Most important for design of products, brands and environments are strategies based on intuitive features.
Each main intuitive strategy splits into a variety of mechanisms, e.g. analogies can be made up in-between basic attributes, like brightness or roughness, or can consider movement, shape, emotion and many other aspects. Some of the main strategies of interaction as listed above are interpreted intuitively and do not need conscious analysis: iconic coupling, cross-sensory analogies and basic connections. Therefore these strategies are of main impact for multi-sensory design.
Other correlations can be constructed with reference to known physical properties or mathematical rules, e.g. the frequency of color light can be related to pitch frequency of a given pure tone by means of appropriate calculations. Common ways to provide constructions can easily be found by programming specific algorithms, as used by audio-visual media players and automated light-shows. If the perceiving subject, however, cannot find any correlations by use of the abovementioned intuitive strategies, it will not be able to find any match of data provided by different senses.
A minority of subjects additionally experiences sensations in sensory channels which are not instantaneously stimulated. In those cases tones can have a color, or taste can cause audible sound. Any kind of sensation can secondly appear in connection to a primary, stimulated sensory feature, but each subject has its very individual, fixed connection. Although rare and strictly individual, such phenomena of genuine synesthesia are assumed to be important to understand the multi-sensory capabilities of the brain. Various research projects are directed towards synesthetic perception (Harrison, 2001; Cytowic 2002; Day, 2005; Robertson, 2005).
In contrary to genuine synesthesia, basic multi-sensory connections are evident in each individual. They enable performance of generic tasks like body movement, which requires perception of muscle forces, tension and displacement, visual orientation in space, vestibular information, haptic feedback and others. Cross-sensory data are integrated without specific attention of consciousness. Merging of taste and smell is another common example of basic connections.
It is evident that the listed main strategies of interaction are acting in-parallel, showing mechanisms that initially are independent from each other. Therefore, during the first step of perception the data processing is separately done according to these strategies. The next step then integrates the results of all strategies to provide a map of correlations which is consistent, with contradictions reduced to a minimum. Shaping of perceptual objects from a variety of data requires integration of all perceived elements. This includes various mechanisms of grouping and segregation. First approaches which are also valid today have been provided by Gestalt-psychology (Werner ,1966). Recently, theories of scene analysis have added essential insight into integration processes (Bregman, 1999). The model presented in figure 3 is suitable for classification of all multi-sensory phenomena.
Each strategy contains various processes of cross-sensory interaction with references to single features of each sensory space. As mentioned above, main strategies which are intuitively processed within every subject are cross-sensory analogies, iconic and symbolic coupling. Those strategies depend on the context of perceived features. Therefore analysis of contextual relations is part of each strategy during cognitive processing, and must be considered by any design process.
The analysis of all single elements and attributes of an object or a complex sensory environment can be understood by means of a multi-modal network, as sketched in figure 4. This network can contain both iconic (concrete associative) and symbolic elements. The iconic elements are stored in memory as shapes/images of sensory perception, i.e. as visual image, sound, odor etc. The number and grade of cross-linking of elements is determined by the level of experience (number and intensity of perceptional events) and by the mental und emotional involvement of the perceiving subject.
Figure 4. Multi-modal network (extract) – Example: ”bell”
With reference to the aforementioned perceptual objects, they can be understood as connected clusters of attributes, for which a certain probability exists that they belong to a specific physical object (or term/item). If this cluster is intuitively integrated within a single modality, it defines a perceptual object of one sensory channel. Attributes of single modalities can be connected via analogies, i.e. correlation functions.
Complete perceptual objects of single modalities can be coupled via iconic or symbolic coupling. The described strategies are performed in-parallel. This principle of parallel processing is in accordance to recent findings of brain functionality. It also refers to daily experience that – e.g. during perception of music – it is possible to experience a concrete association at one time with a sensation of brightness and with conscious analysis of the musical structure.
If activated, each strategy gains an independent result that can be contradictory to other processes. In this case, perceptual conflicts (cognitive dissonances) can occur which can cause unwell feelings or negative rating of perceptions. On the other hand, perceptual conflicts can attract attention. An example is the colouring of colour terms in contrary to their meaning, like “green” is displayed with red letters.
While levels are acting independently, various levels can provide similar results and thus amplify their effects. This redundancy supports reliability of multi-sensory perceptual objects, e.g. multi-sensory brand symbols. Attention can consciously be focused on a specific strategy. During daily life, however, all available methods of cross-sensory coupling contribute to multi-sensory perception. Therefore, an isolated, focused experimental environment can cause artificial results with limited relevance to common perception.
Cross-Sensory Analogies
Cross-sensory analogies refer to the capability of the perceptional system to detect correlations of specific attributes and to analyze them for identification of physical objects and atmospheric features (figure 5, see also Haverkamp 2004, 2007b). The analysis of analogies can refer to:
– generic attributes (intensity, sharpness, brightness …)
– motion (straight, rotational, irregular, expanding …)
– body perception (tense, relaxed, floating …)
– emotion (calm, troubled, angry …)
Figure 5. Correlation of cross- sensory analogies, exemplified in case of perceived visual and auditory attributes.
A series of generic attributes is valid for correlation of each sensory channel. Werner explicitly named intensity, brightness, volume, density und roughness (Werner, 1966). Correlations of generic attributes have been comprehensively discussed by Stevens (Stevens 1961, 1966). For the aspects of motion as a visual and auditory attribute see (Shove, 1995) and (Eitan, 2004).
The capability to detect cross-sensory correlations as well as correlations of various attributes within one sensory space is an essential feature of the perceptional system. Without the ability of evaluating analogies, perception would not be possible because the build-up of perceptual objects would not be enabled. A jingling bell can only be recognized if the jingling noise (auditory space) is connected to the image of the bell (visual space).
If the necessary integration process fails, only a noise with unknown source is detected in-parallel to a moving visual object that cannot be determined as sounding or not sounding. Identification of physical objects and analysis of the complex environment which surrounds the individual is not possible without determining correlations. Analogies are capable to consolidate unknown or unexpected perceptual objects by correlating the perceived attributes.
A sound source localized in a specific angle and distance will be coupled to a physical object seen near its location. A connection will also be presumed in case of different location, but with accurate temporal correlation (synchronicity; Kohlrausch, 2005). Therefore spatial and temporal correlations must primarily be considered. In example, a heard speech can be allocated to a specific person via synchronic sound and motion of the mouth, additionally supported by correlated visual and auditory localization.
The correlation of the visual and auditory logo of Deutsche Telekom provides a good example of cross-sensory coupling via the generic attributes visual height and tone pitch (figure 6). The signet consists of 4 dots and one “T”. The sound logo consists of five single tones related to the five visual objects, with the one tone related to “T” increased in pitch. Like a system of musical notation, tone pitch is correlated to the height of these objects, with time scaled from left to right-hand-side. Additionally, prosody of the term “Deutsche Telekom” with its five syllables correlates to jingle and image.
Figure 6. Visual logo of Deutsche Telekom AG, as used until August 2007.
Cross-sensory analogies are a main topic of recent psycho-physical research on multi-sensory relations (Calvet, 2004). Beside coupling of auditory and visual stimuli, analogies are also suitable to include tactile, olfactory and gustatory features into any multi-sensory design concept.
Iconic coupling
This strategy to establish cross-modal connections is based on associations suitable to identify a known physical object (figure 7).
Figure 7. Perceptual objects of single modalities, grouped from the visual and auditory attributes of figure 5 as base of iconic coupling.
A single stimulus can refer to a multi-modal perceptual object stored in memory. Thus a specific sound stimulus can evoke imagination of the sound source with all of its cross-sensory attributes, if the variety of properties was experienced before. In example, the sound of the siren of an ambulance refers to the image of emergency light and of the whole vehicle. The sound of a bell refers to its visual image as well as its tactile feeling.
Iconic coupling is therefore processed via an identification of the source of stimuli. In case of audition, Chion uses the term causal listening, while Flückiger refers to it as semantic of first order (Chion, 1994; Flückiger, 2002). The term concrete association underlines the fact that this strategy is based on assignment of perceived features to known objects (Haverkamp, 2004).
Listening to music often evokes imagination of landscapes or interiors which fit to the associative content of the music or a comparable atmosphere. More directly, images of musical instruments can be recalled. Onomatopoetics in speech and music is also a common application of iconic features, while the imitation of natural sound generates an intuitive connection to multi-sensory objects or to the atmosphere of an environment. While iconic coupling refers to objects in memory, it is based on learning and experience of the subject. It therefore depends on living environment and cultural background of an individual. Design of multi-sensory objects must always refer to the perceptual context and experience of the recipient (customer/user).
In case of the visual logo of the manufacturer of milk produces Milch-Union Hocheifel eG, the German onomatopoetic expression of the cows moo (“Muh”) is used instead of a sound logo (figure 8). Therefore, the sound logo is included by visual means, beside the image of a cow. Therefore the high degree of sympathy of the cow as the initial producer of the milk is utilized for brand identity.
Figure 8. Visual logo of Milch- Union Hocheifel eG, including the visual image and the onomatopoetic represented sound of a cow.
Iconic coupling is the main strategy of correlating gustatory and olfactory stimuli to multi-sensory perceptual objects. Therefore most terms used for description of smell and taste refer to physical objects which are identified as the source (e.g. “citrus flavour”). In rare cases, however, cross-sensory analogies are used (e.g. “sharp odour”).
Semantic relations / symbolic coupling
Stimuli of a specific modality can also refer to semantic codes (figure 9), which e.g. are given in the visual modality by established signals and logos, sometimes based on aspects of ancient heraldry. Those symbols can only be understood if sender and recipient of a message are based on the same context. Therefore the functionality of signs and symbols is limited to specific aspects of culture and era. Semiotics provides the basic understanding of decoding meaning (Jekosch, 2005). Flückiger describes symbolic coupling as semantics of higher order (Flückiger, 2002).
Figure 9. Additional relations of sensory objects are established using a semantic analysis, enabling multi-sensory grouping by correlations of meaning as base of symbolic coupling.
Synaesthetic terms as a special kind of metaphor are an important element of literature, especially of lyrics. Synaesthetic metaphors are also applied for description of sensations during psycho-physical experiments. In example, a sound can be described as warm, bright or heavy. Those terms are primarily conscious constructions, but need the recipient’s capability of imagination via cross-sensory analogy or association. Semantic relations refer to a known content of meaning, while the source of stimuli may be unidentified.
Speech that is understood includes semantic information, but usually iconic features are added, which are suitable to identify the source, e.g. a person. Speech of a language unknown to a subject, however, only refers to iconic coupling and cross-modal correlations. In this case, a speaker can still be identified and onomatopoetic and emotional information can be interpreted via associations, but meaning cannot be decoded.
A popular example of symbolic coupling of auditory and visual symbols is given by sound logo and visual symbol of Eurovision, a joined venture of the European Broadcasting Union, which started 1954. It initially included a visual image which was combined from the star banner of the European Union with floodlights, providing the impression of a brilliant festival event (figure 10). The sound logo was taken from the Ouverture of Marc-Antoine Charpentier Te Deum in D-major, which has been composed around 1690.
This colourful sacred music includes musical symbols to honour the King, whereby God and the Sun King Ludwig XIV are put on the same level. It also shows martial aspects, while it was possibly performed to celebrate the French victory in the battle of Steinkerque (1692). The Eurovision sound logo keeps the original mood of festivity, but removes references to military (no drum roll at the beginning) and to the King (reduced dotted rhythms).
Figure 10. First visual logo of the Eurovision, established 1954. The visual logo has been updated, while the sound logo as popular ”Eurovision-Hymn” is still in use.
The coupling strategies genuine Synesthesia and mathe-matical/physical correlation are of minor importance for multi-sensory brand design, but shall be included to complete this comprehensive overview.
Genuine Synesthesia
In contrary to analogies as well as iconic and semantic coupling, genuine Synesthesia is not a common cross-sensory process. It is very individual. Only relatively few people report this way of perception. It is induced by a primary, common perception in one sensory channel. This induces perception in a secondary channel without need of a specific stimulus. The correlation is fixed (absolute) and is neither influenced by the perceptual context nor by learning or experience.
Every subject has its own schema of correlation, which remains constant over the whole life. This definition of genuine synesthesia has at first been described by Cytowic (Cytowic, 2002). Many reports and earlier investigations throughout the last century show that genuine synesthesia is widely independent from the cultural, scientific and historical context (see e.g. Bleuler, 1881; Anschütz, 1927; Mahling, 1927; Baron Cohen, 1997; Emrich, 2001; Robertson, 2005).
The most common phenomena of genuine synesthesia are visual phenomena induced by auditory events (colour hearing). Additionally, correlations of visual features, e.g. colour, to visual shapes are often reported, especially graphems can appear in perception with synesthetic colours added. Those phenomena are often, but not always related to the semantic content of stimuli. In this manner, series of numbers or terms can show specific shapes (number forms). In principle, all types of stimuli can induce all types of secondary features. In fact, most secondary perceptions show visual features (Day, 2005).
Although the scientific interest to genuine synesthesia is actually high, it is difficult to draw conclusions which are relevant for common multi-sensory perception. Some preliminary results:
- All parts of the brain can couple sensory information back to specific perceptual areas, especially to visual fields. It does not seem to matter whether the areas are adjacent or not. A variety of feedback loops transfers results of high cognitive processing back to primary fields of perception.
- Basic shapes play an important role in processing of sensory data. Lists of elementary visual shapes have been collected by various authors (e.g. Horowitz, 1970). For design purposes it seems reasonable to search for elementary shapes within all sensory spaces which serve as “atoms” from which perceptual objects are composed.
- Ideal, generic combinations of sensory features may exist, e.g. of visual shape and colour, timbre and tactile surface features etc.
- Semantic relations can be transformed to visual shapes.
In the near future, research on synesthesia will definitely improve knowledge of cross-sensory connections and thus offer further approaches on multi-sensory design.
Conscious Construction (Mathematical and physical Correlation)
Beside the intuitive ways of cross-modal coupling, concepts can be made up by conscious construction to correlate attributes of different sensory spaces. Many of those concepts have been discussed during the past, using mathematical expressions and/or references to physical facts, like relations of frequencies of coloured light to pitch of audible tones. It was often assumed that a physical relation as an objective fact will have specific relevance for human perception. In contrary, however, there are deviations: The intuitive way to choose a colour scale is to sort colours by brightness, not by frequency. Therefore, all mathematical or physical relations used for design of multi-sensory objects have to be proven for intuitive validity.
Today, computational tools offer a large variety of algorithms to correlate stimuli of various modalities with each other. Media-players use simple expressions to generate visualisations of music, using correlations of visual features to auditory attributes. The spectator/listener, however, will only feel an alignment of vision and sound if the result includes at least one of the intuitive correlations described above. If cross-modal correlations are used which cannot be perceived, the mathematical/physical coupling will remain abstract and will not show any alignment to subjective observations.
The physical analysis of auditory and visual stimuli shows some commonness. In both cases energy is transported to the sense organs via physical waves, which are of mechanical (sound) or electro-magnetic (light) nature. A main difference is given by frequency (sound: 16-16000 Hz; visible light: 390-790 THz). Several trials have been made to directly relate frequencies (or wavelengths) of light to those of sound in order to construct light events similar to audible music. A musical system based on various octaves, however, cannot be established with visible light. The relative bandwidth of audible sound is more than 12 octaves, whereas visible light covers less than one octave (!).
Another big difference is given by different resolution of auditory and visual perception: auditory resolution of frequency and time is high compared to vision, whereas spatial resolution of the eye is much higher than that of the ear. A complex spectrum of sound can simultaneously lead to various perceptions like several tones with pitch and timbre independent from each other, while every light spectrum only induces sensation of one single colour. Therefore, a polyphonic structure cannot simply be transferred from sound to vision by use of spectral properties, but must include transformation from the spectral domain to spatial properties.
Jewanski gives a detailed analysis of theories and trials up to the beginning of the 19th century (Jewanski, 1999). Many further trials were made: Among others, Rimington wrote the first comprehensive book regarding correlations of music and colours, followed by László and Klein (Rimington, 1911; László, 1925; Klein, 1926). All three authors have build up colour organs that successfully were presented to public.
Technical progress during the 20th century improved availability of lighting systems and computational tools (Sidler, 2006). The interest in theoretical approaches in colour music, however, decreased after a euphoric phase within the first half of the century, and changed to more creative and free use of capabilities provided by light show, movie and video-clip.
Combinations
The perceptual system collects the information of all modalities and performs sophisticated integration processes to merge the naturally incomplete data into a multi-sensory image of the world. The individual’s world includes exterior as well as interior body perception. The aim of the integration processes is to generate subjective representations of objects which are free of contradictions and gaps. An optimized design must take into account various processes of multi-sensory integration.
The perceptual system is able to analyze analogies, iconic and semantic strategies of cross-modal coupling in-parallel. Therefore, with view on a multi-sensory design process, it appears to be reasonable to focus on a structure with 3 layers for multi-sensory alignment of object features, as shown in figure 11. In the first step, it implies that characteristic attributes of perceptual objects are clearly perceived and segregated from each other. On the level of analogy, correlations of single properties are evaluated. The features with the best capability for a given task must be selected and optimized. For example, correlations of visual shape of a logo can be aligned to dynamic features of a melody.
Figure 11. Three-layer model of in-parallel processing of cross-modal analogies, iconic coupling and semantic correlations. Examples for audio-visual features of a brewery ́s brand image are added
As the next step, the analysis of iconic content identifies and connects attributes which are recognized as elements of known perceptual objects (causal listening). Even a rudimentary stimulus within one modality is capable to initiate the reconstruction of a complex multi-sensory perceptual object by adding all features stored in memory. The main problem of a design process is that a variety of features has to be considered that is only available in memory of the perceiving individual.
The set of subjective qualities (qualia) related to a stimulus also depends on past experience and external context. Within a known cultural and environmental context, however, experience and learned symbols can be evaluated. Especially global brand design is a challenge due to a large variety of cultural expectation and perceptual experience of the customer in various regions of the world. On the semantic level, perceptual objects are coupled according to additional information which is learned separately, like e.g. language, traffic signs, philosophical, political or religious interpretation and thinking.
The meaning of symbols can be very different within various cultures – therefore a global design approach must be based on comprehensive, cross-cultural studies. A basic hierarchy of the three strategies shown in figure 11 can be stated. Within the recipients perceptual system, unknown objects can be loaded with information about the physical source and meaning by correlating their cross-sensory features. Well known objects are already defined by iconic coupling during presentation of single features. Meaning can be added by observation of known objects and by learning of additional, abstract information.
The parallel processing of various strategies gains independent results which can be contradictory or similar. Friendly sentences e.g. can be vocalized with aggressive facial expression, or aggressive sentences can be supported by slight body movement and gentle voice. Different strategies are used by the perceptional system to eliminate those contradictions. If visual and auditory signals contain different data, the result can be dominated by one sensory channel, while the other is inhibited. Ventriloquists make use of the dominance of the visual modality: the mimic action of their doll indicates that the source for its speech is at its mouth, while information about the location of the true sound source is suppressed.
Another possibility is that contradictory sensory information is merged to shape a different result. This has been exemplified by means of the McGurk-effect, where the mimic action of a speaker modifies the perceived sound of syllables (McGurk, 1976). If contradictions of data provided by different modalities cannot be matched, this can cause negative feelings like indisposition or cognitive dissonance. The fact that this attracts specific attention of the subject is nowadays well known and used by print and screen advertising.
Cognitive dissonance can also be used for brand design, but shall be carefully proven to avoid negative feelings induced to the customer. For example, a visual logo can use aggressive colours, but the sound logo supports a peaceful and calm feelings. The general approach, however, should prefer an appropriate alignment of several strategies, e. g. a similar direction of cross-sensory correlations, associations (iconic content) and meaning.
Figure 12. Synesthetic design compared to conventional pro- cess of multi-sensory design.
Conclusion: Ways to Synesthetic Brand Design
A conventional process of brand design is based on single modalities (figure 12, left hand side): auditory elements are designed separately to other sensory spaces. Only during a late phase, some features may be correlated to each other. In contrary, the synesthetic design is targeted to cross-sensory alignment in an early phase of the development process (figure 12, right hand side; Haverkamp, 2008). At first, the intuitive coupling strategies analogies, iconic coupling and semantic relations are considered.
Then, appropriate attributes (like pitch, auditory dynamics, colour, symbol shape, associative elements, symbols) are chosen to fulfill the alignment of these strategies. In the last step, the chosen attributes have to be adjusted to provide an optimized integration of the selected features. Final integration must be based on the relevance of each coupling strategy and each attribute. It shall include processes for minimization of contradictions, to avoid negative emotions induced by cognitive dissonances. First steps were taken to find cross-sensory design of brand elements – but much more is possible if all opportunities are addressed by a systematic approach of synesthetic design.
References
Anschütz, Georg (ed.). (1927). Farbe-Ton-Forschungen. Bd.1. Leipzig: Akademische Verlagsgesellschaft.
Bleuler, Eugen and Lehmann, Karl. (1881). Zwangsmässige Lichtempfindungen durch Schall und verwandte Erscheinungen auf dem Gebiete der anderen Sinnesempfindungen. Leipzig: Reisland.
Baron-Cohen, Simon and John E. Harrison (eds.). (1997). Synaesthesia. Classic and Contemporary Readings. Oxford und Cambridge: Blackwell Publishers.
Bregman, Albert S. (1999). Auditory scene analysis. The perceptual organization of sound (2nd edition). Cambridge, Massachusetts: The MIT press.
Calvert, Gemma A, Spence, Charles & Stein, Barry E. (eds.). (2004). The handbook of multisensory processing. Cambridge (MA): The MIT press.
Campenhausen, Christoph von. (1993). Die Sinne des Menschen. (2nd edition). Stuttgart: Thieme.
Chion, Michel. (1994). Audio-vision. Sound on screen. New York: Columbia University Press.
Cytowic, Richard E. (2002). Synesthesia, a union of the senses (2nd edition). New York: Springer Verlag.
Day, Sean A. (2005). Some demographic and socio-cultural aspects of synesthesia. In: Lynn C. Robertson & Noam Sagiv (ed.), Synesthesia. Perspectives from cognitive neuroscience. Oxford: Oxford University Press.
Eitan, Zohar and Granot, Roni E. (2004). Musical parameters and images of motion. In: R. Parncutt et al. (eds.), Proceedings of the conference on interdisciplinary musicology. Graz.
Emrich, Hinderk M; Schneider, Udo and Zedler, Markus. (2001). Welche Farbe hat der Montag? Synästhesie: das Leben mit verknüpften Sinnen. Leipzig: Hirzel.
Filk, Christian; Lommel, Michael and Sandbothe, Mike (eds.). (2004). Media Synaesthetics. Konturen einer physiologischen Medienästhetik. Köln: Herbert von Halem Verlag.
Flückiger, Barbara. (2002). Sound Design. Die virtuelle Klangwelt des Films (2nd edition). Marburg: Schüren.
Harrison, John. (2001). Synaesthesia. The strangest thing. Oxford: University Press.
Haverkamp, Michael. (2004). Audio-visual Coupling and Perception of Sound-Scapes. Oldenburg: DEGA, pp. 365-366.
Haverkamp, Michael. (2007a). Percezione sinestesica e design del rumore. Progetto Graphico, Anno 5, Numero 10. Roma: Aiap, pp. 162-169.
Haverkamp, Michael. (2007b). Essentials for description of cross-sensory interaction during perception of a complex environment. Proc. of Inter-Noise 2007. Istanbul: Turkish Acoustical Society.
Haverkamp, Michael. (2008). Synästhetisches Design – kreative Produktgestaltung für alle Sinne. München: Carl-Hanser Verlag.
Horowitz, Mardi Jon. (1970). Image Formation and Cognition. London: Butterworths.
Jekosch, Ute. (2005). Assigning meaning to sound – Semiotics in the context of product-sound design. In: Jens Blauert (ed.), Communication acoustics, pp. 193-221. Berlin: Springer.
Jewanski, Jörg. (1999). Ist C = Rot? Eine Kultur- und Wissenschaftsgeschichte zum Problem der wechselseitigen Beziehung zwischen Ton und Farbe. Sinzig: Studio, Verlag Schewe.
Klein, Adrian Bernard. (1926). Colour-Music. London: The Technical Press Ltd.
Kohlrausch, Armin and van de Par, Steven. (2005). Audio-visual interaction in the context of multi-media applications. In: Jens Blauert (ed.), Communication acoustics. Berlin: Springer, pp. 109-138.
László, Alexander. (1925). Die Farblichtmusik. Leipzig: Breitkopf & Härtel.
Mahling, Friedrich. (1927). Das Problem der „Audition colorée“. In: Georg Anschütz, (ed.), Farbe-Ton-Forschungen. Bd.1. Leipzig: Akademische Verlagsgesellschaft.
McGurk, H. and MacDonald, J. (1976). Hearing lips and seeing voices. Nature 264, pp. 746-748.
Rimington, Alexander Wallace. (1911). Colour-Music. The art of mobile colour. London: Hutchinson.
Robertson, Lynn C. and Sagiv, Noam (ed.). (2005). Synesthesia. perspectives from cognitive Neuroscience. Oxford: Oxford University Press.
Shove, Patrick and Repp, Bruno H. (1995). Musical motion and performance. In: J. Rink (ed.), The practice of performance. Cambridge: University Press.
Sidler, Natalia & Jewanski, Jörg (eds.). (2006). Farbe-Licht-Musik. Synästhesie und Farblichtmusik. Bern: Peter Lang.
Stevens, Staley Smith. (1961). The psychophysics of sensory function. In: W. A. Rosenbith (ed.), Sensory communication. Cambridge: M.I.T. Press.
Stevens, Stanley Smith. (1966). Matching functions between loudness and ten other continua. Perception & Psychophysics l (l), 5-8.
Werner, Heinz. (1966). Intermodale Qualitäten. In: Handbuch der Psychologie, Bd. 1 , pp. 278-303. Göttingen: Hogrefe..
