Crossmodal correspondences between linguistic tones and complex 3-D shapes and their low-level sensory properties (Pitch & spatial frequency)

• Main Author: Shang, Nan
• Other Authors: Styles, Suzy
• Format: Theses and Dissertations
• Language: English
• Published: 2017
• Subjects: DRNTU::Social sciences::Psychology::Experimental psychology
• Online Access: http://hdl.handle.net/10356/72028
• Institution: Nanyang Technological University

This thesis investigates tone-related audiovisual cross-modal correspondences using lexical tones (i.e., Mandarin tones) for the first time. We used both 2AFC (two-alternative forced choice) tasks and IAT (Implicit Association Test) tasks involving three groups of participants (Chinese dominant bilinguals (C), Chinese-English balanced bilinguals (C/E), and English-speaking participants with no Chinese (E) (total N = 618 participants). Chapter 1 outlines a summary of the literature on phonetic symbolism, ideophones, sound symbolism and synesthesia, neural evidence for sound symbolism and theories about the mechanisms of sound symbolism. Chapter 1 also reviews non-linguistic effects of pitch on cross-modal processing and how language experience tunes speech and tone perception. Throughout this literature, it is clear that lexical tones have rarely been examined in cross-modal correspondence studies. In Chapter 2, using online 2AFC surveys, we replicated the ‘u-curvy, i-pointy’ vowel-angularity cross-modal correspondence and found that English-speaking participants (E) matched visual angularity with Mandarin tones in terms of pitch height (i.e., high-pointy, low-curvy). The Chinese dominant bilinguals (C) matched angularity with Mandarin tones in terms of pitch change (more change-pointy, less change-curvy). While Chinese-English balanced bilinguals (C/E) swung between the two strategies depending on task types. To avoid possible effects of conscious strategy, in Chapter 3, we used IAT tasks to investigate crossmodal processing for lexical tone stimuli (i.e., /i1/, /i4/, /u1/, /u4/, /i3/, /u3/). We found significant group difference in this tone-shape crossmodal correspondence. The two Chinese groups consistently replicated the Pitch-Change Hypothesis (i.e., more change-pointy, less change-curvy) found in our 2AFC studies for the both tone contrasts (i.e., T1-T4, T1-T3) without significant group difference. On the other hand, the non-Chinese English-speaking group showed a significantly different tone-shape mapping pattern from that of the Chinese groups. The English speakers showed a ‘high-pointy, low-curvy’ mapping pattern for both of the tone contrasts. This ‘high-pointy, low-curvy’ crossmodal correspondence pattern was consistent with their preference observed in Study 1, where low tones go with curvy shapes and high tones go with pointy shapes. These results were also consistent with the fact that pitch height is important in English speakers’ lexical tone perception (Gandour, 1983). To see whether the spectral context of human voice makes difference in cross-modal correspondence between tones and shapes, in Chapter 4, we used sine waves with measured pitch (or both measured pitch and measured intensity) from the auditory stimuli used in the last chapter and IAT tasks. We found that all participant groups (i.e., C; C/E; E) matched visual angularity with tones in terms of pitch height without significant group difference. That is, pointy shapes were matched with high-pitched tones and curvy shapes with low-pitched tones. In addition, the comparisons between Study 2 and Study 3 demonstrated that the difference in language-dependent effects between speech tones and non-speech tones was significant in the present sound symbolism tasks. That is, language specificity disappeared in cross-modal correspondence between visual angularity and non-lexical tones. To see whether the spectral context of human voice makes a difference in cross-modal correspondence between spatial frequency and tones, in Chapter 5, we used both lexical and non-lexical tones to examine the relationships between tones and spatial frequency using the IAT paradigm. The Gabor patches used here were created using the modal spatial frequency information of the visual stimuli used in our previous studies. We found, for the first time, a pattern of ‘u-low, i-high’ vowel-spatial frequency correspondence. When lexical tones were used, and tones were attended, both Chinese groups matched tones with spatial frequency in terms of pitch change (i.e., more pitch change with high spatial frequency, less pitch change with low spatial frequency) without significant group different. However, the non-Chinese English-speaking group showed a significantly different mapping pattern from that of the two Chinese groups by pairing high spatial frequency with high pitch and low spatial frequency with low pitch. By contrast, when non-lexical tones were used, as seen in Chapter 4, all participants showed the same pitch-height-oriented cross-modal correspondence between tones and spatial frequency (i.e., high-high, low-low) without significant group difference, which was consistent with the previously observed cross-modal correspondence between pure tones and spatial frequency in English speakers (Evans and Treisman, 2010). Again, the difference in language-dependent effects between speech tones and non-speech tones was significant. Similar to what was observed in our angularity studies, language specificity again disappeared in tone-related audiovisual cross-modal correspondence when non-lexical tones were used. In Chapter 6, we summarize that Chinese-speaking participants show ‘curvy-steady, pointy-dynamic’ mappings between shape and pitch only when linguistic stimuli are used, while the rest of time they show the ‘curvy-low, pointy-high’ pattern as non-Chinese participants do all the time. For mappings between spatial frequency and pitch, Chinese-speaking participants show ‘low-steady, high-dynamic’ pattern only when linguistic stimuli are used, while the rest of time they show the ‘low-low, high-high’ pattern as non-Chinese participants do all the time. For the first time, we examined cross-modal correspondences between shape and linguistic use of pitch. Theoretical and practical contributions of the present study are also highlighted in Chapter 6, along with some limitations of the present study and future research directions.