The Effect of Induced Emotional States on The Magnitude of Cross-Modal Correspondence Effect

Occupation: Associate Professor of the Department of General Psychology
Affiliation: St Petersburg University
Address: Sankt-Petersburg, Saint Petersburg, Admiral Makarov Embankment, 6, Russia

E. A. Andriushchenko

Occupation: Research engineer, Laboratory of behavioral neurodynamics
Affiliation: St Petersburg University
Address: Russian Federation, Saint-Petersburg

K. Miroshnik

Occupation: Master’s student
Affiliation: Saint Petersburg State University, Faculty of Psychology
Address: Saint Petersburg, Admiral Makarov Embankment, 6, Russia

E. Blinova

Occupation: Research Engineer, Laboratory of Behavioral Neurodynamics
Affiliation: Saint Petersburg State University
Address: Universitetskaya nab., 7-9

Yu. Yu. Shtyrov

Occupation: chief researcher, Laboratory of behavioral neurodynamics
Affiliation:
St Petersburg University
Aarhus University
Address: Russian Federation, Saint-Petersburg

Journal name

Psikhologicheskii zhurnal

Edition

Volume 44 Issue 1

Pages

30-42

Abstract

Cross-modal correspondence effect (i.e., facilitated processing of congruent stimuli from different modalities) occurs not only when simple multi-modal sensory stimuli are processed together, but also during their simultaneous processing with words with emotional and spatial connotations. We tested a hypothesis that the magnitude of cross-modal correspondence effect, arising from concurrent processing of basic sensory and verbal stimuli, is differentially modulated by individual’s emotional state. Thirty-six volunteers (26 females, 18–34 years old) watched videos that evoked positive, negative, or neutral emotional states. This was followed by the main task in which they were presented with sounds of different pitch (low: 1000 Hz; high: 2000 Hz) simultaneously with words that differed in their emotional valence and were associated with different parts of space (low/high). The participant’s task was to identify the pitch (low/high) of the non-verbal sound stimuli. Two-way mixed ANOVA and subsequent pairwise comparisons (Student's t-test for dependent samples) were used to compare both mean reaction times and estimated parameters of the ex-Gaussian distribution. The results showed that the audiovisual correspondence effect became manifested in faster responses to congruent stimulus combinations compared with non-congruent ones (t(35) = -3.20, p = .005, dz = -0.53, 95% CI [-0.89, -0.18]). However, we did not find a large size effect of the induced emotional state on the magnitude of this correspondence effect (F(4, 68) = 0.49, p = 0.744, = .001). This result may be explained either by robustness of cross-modal correspondence effect and its resilience to emotional influence or by specific limitations of present study design.

Keywords

Cross-modal correspondence, multisensory integration, emotional states, sensory stimuli, verbal stimuli, emotional semantics, embodied cognition

Acknowledgment

The study was supported by a grant from the Russian Science Foundation “The role of emotional regulation in the multisensory integration of verbal and non-verbal information: psychological and psychophysiological aspects”, project No. 22-28-01020

Publication date

26.02.2023

Number of characters

29639

Cite

GOST	Shcherbakova O., Andriushchenko E., Miroshnik K., Blinova E., Shtyrov Y. The Effect of Induced Emotional States on The Magnitude of Cross-Modal Correspondence Effect // Psikhologicheskii zhurnal – 2023. – V. 44. – Issue 1 C. 30-42 [Electronic resource]. URL: https://psy.jes.su/S020595920023642-2-1 (circulation date: 20.04.2024). DOI: 10.31857/S020595920023642-2
MLA	Shcherbakova, O., Andriushchenko, E. A., Miroshnik, K., Blinova, E., Shtyrov, Yu. Yu. "The Effect of Induced Emotional States on The Magnitude of Cross-Modal Correspondence Effect." Psikhologicheskii zhurnal 44.1 (2023):30-42. DOI: 10.31857/S020595920023642-2
APA	Shcherbakova O., Andriushchenko E., Miroshnik K., Blinova E., Shtyrov Y. (2023). The Effect of Induced Emotional States on The Magnitude of Cross-Modal Correspondence Effect. Psikhologicheskii zhurnal 44(1), pp.30-42 DOI: 10.31857/S020595920023642-2

100 rub.

When subscribing to an article or issue, the user can download PDF, evaluate the publication or contact the author. Need to register.

Number of purchasers: 1, views: 196

Readers community rating: votes 0

1. Anan'ev B.G. Teorija oshhushhenij. Leningrad: Izd-vo Leningr. un-ta, 1961. (In Russian)

2. Andrjushhenko E.A., et al. Na povyshennyh tonah: rol' prostranstvennogo poznanija v kross-modal'nom vzaimodejstvii jemocional'noj semantiki i audial'nogo vosprijatija. Psihologija. Zhurnal Vysshej shkoly jekonomiki, 2022 (v pechati). (In Russian)

3. Kravkov S.V. Vzaimodejstvie organov chuvstv. Moscow: Izd-vo Akademii nauk SSSR, 1948. (In Russian)

4. Pankratova A.A., Ljusin D.V. Videoroliki dlja indukcii jemocij v laboratornyh uslovijah: normativnye dannye i kross-kul'turnyj analiz. Jeksperimental'naja psihologija. 2018. V. 11. № 2. P. 5–15. (In Russian)

5. Armitage J., Eerola T. Cross-modal Transfer of Valence or Arousal from Music to Word Targets in Affective Priming?. Auditory Perception & Cognition. 2022. V. 5. № 3-4. P. 192–210.

6. Berger A., Kiefer M. Comparison of different response time outlier exclusion methods: a simulation study. Frontiers in Psychology. 2021. V. 12. Article 2194.

7. Blanca M. J., et al. Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit?. Behavior Research Methods. 2018. V. 50. № 3. P. 937–962.

8. Brunetti R., et al. Are crossmodal correspondences relative or absolute? Sequential effects on speeded classification. Attention, Perception, & Psychophysics. 2018. V. 80. № 2. P. 527–534.

9. Cohen J. Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum. 1988.

10. Dolscheid S., et al. Space-pitch associations differ in their susceptibility to language. Cognition. 2020. V. 196. Article 104073.

11. Ernst M.O. Learning to integrate arbitrary signals from vision and touch. Journal of Vision. 2007. V. 7. № 5:7. P. 1–14.

12. Estrada C., Young M., Isen A.M. Positive affect influences creative problem solving and reported source of practice satisfaction in physicians. Motivation and Emotion. 1994. V. 18. P. 285–299.

13. Evans K.K., Treisman A. Natural cross-modal mappings between visual and auditory features. Journal of Vision. 2010. V. 10. № 1: 6. P. 1–12.

14. Fernandez-Prieto I., Spence C., Pons F., Navarra J. Does language influence the vertical representation of auditory pitch and loudness?. i-Perception. 2017. V. 8. № 3. P. 1–11.

15. Fredrickson B.L., Branigan C. Positive emotions broaden the scope of attention and thought‐action repertoires. Cognition & Emotion. 2005. V. 19. № 3. P. 313–332.

16. Gallace A., Spence C. Multisensory synesthetic interactions in the speeded classification of visual size. Perception & Psychophysics. 2006. V. 68. № 7. P. 1191–1203.

17. Gu S.L.H., Gau S.S.F., Tzang S.W., Hsu W.Y. The ex-Gaussian distribution of reaction times in adolescents with attention-deficit/hyperactivity disorder. Research in Developmental Disabilities. 2013. V. 34. № 11. P. 3709–3719.

18. Haverkamp N., Beauducel A. Violation of the sphericity assumption and its effect on type-I error rates in repeated measures ANOVA and multi-level linear models (MLM). Frontiers in Psychology. 2017. V. 8. Article. 1841.

19. Hein G., et al. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. The Journal of Neuroscience. 2007. V. 27. P. 7881–7887.

20. Isen A.M., Daubman K.A., Nowicki G.P. Positive affect facilitates creative problem solving. Journal of Personality and Social Psychology. 1987. V. 52. № 6. P. 1122–1131.

21. Janyan A., et al. Look and ye shall hear: Selective auditory attention modulates the audiovisual correspondence effect. i-Perception. 2022. V. 13. № 3. P. 1–10.

22. Jonas C., Spiller M. J., Hibbard P. Summation of visual attributes in auditory–visual crossmodal correspondences. Psychonomic Bulletin & Review. 2017. V. 24. № 4. P. 1104–1112.

23. Luce R.D. Response times: Their role in inferring elementary mental organization. New York, New York: Oxford University Press. 1991.

24. Makovac E., Gerbino W. Sound–shape congruency affects the multisensory response enhancement. Visual Cognition. 2010. V. 18. P. 133–137.

25. Molholm S., Ritter W., Javitt D.C., Foxe J.J. Multisensory visual–auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex. 2004. V. 14. P. 452–465.

26. Nasby W., Yando R. Selective encoding and retrieval of affectively valent information: Two cognitive consequences of children's mood states. Journal of Personality and Social Psychology. 1982. V. 43. P. 1244–1253.

27. Olejnik S., Algina J. Generalized eta and omega squared statistics: measures of effect size for some common research designs. Psychological Methods. 2003. V. 8. № 4. P. 434–447.

28. Parise C., Spence C. Synesthetic congruency modulates the temporal ventriloquism effect. Neuroscience Letters. 2008. V. 442. P. 257–261.

29. Parise C.V., Spence C. ‘When birds of a feather flock together’: synesthetic correspondences modulate audiovisual integration in non-synesthetes. PloS One. 2009. V. 4. № 5. Article e5664.

30. Patching G.R., Quinlan P.T. Garner and congruence effects in the speeded classification of bimodal signals. Journal of Experimental Psychology: Human Perception and Performance. 2002. V. 28. P. 755–775.

31. Puigcerver L., Rodríguez-Cuadrado S., Gómez-Tapia V., Navarra J. Vertical mapping of auditory loudness: loud is high, but quiet is not always low. Psicológica Journal. 2019. V. 40. № 2. P. 85–104.

32. Scherer L.D., Larsen R.J. Cross-modal evaluative priming: Emotional sounds influence the processing of emotion words. Emotion. 2011. V. 11. № 1. P. 203–208.

33. Spence C. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics. 2011. V. 73. № 4. P. 971–995.

34. Steinhauser M., Hübner R. Distinguishing response conflict and task conflict in the Stroop task: evidence from ex-Gaussian distribution analysis. Journal of Experimental Psychology: Human Perception and Performance. 2009. V. 35. № 5. P. 1398–1412.

35. Suchkova E., Lyusin D. The Influence of Natural and Induced Emotional States on the Recognition of Emotional Facial Expressions (November 6, 2020). Higher School of Economics Research Paper No. WP BRP 121/PSY/2020. SSRN: https:/ssrn.com/abstract=3732065 (дата обращения: 03.07.2022).

36. van Atteveldt N., Formisano E., Goebel R., Blomert L. Integration of letters and speech sounds in the human brain. Neuron. 2004. V. 43. P. 271–282.

37. Vatakis A., Spence C. Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli. Perception & Psychophysics. 2007. V. 69. P. 744–756.