Deconstructing unpleasantness in videogame sound effects
Part 1: Theory
After I wrote a post about disruptive audio in games, a few people asked why didn’t I mention any known acoustic features of annoying or unpleasant sound effects. Back then, I did it on purpose. I think it is a vast topic that deserves a separate, in-depth exploration.
There are some universally hated sounds like the scratching of fingernails over a chalkboard. Nobody in the entire world perceives them as pleasant, disregarding the context. And I think it is fascinating! It implies the existence of certain acoustic features that trigger negative emotional reactions in every human being. And if such features exist, we can analyze, measure, and control them. Before we continue, I need to warn you. This two-part post will be much nerdier than anything I have written before or plan to write in the foreseeable future. It comes with a list of references, so you can follow either links or footnote numbers to see where the information came from.
Disclaimer: I am not an academic researcher, and I should not be considered a credible source of information about the majority of the topics I’m touching here. I’m linking all my sources for the sake of transparency, but remember: this blog is written by a curious idiot with no supervision. Doubt everything, and please reach out to me if you spot any mistakes. I’ll be happy to correct myself.
Similar neural mechanisms can trigger different emotions. There is evidence that the amygdala, the part of the brain that is frequently associated with threat detection and sensation of fear, takes part in processing acoustic information¹ ². Sometimes the line between negative emotional responses becomes too thin to separate them. So instead of speaking about sounds that make us scared, disgusted, irritated, or uneasy, I decided to stick with a more general concept of unpleasantness.
Of course, we can not reduce the entire spectrum of sound-induced negative emotions to a single universal pattern. Our reactions are very complex. Some sounds feel unpleasant because of external meanings that we associate with them. Vomiting noise is one of the most disgusting sounds you can hear³, but it doesn’t seem to have any acoustic properties to make it intrinsically aversive⁴. Other sounds only become unpleasant in specific contexts. You may love the sound of your car’s engine, but not if it wakes you up in the middle of the night. The unpleasantness of a sound may depend on other modalities. For instance, the sound can become more or less unpleasant when it plays on top of a video⁵. There is a variety of contextual factors: misplacement, unexpectedness, lack of synchrony with other sensory information, and so on.
That’s why I’m not attempting to build a universal framework to explain every possible case — to me, it sounds too ambitious and impractical. Instead, I’d love to discover a set of reliable, context-agnostic techniques to analyze and directly manipulate the perceived unpleasantness of sound effects.
Such techniques could be useful not only to instigate negative emotions in the player but also to monitor and reduce irritation from sounds that are supposed to be pleasant or unnoticeable. Ultimately, if proven to be reliable, they could become a foundation for a real-time processing tool or a procedural technology — something like a biofeedback-driven “emotioneering” system mentioned by Garner et al. back in 2010⁶.
To start, I explored some scientific and nearly scientific literature on the topic and tried to list the acoustic properties that make sounds unpleasant to our brains.
Loudness
We all know that sound can be uncomfortably and dangerously loud. Different authors frequently mention loudness as one of the factors of auditory unpleasantness⁷ ⁸. From an evolutionary perspective, a loud sound means that something is either fairly big or too close to you. Such objects naturally demand more attention to ensure survival. Also, loud noises mask other sounds, which makes it harder to identify a different threat, so they make us more vulnerable.
Even though loudness is a factor, it might not be the most important one. At least in the videogame context, where we deal with a limited dynamic range. One experiment has shown that the volume level doesn’t significantly alter the intensity of the response of the players⁶. A larger-scale study by Ellermeier et al. has also demonstrated no significant effect of loudness on the auditory unpleasantness of the context-independent sounds⁹.
Sharpness
Sharpness is a psychoacoustic measure of high-frequency content in sound. Since my early studies in sound engineering, people have told me that too much treble will make everything sound irritating. A plausible evolutionary explanation could be that lots of energy in the higher end indicates proximity, and the objects right next to you deserve more attention.
A study on textiles has shown that fabric feels less pleasant to interact with if it makes loud and sharp rustling noises¹⁰. In the videogame context, Garner and Grimshaw report a correlation between sharpness and biometric data they used to assess fear response in players¹¹. Another paper suggests that unpleasantness is a product of sharpness and roughness combined¹².
The study by Ellermeier et al⁹. comes to a similar conclusion but states that the role of sharpness is relatively small compared to roughness. Another study results suggest that the unpleasantness is “more level dependent but less frequency dependent than sharpness” ¹³.
Ambiguity
Sounds become cognitively demanding if they are hard to identify. If we can’t extract crucial information from the sound we hear, our mental image of the surrounding reality is dangerously incomplete. Evidence shows that our auditory system is sensitive to ambiguity and reacts to it at a very early stage of processing¹⁴. If the signals we receive lack in clarity, a part of our brain boosts vigilance, making us more attentive¹⁵.
In the videogame context, ambiguity is a vital element of horror game soundscapes¹⁶. But the concept of ambiguity in sound is somewhat ambiguous on its own: there are several ways in which we might not get enough information from an auditory cue.
The sound may come from an unexpected place or point to an unusual source. Such ambiguity is contextual and goes beyond the scope of this post. There is also an awkward case of the inability to localize the sound source in space where sound can be ambiguous because of either contextual or acoustic factors. One study examined the effect localization has on the sensation of fear and terror in horror games. The researchers concluded that harder to localize sounds are perceived as more scary¹⁷. However, another experiment did not show any significant effect of binaural processing on the intensity of fear in the videogame⁶.
Noisiness
Noisiness or lack of clarity also falls under the concept of ambiguity. Noisy sounds challenge our auditory attention: remember listening to a heavily distorted speech or a cluttered mix in a videogame, trying to make sense of what you hear. Evidence suggests that wide-bandwidth sounds tend to be more annoying⁸, probably because they carry too much useless information. So, the signal-to-noise ratio could be another factor of unpleasantness¹⁸.
This statement is also supported by the idea that aesthetic pleasure is a function of processing fluency¹⁹, which on its own, aligns with Daniel Kahneman’s concept of cognitive ease²⁰.
Onset time or attack
Amplitude envelope describes how the volume of a sound changes in time. The very first stage of the envelope, when the sound rises in intensity, is called an attack. Sudden sounds with a very sharp attack can startle us²¹ ²². Such sounds are generally perceived as urgent, demanding immediate attention.
In the natural environment, even gradually raising intensity can mean that something is approaching us, and the faster it gets louder, the faster it moves. So, even a slower attack time may put us into an alarm state²² ²³ ²⁴.
I doubt that onset time is a significant factor of unpleasantness. If on its own, it was enough to stress us, we would never leave this state, given how diverse the modern soundscape is. But subjectively, it can boost unpleasantness for the sounds that already have some other feature from this list.
Low pitch
Seth Horowitz, in his book The Universal Sense²¹, says that low-pitched noises scare us because they remind of big predators or dangerous natural events. I’ve seen this idea repeating quite a few times in different literature, but I’ve never found an in-depth description of this effect.
Many believe that ultra-low sound beyond the range of human hearing (infrasound) makes us uncomfortable and frightened. Still, even if true, it has no practical implications in the videogame context. A recent study on low-frequency noise does not support either of those ideas²⁵.
Even though subjectively boomy sounds drag more attention, and we can easily get overwhelmed with prominent low frequencies, I could not find much evidence about the relation between low pitch and negative emotional responses.
Roughness
Roughness reflects the power of amplitude modulations of a sound in the range between 15 and 150 Hz. Important notice: those are modulation frequencies, not the audio frequencies we usually mean when talking about sounds. Amplitude modulation of 15 Hz means that the amplitude of the sound changes 15 times per second, while the carrier (audio) frequency can be any. Some sources tell that sensation of roughness starts from a higher modulation frequency, but in this post, I’ll stick with a definition from the book Psychoacoustics: Facts and Models by Fastl and Zwicker⁷. Many researchers see roughness as a significant factor that makes sounds unpleasant⁹ ¹².
In 2015 a group of researchers led by Luc H. Arnal found out that roughness is a distinct feature of human screams²⁶. Screams were crucial for the survival of our ancestors. And they remain an efficient way to inform others about a threat. That’s why Arnal et al. suggest that our auditory system has a mechanism to segregate screams from other communication signals.
According to them, rough temporal modulations between 30 and 150 Hz specifically communicate danger. In the same study, they found out that artificial alarms also follow the same pattern. And even more, a recent study demonstrated that the same acoustic property is typical for scary film music²⁷. It implies that horror movie composers have intentionally or unintentionally mimicked human screams and other alarm sounds to make their music more terrifying.
Opposing to all that, the study by Kurakata et al. didn’t show that roughness has a significant effect on unpleasantness¹³.
Beating or Fluctuation strength
Fluctuation strength is a “subsonic roughness”: it is essentially the same phenomenon, but slower. It refers to amplitude modulations starting from 1 Hz and ending at the beginning of the roughness region. In other words, roughness begins on a frequency where you can’t hear individual fluctuations and start detecting a change in the timbre instead. Both beating and roughness are strongly associated with the concept of dissonance in music.
There is a study from 2009 that strongly links the unpleasantness with fluctuation strength, even though the specific term never appears in the text¹. The researchers selected famously unpleasant sounds like dragging fingernails down a blackboard, running an electric drill, scraping a steel fork over a glass, etc. and examined them for similar acoustical properties. The results show that all such sounds share two features: they have a lot of energy in the range from 2500 to 5500 Hz and a lot of temporal modulations in the range from 1 to 16 Hz. The authors hypothesize that those ranges are the ones our auditory system is the most sensible at for both temporal modulation and spectral content.
Andy Farnell describes a similar beating effect in his book Designing Sound²⁸, even though there he calls the effect roughness:
“Roughness as an unpleasant effect may be better understood as the doom tone. It is an interference rhythm composed of cycles in the 4Hz to 8Hz range resulting from dangerous high-energy phenomenon like stampeding animals and high speed wind. When it modulates higher frequency components, the resulting critical band ambiguity leads to an unsettling effect.”
At the same time, neither Kurukata et al.¹³ nor Ellermeier et al⁹. experiments that I’ve mentioned several times, didn’t show that fluctuation strength significantly influences the perceived unpleasantness of a sound.
In the book Psychoacoustics: Facts and Models⁷, Eberhard Zwicker has proposed a psychoacoustic annoyance model based on four parameters: loudness, sharpness, roughness, and fluctuation strength. The model is frequently referenced as reliable in other literature on psychoacoustics, but I failed to find any tool to test it on video game sounds.
In general, the tools allowing to measure the psychoacoustic properties described above are scarce and not very approachable for the game audio crowd. And still, I managed to find some. If you think this post was too short on game-audio relevant information, stay tuned! In the next part, I’m going to put some of this knowledge into action. Here is a sneak peek:
I selected 27 game sound effects that many people reported as unpleasant, annoying, disturbing, or scary. Then I asked 24 non-gamers to listen to those sounds and rate based on how unpleasant they feel. After that, I compared this “unpleasantness score” to measurements of some features from this post to look for correlations. Stay tuned! Update: read it here.
References:
[1]: S. Kumar, K. von Kriegstein, K. Friston and T. Griffiths, “Features versus Feelings: Dissociable Representations of the Acoustic Features and Valence of Aversive Sounds”, Journal of Neuroscience, vol. 32, no. 41, pp. 14184–14192, 2012. Available: 10.1523/jneurosci.1759–12.2012.
[2]: A. Pannese, D. Grandjean and S. Frühholz, “Amygdala and auditory cortex exhibit distinct sensitivity to relevant acoustic features of auditory emotions”, Cortex, vol. 85, pp. 116–125, 2016. Available: 10.1016/j.cortex.2016.10.013.
[3]: T. Cox, “Scraping sounds and disgusting noises”, Applied Acoustics, vol. 69, no. 12, pp. 1195–1204, 2008. Available: 10.1016/j.apacoust.2007.11.004.
[4]: C. Reuter, M. Oehler and J. Mühlhans, “Physiological and acoustical correlates of unpleasant sounds”, in Joint Conference ICMPC13-APSCOM5, Yonsei University, Seoul, Korea, 2014.
[5]: P. Samermit, J. Saal and N. Davidenko, “Cross-Sensory Stimuli Modulate Reactions to Aversive Sounds”, Multisensory Research, vol. 32, no. 3, pp. 197–213, 2019. Available: 10.1163/22134808–20191344
[6]: T. Garner, M. Grimshaw and D. Nabi, “A preliminary experiment to assess the fear value of preselected sound parameters in a survival horror game”, Proceedings of the 5th Audio Mostly Conference on A Conference on Interaction with Sound — AM ’10, 2010. Available: 10.1145/1859799.1859809.
[7]: H. Fastl and E. Zwicker, Psychoacoustics: Facts and Models.Berlin: Springer, 2007.
[8]: D. Steele and S. Chon, “A perceptual study of sound annoyance.”, in Audio Mostly 2007, 2007.
[9]: W. Ellermeier, M. Mader and P. Daniel, “Scaling the unpleasantness of sounds according to the BTL model : Ratio-scale representation and psychoacoustical analysis”, Acustica United with Acta Acustica, vol. 90, no. 1, pp. 101–107, 2004.
[10]: J. Cho, E. Yi and G. Cho, “Physiological Responses Evoked by Fabric Sounds and Related Mechanical and Acoustical Properties”, Textile Research Journal, vol. 71, no. 12, pp. 1068–1073, 2001. Available: 10.1177/004051750107101206.
[11]: T. Garner and M. Grimshaw, “Psychophysiological Assessment Of Fear Experience In Response To Sound During Computer Video Gameplay”, in International Conference Interfaces and Human Computer Interaction, 2013, pp. 45–53.
[12]: R. Weber and R. Eilers, Combined contribution of roughness and sharpness to the unpleasantness of modulated band pass noise. DAGA ’07, Stuttgart, Deutschland, 2007. ISBN: (978–3–9808659–3–7), p./pp. 565/566. Deutsche Gesellschaft für Akustik.
[13]: K. Kurakata, T. Mizunami and K. Matsushita, “Sensory unpleasantness of high-frequency sounds”, Acoustical Science and Technology, vol. 34, no. 1, pp. 26–33, 2013. Available: 10.1250/ast.34.26.
[14]: L. Gwilliams, T. Linzen, D. Poeppel and A. Marantz, “In Spoken Word Recognition, the Future Predicts the Past”, The Journal of Neuroscience, vol. 38, no. 35, pp. 7585–7599, 2018. Available: 10.1523/jneurosci.0065–18.2018.
[15]: P. Whalen, “The uncertainty of it all”, Trends in Cognitive Sciences, vol. 11, no. 12, pp. 499–500, 2007. Available: 10.1016/j.tics.2007.08.016.
[16]: G. Roux-Girard, “Listening to Fear”, Game Sound Technology and Player Interaction, pp. 192–212. Available: 10.4018/978–1–61692–828–5.ch010.
[17]: I. Ekman. & R. Kajastila, Localization Cues Affect Emotional Judgements–Results from a User Study on Scary Sound, in 35th AES International Conference: Audio for Games, 2009.
[18]: I. Ekman, “Psychologically motivated techniques for emotional sound in computer games”, in AudioMostly 2008, 3rd Conference on Interaction with Sound, Pitetextbackslashaa, Sweden, 2008, pp. 20–26.
[19]: R. Reber, N. Schwarz and P. Winkielman, “Processing Fluency and Aesthetic Pleasure: Is Beauty in the Perceiver’s Processing Experience?”, Personality and Social Psychology Review, vol. 8, no. 4, pp. 364–382, 2004. Available: 10.1207/s15327957pspr0804_3
[20]: D. Kahneman, Thinking, fast and slow. New York: Farrar, Straus and Giroux, 2013.
[21]: S. Horowitz, The universal sense. New York: Bloomsbury, 2013.
[22]: J. Edworthy, S. Loxley and I. Dennis, “Improving Auditory Warning Design: Relationship between Warning Sound Parameters and Perceived Urgency”, Human Factors: The Journal of the Human Factors and Ergonomics Society, vol. 33, no. 2, pp. 205–231, 1991. Available: 10.1177/001872089103300206.
[23]: D. Bach et al., “Rising Sound Intensity: An Intrinsic Warning Cue Activating the Amygdala”, Cerebral Cortex, vol. 18, no. 1, pp. 145–150, 2007. Available: 10.1093/cercor/bhm040.
[24]: D. Bach, J. Neuhoff, W. Perrig and E. Seifritz, “Looming sounds as warning signals: The function of motion cues”, International Journal of Psychophysiology, vol. 74, no. 1, pp. 28–33, 2009. Available: 10.1016/j.ijpsycho.2009.06.004.
[25]: O. Behler and S. Uppenkamp, “Activation in human auditory cortex in relation to the loudness and unpleasantness of low-frequency and infrasound stimuli”, PLOS ONE, vol. 15, no. 2, p. e0229088, 2020. Available: 10.1371/journal.pone.0229088.
[26]: L. Arnal, A. Flinker, A. Kleinschmidt, A. Giraud and D. Poeppel, “Human Screams Occupy a Privileged Niche in the Communication Soundscape”, Current Biology, vol. 25, no. 15, pp. 2051–2056, 2015. Available: 10.1016/j.cub.2015.06.043.
[27]: C. Trevor, L. Arnal and S. Frühholz, “Terrifying film music mimics alarming acoustic feature of human screams”, The Journal of the Acoustical Society of America, vol. 147, no. 6, pp. EL540-EL545, 2020. Available: 10.1121/10.0001459.
[28]: A. Farnell, Designing sound. Cambridge, Mass.: MIT Press, 2010.
