Neither widely used pictorial emotion stimuli sets (i.e., International Affective Pictorial System, Lang, Bradley, & Cuthbert, 1999) nor more recently developed sets of emotional pictures (i.e., Geneva Affective Picture Database, Dan-Glauser & Scherer, 2011; Nencki Affective Picture System, Marchewka et al., 2014) have ever been tested for stability across time, an important consideration when presenting such stimuli more than once within the same individual. The purpose of the present study was to develop a new set of reliable pictorial stimuli, which elicited target emotions stably over time.

In the first screening phase, we collected 10,696 images from the biggest image hosting website, `flickr.com' based on their emotional labels and categorization. In this phase, 179 participants rated these images twice (across a 1-week interval) along four emotional dimensions (i.e., valence, dominance, arousal and likeness). Rating scales consisted of 7-point Likert scales adopted from Lang (1980)'s Self-Assessment-Manikin Scale. All images, instructions, and scales were presented on the same monitors and all participants' responses were saved onto a passcode protected server. In each session, individual images were rated an average 3.14 times ranging from 1 to 11. Based on initial retest reliability (correlation coefficient greater than or equal to .60), at least 5 ratings per image, and distribution across emotion combinations, a total of 1,080 images were selected as the preliminary set of stimuli. In the second phase, more ratings were conducted for each of these images. In each session, each image received an average of 62 ratings ranging from 26 to 92. Based on their retest reliability (correlation coefficient greater than or equal to .60) and degree of emotion induction (high: higher than 67th percentile; low: lower than 33rd percentile), a total of 356 images were chosen as the final group of reliable pictorial stimuli.

Although it has fewer images than prior pictorial sets in the number of images, the stimuli set developed in the current study has advantages of eliciting target emotions more stably over time. Another advantage over prior stimuli sets is that the initial images were selected based on computing methods as opposed to using face validity. Through applying computer-based approaches, we tried to minimize human bias, which can operate when the initial images are selected by experimenters based on subjectivity.

Full color PDF file (0.9 MB)