Unlocking the Emotional World of Visual Media:
An Overview of the Science, Research, and Impact
of Understanding Emotion
James Z. Wang (1), Sicheng Zhao (2), Chenyan Wu (1),
Reginald B. Adams, Jr. (1), Michelle G. Newman (1), Tal Shafir (3), Rachelle Tsachor (4)
(1) The Pennsylvania State University, University Park, USA
(2) Tsinghua University, Beijing, China
(3) University of Haifa, Israel
(4) University of Illinois, Chicago, USA
Drawing insights from psychology, engineering, and the arts, this
article provides a comprehensive overview of the field of emotion
analysis in visual media, and discusses the latest research, systems,
challenges, ethical implications, and potential impact of artificial
emotional intelligence on society.
Abstract:
The emergence of artificial emotional intelligence technology is
revolutionizing the fields of computers and robotics, allowing for a
new level of communication and understanding of human behavior that
was once thought impossible. Whereas recent advancements in deep
learning have transformed the field of computer vision, automated
understanding of evoked or expressed emotions in visual media remains
in its infancy. This foundering stems from the absence of a
universally accepted definition of ``emotion,'' coupled with the
inherently subjective nature of emotions and their intricate
nuances. In this paper, we provide a comprehensive, multidisciplinary
overview of the field of emotion analysis in visual media, drawing on
insights from psychology, engineering, and the arts. We begin by
exploring the psychological foundations of emotion and the
computational principles that underpin the understanding of emotions
from images and videos. We then review the latest research and systems
within the field, accentuating the most promising approaches. We also
discuss the current technological challenges and limitations of
emotion analysis, underscoring the necessity for continued
investigation and innovation. We contend that this represents a ``Holy
Grail'' research problem in computing and delineate pivotal directions
for future inquiry. Finally, we examine the ethical ramifications of
emotion-understanding technologies and contemplate their potential
societal impacts. Overall, this article endeavors to equip readers
with a deeper understanding of the domain of emotion analysis in
visual media and to inspire further research and development in this
captivating and rapidly evolving field.
Full Paper
(IEEE, open access, 6.4MB)
Full Paper
(high-resolution PDF, 19MB)
Full Paper
(normal-resolution PDF, 4.0MB)
More information
Citation:
James Z. Wang, Sicheng Zhao, Chenyan Wu, Reginald B. Adams, Jr.,
Michelle G. Newman, Tal Shafir and Rachelle Tsachor, ``Unlocking the
Emotional World of Visual Media: An Overview of the Science, Research,
and Impact of Understanding Emotion,'' Proceedings of the IEEE,
vol. 111, no. 10, pp. 1236-1286, 2023.
© 2023 IEEE. Personal use of this material is permitted. However,
permission to reprint/republish this material for advertising or
promotional purposes or for creating new collective works for resale
or redistribution to servers or lists, or to reuse any copyrighted
component of this work in other works must be obtained from the IEEE.
Last Modified:
October 18, 2023
© 2023