Keynote Speech
Modeling Aesthetics and Emotions in Visual Content:
From Vincent van Gogh to Robotics and Vision
James Z. Wang
The Pennsylvania State University, USA
Abstract:
As inborn characteristics, humans possess the ability to judge visual
aesthetics, feel the emotions from the environment, and comprehend
others’ emotional expressions. Many exciting applications
become possible if robots or computers can be empowered with
similar capabilities. Modeling aesthetics, evoked emotions, and
emotional expressions automatically in unconstrained situations,
however, is daunting due to the lack of a full understanding of
the relationship between low-level visual content and high-level
aesthetics or emotional expressions. With the growing availability
of data, it is possible to tackle these problems using machine
learning and statistical modeling approaches. In the talk, I provide
an overview of our research in the last two decades on data-driven
analyses of visual artworks and digital visual content for modeling
aesthetics and emotions.
First, I discuss our analyses of styles in visual artworks. Art
historians have long observed the highly characteristic brushstroke
styles of Vincent van Gogh and have relied on discerning these
styles for authenticating and dating his works. In our work, we
compared van Gogh with his contemporaries by statistically analyzing
a massive set of automatically extracted brushstrokes. A
novel extraction method is developed by exploiting an integration
of edge detection and clustering-based segmentation. Evidence
substantiates that van Gogh’s brushstrokes are strongly rhythmic.
Next, I describe an effort to model the aesthetic and emotional
characteristics in visual contents such as photographs. By
taking a data-driven approach, using the Internet as the data source,
we show that computers can be trained to recognize various characteristics
that are highly relevant to aesthetics and emotions. Future
computer systems equipped with such capabilities are expected to
help millions of users with unimagined ways.
Finally, I highlight our research on automated recognition of
bodily expression of emotion. We propose a scalable and
reliable crowdsourcing approach for collecting in-the-wild perceived
emotion data for computers to learn to recognize the body
languages of humans. Comprehensive statistical analysis revealed
many interesting insights from the dataset. A system to model the
emotional expressions based on bodily movements, named ARBEE
(Automated Recognition of Bodily Expression of Emotion), has also
been developed and evaluated.
Full color PDF file (1.2 MB)
Citation:
James Z. Wang ``Modeling Aesthetics and Emotions in Visual Content:
From Vincent van Gogh to Robotics and Vision,'' Proceedings of the
Joint Workshop on Aesthetic and Technical Quality Assessment of
Multimedia and Media Analytics for Societal Trends, in conjunction
with the ACM International Conference on Multimedia, pp. 15-16, Virtual,
October 2020.
Copyright 2020 James Z. Wang. Personal use of this material is
permitted. However, permission to reprint/republish this material for
advertising or promotional purposes or for creating new collective
works for resale or redistribution to servers or lists, or to reuse
any copyrighted component of this work in other works, must be
obtained from the author.
Last Modified:
September 14, 2020
© 2020