Synesthesia Machine

Synesthesia Machine

AI Interactive Installation, 2021







Synesthesia is a special phenomenon, which refers to the stimulus of one channel that can not only cause the sensation of that channel but also cause the sensation of another channel at the same time. For example, when a synesthete hears the sound of a piano, he will feel as if he smells cucumber. "Synesthesia Machine" is an artificial intelligence with synesthesia, it will emitting corresponding unique scents while perceiving human facial expressions.





Through the facial expression recognition algorithm, it can analyze the facial expressions of the audience and output the values of different emotions. At the same time, based on these emotion values, the installation is driven to extract different proportions of scent liquid and atomized and mixed to convert the audience's facial expression into a unique smell.



"Synesthesia Machine" is consists of two parts: the image acquisition part on the left and the odor generating part on the right. When the audience walks into the image collection area and stays for 3 seconds, the camera will take a photo of the audience's face and input it to the artificial intelligence program to analyze the emotion of the audience in the photo. The test tubes in the odor-generating installation on the right are respectively filled with wood scent, chemical odor, fruity, and bad-smelling liquids. The program will drive the installation to operate and draw different proportions of odorous liquid to the upper test tubes based on the results of the facial expression analysis, spray the odor through the atomization installation and mix it in the hemisphere. Finally, the fan turns on to blow out the mixed smell based on the audience's facial expression data.


Customized circuit components communicated by HC-06


Test tubes containing liquids with different odors


The audience’s expressions are converted into scents


The facial expression recognition code in the work is completed by combining TensorFlow and OpenCV. It can analyze the facial expressions in the image and output the proportion of different emotions. These values are linked to the odor generation part of the installation, and the Arduino receives the different emotional data from the audience’s facial expressions through the Bluetooth module, and uses it to drive the operation of the installation.



Python library installation and update


Facial expression recognition program test