Tele

Empathy

This project was technically challenging because multiple software and hardware components must operate simultaneously in real-time. Machines must simultaneously capture and share data across all connected devices, perform real-time analysis, output data to audio engines, and visualize complex data using GPU particles.

We capture raw EEG data and analyze it using Python - the most robust and versatile tool for connectivity and scientific analysis. In this scenario, we employ z-Score analysis to examine frequency band shape features. By comparing phase band shapes, we can identify similarities and understand how brain regions interact within a single user. We apply this technique to all bands at the same frequency across all electrodes while also analyzing inter-brain synchronization to determine where patterns occur. By utilizing the power of LSL Lab Streaming Layers, we enable seamless sharing of raw and processed data across our local ethernet network. This empowers us to work together towards our goal.

Unreal Engine UE5 plays a significant role in our project. All visuals are created using Unreal Engine. With the help of the LSL plugin for UE5, we are able to receive LSL data streams, apply logic, and visualize them in various ways. Unreal Engine has a built-in GPU-based particle system that can render hundreds of thousands of particles in one eyeblink. This is exactly what we are looking for when we want to visualize complex data, and EEG data can be extremely complex easily. Did you know that there is a Python plugin for Unreal Engine that can run in runtime? Check out the Yothon plugin for UE5. Currently, it does basic tasks, but in the future, it promises multi-threading and more.

Let's take a look at the back screen that displays the inter-brain synchronization. We are overlaying a graphic on top of the camera feed, which can be achieved in various ways. We tried out the ZED stereo camera, which provides a color image and depth image, and tracked, segmented bodies with a skeleton.

Working with the ZED stereo camera is not easy. Camera initialization can be complex, and body tracking can take some time. If you need to track a specific number of users, this feature needs to be implemented manually. Currently, it is not possible to segment only human bodies; it uses the depth image, and segmentation is done precisely in centimeters. While precision is nice, your physical space needs to be large enough for the composed graphics to not overlap with walls and objects. It would be a nice feature to be able to segment only the human body and move the camera feed somewhere further away so that all the graphics can be in between.

The particles that you can see in the swimming pool were created using UE5 Niagara particle system. We designed custom vector fields in Houdini. The vector field works as the path for a particle; particles are guided through this 3D arrow map that whirlpools into the center. This gives us an artistic and visually pleasing output while also helping us understand inter-brain synchronization. Each color represents a specific frequency band, and we can see how long the synchronization occurs over time.

The visualizations that you see on the screen behind us and on the floor are running from separate computers. We use LSL layers to ensure that data synchronization happens seamlessly between multiple machines.

The Unreal Engine is a great tool that now supports OSC Open Sound Control, Midi, and ArtNet DMX protocol. This makes it ideal for running all the gear on the stage. We are using OSC and Midi to share analyzed data with other machines and control sound parameters in Logic Pro and Ableton Live. In addition, DMX is the industry standard for driving light colors, positions, animations, and other effects for fixtures. Although we have found that there may be room for improvement in how we utilize DMX, we are confident that we can make the most out of this powerful technology.

"Together with 27 fellow performance artists, we have created a unique performance that is also an experiment in exploring the depths of inter-brain synchronization. By pushing our boundaries and discovering what triggers synchronization, we hope to inspire a new wave of creativity and innovation. We believe that through collaboration and experimentation, we can unlock the full potential of our minds and achieve greatness beyond our wildest dreams."

Thank you to all participants for contributing to our success. Your participation has been invaluable, and we appreciate your time, effort, and artistic input. Once again, thank you for your participation.

Anna Segliņa, Elza Jančuka, Rūta Krūskopa, Alise Elizabete Ipatova, Evelīna Zariņa, Adele Atlāce , Toms Baldonis , Elza Jančuka ,Alise Jaunkalne, Amēlija Luīze Lepse , Estere Lutce , Enija Pastva , Adrija Plūme, Elza Riekstiņa, Patrīcija Sokolovska, Dr. Agnese Dāvidsone, Lāsma Brūvere, Līva Rēzija Vanaga Kerija Mugina, Denīze Ozola-Kumeliņa, Frīda, Lācēns - Braiens Brauķis, Lāču mamma - Lāsma Bērziņa, Lāču tētis -Ernests Ozoliņš, Pelīte - Annija Kate Balode, Dzenis - Elīza Zvirbule, Briedis - Annija Kate Balode, Zaķis - Kate Zvārgule, Lapsiņa - Estere Bāliņa, Vadītāja - Anna Paula Gruzdiņa, Sigita Skrabe, Melānija Fimbauere, Lizete Bērziņa, Loreta Šperliņa-Priedīte, Ieva Freimane-Mihailova, Anna, Liene Brokāne, Samirs Laurs un Tomass Alksnītis, Andris Taurītis, Braiens Brauķis, Lāsma Bērziņa, Ernests Ozoliņš, Skarleta Kalniņa, Elīza Zvirbule, Annija Kate Balode, Estere Bāliņa, Anna Paula Gruzdiņa, Amēlija Eglīte, Lote Mincāne, Kate Jursone, Kristiāna Putniņa, Toms Treimanis, Rūdolfs Segliņš, Jurģis Segliņš, Adrians Gerasimovs, Niks Judenkovs, Elīza Petruseviča, Artūrs Eglītis, Elija Stolere, Vokālais ansamblis "IMERA"