Sense-IT@iet.NTNUOct. 6 @ Gløshaugen
A-me : Augmented memories
http://picturingthebrain.org/a-me
A-me is a hypothetical memory recall device that enables the user to immersively experience human memories. It aims at criticizing paradigms related to brain/mind imaging that are currently under discussion in the scientific community. A-me investigates the implications of approaches to brain science, such as reductionism, and phenomenology. Thus, it is a symbolization of the current scientific discussions on human behavior through brain activity analysis. A-me is also a technical research on Augmented Reality techniques applied to brain visualization and auralization.
A-me is a hypothetical memory recall device that enables the user to immersively experience human memories.
This contribution presents the concept of an optical see-through Augmented Reality system to overlay a volume rendered MRI scan onto a medical head phantom. In addition, the installation features a binaural rendering system to auralize memories as enveloping 3D soundscapes. The user will be able to navigate the brain by using a tracked probe in a similar way neurosurgeons examine brain injuries during preoperative planning. While navigating the brain, the user will find active areas in specific parts of the nervous structure. Pointing at them with the probe will trigger an stored emotional experience in the form of a visual and auditory representation of its neural activity.
This project started from the need to discuss and criticize the above mentioned concepts and theories related to brain / mind definitions. To this end we propose the development of an interactive art installation exemplifying a hypothetical memory reading apparatus. The artwork aims at exposing the current state of the art on brain mapping while offering the possibility to activate an experience. It is known that long term memories are located in the hyppocampus which is an area in the medial temporal lobe of the brain. The device is using real tomographic data from a scanned corpse, which can be navigated by manipulating a tracked probe. The interaction led by the visitor will determine a selection of a memory which will trigger an audiovisual response.
The installation works as follows. In the exhibition space a there will be an area properly equipped to reveal the experience. One stereo 3d screen, six tracking cameras, a half-silvered glass and a head manikin will be standing on a table (see Fig.1). The visitor will be equipped with high-end wireless headphones, tracked shutter glasses and a tracked probe. Looking through the glass, the visitor will see an MRI volume visualization registered on the dummy-head. The visitor will be able to navigate different areas of the brain by manipulating the probe. Visually merged with the real data there will be some active hotspots indicating the location of a memory. By pointing precisely at one of them an immersive audio-visual response will be triggered in the form of brain activity registered on the actual tomographic data. When moving further away from such a hotspot, the device will merge more and more soundscapes of neighbouring memories resulting – from a certain distance on – in a complete auditory chaos.
In collaboration with:
Dirk Schröder Immersive – Sound Design and Auralization Concept
Tore Landsem – Industrial design
Rune Svensrud – Image processing
Carles Gutierrez – Coding support
Developed at SenseIT, funded by Q2S and Picturing The Brain.
Chroma Space
We are presenting an installation called Chroma Space to serve as a platform for experimenting the novel usage of affective colors in an immersive environment. Chroma Space shares Flick Flock`s (2009) technology, which explores a transformative enclosure for the viewers through a combination of 3D game engine, computer vision, and ambisonic 3D. The challenge in the implementation of stylistic presentation of colors in Chroma Space involves not only game engine restrictions but also the sensually destructing visual continuity issues (both in visual presentation and in animation) that are difficult to resolve in a synthetic world. This installation is aimed at demonstrating the effective impacts of using a stylistic approach to address emotional sensations, by making colors move in space.
We envision Chroma Space in muted palettes, depriving each rendered frame with the richness of colors. With carefully chosen palettes to only direct the sensations of the spectator, entering Chroma Space, the performers find themselves influencing the behaviour, colors, and presentation of the virtual objects. Chroma Space presents muted colors that are temporally interacting and communicating to our senses. The installation also features virtual objects that are moving in harmony with colors and sound. This is the abstract world that defines Chroma Space.
Chroma Space creates its immersive environment through light and wall projection and is driven by the immersant`s own body. The immersant`s body affects the imagery and sound is acting like a solid geometry mapped in a coordinate system. By raising the viewer`s arms far apart from the body, the viewer signals the system to go forward in space. By moving the viewer`s body some degrees to the left or right, the system shifts the virtual camera in that direction. The direction of the movement of the viewer also affects the panning and playback of sounds.
Networked Decollage
http://www.q2s.ntnu.no/~wendyann/decollage
Networked Decollage serves as a collective ground for art interventions in a networked space. We started with a simple objective of allowing participants to annotate, select and share information from the social network (preferably their own) that they think relevant for a particular social space. To achieve this, our work expands social computing in serious gaming to include two aspects in game design: the usage of art intervention and crowd sourcing.
Networked Decollage is created as a massively multiplayer online game (MMOG). In MMOGs, multiple players interact and communicate with each other using various communication networks and multimedia infrastructures. However, despite the robustness of MMOGs, the satisfactions of MMOG players are always challenged by issues on social realism and aesthetic aspects of the environment. To account for the multi-disciplinary research objectives within our centre, the current work is designed to support several kinds of media compared to traditional MMOGs. This brings up several important research topics, particularly in the Quality of Experience (QoE) domain to guarantee better user experience. The current platform is envisioned to serve further investigations in an immersive environment context, and on the effects of new aesthetic forms and other modes of implicit experiences affecting our quality perception.
Candy
http://www.q2s.ntnu.no/~wendyann/candy
Candy is an audio-visual experimental performance piece that displays metaphorical homogenization of individuals brought by our evolving consumption and technology. Reflects on the triumph of the manipulative campaigns in the culture industry, and the challenges of the innate human pleasure circuits. Circuits that are driven by our primitive desires and supernormal stimulants that are capable of pressing our pleasure buttons. Candy gathers information from the social space for beauty and truth, which includes visual artefacts, personal statements and consumer campaigns and projects them on a physical multi-sensory space.
The artwork invites the participant to reflect, sit, relax and to physically explore a dining table as a diner. A menu is served and the diner chooses a dish. When the dish is presented on the table, the cover is lifted which results to a release of aroma that serves to tease the gustatory nerves and to lift the sense of imagination and craving of the participant. The diner experiences a multi-sensory narrative medium that visualizes the social networking world and electronic mass media for its influences on human preferences, consumption and pleasure in real-time. In real-time, the piece also amplifies the auditory sounds and noise associated with the direct interaction with food and the associated narrative medium.
Q2S:
The work on all installations is carried out at the Sense-IT lab at Q2S – Centre for Quantifiable Quality of Service in Communication Systems, a Norwegian Centre of Excellence at the Norwegian University of Science and Technology in Trondheim. Q2S is appointed by the Research Council of Norway and funded by the Research Council, NTNU and UNINETT.
The main activities at Q2S are the study of principles, mechanisms, methods and technical solutions related to quality and assess their properties and performances by means of experiments and models. Performances relate to perceived quality of streamed speech/music and video, delays and throughput of elastic traffic, reliability and availability of services, and information security with encryption and user authentication.
Immersive Media Technology Experiences:
The intersection between arts and sciences in Information and Communication Technology (ICT) has intrigued artists, technologists, and scientists since the 60’s. Computer-based art has existed since the 60’s with artists like Michael Noll and Frieder Nake. The first computer art exhibitions were held in 1965. The Leonardo journal was found in 1968. Photoshop 1.0 was released in 1990 and in the next years digital cameras appeared on the market. Pictures could be uploaded to computers and manipulated. The first multimedia Personal Computer was released in 1991. With the advent and the evolution of the personal computer, the web, software tools like Adobe Photoshop, and electronics tools such as cameras and mobile telephones, there has been an explosion of both production and sharing of digital content. Artists need software technology for creating and evolving their artwork. Theoreticians aim at understanding the consequences of software to art practices. Technologists see the contact with artists as a source of innovation.
Immersive Media Technology Experiences (IMTE) is a key concept incorporating several disciplines including Media Technology, Information and Communication Technology, and Media Studies. In this way IMTE can encompass diverse core competencies covering fields such as communications, information retrieval, entertainment and social networks. The scientific platform of Immersive Media Technology Experiences makes use of today’s new digital media and merges these into social networks by connecting people in a whole new way creating collaboration spaces. A strong emphasis on experiences achieves full focus on the users and their requirements and needs, enabled by the developed experience model describing the complete media technology eco system from content representations to its usages and social, economic and cultural impact.
The areas of research and innovation are linking together three main Focus Areas: User, Content and Infrastructure. The glue in the linking is by the Users interacting with the Content through devices connected to the Infrastructure. Such Interactions and Devices demands New Digital Media and abilities for the Content to Adapt to User requests and the available Infrastructure (Networked Media Handling). This structure allows for piloting new advanced applications and services such as Digital Storytelling, Digital Art, Serious gaming, Presence and immersive experience (interactivity) etc
The IMTE team spreads over 4 departments over 2 faculties and is coordinated by:
• Andrew Perkis (NTNU IET/Q2S)
• Leif Arne Rønningen (NTNU/ITEM)
• Yuming Jiang (NTNU ITEM/Q2S)
• Aud Sissel Hoel (NTNU IKM)
• Letizia Jaccheri (NTNU IDI)
The works within IMTE presented at Meta.Morf are done under the supervision of Andrew Perkis, Touradj Ebrahimi and Aud Sissel Hoel.
Sense-IT
In our laboratories we explore the intersection of art and technology through several installations. Candy, Chromaspace, Networked decollage and A-me our research in QoE and art related activities. We hold a permanent exhibition in Sense-IT laboratory where visitors can learn from our research and directly experience the latest developments in various fields such as 3DTV, full body interaction techniques, augmented reality and immersive media applications amongst others.
Picturing the brain
A-me is an installation developed under the research project Picturing the Brain: Perspectives on Neuroimaging. This project seeks to deepen our understanding of the epistemological roles neuroimaging technologies play in the conduct and communication of medicine and science. The primary objective is, more precisely, to develop a fine-grained understanding of socio-cultural and ethical issues that arise in relation to current applications of these technologies, as they are put to use as cognitive tools, as perceptual prostheses, and as visual rhetoric.
Publication list
Exhibitions, shows and Art:
Mansilla, Wendy Ann; Puig, Jordi; Perkis, Andrew; Ebrahimi, Touradj. Chroma Space. [Exhibition catalogue]. Colorito: An Interactive Renaissance of Colours. ACM & Titivillus Mostre Editoria; Pisa, Italy, Oct, 2010
The NEM Art Jury is pleased to announce that your artwork – WON: Networked Decollage – is selected to participate in the 2011 NEM Art & Design Contest, which took take place on 27-29 September 2011, in Torino, Italy.
Refereed conference publications:
2012
Jordi Puig, Andrew Perkis, Frank Lindseth and Touradj Ebrahimi, Towards an efficient methodology for evaluation of Quality of Experience in augmented reality, Accepted for publication in Proc. Of IEEE Qomex 2012, Melbourne, July 5-7, 2012
Wendy Ann Mansilla, Touradj Ebrahimi and Andrew Perkis, Considering the Changing Aesthetic Form and Social Media Dynamics in an Immersive-Networked Environment , accepted for Poster/Demonstration IADIS Game and Entertainment Technologies 2012 (GET 2012), Lisbon, Portugal, 18-21 July 2012, http://www.gaming-conf.org/
2011
Wendy Ann Mansilla, Andrew Perkis and Touradj Ebrahimi, Implicit Experiences as a Determinant of Perceptual Quality and Aesthetic Appreciation, Accepted for publication in the Proceedings of ACM MULTIMEDIA 2011 (long paper)
2010
Mansilla, Wendy Ann; Puig, Jordi; Perkis, Andrew; Ebrahimi, Touradj. Flick Flock: the distant and distinct characteristics of the masses in immersive aesthetic space. In Siggraph Asia ’10 ACM SIGGRAPH ASIA 2010 Posters. No. 64, ACM New York, NY, USA, Dec, 2010.
Mansilla, Wendy Ann; Puig, Jordi; Perkis, Andrew; Ebrahimi, Touradj (2010). Chroma Space: Affective colors in interactive 3D world. I: Proceedings of the 18th ACM International Conference on Multimedia 2010 (MM 2010). Association for Computing Machinery (ACM) 2010 ISBN 978-1-60558-933-6. p. 1441-1444.
2009
Mansilla, Wendy Ann; Perkis, Andrew; Ebrahimi, Touradj. Exposure effect on experience and visual perception in stereoscopic visual presentations. In Proceedings of the International Conference on Advances in Computer Entertainment Technology, ACE 2009, Athens, Greece, October 29-31, 2009. ACM 2009, ISBN 978-1-60558-864-3, ACM Press, 2009
Bios
Andrew Perkis was born in Norway 1961. He received his Siv.Ing and Dr. Techn. Degrees in 1985 and 1994, respectively. In 2008 he received an executive Master of Technology Management in cooperation from NTNU, NHH and NUS (Singapore). Since 1993 he has held the position of Associate Professor at the Department of Telecommunications at NTNU and as full professor since 2003. In 1999/2000 he was a visiting professor at The University of Wollongong, Australia and in 2008 a visiting professor at the National University of Singapore. He is responsible for “Network Media Handling” within the National Centre of Excellence – Quantifiable Quality of Service in communication systems at NTNU. Currently he is focusing on Multimedia Signal Processing, specifically within methods and functionality of content representation, quality assessment and its use within the media value chain in a variety of applications. He is also involved in setting up directions and visions for new research within media technology and art. Within applied research he is heavily involved in multi platform publishing, especially to handheld devices. He has been involved in the start-up company Adactus and commercial aspects of Digital Cinema role out through running the Norwegian trial project NORDIC. He is member of The Norwegian Academy of Technological Sciences (NTVA), senior member of the IEEE, member of ACM and member of The Norwegian Society of Chartered Engineers (TEKNA).
Touradj Ebrahimi is a Professor at EPFL and head of its Multimedia Signal Processing Group (MMSPG). He is also an adjunct professor at NTNU and with its Quantifiable Quality of Service Center (Q2S). His research activities span over three main areas of multimedia signal processing, namely, image and video compression (notably advanced compression algorithms for 3D, Ultra High Definition, and High Dynamic Range content), media security (in particular privacy protection, trust models, content integrity verification and conditional access) , and multimedia content analysis and interpretation (including subjective and objective assessment of quality of experience, content annotation and propagation, visual search, and brain computer interface). He is head of Swiss delegation to MPEG, JPEG and SC29, and has been an active participant and contributor to these standards. Prof. Ebrahimi first coined the notion of Quality of Experience in 2001 as a user-centric concept as opposed to Quality of Service in multimedia communication. He is the Chair of the COST Action IC1003 (Qualinet) and active in various research and educational activities concerned with how to measure user experience in multimedia applications.
Aud Sissel Hoel is an Associate Professor of Visual Communication in the Department of Art and Media Studies at NTNU. She is leader of the priority research area PerFormativity at NTNU’s Faculty of Humanities, and currently she is heading the interdisciplinary research project Picturing the Brain: Perspectives of Neuroimaging (2010-2013). Sissel’s research interests revolve around science images, imaging, and technologies of vision, and branch out to questions concerning the role of symbols and tools in cognition. She has done research on the use of photography for the purposes of surveillance, control and classification of human beings and, more recently, on medical imaging and neuroimaging. In her ongoing attempts to conceptualize photography and neuroimaging, she draws on phenomenology, embodied perception and extended mind approaches, STS approaches, and contemporary philosophy of technology.
Jordi Puig is interested in the intersection between arts and technology. He studied Electronic Art in ESDI and he focused on interactive installations. He became a instructor on that field in the same university for three years. Jordi co-established a creative advertising company called Cuatic, specialized in the development of physical interaction where he acted as a director of new media technologies for two years. He won several advertising prizes in different festivals, some of them are El Sol Festival, New York Festival, Laus and Europrix. Jordi also has a career in the artistic scene, exhibiting in many countries around the world, most of the times for media arts organizations like FILE, MedialabPrado, Sonar, Baltan Laboratories, Piksel or Le Fresnoy. He worked in Ars Electronica Futurelab as a creative engineer. Also worked as a sound programmer for electroacustic concerts and theater pieces. He is co-founder of Lummo, a studio for the creation of active spaces. He is currently a Ph.D candidate at the National Center of Excellence for Quantifiable Quality of Service in communication systems at the Norwegian University of Science and Technology (NTNU).
Wendy Ann Mansilla
Wendy Ann Mansilla works on the interdisciplinary field of digital media, experimental art and science. Her research interests and art productions explore the implicit experiences, pleasure technologies synthetic reality, social media and the aesthetics of failure. Some of her latest works or performances were presented and exhibited at Piksel Bergen, STRP Technology Festival Eindhoven, Siggraph, ACM Multimedia, Campus Party Brazil, Factory of Art and Design Copenhagen and LABoral Centro de Arte y Creación Industrial. She has also won several artistic grants on collaborative digital media and contemporary art research. She is currently a Ph.D candidate at the National Center of Excellence for Quantifiable Quality of Service in communication system at the Norwegian University of Science and Technology (NTNU).
English