Mixed Reality and the Interactive Imagination: Adding the art to the science and technology of Mixed Reality for training, education and entertainment

Technology has advanced to the point where realism in virtual reality is very achievable. However, in our obsession to reproduce the world and human experience in virtual space, we overlook the most important aspects of what makes us who we are—our reality. Yet, it isn’t enough just to trick the eye or fool the body and mind. One must capture the imagination in order to create truly compelling experiences. Mixed Reality (MR)) allows for the merging of our imagination with our physical self and virtual presence. With the additional convergence of entertainment simulation, artistic convention is playing an ever-growing importance in igniting the imagination to achieve compelling and effective simulations. This paper explores the artistic and technical research challenges of mixing these realities with the imagination based on the work at the Media Convergence Laboratory (MCL) presently being conducted at the at the University of Central Florida (UCF). Introduction: Putting the stimulation in simulation The Chief Scientist from the US Army’s Simulation Training and Instrument Command (STRICOM) remarked to the creative producer of Universal Studio’s Spiderman, The (simulation) Ride, “You know, we are in the same business really. We are in the business of making memories that people will never forget.” During the same panel discussion at the 2000 ACM/SIGGRAPH conference, leaders and visionaries of the future of military and entertainment simulation found they had more in common than differences. It is not our intent to confuse the importance of training the next generation soldier for life and death scenarios with extravagant amusement park attractions, but to reflect on the importance that simulations need to be more than realistic. They need to be compelling and engaging to make a difference in one’s life and perhaps save lifes. It is when you impact the imagination that you can seal a simulated experience into one’s memory in preparing for war, learning or just for thrills. In the study of Mixed Reality at the Media Convergence Laboratory, the creative techniques and technology of both experience-based entertainment (theme parks, video games, virtual sports, etc.) and simulation training, are increasingly more dependent upon one another in extending the art and science of simulation to reach the next generation of simulation experience in order to achieve its purpose in entertainment, education and training. Background: Evolution of Mixed Reality Since late in the last century, have computer-generated virtual simulation has been able to engage successfully with live simulation. Both have had their own advantages, but the choice of the user was to use either Virtual or Live simulation. Virtual Reality (VR) was extremely pliable to one’s purpose, but consisted of a disembodied experience. Live Simulation was significant in providing to a soldier the “ground truth” of Combat Reality (CR) in order to properly test the skills under fire.