All posts by admin

AR applications in medical training

Medical education is one of the application fields of augmented reality. As mentioned in the use-case section, the head-mounted device HoloLens released by Microsoft can be used to visualize a fully virtual body and its parts in a room for studying anatomy. But there are approaches in the scientific world in this area as well.

In 2014, a study about the skills assessment in minimal invasive surgery with AR was conducted. A system consisting of a box-trainer, a camera and surgery tools equipped with sensors was developed by the scientists. Basic skills, hand-eye coordination and bimanual operation were tested in this survey. Therefore, experienced surgeons and novices in this field were compared. Differences between the two groups regarding smoothness and economy of motion were found. The authors highlighted the potential of the technology for medical training as a conclusion. [1]

The study of Kim, Chan, & Du did not deal with the potential of AR for surgical training, but for interaction with a radar screen. Also for this study, an AR system was developed. In addition, a training textbook for the same task was written. Two different groups of undergraduates trained either with the AR system or the textbook. During a practice session, the performance of the students was measured. Significant differences were found between the two groups. Students practicing with the augmented reality application were more aware of the situation than students training with the textbook. This indicates that AR can have a positive learning effect on trainees in the medical field. [2]

The above studies suggest that the potential of augmented reality in medical training is high. In 2016, Lahanas el al. made a survey of the educational opportunities of this technology. Thus, the authors looked for relevant articles and evaluated them. Up to August 215, they found twenty-seven relevant studies. The articles could be assigned to the following categories:

  • Laparoscopic surgical training
  • Neurosurgical procedures
  • Echocardiography

Even though face- and construct-validity was proven for most training methods, none of the applications was proven to have the ability to transfer information to the user. [3]

In conclusion, there are promising possibilities of applying AR to medical training. Possible areas of application in the medical education field are minimal invasive surgery, interaction with a radar screen or anatomy. It seems like AR applications have a positive learning effect. However, there is no evidence in literature that AR systems are able to transfer information to medical personnel.

 

[1] Lahanas, Loukas, Smailis, & Georgiou, 2015

[2] Kim, Chan, & Du, 2015

[3] Lahanas et al., 2015

3D real time capturing combined with AR

Introduction

Communication between two remote parties has always been an interesting research topic. With the emerging technologies of augmented reality and the devices based on it, the opportunities have again reached a new height. The aim to create a virtual face-to-face experience is now possible with the help of 3D capturing and augmented reality.

Holoportation

“holoportation is a new type of 3D capture technology that allows high-quality 3D models of people to be reconstructed, compressed and transmitted anywhere in the world in real time. When combined with mixed reality displays such as HoloLens, this technology allows users to see, hear, and interact with remote participants in 3D as if they are actually present in the same physical space. Communicating and interacting with remote users becomes as natural as face-to-face communication.” [1]

Room2Room

The room2room technology makes it possible to create a telepresence between two remote participants. Its aim is to recreate the experience of a real face-to-face conversation with the help of 3D capturing and augmented reality technologies. Other than in the holoportation approach stated above, the room2room telepresence does not relay on any wearable device. The local user is being captured by colour and depth cameras which allows to create a virtual copy of the user projecting an image with the correct perspective. This setup makes it possible to enhance the non-verbal communication and allows to perform physical collaborative tasks more efficiently proven in a research environment. [2]

Fig. 1: Room2Room Setup
Fig. 1: Room2Room Setup

 

[1]         http://research.microsoft.com/en-us/projects/holoportation/

[2]         T. Pejsa, J. Kantor, H. Benko, E. Ofek, and A. Wilson, “Room2Room : Enabling Life – Size Telepresence in a Projected Augmented Reality Environment,” 2016.

Augmented Reality – Real Estate

Introduction

Augmented reality is adding value to a varity of aspects in the real estate sector. The following chapters introduce the most common fields.

AR for Planning and Constructing

When planning an architectural object many different people are involved during the whole processes. Since not individuals are able to understand constructing plans easily, augmented reality can be a very useful support tool.

The system setup to provide the mentioned function consists of two elements.

  • Collaborative Design Platform (Fig 1, left)
  • On Site AR Application (Fig. 1, right)
AR_realestate
Fig 1: System set up

Collaborative Design Platform

The Collaborative Design Platform (CDP) is based on a multi touch screen surface enhanced with real 3D object recognition. In the planning phase 3D models can be placed on the CDP which automatically get recognized and digitally replicated in the central system.

On Site AR Application

The On Site AR Application is linked to the central system with wireless communication. This enables an operator on site to receive changes in the model in real-time. With the help of a mobile device (e.g. tablet, phone or augmented reality glasses) and AR technology, the virtual objects can be integrated in the physical environment

The concept provides a possible solution which allows to reflect design changes directly on site with the capabilities of Augmented Reality. [1]

 

AR for Real Estate Agents and Buyers

Augmented Retail can be used as a sales support tool for estate agents as well as for the buyers. Agents benefit from a faster communication with potential customer. Furthermore it is possible to get real time feedback whenever a potential buyer is looking at an AR advertisement. This can help to generate new customers with the right targeting.

Buyers can get a virtual tour of the property before the construction is even finished. It is also possible to change the interior design without the existence of actual furniture. This is huge improvement compared to looking at an outline design. Taking a virtual tour is a helpful decision support system for the process of buying new property. [2]

 

 

 

[1] R. Velasco, A. P. Brakke, and D. Chavarro, “Computer-Aided Architectural Design Futures. The Next City – New Technologies and the Future of the Built Environment,” Springer-Verlag Berlin Heidelb., vol. 527, pp. 172–191, 2015.

[2] http://www.augmentedrealitytrends.com/augmented-reality/real-estate-industry.html

Application Areas of Virtual Reality

With this article the different application areas of virtual reality are introduced. Other than augmented reality virtual reality is not extending the physical environment with virtual elements. The aim is to create a completely independent virtual reality a user is able to interact with. The sensory experience can include sight, touch, hearing and even smell. [1]

Application Areas

The most popular fields in which virtual reality is present are:

  • Entertainment
  • Urban Design and Architecture
  • Training and Education
  • Medicine

The following chapters cover example applications of virtual reality to give a high level overview.

Entertainment

The first product which really caught the attention of the public eye was definitely the crowdfunding campaign on Kickstarter for the Oculus Rift in 2012. [2]

In the meantime there have been many different products introduced to the market. In Tokyo, April 15th 2016 the first virtual reality exception opened for a limited time. It allows anyone to experience the fascination of virtual reality.  [3]

Urban Design and Architecture

Virtual reality can be used to plan large scale urban project as well as individual buildings. The core element is a three dimensional digital model of the planned development. This is especially useful for long term project and has been used for the development of the campus of the University of Cambridge. [4]

3D Model University Campus
3D Model University Campus [4]
Training and Education

The US Army is using virtual reality setups to train combat situations in an enclosed environment. This minimizes potential risks and allows to reset and simulate specific situations in a short amount of time. This minimizes costs and increases efficency.[4]

US Army – Virtual Reality Training [5]
Medicine

Virtual Reality is used to help individuals with a stroke or neurological disorder to recover. A patient is able to manipulate objects in a virtual environment which can be structured to provide audio-visual feedback. This promotes motor learning to improve the rehabilitation process. Specific living tasks are being practiced within the virtual environment, e.g. making a cup of coffee in a kitchen. Real-time motion capture helps the therapist to document change with the support of a software. [6]

Virtual Kitchen Setup [6]
US Army – Virtual Reality Training [5]

[1] https://en.wikipedia.org/wiki/Virtual_reality

[2] http://video.golem.de/games/8816/oculus-rift-virtual-reality-brille-%28kickstarter%29.html

[3] http://gematsu.com/2016/03/bandai-namco-opening-vr-zone-tokyo-april-15

[4] S. Roudavski, “VIRTUAL ENVIRONMENTS AS SITUATED TECHNO- SOCIAL PERFORMANCES Virtual West Cambridge case-study,” 2010.

[5] http://www.army.mil/article/84453/

[6] D. White, K. Burdick, G. Fulk, J. Searleman, and J. Carroll, “A virtual reality application for stroke patient rehabilitation,” IEEE Int. Conf. Mechatronics Autom. 2005, vol. 2, no. July, pp. 1081–1086, 2005.

 

 

Medical Augmented Reality

Health care is one of the application areas where augmented reality can be applied by all actors of the field. AR is already widely used in training of new doctors. There are already lots of applications for medical education. Also, patients are benefiting from therapy applications, helping them to regain full functionality of their body. In the future, AR will have a more profound impact on the health care sector. Doctors will use AR to conduct surgeries more efficient and with more accuracy than today. [1]

Medical education

Learning how the human body works is difficult. Medical students not only need to know how all the parts of the body function, but where they are located. Since the human corps is a 3 dimensional object, augmented reality makes it easier to visualize it compared to a 2 dimensional screen or piece of paper.

At the Build 2016 Keynote, Microsoft demonstrated an application for its AR glasses called “HoloLens”, which is used in medical education. Students are able to see a model of the human body including the relevant parts for the course. Through gestures the model can be manipulated. For example, it is possible to zoom in/out, rotate the model or switch between the parts, which should be visible. Besides interaction, it is possible to collaborate in a group. A model can be manipulated by more than one person. This person not necessary needs to be in the same room. With the application for HoloLens, it is possible to work with people in different geographical locations. Learning the functions of the human body with a 3D model compared with studying with books saves valuable time and allows students to understand functions more efficient. [2]

Video: Microsoft HoloLens: Build 2016 Keynote. Medical education application demonstrated at 6:00 to 10:40.

Rehabilitation

Patients are an additional user group, which can benefit from AR in the health care are. For example, AR can be used in the stroke rehabilitation process. Patients need intensive treatment, usually with a therapist. To lower therapy costs, patients practice arm and hand movements with an AR application instead of with a therapist. The app cannot fully replace a therapist. Compared with virtual reality solutions, an AR system feels more natural because the patient is able to see its own body and interact more intuitively. [3]

In 2010 already, a group of researchers developed a prototype of a game for rehabilitation of the upper-limb. Objects on a table and the hand movements are recognized by a webcam. Different movements can be practiced with various objects. The equipment for the training is affordable and is not only suitable for rehabilitation in a hospital, but also for home use.

[1] http://www.healthtechevent.com/technology/augmented-reality-revolutionizing-medicine-healthcare/

[2] http://medicalaugmentedreality.com/2016/03/microsoft-hololens-build-2016-keynote-medical-demo/

[3] Mousavi, H., Khademi, M., & Dodakian, L. (2013). A Spatial Augmented Reality Rehab System for Post-Stroke Hand Rehabilitation, 279–285.

Application Areas of Augmented Reality

Augmented reality can be applied in numerous areas. While the first AR systems were developed for military, industrial and medical fields, they also appear in commercial and entertainment. [3] suggest to classify the possible areas of application of AR as follows:

  • Personal information systems
  • Industrial and military applications
  • Medical applications
  • AR for entertainment
  • AR for the office
  • Education and training

Personal information systems

Personal wearable computing is one of the biggest potential markets for AR [3]. Firstly, they can be used for personal assistance and advertisement. Devices to automatically store inputs from the real world like conversations, people or just any information which might be useful for the user are stored and displayed at a convenient time. Navigation is another field of appliance for AR. For example, virtual signs lead the driver of a car or provide warnings when potential hazards arrive.

Industrial and military applications

Besides design, assembly and maintenance, AR is useful for combat simulation, too. To experience a product before building a prototype is an advantage of augmented reality. Using this technology in the design phase of a car therefore helps to cut costs. It is also used in the manufacturing process of cars as well as aircrafts. Important information is displayed to workers in real time and without interrupting the working progress.

Medical applications

As stated in the section before, information can be displayed to medical personnel, too. AR can be used during surgeries, showing information about a patient to a nurse or for visualizing CT and MR scans.

Augmented-Reality-in-Healthcare
[4]
AR for entertainment

There are two main areas in entertainment where AR can be applied. These areas are games as well as sports broadcasting. While VR is mainly used for indoor gaming, AR games can also take place outdoors. Examples for such games are “Ingress” where the competitors have to hack and build up virtual portals in the real world or “Zombie run!” in which the player has to find a way to get away from zombies. [2] In sports broadcasting objects like players or a ball can be tracked and highlighted with AR technology. This technology is used especially for analyzing certain key plays and tactical behavior.

AR for the office

In the office, AR can be useful for collaboration. Working together on a subject might be more comfortable if the workers see a 3D model of the object. Also remote collaboration becomes easier and more efficient.

Education and training

The last area of application is education and training. As mentioned in the previous section, virtual models also support the learning process and help to understand complex matters. Math, geometry or engineering are only a few courses which could benefit from AR technology.

 

Game-based learning is only one example of how AR is used in different of the above mentioned areas and shows that there are no exact boundaries between these areas of application. As augmented reality is being developed further, the possibility for its usage become more and more diverse.

 

[1] D.W.F. van Krevelen, R. Poelman. A Survey of Augmented Reality – Technologies, Applications and Limitations. The International Journal of Virtual Reality, 2010, 9(2):1-20

[2] http://www.androidauthority.com/best-ar-apps-and-games-for-android-augmented-reality-584616/. Accessed 28.03.2016

[3] T. H. Höllerer and S. K. Feiner. Mobile Augmented Reality. In H. Karimi and A. Hammad, editors, Telegeoinformatics: Location-Based Computing and Services. CRC Press, Mar. 2004. ISBN 0-4153-6976-2.

[4] http://www.augmentedrealitytrends.com/augmented-reality/healthcare-industry.html. Accessed 28.03.2016

How does AR work? Different approaches

The aim of augmented reality is to bring graphics, audio etc. to the real world in real time. For example for apps on a smartphone are two different approaches in use, the marker-based or image recognition and location or GPS-based approach.[1]

Marker-based

Marker-based applications are using a camera to overlay the image with content or information. This is happening in real time. The camera feed is processed and the real image is augmented by some content or information. Markers are for example QR codes. If you are scanning an QR-code and you are getting additional information on your screen than you are using a marker-based augmented reality application. Marker based applications are most common in augemented reality because it is much more simple to recognize something what is hard coded in your application.

For example Nissan made a application where you can see one of their cars in augmented reality by just using your webcam and a printed wheel. The demo is under the following link:

http://takethewheel.nissan.com.au/

Location-based[2]

Location-based applications are using the GPS tools in the device to determine the users position. The position and the direction of the user are the input parameter for the app to show at what the user is looking at.

For example:

[1] https://www.appfutura.com/blog/whats-augmented-reality-and-how-does-it-work/

[2] Location-based augmented reality on mobile phones, R. Paucher ; M. Turk IEEE, DOI 10.1109/CVPRW.2010.5543249