Augmented reality

Augmented reality (AR) is a live, direct or indirect, view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data. It is related to a more general concept called mediated reality, in which a view of reality is modified (possibly even diminished rather than augmented) by a computer. As a result, the technology functions by enhancing one’s current perception of reality. By contrast, virtual reality replaces the real world with a simulated one. Augmentation is conventionally in real-time and in semantic context with environmental elements, such as sports scores on TV during a match. With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally manipulable. Artificial information about the environment and its objects can be overlaid on the real world.

Hardware
The main hardware components for augmented reality are: processor, display, sensors and input devices. Modern mobile computing devices like smartphones and tablet computers contain these elements which often include a camera and MEMS sensors such as accelerometer, GPS, and solid state compass, making them suitable AR platforms.

Display
Various technologies are used in Augmented Reality rendering including optical projection systems, monitors, hand held devices, and display systems worn on one's person such as head mounted displays.

Head-mounted
A head-mounted display (HMD) is a display device paired to a headset such as a harness or helmet. HMDs place images of both the physical world and virtual objects over the user's field of view. Modern HMDs often employ sensors for six degrees of freedom monitoring that allow the system to align virtual information to the physical world. The main advantage of HMD AR is the user's immersive experience. The graphical information is coupled to the view of the user and adjusts accordingly with their head movements.

Eye Glasses
AR displays can be rendered on devices resembling eyeglasses. Versions include eye wear that employs cameras to intercept the real world view and re-display it's augmented view through the eye pieces and devices in which the AR imagery is projected through or reflected off the surfaces of the eye wear lens pieces.

Contact Lenses
Contact lenses that display AR imaging are in development. These lenses contain the elements for display embedded into the lens including integrated circuitry, LEDs and an antenna for wireless communication. Another version of contact lenses, in development for the U.S. Military, is designed to function with AR spectacles, allowing soldiers to focus on close-to-the-eye AR images on the spectacles and distant real world objects at the same time.

Virtual Retina Display
A virtual retinal display (VRD) is a personal display device under development at the University of Washington's Human Interface Technology Laboratory. With this technology, a display is scanned directly onto the retina of a viewer's eye. The viewer sees what appears to be a conventional display floating in space in front of them.

Handheld
Handheld displays employ a small display that fits in a user's hand. All handheld AR solutions to date opt for video see-through. Initially handheld AR employed fiduciary markers, and later GPS units and MEMS sensors such as digital compasses and six degrees of freedom accelerometer–gyroscope. Today SLAM markerless trackers such as PTAM are starting to come into use. Handheld display AR promises to be the first commercial success for AR technologies. The two main advantages of handheld AR is the portable nature of handheld devices and ubiquitous nature of camera phones. The disadvantages are the physical constraints of the user having to hold the handheld device out in front of them at all times as well as distorting effect of classically wide-angled mobile phone cameras when compared to the real world as viewed through the eye.

Spatial
Spatial Augmented Reality (SAR) augments real world objects and scenes without the use of special displays such as monitors, head mounted displays or hand-held devices. SAR makes use of digital projectors to display graphical information onto physical objects. The key difference in SAR is that the display is separated from the users of the system. Because the displays are not associated with each user, SAR scales naturally up to groups of users, thus allowing for collocated collaboration between users. SAR has several advantages over traditional head-mounted displays and handheld devices. The user is not required to carry equipment or wear the display over their eyes. This makes spatial AR a good candidate for collaborative work, as the users can see each other’s faces. A system can be used by multiple people at the same time without each having to wear a head-mounted display.

Examples include shader lamps, mobile projectors, virtual tables, and smart projectors. Shader lamps mimic and augment reality by projecting imagery onto neutral objects, providing the opportunity to enhance the object’s appearance with materials of a simple unit- a projector, camera, and sensor. Handheld projectors further this goal by enabling cluster configurations of environment sensing, reducing the need for additional peripheral sensing.

Other tangible applications include table and wall projections. One such innovation, the Extended Virtual Table, separates the virtual from the real by including beam-splitter mirrors attached to the ceiling at an adjustable angle. Virtual showcases, which employ beam-splitter mirrors together with multiple graphics displays, provide an interactive means of simultaneously engaging with the virtual and the real. Altogether, current augmented reality display technology can be applied to improve design and visualization, or function as scientific simulations and tools for education or entertainment. Many more implementations and configurations make spatial augmented reality display an increasingly attractive interactive alternative.

Spatial AR does not suffer from the limited display resolution of current head-mounted displays and portable devices. A projector based display system can simply incorporate more projectors to expand the display area. Where portable devices have a small window into the world for drawing, a SAR system can display on any number of surfaces of an indoor setting at once. The drawbacks, however, are that SAR systems of projectors do not work so well in sunlight and also require a surface on which to project the computer-generated graphics. Augmentations cannot simply hang in the air as they do with handheld and HMD-based AR. The tangible nature of SAR, though, makes this an ideal technology to support design, as SAR supports both a graphical visualisation and passive haptic sensation for the end users. People are able to touch physical objects, and it is this process that provides the passive haptic sensation.

Tracking
Modern mobile augmented reality systems use one or more of the following tracking technologies: digital cameras and/or other optical sensors, accelerometers, GPS, gyroscopes, solid state compasses, RFID and wireless sensors. These technologies offer varying levels of accuracy and precision. Most important is the position and orientation of the user's head. Tracking the user's hand(s) or a handheld input device can provide a 6DOF interaction technique.

Input devices
Techniques include speech recognition systems that translate a user's spoken words into computer instructions and gesture recognition systems that can interpret a user's body movements by visual detection or from sensors embedded in a peripheral device such as a wand, stylus, pointer, glove or other body wear.

Computer
The computer analyzes the sensed visual and other data to synthesize and position augmentations.

Software and algorithms
A key measure of AR systems is how realistically they integrate augmentations with the real world. The software must derive real world coordinates, independent from the camera, from camera images. That process is called image registration which uses different methods of computer vision, mostly related to video tracking. Many computer vision methods of augmented reality are inherited from visual odometry. Usually those methods consist of two parts.

First detect interest points, or fiduciary markers, or optical flow in the camera images. First stage can use feature detection methods like corner detection, blob detection, edge detection or thresholding and/or other image processing methods. The second stage restores a real world coordinate system from the data obtained in the first stage. Some methods assume objects with known geometry (or fiduciary markers) present in the scene. In some of those cases the scene 3D structure should be precalculated beforehand. If part of the scene is unknown simultaneous localization and mapping (SLAM) can map relative positions. If no information about scene geometry is available, structure from motion methods like bundle adjustment are used. Mathematical methods used in the second stage include projective (epipolar) geometry, geometric algebra, rotation representation with exponential map, kalman and particle filters, nonlinear optimization, robust statistics.

Applications
Augmented reality has many applications, and many areas can benefit from the usage of AR technology. AR was initially used for military, industrial, and medical applications, but was soon applied to commercial and entertainment areas as well.

Archaeology
AR can be used to aid archaeological research, by augmenting archaeological features onto the modern landscape, enabling archaeologists to formulate conclusions about site placement and configuration

Another application given to AR in this field is the possibility for users to rebuild ruins, buildings, or even landscapes as they formerly existed.

Architecture
AR can aid in visualizing building projects. Computer generated images of a structure can be superimposed into a real life local view of a property before the physical building is constructed there. AR can also be employed within an architect's work space, rendering into their view animated 3D visualizations of their 2D drawings. Architecture sight-seeing can be enhanced with AR applications allowing users viewing a building's exterior to virtually see through its walls viewing it's interior objects and layout.

Art
AR can help create art in real time integrating reality such as painting, drawing and modeling. AR art technology has helped disabled individuals to continue pursuing their passion. Recently, Alaskan artist Nathan Shafer created a global warming-oriented AR project, called Exit Glacier AR Terminus. In this project, AR technology (is this case, smartphones) walks the viewer through various positions of South Central Alaska's Exit Glacier, going back three decades. Belgian photographer, Liesje Reyskens recently commissioned Augmented Reality Agency Kudanto produce an AR art installation at the Albus Lux gallery - the AR app recognized her photography and overlaid live animation video to enhance the gallery experience.

An item such as a commemorative coin can be designed so that when scanned by an AR enabled device it displays additional objects and layers of information that were not visible in a real world view of it.

Commerce
AR can enhance product previews such as allowing a customer to view what's inside a product's packaging without opening it. AR can also be used as an aid in selecting products from a catalog or through a kiosk. Scanned images of products can activate views of additional content such as customization options and additional images of the product in its use. AR is used to integrate print and video marketing. Printed marketing material can be designed with certain "trigger" images that, when scanned by an AR enabled device using image recognition, activate a video version of the promotional material.

Education
Augmented reality applications can complement a standard curriculum. Text, graphics, video and audio can be superimposed into a student’s real time environment. Textbooks, flashcards and other educational reading material can contain embedded “markers” that, when scanned by an AR device, produce supplementary information to the student rendered in a multimedia format. Students can participate interactively with computer generated simulations of historical events, exploring and learning details of each significant area of the event site. Augmented reality technology also permits learning via remote collaboration, in which students and instructors not at the same physical location can share a common virtual learning environment populated by virtual objects and learning materials and interact with another within that setting.

Industrial Design
In the area of industrial design AR can provide crucial help, AR can help designer experience the final product before is complete or can help with the testing part of it. Volkswagen is already using AR for comparing calculated and actual crash test imagery. But AR can also be used to visualize and modify a car body curvature or the engine layout of it. AR can also be used to compare digital mock-ups with physical mock-ups for efficiently finding discrepancies between them.

Medical
AR can provide the surgeon with information of the heartbeat, the blood pressure, the state of the patient’s organ, etc. It can also help the doctor identify the problem with the patient right away. This approach works in a similar as the technicians doing maintenance work. Examples include a virtual X-ray view based on prior tomography or on real time images from ultrasound and confocal microscopy probes or open NMR devices. AR can enhance viewing a fetus inside a mother's womb. See also Mixed reality.

Military
In combat, AR can serve as a networked communication system that renders useful battlefield data onto a soldier's goggles in real time. From the soldier's viewpoint, people and various objects can be marked with special indicators to warn of potential dangers. Virtual maps and 360° view camera imaging can also be rendered to aid a soldier's navigation and battlefield perspective, and this can be transmitted to military leaders at a remote command center.

Navigation
AR can augment the effectiveness of navigation devices. Information can be displayed on the car's windshield indicating information of where the user is going. Not only can information of how to get to the place be offered but so can information about the weather or the terrain. AR can provide traffic information to drivers as well as alert the driver in case of an emergency or highlight objects on the road that might not be caught by the driver’s eyes at a first glance. It can also be used in the sea where fishermen can use the technology to display information about the amount of fish that are in the area and how to get to them.

Currently some car manufacturers (e.g. BMW and GM) are using this technology in car windshields to display meter information and traffic information.

Aboard naval and maritime vessels, AR can allow bridge watch-standers to continuously monitor important information such as a ship's heading and speed while moving throughout the bridge or performing other tasks.

Office Workplace
AR can help facilitate collaboration among distributed team members in a work force via conferences with real and virtual participants. AR tasks can include brainstorming and discussion meetings utilizing common visualization via touch screen tables, interactive digital whiteboards, shared design spaces, and distributed control rooms.

Sports & Entertainment
AR has become common in sports telecasting. Sports and entertainment venues are provided with see-trough and overlay augmentation through tracked camera feeds for enhanced viewing by the audience. Examples include the yellow "first down" line seen in television broadcasts of American football games showing the line the offensive team must cross to receive a first down. AR is also used in association with football and other sporting events to show commercial advertisements overlayed onto the view of the playing area. Sections of rugby fields and cricket pitches also display sponsored images. Swimming telecasts often add a line across the lanes to indicate the position of the current record holder as a race proceeds to allow viewers to compare the current race to the best performance. Other examples include hockey puck tracking and annotations of racing car performance and snooker ball trajectories.

AR can enhance concert and theater performances. For example, artists can allow listeners to augment their listening experience by adding their performance to that of other bands/groups of users.

The gaming industry has benefited a lot from the development of this technology. A number of games have been developed for prepared indoor environments. Early AR games also include AR air hockey, collaborative combat against virtual enemies, and an AR-enhanced pool games. A significant number of games incorporate AR in them and the introduction of the smartphone has made a bigger impact.

Task support
Complex tasks such as assembly, maintenance, and surgery can be simplified by inserting additional information into the field of view. For example, labels can be displayed on parts of a system to clarify operating instructions for a mechanic who is performing maintenance on the system. Assembly lines gain many benefits from the usage of AR. In addition to Boeing, BMW and Volkswagen are known for incorporating this technology in their assembly line to improve their manufacturing and assembly processes. Big machines are difficult to maintain because of the multiple layers or structures they have. With the use of AR the workers can complete their job in a much easier way because AR permits them to look through the machine as if it was with x-ray, pointing them to the problem right away.

Tourism and sightseeing
Augmented reality is used in applications for tourism, including highlighting information of important places and providing connections between the real world and historic events, such as rendering historical events into a view of a current landscape. The use of AR in this area has enhanced the experience of users when they go traveling by providing information of the place they are at as well as comments made by other users that have been there before. AR in the tourism industry can connect with different platforms to provide a richer experience to the final user of the system.

Translation
AR systems can interpret foreign text on signs and menus and, in a user's augmented view, re-display the text in the user's language. Spoken words of a foreign language can be translated and displayed in a user's view as printed subtitles.

Notable researchers

 * Ivan Sutherland invented the first AR head-mounted display at Harvard University.
 * Steven Feiner, Professor at Columbia University, is a leading pioneer of augmented reality, and author of the first paper on an AR system prototype, KARMA (the Knowledge-based Augmented Reality Maintenance Assistant), along with Blair MacIntyre and Doree Seligmann.
 * L.B. Rosenberg developed one of the first known AR systems, called Virtual Fixtures, while working at the U.S. Air Force Armstrong Labs in 1991, and published first study of how an AR system can enhance human performance.
 * Dieter Schmalstieg and Daniel Wagner jump started the field of AR on mobile phones. They developed the first marker tracking systems for mobile phones and PDAs.
 * Bruce H. Thomas and Wayne Piekarski develop the Tinmith system in 1998. They along with Steve Feiner with his MARS system pioneer outdoor augmented reality.
 * Reinhold Behringer performed important early work in image registration for augmented reality, and prototype wearable testbeds for augmented reality. He also co-organized the First IEEE International Symposium on Augmented Reality in 1998 (IWAR'98), and co-edited one of the first books on augmented reality.

History

 * 1901: L. Frank Baum, an author, first mentions the idea of an electronic display/spectacles that overlays data onto real life (in this case 'people'), it's named a 'character marker'.
 * 1957–62: Morton Heilig, a cinematographer, creates and patents a simulator called Sensorama with visuals, sound, vibration, and smell.
 * 1966: Ivan Sutherland invents the head-mounted display and positions it as a window into a virtual world.
 * 1975: Myron Krueger creates Videoplace to allow users to interact with virtual objects for the first time.
 * 1989: Jaron Lanier coins the phrase Virtual Reality and creates the first commercial business around virtual worlds.
 * 1992: L.B. Rosenberg develops one of the first functioning AR systems, called Virtual Fixtures, at the U.S. Air Force Research Laboratory—Armstrong, and demonstrates benefits to human performance.
 * 1992: Steven Feiner, Blair MacIntyre and Doree Seligmann present the first major paper on an AR system prototype, KARMA, at the Graphics Interface conference.
 * 1993 A widely cited version of the paper above is published in Communications of the ACM - Special issue on computer augmented environments, edited by Pierre Wellner, Wendy Mackay, and Rich Gold.
 * 1993: Loral WDL, with sponsorship from STRICOM, performed the first demonstration combining live AR-equipped vehicles and manned simulators. Unpublished paper, J. Barrilleaux, "Experiences and Observations in Applying Augmented Reality to Live Training", 1999.
 * 1994: Julie Martin creates first 'Augmented Reality Theater production', Dancing In Cyberspace, funded by the Australia Council for the Arts, features dancers and acrobats manipulating body–sized virtual object in real time, projected into the same physical space and performance plane. The acrobats appeared immersed within the virtual object and environments. The installation used Silicon Graphics computers and Polhemus sensing system.
 * 1998: Spatial Augmented Reality introduced at University of North Carolina at Chapel Hill by Raskar, Welch, Fuchs.
 * 1999: Hirokazu Kato (加藤 博一) created ARToolKit at HITLab, where AR later was further developed by other HITLab scientists, demonstrating it at SIGGRAPH.
 * 2000: Bruce H. Thomas develops ARQuake, the first outdoor mobile AR game, demonstrating it in the International Symposium on Wearable Computers.
 * 2008: Wikitude AR Travel Guide launches on 20 Oct 2008 with the G1 Android phone.
 * 2009: ARToolkit was ported to Adobe Flash (FLARToolkit) by Saqoosha, bringing augmented reality to the web browser.
 * 2009: project SixthSense from MIT showcased projection based wearable AR pendent device.
 * 2011 : LASTER Technologies, a French start-up from University of Paris Sud (Orsay), has developed first augmented reality ski goggle in the market, demonstrating it at SIGGRAPH.