Augmented Reality UX Principles we can learn from Fragments for Hololens

AR UX Principles we can learn from Fragments

Fragments provides one of the most delightful user experiences for gaming in AR, and the following are UX principles we can learn from what the game does well that can be applied to other augmented reality experiences.

Physical adaptation of digital assets

User experience in augmented reality games generally follows the pattern set by console games, with some spatial adjustments. Menus in console games are displayed via a start or select button, and generally pull the player out of the experience by pausing the game. In Fragments, menus are always visibly pinned to the wall during gameplay, and are accessible at any point. This means that the player never truly “leaves” the game, and participates in a much smoother experience.

Occlusion of digital assets behind real-life objects

Fragments makes fantastic use of the Hololens’ occlusion capabilities. Occlusion is the hiding of digital assets behind real-world objects. This is no easy feat, as it requires an incredibly accurate map of the environment before the game begins, and Fragments does an excellent job of getting this right. It makes the experience of augmented reality that much more real, forcing the player to look for clues around obstructions, beneath tables, and in crevices, much like a real-life detective.

Giving the user autonomy in the space

While console games provide limited controls for how and where a player interacts with objects, AR games give players much more autonomy. Using Uncharted 4 as a comparison, it’s obvious that object interaction is significantly more structured in a console game, with limited viewing and selection capabilities. Once a player selects an object, a menu appears with a set of control options to indicate ways for the player to interact with and exit viewing the object.

In Fragments, object interaction is simpler and more natural, as it involves only gazing and air-tapping. These simplified gestures mimic the experience of lifting and analyzing objects in real life. Additionally, objects are anchored to a physical location in space in AR, so the player may step away from the object or walk through it. This means that object views require an extra layer of intelligence, and Fragments automatically exits the view mode of objects which the player walks away from. By accounting for unusual player behavior, AR games give the player more autonomy in how they navigate through the space, and thus provide for a more realistic and natural user experience.

Mimicking natural experiences and gestures

The unique controls of the Hololens mimic and simplify real-life experiences. This is perhaps best depicted by the player’s experience of piloting a drone. The player’s gaze controls the drone’s vertical and horizontal position in the plane of the game, while air-tapping controls its position relative to the player. Holding a left-hand air-tap brings the drone closer to the player, while holding a right-hand air-tap moves it farther away.

This was the only section that required two hands to play, and created a complex mechanic that had players navigate both physical and digital obstacles. I was particularly impressed with the inventive way that this was handled in Fragments. Using a movement system that focused entirely on the position relative to the player provided for a delightful user experience that mimicked controlling a drone or remote-controlled plane in real life.

Guiding the user with help text and audio cues

Because of the autonomy it gives the player in moving through the game environment, Fragments employs several strategies to orient the player and ensure that they are able to navigate the game and follow the story.

Using arrows that follow the field of view, it prompts the player to move through the space with help text such as “Look Here” and “Go Here,” even specifying if the desired location is behind a physical obstruction. The game also provides audio feedback on the majority of interactable objects, providing greater backstory and hints to keep the player on track. These features enable Fragments to give the player freedom in navigating through the story, while providing guidance to help them adapt to augmented reality.

Indicating interactions with visual feedback

Similar to traditional console games, Fragments provides visual feedback when objects are able to be interacted with. The 4-arrow cursor expands to surround an interactable object with 8 arrows, sometimes even spinning to direct attention to something that can be further analyzed. This selection technique takes advantage of the familiar interface console gamers are used to, and adds an elegant layer of animated feedback to make it more useful for the player in AR.

Providing voice control

Though replacing a controller with the player’s body simplifies the core aspects of gameplay in Fragments, using the head as a cursor and constantly lifting an arm to air-tap can get tiresome over longer periods of time.

In this regard, voice plays a major role in the game, giving the player the option to make their way through it using vocal cues. Voice control is used extensively in Fragments, with a mic icon indicating actions that can be activated vocally appearing near almost every action. This also comes in handy for activating tools and menus used to gather clues, as navigating the space via voice is much more convenient than utilizing the menu pinned to the wall for each new action.

Improving on the UX of Fragments

Increasing accessibility with background contrast

When an object is being viewed in Fragments, key evidence discovered in the game is highlighted in orange and identified with annotations. This is useful when drawing conclusions from clues, as when a player interacts with each piece of evidence, relevant clues they have already gathered are automatically displayed.

Though these clue annotations are no doubt effective in aiding the player through the game, they would be easier to read if they appeared in bolder text against a darker background. Additionally, selection arrows in a different color could indicate that whether evidence can be further analyzed, and more annotations could appear while the object is selected with the player’s gaze. This could be implemented for an easier and more accessible version of the game, as it simplifies reviewing evidence to reach conclusions. These changes would retain the sleek and professional UI of the game, while increasing the contrast and providing the player with a better experience.

Limitations of AR Technology

Some of the major pitfalls of the Hololens come from an incorrect scan of the room before the game begins, which can lead to improper placement of AR assets and thus an unsatisfying game experience. Mirrors, windows, and other non-opaque surfaces also cause glitches, sometimes leading the game to expand the space to undesirable areas.

More general disadvantages include using the head as a cursor, using air-tap as the main interaction tool, and the weight of the Hololens, which can cause a minor headache when worn for extended periods of time. As AR becomes more and more mainstream, these problems will all be solved with advances in spatial mapping, eye-tracking technology, and AR headsets.