UX consultant
project
AR testing and research for Sky Glass
Role
Skills
User interviews
Qualitative research
Interview analysis
Time
2 months
Year
2023
challenge
Creating the right conditions to test and evaluate AR games
Sky had developed a series of AR games, designed for 5-12-year-olds, to be integrated into their new Sky Live platform. Our team was approached to evaluate the games and provide key insights to inform Sky's investment decisions, determining which gaming experiences were worth pursuing and which ones needed refinement or discontinuation. At the core of this project was a series of usability tests focusing on the following questions:
- Did participants understand the objectives of the game?
- Did they respond well to the game mechanics?
- Did the camera accurately map to the body and head movements of the participants?
- What was the game experience like?
- Were they engaged?
- Are the games set at the right level?
- What aspects did they like and dislike?
- What changes could be made to improve the experience?
the framework
Developing an evaluative framework to identify the right issues and areas for improvement
Creating AR games for a new generation of TV boxes is a unique challenge, requiring a tailored evaluative framework. I needed to create a structured approach to evaluate both participants and the findings.
The framework revolved around three main areas:
The framework revolved around three main areas:
- Did the user understand the game, instructions and required actions/movements?
- How did the game mechanics work?
- What was the overall game experience like?
As the Sky product team were looking to gain quantitative insight as well as qualitative information from this study, these areas had a series of features that were each rated according to "Success", "Partial Success", or "Fail", with corresponding numeric scores used to assess each aspect of the game. Once the framework was in place, we were ready to begin testing.
usability testing
The nuances of in-person testing with children
Based on previous experience, we knew that testing products with children often ran better when the children came along as friendship or sibling pairs or groups (e.g., siblings with siblings or best friends with best friends). When together, children tend to be more relaxed, honest, and interact with products more naturally. It also creates an environment where their interactions mirror real-life scenarios, offering organic discussions and revealing nuances that might otherwise remain undiscovered in one-on-one settings.
As a result, we tested the games in twelve separate user sessions, with participants coming in pairs and sometimes trios. A combined total of twenty-six participants were involved.
As a result, we tested the games in twelve separate user sessions, with participants coming in pairs and sometimes trios. A combined total of twenty-six participants were involved.
analysis and findings
The intricacies of designing for AR
Soon into the sessions it was clear that designing Augmented Reality (AR) games presents unique challenges.
Unlike traditional games, AR games require examining body movements and shapes to ensure accurate mapping of real-world elements to a virtual world. Factors such as room size and the positioning of cameras or sensors can significantly impact how the games are played and experienced. For instance, if a room is too short or narrow, certain body movements can be restricted, preventing the game from functioning as intended. Additionally, if a person is particularly short or tall or has very short or long arms, this can influence how sensors pick up their body movements. Other factors, such as the presence of non-players moving behind or in front of the cameras, can also affect the experience. Sometimes, sensors or cameras mistakenly lock onto non-players, disrupting the gaming experience for the user.
These factors influenced how participants experienced the games. Some pairs struggled to join the games together because the sensor did not recognise a second player standing in front of the camera, particularly if they were relatively small. Certain games failed to pick up some body movements, while others incorrectly registered body movements, not recognising them as the correct technique. In some cases, in-game objects that participants were supposed to "touch" or "grab" were impossible to reach, either because the participants’ arms were too short or because they required hitting a wall to grab them.
Unlike traditional games, AR games require examining body movements and shapes to ensure accurate mapping of real-world elements to a virtual world. Factors such as room size and the positioning of cameras or sensors can significantly impact how the games are played and experienced. For instance, if a room is too short or narrow, certain body movements can be restricted, preventing the game from functioning as intended. Additionally, if a person is particularly short or tall or has very short or long arms, this can influence how sensors pick up their body movements. Other factors, such as the presence of non-players moving behind or in front of the cameras, can also affect the experience. Sometimes, sensors or cameras mistakenly lock onto non-players, disrupting the gaming experience for the user.
These factors influenced how participants experienced the games. Some pairs struggled to join the games together because the sensor did not recognise a second player standing in front of the camera, particularly if they were relatively small. Certain games failed to pick up some body movements, while others incorrectly registered body movements, not recognising them as the correct technique. In some cases, in-game objects that participants were supposed to "touch" or "grab" were impossible to reach, either because the participants’ arms were too short or because they required hitting a wall to grab them.
Additional insights
From the importance of intelligent sensors to the competitive advantage AR games hold
Beyond the idiosyncratic factors involved in AR games, there were several other valuable insights from the tests. These included:
- Clear instructions, tutorials and guides to how sensors track body movements and the techniques or movements that would enable the gamer to master the game
- Finding the right difficulty level. Games that were too easy failed to maintain the gamers' interest over time, while games that were too difficult caused frustration.
- The importance of camera sensors maintaining focus on the active player, even if other people were in the camera’s line of sight. This is especially crucial when onlookers are sitting in front of the TV, as asking them to move out of their seats reduced their enjoyment of watching their friends or siblings play.
- Participants found the ability of AR games to facilitate movement and exercise particularly enjoyable, which seemed to be a key competitive advantage of AR games over traditional games.
