Planetrek - Providing an immersive learning experience for students with visual or hearing impairments

The inclusive education app about astronomy offers an immersive learning experience

 CLIENT Institute of Design (Educational purpose)

Challenge

Create a mobile experience that provides a more immersive astronomy class while addressing the needs of a variety of students, including those with visual and hearing impairments.

Solution

A smartphone app interface with AR, AI, VoiceOver, and real-time subtitles. Not only does it provide an equitable learning experience optimized for the needs of different students, but it also offers an immersive experience to observe the planets and stars virtually and closely.

As a part of the Introduction to Interaction Design class, we were tasked to design an interactive and accessible educational experience on topical causes, characteristics, and effects of seasonal change in Chicago. We were assigned to work on the topic of astronomy and crafted one educational experience, aimed at a 5-10 minute experience for sixth through eighth-grade life science students, provided through several different interactive experiences to support diverse audiences. Our team was assigned to work on astronomy.

Approximately 6.8% of children under 18 have vision-related challenges, and an estimated 1 in 5 American teens experiences some degree of hearing loss. The traditional classroom structure, based on textbooks and lectures, does not accommodate for students with these challenges. Only a small number of students with these challenges have access to an individualized education program.

How might we provide an immersive learning experience about astronomy for students with visual or hearing impairments?

Role

UI/UX Designer

Collaborator

Sunaina Kuhn

Instructor

Zach Pino

Project Duration

October 2021 - November 2021 (in Chicago)

Outcome

With AR, AI, VoiceOver, and real-time subtitles, the app enables students to identify specific stars in the sky, understand how old the star you are looking at is, and move through distance (exploring stars you cannot see with your naked eyes) and time (exploring how stars looked/will look in the past/future).

Click/tap chevrons (< >) to switch images

Approach

1. Understand 

Literature reviews

To better understand the concept of Inclusive Design and Accessibility, we read “Mismatch : How Inclusion Shapes Design” by Kat Holmes and “Accessible America : A History of Disability And Design“ by Bess Williamson and then discussed key take-aways and questions in class.

Product reviews

We studied existing apps or physical products that offer immersive learning experiences. Our team analyzed the Museum Alive AR app. Then we shared and discussed our insights and hypothesis of what those products are trying to achieve.

2. Define 

Defining our app concepts and features

Thanks to contributions from my teammate Sunaina, who has a background in teaching and is familiar with potential users’ desires and needs, we came up with the following App concepts and specific features.

• Making the abstract directly observable

  Students may find written materials unrealistic and difficult to understand for a topic like astronomy due to the foreign and abstract nature. Using AR to address such stressors, students observe celestial bodies in the simulated sky while Planetrek reads explanations out loud.

• Facilitating cross-topic exploration

  We connect astronomy to other subjects like history and the environment. Through celestial exploration and time manipulation, students learn about the star’s history and formation, while simulating reduced visibility due to air or light pollution. Students can deepen their knowledge in one subject while extending their learning into others.

• Making learning more equitable for those with visual/hearing impairments

  Planetrek uses AI to interactively help guide the student if they struggle during an exercise. Closed captioning benefits non-native English students or those with hearing impairment. For visually impaired students, the system offers personalized spoken support, providing a comparable learning experience as those without. Additionally, students can control text size and color contrast as design features.

Creating Wire Flows

Based on the features we came up with, we drafted the user experiences with screen flows.

3. Develop

Creating UIs with Figma and getting them reviewed

We moved on to Figma and made series of user interfaces. We also prototyped AR and Voice Over functionalities with Figma animations so that we can better communicate our intents. Once our prototypes are done, we got them reviewed and received feedback from classmates.

4. Deliver

Finalizing our app based on the feedback