Prototype - Mixed Reality
Process Highlights
Design Overview and Challenges
Overview
Through augmented reality, specific knowledge skills can be democratized. Tasks that once required a specialist or long tutorials, can now be learnt and applied hands-on. In order to prove this concept, I designed an AR experience which enables the user to be able to fix any of their home appliances.
Process Highlights
Project Information
Timeline
July 2024 - August 2024
Skills
Figma, Photoshop, Blender, Unity
Research
"Young People Don't Know How To Do Anything Anymore"
Gen Z and Home Appliance Skills
There is a common notion that “young people don’t know how to do anything anymore”. In fact, this problem has been parodied on shows like South Park. My generation does not know the basic “life skills” that past generations grew up with, and this leads to a major problem for Gen Z.
Specific Knowledge
Home repairs and basic mechanical tasks have fallen by the wayside. This leads to even more demand for workers in these specific knowledge fields - leading to costly and lengthy repair times. Moreover, most of these tasks are difficult to automate as they require precision. AI can't help us with this (sort of).
AR/VR Democratizing Specialized Knowledge
By using AR, the skills required to repair home appliances are given to everyone. This firstly saves money for the user, and also encourages them to learn the skills themselves. Using spatial technology also makes it far easier to learn, as the tutorial is overlaid in your environment.
Research
So… The Main Problems:
Why Spatial Computing Transforms Specialized Skills
Many basic home repair skills are absent in younger generations
It is both costly and time consuming to hire a professional to fix appliances
Home appliances are some of the most overcharged repairs
Research
Survey Research
I asked my hometown and college friends a simple question
"What are the common reasons a fridge breaks?"
"I don't know, the battery?"
Computer Science Student
"Something to do with the fan inside that cools the fridge?"
Economics Student
"I'd just google it"
My Little Brother
"I'd call a professional to come help out"
Recent Grad
The Main Pain Points:
The research showed me that most of my generation doesn't know practical knowledge the same way other generations might have. However, we are more willing to learn through online resources. This means an AR solution to this problem would be welcomed.
Lack of Specific Knowledge
The practical knowledge of young people regarding household appliances has declined due to a shift away from mechanical engineering education. This has created a shortage of skilled repair professionals, disrupting market equilibrium. With fewer people trained in these essential skills, the demand for repairs exceeds the supply, leading to increased costs and inefficiencies in the service industry.
Home Repairs are Costly
Repairing household appliances has become a significant financial burden due to the high cost of labor and parts, exacerbated by a scarcity of qualified technicians. Homeowners often face the tough decision of whether to repair or replace an appliance, with repairs frequently being cost-prohibitive. This highlights the need for more affordable and accessible solutions in the home repair market.
Online Tutorials Don't Provide Enough Context
Online tutorials, while popular, often lack the interactive depth needed for successful home repairs. These resources are typically static, offering limited contextual guidance. Without real-time feedback or 3D visualization, learners may struggle to fully grasp complex mechanical tasks, leading to frustration and errors. This reveals the limitations of online tutorials for comprehensive learning.
AR is The Best Medium for Home Repairs
Augmented Reality (AR) offers a hands-free, immersive experience for home repairs, providing step-by-step guidance directly in the user's physical environment. AR overlays digital instructions with depth and spatial context, making repairs more intuitive and accessible. This technology empowers users to tackle home repairs independently, reducing reliance on costly professional services.
Research
The User Journey
Understanding the User Journey Through a Personal Problem
Recently, my mini fridge broke. I decided to try and solve this problem on my own, which led me to discovering the industry of home appliance repairs. I believe that AR applications will enable anyone to be able to fix their home appliances through guided repairs.
Research
Developing a Solution
AR and AI Integration into The Software
Integrating AI with AR in home repairs opens up new possibilities for real-time, interactive support. Multi-modal assistants powered by AI can provide instant feedback and answer questions as users work on repairs, enhancing the learning experience.
As AI technology improves, it will be able to generate more detailed 3D models, offering users even more precise guidance and context. This synergy between AR and AI transforms home repairs into a more accessible and less intimidating task, empowering users to tackle even complex issues with confidence.
Accessibility & Tutorial Integration
Advanced AR software, equipped with object detection, offers a highly personalized approach to home repairs. This technology acts as a "second brain," helping users understand the specialized skills needed for specific tasks. By recognizing the objects being repaired, AR can tailor tutorials and guidance to the user's exact situation, making the learning process more efficient and effective. This level of personalization ensures that even those with little technical knowledge can successfully complete repairs, bridging the gap between professional expertise and DIY efforts.
Design Goals
Enable hands-free repair assistance
Use of advanced software to provide personalization
Convey specialized skills easily
Synergy between AI assistant and AR environments
Ideation
Moodboard
Design System
I used Apple's VisionOS UI.
Gestural Interactions that Already Exist in Technology
A home repair tutorial would be most efficiently navigated through gestural interactions.
Repair Augmented Experiences
Investigating different repair apps and technical apps.
Ideation
Low Fidelity
Low Fidelity Phase
Here are the preliminary thoughts on the research and the app's goals. The low-fidelity prototypes offer an early view of the app's possible functions, highlighting how the essential features could be integrated.
Ideation
Mid Fidelity
Mid Fidelity
This stage was focused on ensuring the user experience of cooking in spatial environments aligned with the user needs of accessibility, multitasking, and hands-free interaction.
Inventory Window
An initial window that the user calls showing all the parts involved in the repair.
Steps Window
This window would be present as users navigate the steps in the tutorial. They could see detailed steps through written text and also the components involved.
Simplifying the Steps Window
I found that the user doesn't need extra written information about the steps. Most users do not care what a repair part is called and therefore would prefer a simpler interface.
Hand Menus
The user should be able to navigate through tools using a hand menu.
Control Through Gestures
Users can control components transformations through gestures.
Physical AI
Response animation
Physical AI
User animation
Ideation
High Fidelity
High Fidelity
This stage was to explore what the (few) windows would look like in the app.
Inventory Window
This window aimed to relay information about all the parts involved in a home appliance repair. It also allowed the user to "Add to Cart" a part they might be missing.
Steps Control
After deciding on a smaller size steps window, I iterated a couple ideas of how this would look.
Icons
Both filled and unfilled icons, alongside active and inactive states.
Ideation & Prototypes
Design Decisions & Prototypes
Voice Assistant
As generative AI becomes more common place in applications I thought that the app should include it as well. I experimented in 2D using some React code to understand how this generative AI could look.
3D and Text
However as I built the 2D UI in Unity, I realized that the sound waves should be physical and take up space. This led me to designing it in 3D instead, through a C# script. I also though the user should have visual text feedback to understand what the AI is saying.
3D Models (Icons and Components)
I built the 3D models of the fridge components that I needed for the repair. I also made the icons for the hand menu 3D, so that they had more weight.
Hand Menu
I designed a hand menu for hover effects when a user was underneath an icon. This made the hand menu far more dynamic than just a regular pop-up icon. It also enabled quick interactions for navigation accessibility purposes. When the hover state is above 60% of its max height, the icon is selected and the material turns grey.
Drawing and Hand Menus
In cases where a human assistant is required, the user can help direct the assistant to the problem through the use of 3D drawing. The user can also direct a multi-modal AI's attention to an area through drawing as well. The hand menu provides easy accessibility to the draw function.
Prototype
Prototyping in Unity
Prototyping the app design using various gestures and actions
Demo in Quest 3
The Inventory Window
Benefits
The user can clearly see all the parts involved in a repair. They can also investigate more into specific components they may be missing in their home appliances. Furthermore, users can see the components from all angles.
Considerations
In order to integrate this system, there would need to be a vast library of 3D models for all of these mechanical components. I believe generative AI will be able to make these models based on commonly found online diagrams and patents.
Prototyping Switching Between Components
Benefits
The user's navigation through the component menu is swift with the use of toggles.
Considerations
Cannot compare and contrast models. The animation may be considered jerky.
Rotation and Positioning
Benefits
Allows the user to easily explore possible components involved in the repairs. The user can visually understand a part by rotating or positioning it using hand gestures. This is faster and better for hands free repair guidance.
Considerations
The user may find this unintuitive and difficult to understand. The user may also find themselves accidentally interacting with these models, although this is solved through a hand menu toggling. Finally, a user can also customize these gestures and sensitivities
Snap to Position
Benefits
User can snap the location of the object they are inspecting to their hand by pinching the middle finger and thumb.
Considerations
The user can occasionally pinch the thumb and middle finger instead of the index, or possibly the hand tracking misreads the gesture.
Steps Window
Benefits
The steps window is small and doesn't detract from the UX. It also has a dynamic text animation.
Considerations
Lack of information for the user. Although, the user can always call the inventory for more information.
Visual Tutorial Steps
Benefits
The user can visually see (from all angles) a step in the tutorial. The animation also allows for consistent playback for more information.
Considerations
Load on the system. However, this can be solved through lower poly models. The animations can also be standardized across many different appliances.
Conclusions
Main Takeaways
Design Takeaways
Through this project, I explored a more futuristic design for spatial computing applications. I wanted to design something that may not be able to be built right now, but could be built in a couple years.
This project also reinforced my belief that spatial computing devices provide domain specific skills to everyone - and how this will be powerful in upskilling people worldwide.
Technical Takeaways
This was by far my most technical mixed reality prototype. All in all, I wrote over 6 different scripts, and hundreds of lines of code. I learned how gestural interactions work, how scripts can influence 3D objects, and animating through C#.
I realized that to be a product designer in Mixed Reality, the designer must be able to code. Otherwise, they are limiting their prototyping skills.
Next Steps
I wanted to institute object recognition to the program, although I was hesitant to train a ML model. Object recognition would allow home repairs to be even easier, as you would not need to interact as much with the AI agent. Also, companies could integrate these tutorials into their products for easier repairs.
I wanted to explore more how human assistants would interact with the user in this app. How would the user be guided by another person in mixed reality? What would this interface look like?
Thanks for checking it out!