Prototype - Mixed Reality
Process Highlights
Design Overview and Challenges
Overview
I developed an augmented reality solution that simplifies the cooking process via hands free navigation and integrated AI support. Everyday tasks that utilize both hands lend themselves to 3D solutions as they can be interacted with by the user without the need of a physical object.
Process Highlights
Project Information
Timeline
June 2024 - July 2024
Skills
Figma, Photoshop, After Effects, Unity
Research
Cooking with Physical Devices is a Problem
Cooking is a Messy Experience
So How Can Your Kitchen Experience be Transformed?
By viewing mixed reality headsets as a tool for supplementing daily activities, the kitchen is a perfect place to begin developing new forms of technology to help aid us in our daily lives. It acts as a new medium which solves problems previously unsolvable in 2D.
Research
So… The Main Problems:
Cooking in 2D sucks because there is…
Absence of hands free navigation lending itself to the cooking process
Difficulty in multi-tasking
Lack of user accessibility as devices are constrained by their physicality
Research
Survey Research
I asked a few of my friends and family about cooking using recipes and devices.
They had no idea I was thinking about this idea (yet) and had some interesting perspectives on cooking. The main problems I had initially observed were more prevalent than I thought, and the user research presented more I hadn't thought of.
"I often find myself needing to check the recipe, set a timer, and still watch [removed name]’s rugby game all at once. It's a nightmare switching between apps"
Parent, Age 50-60
“I'm just starting to learn how to cook, and it's overwhelming. I find myself constantly checking the recipe for steps, setting timers, and Googling techniques. My phone screen gets greasy, and I hate having to wash my hands every time I need to scroll”
College Student, Age 19
"It's frustrating that my phone stays in one place — I end up walking back and forth just to follow the recipe, which makes cooking stressful and less enjoyable"
College Student, Age 20
"Switching between apps for timers and shopping lists is annoying, and I often forget to check the recipe because my phone is stuck in one place. It's even worse when my roommates are cooking too; I can't always get to my phone without bumping into someone"
Recent Grad, Age 23
The Main Pain Points:
The pain points for user's varied based on their age and profession. I saw that the issues that plagued some of the younger users were not present in older chefs.
Multitasking Difficulties
Users find it challenging to manage multiple applications simultaneously while cooking. They often need to switch between different apps for recipes, timers, shopping lists, and instructional videos. This constant toggling disrupts the cooking process, leading to inefficiency and frustration.
Greasy Hands
Cooking can leave users with greasy or messy hands, making it difficult and unhygienic to interact with touchscreens. Washing hands is frustrating and interrupts the cooking flow.
Stationary Devices
Traditional 2D devices do not move with the user around the kitchen. This limitation forces users to stop what they are doing and walk to their device to check the recipe or adjust settings.
Limited Visibility and Access
In shared or small kitchen spaces, users may struggle with placing their devices in a convenient location where they can easily see and interact with them. This is worsened when there are multiple cooks in the kitchen
Research
Personas - Who is this made for?
As a beginner, Taylor finds it difficult to follow recipes while cooking, especially when it involves multitasking or using different kitchen tools. They often encounter issues with greasy hands, which complicates using their phone for recipe steps. Additionally, Taylor has limited kitchen space, which makes managing their phone or tablet cumbersome and inconvenient. Taylor also requires short tutorials on some aspects of cooking, which means she needs more interactions with her phone
Jordan often struggles with multitasking in the kitchen, especially when trying to prepare meals while managing household chores or supervising children. They frequently face the issue of having to use multiple apps or devices to manage recipes, timers, and other cooking tasks, which can be frustrating and time-consuming
Considering All Skill Levels
Since cooking is such a ubiquitous and perpetual activity, a solution using technology must appeal to all cooking abilities. Therefore, we must design with all users in mind, and try not to be too niche or too broad in our approach to solving this problem.
Research
Developing a Solution
Cooking in 3D. How?
Cooking in 2D environments clearly leads to pain points due to their medium. By placing aspects of cooking into limitless 3D space, users can move, scale, and interact with cooking features on their own terms.
Accessibility & Seamless Integration
Cooking is never going to change (at least for now) and therefore the solution proposed to the issue presented cannot be too radical. They must rather fit into our daily lives - just like food does. So, I based my design goals from that notion.
Design Goals
Enable hands-free interaction with cooking apps
Enhance multitasking for key features related to cooking
Improve accessibility for users
Integrate smart recipes and suggestions
Ideation
Moodboard
Considering XR Design Systems
I researched different cooking apps and their designs in order to make the app in an unfamiliar environment (mixed reality) feel familiar. I also included a limited color scheme in order to make the app versatile across devices.
I used a design system that I had created in another project (you can see it in my portfolio).
What does cooking already look like?
What does cooking look like digitally?
Ideation
Low Fidelity
Low Fidelity Phase
Here are the preliminary thoughts on the research and the app's goals. The low-fidelity prototypes offer an early view of the app's possible functions, highlighting how the essential features could be integrated.
Ideation
Mid Fidelity
Mid Fidelity
This stage was focused on ensuring the user experience of cooking in spatial environments aligned with the user needs of accessibility, multitasking, and hands-free interaction.
Ideation
High Fidelity
High Fidelity
This stage explored the high fidelity design concepts to visualize how the cooking experience in spatial environments would feel like. I readjusted some of the designs after receiving feedback from users that some of the UI felt overwhelming.
I received user feedback and realized the blurred cards in the background of the recipe card stack was overwhelming. Also, it took up a lot of screen space - so I cut it and reverted back to simplicity.
Using my design system of placing key navigation features on every panel, it allows the user to navigate the application on their own terms. No more searching for the navigation window in their environment.
I decided the timer was better with more simplicity.
Ideation
Design Decisions
By separating the recipe into step by step cards, the cooking experience becomes far less overwhelming. Cards also are easier to follow since they are designed for your individual actions. Recipe cards also improve accessibility for the user, since the text is large and the user does not need to scroll nor squint to see the instructions.
The progress bar acts as a breadcrumb for the user. The design allows for the user to know how far they are in the recipe subtlety, alongside the numbering on the top of the recipe cards. The use of filled versus unfulfilled checkboxes also communicates what has been completed to the user.
Using Crouton as inspiration, ingredients and measurements are colored in blue and orange respectively. This helps both guide the user to important information, but also ensures that when interacted with, the user can get more information about their ingredients. Also, since I use the metric system, I liked how you can swap between whichever unit you prefer.
For both beginner and busy cooks, suggesting recipes based on ingredients they have in their homes can be useful in minimizing cognitive load. By integrating an AI api, the user could provide what ingredients they have in their homes, and get a recipe suggested in seconds. Recent recipes cooked also allows for users to have quick access to dishes they may often cook.
I believe that gestural interactions should be utilized more in spatial environments, and thus place the navigation of the cooking app in the user’s hand. However, this can easily be placed back in a windowed format for accessibility reasons. The menu provides quick access to the home panel, recipe panel, and timer panel.
Prototype
Prototyping in Unity
Prototyping the app design using various gestures and actions
Demo in Quest 3
The home screen is large which allows the user to easily interact with the window. Calling a recent recipe or using the AI integrated search function is very accessible.
The home screen could be a bit overwhelming to new users as it is large and throws a lot of options for interactions. There is also a slight transparency problem to the window hindering accessibility. This is due to the way my Unity file is built on the Quest.
The user can interact with the arrows using both touch and pinch and point. This allows for accessibility in controls anywhere in the kitchen. The user can go to the next card while still being far away from the window.
The buttons may still be a bit too small for accessibility reasons. It also may be difficult to point and pinch accurately. A swipe function could solve these issues although it hinders the access to the home navigation bar.
The navigation bar can be called on every window, which improves accessibility for the user. The user does not need to search for the navigation bar as a window in their environment.
The action of calling the navigation bar can be unintuitive to anyone but Arc Browser users. Furthermore, the use of a touch interaction could be improved through a swipe interaction instead.
Faster accessibility through hand interactions. Bringing the home screen and control to the user.
May be called accidentally. Unintuitive to users who have never used headsets with hand tracking.
Conclusions
Main Takeaways
Design Takeaways
I learned in this project the strengths and weaknesses of a spatial computing application. This medium prioritizes handsfree interactions and democratizing specific domain knowledge.
I can see how a future using spatial computing can provide access to information to everyone.
Technical Takeaways
I learned how interactive animation is critical to making an application feel alive. I also learned that simplicity is difficult to get right, so that for the user it is easy.
I also learned the power of scripts in Unity and that C# unlocks this application.
Next Steps
Going forward, I can see how this app could use object recognition to detect and generate recipes in real time.
This app could also integrate artificial intelligence in a visual way, so that you could always have a consistent cooking companion.
Thanks for checking it out!