Back

Spatial Experience

IoT

Co-Design

User Research

Iterative Design

Advanced Prototyping

Usability Testing

A project sponsored by

Live music, fully unleashed in your space.

Made for your space, and built to move you.

Live music, fully unleashed in your space.

Made for your space, and built to move you.

SonySphere

Your front row ticket just got a whole lot closer

An AR dome projector, with headset-grade immersion, nothing worn.

Solution Showcase Video

Interactive Mobile App Prototype

01_

Overview

Before we jump in..

Skills

Interaction Design

Co-Design

User Research

Iterative Design

Spatial Experience Design

Advanced Prototyping

Usability Testing

Project timeline

December 2023 - June 2024

Stakeholders

Sony Product Planning Team

Sony Human Factor Engineering Team

Team

Dave Cai - UX Designer, Prototyping Specialist (Me)

Amodini Khade - UX Researcher

Marianna Nam - UX Generalist, Product Manager

Winnie Hou - UX Designer

Tracy Barnes - Human Factor Engineer, Advisor

Marvin Demerchant - Human Factor Engineer, Advisor

Contribution

Conducted user research, led competitor analyses, and distilled key data points into actionable insights that shaped our design direction.

Led the UX design process and engineered advanced prototypes, transforming a futuristic tech concept into a tangible, interactive experience.

Introduced the solution to stakeholders, fielded their questions, and ensured alignment on the project’s vision and potential.

Status

Presented research findings and interactive prototype to Sony’s Product Planning teams in the US and Japan, gathered cross-regional feedback informing our next iteration.

Additionally, the team compiled all key insights, design rationale, and solution details into comprehensive documentation shared with stakeholders for a clear, unified understanding of the project’s potential.

Solution Preview

Quick glance

SonySphere helps Elena enjoy immersive live music right at her home

Elena is a huge Taylor Swift fan who’s been following her music career since the beginning.

See how SonySphere helps Elena and her friends feel like they’re a part of the Eras Tour performance, even from home.

👩🏻‍💻

Elena’s pain point #1

Scarce Ticket

“As a Swiftie, I was furious when my presale code got rejected and I was stuck in a crashed Ticketmaster queue, surrounded by bots and scalpers. It’s maddening that millions of us fans ticketless.”

Elena is disappointed that she could not get tickets to the Taylor Swift concert. She wanted to experience the concert in-person along with her friends.

Image: The Michigan Daily

🎫

With sonysphere

Easily Get the “Ticket” at a Fraction of Price

Elena skips the headaches of traditional ticketing. No lines, no scalpers, no hidden fees. With just a few taps on the SonySphere App, she can buy a single-event ticket or subscribe to the SonySphere Event Pass, all while saving time and money.

🤦🏻‍♀️

Elena’s pain point #2

Compromised Views Causing Letdown Experience

“As a 5-foot-tall person, I’m stuck at concerts staring at the floor or someone’s phone screen, unable to see the stage, even pushed by people. It’s frustrating to pay the same price for a ticket but get shoved around and blocked by taller people, it’s like a nightmare”

Elena often felt crushed at the concert as taller fans blocked her view, leaving her staring at the floor or phone screens instead of the stage. Despite her excitement and full ticket price, the experience fell flat, overshadowed by her height disadvantage.

🏠

SOLUTION

Front-row experiences

effortlessly at-home

Front-row experiences

effortlessly at-home

Elena’s friend introduces her to SonySphere which creates unique interactive experiences for at-home audiences to augment their live music experience.

Now Elena can immerse herself in Taylor Swift’s music from the comfort of her home.

😪

Elena’s pain point #3

Bland, Disconnected Online Live Music Experience

“I’m out here, tryna feel something, but it’s just me, my crusty laptop, and a weird vibe ... it’s all fake, laggy streams, awkward chat boxes spamming “WOOOO,” and a screen that’s got zero soul. I’m over it. ”

“I’m out here, tryna feel something, but it’s just me, my crusty laptop, and a weird vibe … it’s all fake, laggy streams, awkward chat boxes spamming “WOOOO,” and a screen that’s got zero soul. I’m over it. ”

Elena finds most online live music events are disappointing with disengagement with the stage, laggy streams and clunky chats, leaving her detached and craving the real deal.

Image Generated by “Grok” AI

🔮️

SOLUTION

Transport to Immersive Worlds with Besties

With SonySphere, Elena can enjoy custom-made, artist-curated visuals and interactions that elevate her favorite artist’s performance.

Her apartment can transform into different music worlds that she can explore while staying connected with her friends and physical surroundings.

With SonySphere, Elena can enjoy custom-made, artist-curated visuals and interactions that elevate her favorite artist’s performance.

Her apartment can transform into different music worlds that she can explore while staying connected with her friends and physical surroundings.

02_

Problem Space

Hello,

Immersion.

Hello,

Immersion.

Hello,

Immersion.

What we trying to figure out

The Background

Building an innovative solution on where immersive spatial technology & home entertainment intersected

The team is in partnership with Sony to explore the next generation of home entertainment experiences that consider the role of augmented reality and spatial computing.

Sony explicitly asked the team to construct a solution besides gaming, and it would be a bonus if the design solution can incorporated with the existing Sony product lineup in some extent.

Sony stated that the project is a “blue-sky project”, in which the team is encouraged to think outside of current implications of spatial technologies and build beyond technical limits, but the team needs to take the product-market-fit of the solution into account.

The team is in partnership with Sony to explore the next generation of home entertainment experiences that consider the role of augmented reality and spatial computing.

Sony explicitly asked the team to construct a solution besides gaming, and it would be a bonus if the design solution can incorporated with the existing Sony product lineup in some extent.

03_

Identify user pain points & needs through research

Peeking

Under the Hood

Peeking

Under the Hood

Peeking

Under the Hood

What do users care the most when it comes to spatial reality?

Key Insights

From analyzing and connecting the findings across all of our research methods, we drew the following 3 major insights from our research.

research Insight #1 - Novelty

Novelty is the key to making remote live music experiences enticing

The energy and atmosphere at an in-person live music event play a large part in making the experience enjoyable and memorable.

Given that these attributes of live music events cannot necessarily be replicated in remote settings, there’s a need to determine what can make live music at home feel special in its own unique way.

💡 How it informed our design solution:

We focused our design on the AR experience of live music rather than trying to replicate the in-person live music experience at home.

This meant we honed in on creating unique augmented spectacles and interactions that complement the music for at-home audiences.

“The sky's the limit of what you can get away with [in virtual events]. And I feel like if you're just going back to the binary broadcaster to listener mode, you're missing out on a lot of the potential”

- Interview Participant 2

“The sky’s the limit of what you can get away with [in virtual events]. And I feel like if you’re just going back to the binary broadcaster to listener mode, you’re missing out on a lot of the potential”

- Interview Participant 2

“It doesn’t make sense to just recreate the live music experience virtually. Instead, you should give fans something new and different that they can’t do in real life.”

- Sam Schoonover, Innovation Lead for Coachella

“It doesn’t make sense to just recreate the live music experience virtually. Instead, you should give fans something new and different that they can’t do in real life.”

- Sam Schoonover, Innovation Lead for Coachella

research Insight #2 - connectivity

AR experiences should aim to feel more connected and less isolating

A mock up of people using AR glasses in public transport (2021),

Sander Veenhof, LinkedIn

Although the ideal level of immersion varies according to personal preferences, there’s a shift in consumer preferences and industry trends away from complete immersion that removes users from their physical space.

As we found through our literature review and interviews, people express interest in immersive experiences that ideally do not come at the expense of sacrificing spatial awareness of their physical surroundings.

Given that most consumer-level AR/VR products of today depend on a head-mounted display (HMD), most use cases in this space are designed to be experienced alone vs. shared with other people in the same space. There may be an opportunity here to explore solutions that help fill this gap.

💡 How it informed our design solution:

Throughout our design process, we aimed to transport users via immersive music experiences while keeping them connected to their social and physical surroundings.

This finding steered us away from designing a solution that relies on wearables like head-mounted displays (HMDs) or glasses that detach users from other people and objects in their space.

research Insight #3 - Social Interactions

Social interactions are a highly valued aspect of entertainment experiences

Lessons from building social experiences in VR, Meta

One of the biggest draws of attending live events is the opportunity to enjoy entertainment with others. People crave social entertainment experiences because of their desire to form and foster connections, feel a sense of belonging, and create a collective memory with others.

This prompted us to consider how to reimagine home entertainment experiences to create a sense of connection and shareability.

💡 How it informed our design solution:

We designed interactions that multiple people can enjoy at the same time. Furthermore, our solution relies on projections and is not wearable -dependent which enables shareability with others in the same space.

04_

Pin-up design directions

Imagining

Possibilities

Imagining

Possibilities

Imagining

Possibilities

How Do We MAKE SURE OUR SOLUTION ADDRESSED USERS NEEDS?

Co-Design Session

To understand user preferences in-depth, we hosted a co-design session with six participants. We drafted three prompts that focused on live music visuals, storytelling, and music’s energy. The six participants were split into two groups. Both groups worked on the prompts individually and came back together at the end for a group discussion.

From the co-design session, we found out that customization and flexibility are the most desired features. For example, participants want to have the ability to choose the level of immersion and set the mood according to their preferences.

During group brainstorming, multiple participants shared ideas that incorporated projections and dynamic lighting. Surround sound was also mentioned a couple of times as an important part of enjoying live music.

What insights dis we get from co-creating with target users?

Multiple participants stated the desire of choosing the level of immersion (low to high)
Multiple design ideas involved with projections and lighting
Participants expressed interests in more immersive sound experience

What insights dis we get from co-creating with target users?

Multiple participants stated the desire of choosing the level of immersion (low to high)
Multiple design ideas involved with projections and lighting
Participants expressed interests in more immersive sound experience

Team members conduct co-design session with participants

How Did we showcase a future technology prototype to stakeholders

High-fidelity Prototyping

Based on feedback from earlier user testing, the team iterated on the solution to better fit users’ patterns and solve their pain points when attending live music events.

To better demonstrate the solution to stakeholders, the team crafted 3 types of deliverables.

SonySphere Companion Mobile App Interactive Prototype
SonySphere Concept Video
Experience Diagram (Requested by stakeholders)

🚀

setting up the immersion

SonySphere Companion App Interactive Prototype

Play Interactive Prototype

SonySphere’s companion app guides users through the process of discovering immersive live music events, setting up the device, and preparing to join their immersive live music event with friends at the palm of their hands and in the comfort of their places.

Mobile prototype demonstrated the user flow from the moment they receive a notification for an upcoming event to the start of the event, and the seamless transition from user’s smartphone to SonySphere Dome Projector.

Read More on the Companion App Design Breakdown

🎬

SONYSPHERE 360 IMMERSION SHOWCASE

Concept Video

Since the AR technology we envisioned doesn’t exist yet, we decided to use video as the medium for our high-fidelity prototype.

We invited individuals who had participated in our low-fidelity prototype testing and matched our target audience to act in scenes that showcase key features of our solution.

Our team wrote the script and screenplay based on previous research on target user, created the storyboard, and developed and compiled the high-fidelity assets that would be featured in the video.

After that, with team member’s help, I filmed and composited VFX to showcased the various touchpoints of our design solution, and conveys the social aspect of the experience, making this virtual journey looks lifelike.

Since the AR technology we envisioned doesn’t exist yet, we decided to use video as the medium for our high-fidelity prototype.

We invited individuals who had participated in our low-fidelity prototype testing and matched our target audience to act in scenes that showcase key features of our solution.

Our team wrote the script and screenplay based on previous research on target user, created the storyboard, and developed and compiled the high-fidelity assets that would be featured in the video.

Watch Video (2 min)

Sketched Video Storyboard

Dave leads the prototyping process

Dave leads the video prototyping production

🔍

HOW USERS WOULD INTERACT WITH SOLUTION

Experience Diagram

A step-by-step interactive flow demonstrates the end-to-end experience of using the SonySphere dome projector.

The experience starts with the mobile companion application and ends at the conclusion of a live music event.

The scope of this diagram focus on the user experience in the moments right before and during the live music experience itself.

A step-by-step interactive flow demonstrates the end-to-end experience of using the SonySphere dome projector.

The experience starts with the mobile companion application and ends at the conclusion of a live music event.

The scope of this diagram focus on the user experience in the moments right before and during the live music experience itself.

Check out Experience Diagram

05_

Final Solution

Reality, Amplified

What is the solution

SonySphere AR Dome Projector

Transforming immersive ✨ live music experiences at home with SonySphere augmented reality (AR) dome projector and its mobile companion app.

HOW IT WORKS

Experience live music from home like never before

SonySphere offers a uniquely immersive way of experiencing live music from the comfort of your home.

Whether it’s Taylor Swift’s Eras Tour performance or a small indie concert, immerse yourself in the artist’s world with projected visuals that fill your space and are personalized to your movement.

With easy setup using the mobile companion application, SonySphere instantly transports you to a whole new dimension of live music.

SonySphere offers a uniquely immersive way of experiencing live music from the comfort of your home.

With easy setup using the mobile companion application, SonySphere instantly transports you to a whole new dimension of live music.

SonySphere Companion Mobile App

SonySphere’s companion app allows users to discover immersive live music events they wish to join and set up the projector for these events on their mobile phone.

🔔

Receive Event Notification

The SonySphere companion app notifies the user when their saved event is about to start. The user can tap on the push notification to set up their dome projector for the event.

💫

Choose Mode for Experience

The user is given the choice to select between Smart Lighting and 360 Immersion according to their preferred level of immersion for the event.

🪜

Complete set up with step-by-step guidance

After confirming their mode selection, the user is guided through a step-by-step process to optimize their experience.

Each step includes instructions and visuals for the user to reference as they prepare for the event.

👀

Review what to expect

Once the user completes setup, they are primed with information on what to expect when the event starts.

In the case of 360 immersion, they are instructed to keep an eye out for prompts that will direct them on how to interact with the projected visuals.

⌛

Waiting Room with Countdown

When the user taps “I’m ready” action button from the previous screen, they will be entering the waiting room where the system displays a countdown to the event.

When the countdown reaches zero, the user can join the event and they are instructed to set their mobile device down while the event is in progress.

SonySphere AR Dome Projector

After concept validation, we brainstormed possible ways to bring our futuristic ideas to life.

Our team wrote the script and screenplay, created the storyboard, and developed and compiled the high-fidelity assets that would be featured in the video.

After concept validation, we brainstormed possible ways to bring our futuristic ideas to life.

Our team wrote the script and screenplay, created the storyboard, and developed and compiled the high-fidelity assets that would be featured in the video.

Projected Guidance for New Users

Interactive Method Example

360 Degree Projections

3D hologram-like visuals transform your surroundings with 360-degree projection mapping, enabling a fully immersive AR experience without the need for wearables.

Motion-Sensored Interactions

Create your own concert experience with novel interactions. Smart sensors allow you to interact with projections that respond and adapt to your movement.

Artist-Curated Visuals

Immerse yourself in the artist’s world through specially curated lighting and visuals that represent their music and vision.

What do target consumers say about sonysphere

“Love this idea, especially because it creates connection and brings the digital experience to the real world. Amazing! :)”

“I like this product - I believe it solves a huge hurdle to making AR an everyday product!”

06_

future Direction

Our Journey,

Our Why

Our Journey,

Our Why

Our Journey,

Our Why

going back in time

What I wish we could do better?

Expanding Spatial Computing Knowledge

Spatial computing was a term we were only vaguely familiar with at the start of this project.

Although we were able to gather a lot of information online through researching this topic, we found it challenging to understand the full breadth and depth of its capabilities, which also led to difficulties of imagining possible design solutions.

Looking back, it would have been helpful if we had the opportunity to talk to experts in this rapidly evolving field before diving into research.

Spatial computing was a term we were only vaguely familiar with at the start of this project.

Looking back, it would have been helpful if we had the opportunity to talk to experts in this rapidly evolving field before diving into research.

Navigating Ambiguity

Sony’s project proposal for our team was highly futuristic and open-ended, so we had to adapt quickly and learn to navigate ambiguity.

It required us to stay agile while being comfortable with continuous scoping and iteration throughout the project.

There were moments when we found ourselves questioning whether we had narrowed our focus enough to make progress towards the final design instead of appreciating the freedom of divergent thinking.

In hindsight, we would tell ourselves to focus on the different steps in the process rather than being overwhelmed by the blue-sky nature of the overall project.

Sony’s project proposal for our team was highly futuristic and open-ended, so we had to adapt quickly and learn to navigate ambiguity.

It required us to stay agile while being comfortable with continuous scoping and iteration throughout the project.

There were moments when we found ourselves questioning whether we had narrowed our focus enough to make progress towards the final design instead of appreciating the freedom of divergent thinking.

In hindsight, we would tell ourselves to focus on the different steps in the process rather than being overwhelmed by the blue-sky nature of the overall project.

Sections

MORE WORK

•

•

•

•

made in Seattle with ❤️