✦ AR/VR/XR, Immersive Experience, Museum, Mobile APP
1 Designer, 2 Researcher
Oct – Dec 2024 (6 weeks)
Figma, Unity, Leap Motion, Adobe Aero
Overview
Key Components
Future City is a Gamified Museum Experience that connects human decisions to urban growth and ecological health.
02
Exploring ＆ Collecting with AR
As visitors navigate the museum, they encounter objects inspired by Pittsburgh’s landscape. Indicators guide them to specific locations where they can collect resources. Scanning these objects unlocks animations, revealing stories about each resource and its connection to Pittsburgh's urban ecosystem.
03
Co-Creating the Future City
After collecting resources, users proceed to the Wall of Future to contribute their items. By standing in a designated spot and following prompts on their phone and the wall, users perform gestures—like hatching an egg or watering a plant—that add their resource to the evolving cityscape. These contributions blend with others, collectively shaping a dynamic urban environment.
04
Sharing the Results
At the end of their visit, users are invited to share their visions for the future on an interactive wall. They also receive a personalized summary of their contributions, including a time-lapse of the Future City’s evolution, showcasing how individual actions collectively shape a shared vision of humanity's relationship with nature.
05
User testing with prototyping
We constructed the real interaction pathway and created the gestural prototypes for individuals to undergo the complete experience.
Research
Why Carnegie Museum of Natural History (CMNH)?
We collaborated with the museum’s exhibitions and community engagement teams to understand their approach to exhibits, challenges, and aspirations. Inspired by the museum’s reflection section, which asks visitors to share their hopes and fears for the future, we sought to move beyond passive viewing of artifacts. Our goal was to create an experience that encourages personal connection and thoughtful introspection.
“
I really value rethinking what a museum is—especially one that gives you facts…think about curiosity, not just pedagogy.
– CMNH staff
Why GenZ
As Pittsburgh's urban landscape continues to evolve, engaging younger generations in environmental stewardship is more important than ever.
Gen Z values purpose, sustainability, equality, and ethical practices. While they embrace technologies like AI and AR, they prioritize authentic human connections and hands-on, creative learning. Their appreciation for visually engaging content and analog elements makes them ideal for innovative and impactful designs.
What We Heard
We explored visitor motivations before their museum visit, their experiences during it, and their key takeaways. We conducted user interviews to understand what the audience wanted from what they said.
“
Dated.
When I walk into this part of the museum, it’s extremely dated. I don’t know... I’m not that interested.
“
Reflection Area?
When I walk into this part of the museum, it’s extremely dated. I don’t know... I’m not that interested.
Through field research and secondary sources, we examined various exhibits, installations, and activations to understand how tools like projection mapping, mixed reality, AI-powered exhibitions, data materialization, and physical computation engage participants. This helped us identify feasible technology setups for museum contexts and evaluate what works effectively in similar experiences.
Define
Based on research insights, we developed a research question and design principles that would frame our explorations:
How might we design an immersive experience to help Gen Z CMNH patrons connect human choices with nature in urban settings?
We followed these principles to guide the experience:
Intuitive and non-intrusive interactions
Engaging users through play
Fostering connections through co-creation
Ideation
Instead of focusing on a single exhibition hall, we want to create a connected experience across all seven permanent exhibition halls. By emphasizing the interconnectedness of the natural world—spanning birds, fossils, mammals, and more—we aim to showcase how each exhibit is part of a larger narrative.
This approach not only deepens the visitor's understanding of the natural world but also extends the experience beyond the museum walls, fostering a sense of connection to nature in everyday life.
Storyboard
We drew storyboards followed by design principles.
Visualizing the Journey
The journey map showcases how our design approach transforms interactions into meaningful connections, linking human actions with the natural world in urban settings.
01
Onboarding
Users scan the QR code on their ticket, create an account, and learn how to collect resources using AR.
02
Exploring & Collecting
Visitors discover objects inspired by Pittsburgh’s landscape in seven exhibition halls. Scanning these unlocks animations and insights about the resource’s origin and connection to urban life.
03
Co-creating
At the Wall of Future, users perform gestures to add resources to the cityscape. For example, hatching an egg introduces a bird to the ecosystem. Contributions appear alongside those of others, forming a collective urban environment.
04
Reflecting & Sharing
Visitors share their visions on the interactive wall and receive a time-lapse of the evolving cityscape. A personalized summary highlights their impact, ready to be shared on social media.
Design
Design 01
Game Rule
We've incorporated game-based elements into the FutureCity Interaction, wherein the guidelines for the game are specified. The illustration elucidates which collectibles the attendees will accumulate and their transition into the ultimate state within the urban landscape.
Design 02
AR Experience
Nine unique resources are scattered across the permanent exhibitions. Users scan artifacts to unlock layered stories and animations. This interaction provides:
Origins and Connections: Insight into the artifact’s natural resource origins.
Real-World Applications: How the artifact is used in practical, everyday contexts.
For example, scanning limestone reveals its role as a raw material for ceramic tiles, which are later processed into bricks—the foundation of modern buildings. This system helps bridge the gap between museum artifacts and their broader implications in the natural and human-made world.
Design 03
The Wall of Future City
After collecting resources, users return to the entrance and stand in front of the wall, which showcases a real-time, dynamic representation of the Pittsburgh cityscape.
To interact, users stand in a designated circle in front of the wall and select one resource from their app. By following the gestures, they can witness their resource transforming and blending into the landscape.
We crafted a system map in alignment with this. It is a dynamic projection system organized into 6 sections that visualizes data contributed by users. Users interact with the system by syncing their app.
The cityscape reflects the current time of day, with the sky transitioning from light to dark accordingly. In the top-right corner, a board shows the total resources contributed throughout the day.
We provide instant feedback by showing users an animation of their resource growing and transforming from its original state to its final form within the cityscape.
Next, follow the interaction prompts on the wall to transform a resource from its initial state to its final form. For instance, mimicking the gesture of hatching an egg will transform it into a bird.
Design 04
Future City App
The FutureCity app ties everything together, users can scan resources, contribute to the co-created cityscape, and view an overview of the resource map.
Onboarding
The app guides user their experience in the museum.
Resource Map
Users can click on resources within the app to see where they’re located on the map.
Hopes from Others
During the collection process, notifications appear, showcasing other users’ progress and hopes, which might influence exploration and collection behaviors. Once users finish collecting, they’re prompted to approach the Future City Wall.
Contribution Mode
When ready, users select a resource to contribute to the future city. For instance, adding an egg reveals both its positive and negative impacts.
Bank
If users don’t have time to gather all the resources or are particularly interested in one specific resource, they can use the Bank to trade for it.
Personalized wrap-up
After contributing, users receive a personalized wrap-up highlighting their impact on the wall. Examples include like birds, plants, and limestone. They can also use a filter to share their experience on social media.
Design 05
Prototyping: Build experience for user testing
To demonstrate the concept, we built an interactive experience using Unity and Leap Motion. Users interacted with three gestures—bird, plant, and gem—watching resources transform from their original state to their final form in the cityscape.
Here's the setup for the user testing. Visitors can experience the AR, gesture and the filter, finally leave some message and reflection.
Design 06
Design System
The project’s design system spans visual guidelines, AR interfaces, app UI, the interactive wall, and physical printouts.
Learning & Reflection
Overcoming Initial Challenges
At the start, we faced significant constraints with the technology available. While many projects gravitate towards AR, VR, or projection mapping, our approach focused on solving the core problem rather than simply fitting it into a predetermined framework. It was challenging for our team to align initially, as we struggled to find common ground. However, revisiting our research question repeatedly helped us diverge and converge until we clarified our goal: to raise awareness and emphasize the critical connection between humans and nature. This iterative process ultimately led us to an innovative and satisfying solution, one that stood out compared to previous projects.
Balancing Concept and Implementation
Designing the experience journey posed another challenge due to the complexity of our process. Early on, communicating our concept through tools like After Effects allowed us to create a "Wizard of Oz" prototype that effectively persuaded audiences. However, for the actual demo, it was important to deliver both a clear concept and an engaging hands-on experience. To achieve this, I focused on creating a fully interactive demo, using Unity and Leap Motion to track gestures and movements. This solution not only saved time but also brought our vision to life in a functional and meaningful way.
Insights from the Demo Day
During the demo, even though we didn’t build a fully functional experience, visitors followed our instructions well. This highlighted the effectiveness of our guidance and interaction design. The positive reception reinforced our confidence in the concept and its execution.
Future Improvements
Reflecting on this journey, I realized that I can achieve more than I initially thought. If given the opportunity to expand this project, I would focus on iterating and refining the gestures to make them more accessible to all users. Enhancing gesture tracking and interaction design would bring the experience to a wider audience and make it even more immersive.
