Case Study
Snowflake
.
Project Overview
Snowflake is a proof of concept of an expanded lookbook creation tool that centers on transmedia production workflows in a collaborative immersive environment using machine learning.
Sponsored by Google, USC's Mobile and Environmental Media Lab was tasked to develop an immersive platform to facilitate user pain points in the cinematic world. After 4 years of research done by the original team, I was hired as a Product Designer to bring the platform to life, and to create a design fiction to present Google and other potential stakeholders our findings.
.
Team
1 Product Designer
1 Product Manager
1 Project Manager
1 Developer
1 3D Modeler
4 Faculty Advisors
2 Undergrad Assistants
Role
Product Designer
UX Researcher
Brand Manager
Video Editor
Actress
Duration
Project was established in 2017
I joined the team in 2021 and finished in 2022
Tools
Figma
Adobe Creative Suite
Tvori
Shapes XR
Unity
uMake
.
Problem
Leon Silverman, General Manager of the Digital Studio at Walt Disney and President of the Hollywood Post Alliance, has remarked that every production is like a snowflake – no two workflows are alike. Overwhelmed with the influx of student projects needing workflow guidance, USC’s School of Cinematic Arts needed to create an automated innovative solution to determine the best choices for these “snowflake productions.”
“When Every Film is Different, How Do Filmmakers Determine Their Path?”
.
Solution
Snowflake is an expanded lookbook creation tool that centers on transmedia production workflows in a collaborative immersive environment using machine learning. Instead of creating lookbooks the conventional way (Slide decks, Pinterest boards,etc.) we created a solution that would allow you to immerse yourself in the mood of your project and get suggestions on how to approach production, all with the help of AI and machine learning.
An AI powered creative workspace
.
Filmmaking Crash Course
What exactly is a film workflow and what are the pain points associated with it? A typical project consists of:
❄️ Development
This is where the script is written, main actors are cast, locations are scouted and a budget for the film is established.
❄️ Pre-production
This stage consists of adjusting and testing everything built in development to prepare for shooting. This includes hiring extra actors and crew, building sets, creating shooting scheudles and costume/prop creation. This can last for years.
❄️ Production
This is the part where the film is actually made. This is usually the shortest of the 5 phases due to extensive planning, yet where most of the budget goes due to selecting equipment, setting up shots, and other needs
❄️ Post-production
This is where the footage is put together and edited into the final film. This includes editing, visual effects, soundtrack creation etc.
❄️ Distribution
This is the part where it is decided how a film is to be shown to audiences. Will it be in theaters? Streaming? What network? How should it be packaged?
The tools and techniques employed in the film workflow are highly dependent on the nature of the production and significantly impact the budget. For students and smaller studios, who may lack the experience and resources of seasoned professionals, navigating this complex decision-making process can prove overwhelming.
.
Research
Understanding the problem space and current solutions
With the knowledge of current workflows, I decided to do a bit more research in the space as well as conduct interviews with those in the industry to figure out how to approach this problem.
Target User
This solution was catered to the pain points of students. However, I found two other user groups that had similar issues and could benefit from this solution. These groups are:
These three groups fundamentally have different needs than those at larger studios that have bigger budgets and access to important resources that an amateur would not. Some potential use cases would be:
❄️ Students
❄️ Small studios
❄️ Hobbyists
Film students trying to bring their project to life
Independent game designers collaborating from different countries
Small studio trying to pitch transmedia virtual projects to investors
Competitive Analysis
After identifying pain points, I researched relevant tools and found no single tool with all required functionality, validating the issue of fragmentation voiced by users. I categorized activities into buckets and added the tools users used to accomplish those activities to gain further insights. These buckets are:
From these buckets I narrowed down the 4 closest competitors and looked for design opportunities. These competitors are:
From this analysis, the optimal approach for distinguishing Snowflake was to prioritize a collaborative and immersive workflow recommendation system.
🪣 Collaborative platforms
🪣 Lookbook platforms
🪣 Moodreel platforms
🪣 Asset creation platforms (2D/3D/Video)
🪣 AI generators
Frame.io
Nightcafe
Adobe Creative Suite
RunwayML
.
Pain Points
Fragmented tools and overwhelming workflow
Hard to articulate and communicate vision
Inefficient collaboration functionality
Limited support for transmedia projects
Moodboards are helpful but I feel like trying to explain mood and emotion isn't something that can be accomplished with words or pictures.
Jennie Auckland
Film student at USC
We had to fully move our collaboration online through Zoom because of COVID and we realized how hard it was to film something as a fully remote team.
Alexander Hill
Producer at a small studio
I make VR films and most programs and tools don't work together which makes my workflow difficult.
Tyler Hill
Amateur producer
We go on Zoom to talk, brainstorm on Google Docs, share files through Dropbox, edit on Premiere and keep track of everything on Asana. We use so many tools it's confusing!
Dakota Wood
Director at a small studio
.
Principles & Features
Defining the guiding principles of the project
I returned to my team with my discoveries and showcased 3 design principles that I deemed essential for directing me in my tasks:
The experience in the space should feel instinctual, despite being powered by cutting-edge technology. Information should be presented with ultimate clarity, brevity, and simplicity.
Part of an intuitive design is ensuring that complex tools and functionalities are organized in such a way that the most important of tools are easily accessible, while others are still presented in a way that will streamline a wide variety of workflows.
With an immersive project, this can mean two things. Not only is the scale of the workspace important, but having a system that is able to scale with each project was important in the design of the space.
❄️
❄️
❄️
Intuitive
Scalable
Organized
Defining the best balance of features
After analyzing research data and holding discussions, the team and I collaborated on core features for the solution. Despite encountering scope creep, I focused on feasible and useful features that could be portrayed in a design fiction.
Flakey, your AI-powered assistant
Flakey will be there at every step of the process, if you prefer. Activated by voice commands, Flakey will help users search and edit assets, as well as navigate the rest of the Snowflake space with ease.
Asset Manipulation
Using AI and GAN technology, assets can be manipulated to fit the specific look and feel requirements of your project.
Moodboard creation
The main function of the first iteration of this space, users are able to create a transmedia lookbook that encompasses 2D and 3D assets, as well as audio and video.
Exporting
To cater to the needs of a wide range of projects, Snowflake offers multiple ways of exporting a project such as a PDF, OBJ, or a GAN generated mood reel.
Crowdsourcing
Adding assets and projects to the Snowflake repository that was made in the Snowflake platform itself is a great way to build a community, as well as facilitate more creative thinking by including work outside of established mainstream projects.
Why VR?
Due to COVID-19, virtual collaboration in filmmaking became essential. However, current platforms have limitations that hinder creativity and efficiency. The best solution is to recreate the physical collaboration experience in a virtual space, which is timely with Facebook's shift towards building the metaverse. In less than a year since its launch, Oculus' App Lab for the Quest has almost 900 virtual reality apps, validating the need for immersive workspaces like Snowflake.
Why AI?
The content we consume and create has evolved. GAN (General Adversarial Network) technology has been used to create AI-powered content for a while now, and last year we were introduced to more even more machine learning models such as:
DALL·E - Creates images from text captions.
OpenAI CLIP - Uses natural language to learn visual concepts.
VQGAN - Can generate new images based on training images.
Guided Diffusion - Process that reverses noise to create a clearer and more detailed image.
While initially these models could only produce outputs that were surrealist in nature, the technology is advancing in a way where it is increasingly producing more realistic and accurate content. These new technologies open the door for new conversations on how we can gather and build on content that we use for inspiration.
.
Immersive
Wireframing
My team and I were working remotely, and it was hard to communicate through 2D sketches what I was envisionoing. Thanks to the app uMake, I was able to create 3D sketches of how we wanted this space to look. I started out with sketching out what tasks we wanted to accomplish to get a better understanding of how to organize the space.
It was hard to sketch a 3D concept in 2D so I found a new way to wireframe
.
Immersive
Prototyping
This was my first time designing something in an immersive space, and the vastness was overwhelming. It felt like I wasn't utilizing the limitless space like I should. To get a better understanding of how this space would work, I shot and edited a trailer to bring my previous sketches to life, as well as show how AI/ML can assist users in the space. This video was created entirely in Tvori, with me doing in-world rigging in real time to make all interactions occur in realtime through Tvori's keyframe feature.
Storyboard feature that allows you to physically place sketches/footage in the order you want, or with the help of AI.
Image aggregator and manipulation feature that allows you to sort through images for inspiration by color, mood, theme, and more.
Location scouting feature that allows users to immerse themselves in potential filming locations.
3D wireframes means 3D prototypes
3D wireframes means 3D prototypes
Teaser Trailer
.
Changing My Approach
Upon completing my initial design of the space, I realized that there was a distinct lack of structure. Various elements appeared to be floating aimlessly in space, without any clear organizational scheme. When I presented the video trailer to users, they expressed confusion regarding the functionality of each section and how they could operate within it. Consequently, I revisited my original idea and made revisions that included dedicated workspaces to better clarify and define the distinct sections of the space.
Creating structure in a limitless space
.
2D Wireframes
I began by sketching out the essential elements that would be necessary for an immersive moodboard. From there, I brainstormed ideas on how users could create moodboards for each aspect of their project. For instance, instead of creating a single moodboard for all characters, users could delve into the details and create individual moodboards for each character, allowing them to fully develop each personality and style. It was from this idea that I came up with the concept of "flakes", having a section dedicated to one concept but contributing to the bigger project as a whole.
Exploring dedicated workspaces
.
3D Prototype
I built these wireframes in ShapesXR, a great worldbuilding prototyping tool that I call “the Figma of VR” The first iteration posed a problem: With no foundational flake, it was hard to stay organized. It also posed these questions:
To improve efficiency and collaboration in our workspace, we shifted to a hexagonal design, providing individual workspaces for each team member while facilitating communication. This scalable approach can easily accommodate larger teams or projects and the hexagons can be connected to create a modular system that can be expanded or reconfigured. This new approach addressed our previous issues and created a functional and flexible workspace.
Bringing the wireframes into an immersive space
❄️ Where and how are we getting assets to put in the flakes?
❄️ Does each flake come with its own board or screen?
.
Avatars
To promote collaboration, we needed to determine how users would be represented in the space. Through testing, we discovered that full body avatars were preferred over more distracting options like floating heads or disembodied voices. Realistic avatars provided clarity and gave users the opportunity to personalize their virtual presence. To create these avatars, we used Ready Player Me, a cross-platform metaverse avatar generator that integrated seamlessly with various game engines and had an intuitive platform.
Full-body avatars for immersive collaboration
.
Flakey
As a team, we deliberated on how to represent the AI in the platform. We considered various options such as a human, a voice-only interface, or no representation at all. Ultimately, we decided that having a visual representation would make the platform more accessible, and we drew inspiration from Clippy, the old Microsoft Paperclip. We transformed this concept into a snowflake to align with our overall theme and created Flakey as our AI representation.
Getting inspiration from a paperclip
First
Iteration
Final iteration
Logo
.
Export
It was important that there was a way to export work done in Snowflake, since sharing and pitch to other stakeholders was a big part of the production workflow. I created a more effective way to aggregate work done in a space by exporting it in different file types and frames, including a moodreel that combines assets into a GAN-generated video to convey mood and emotion beyond 2D moodboards.
Example of GAN generated video
Creating ways to efficiently share work
.
Immersive Hand off
In our workflow, we needed to import everything created in ShapesXR into Unity for interactions, which was not possible in ShapesXR itself. However, doing this remotely was challenging, especially when working in VR. To address this, the engineer and I collaborated synchronously in VR, where I could point out how I wanted things to look and the engineer applied my notes to Unity in real-time. Despite the need for creativity to facilitate a VR handoff, this collaboration method worked well for us and we enjoyed the process.
Figuring out how to hand off an environment
.
Filming
To present our vision to Google, we created a design fiction using a film and a virtual space. Half of us acted as avatars, while the other half served as crew. We filmed remotely using OBS and edited asynchronously using Adobe Premiere Pro.
Collaborating with our remote teammates
Half of our production team
Half virtual half physical production
.
Final Design Fiction
.
Branding & Visual Design
Much of the branding and visual design was modeled after a snowflake, from Leon Silverman's quote on unique projects. I wanted a balance of simplicity and modernism while incorporating the wintery snow theme.
Snowflake design elements shape the brand
.
Business Implications
In order to provide structure to my design process and explore potential revenue streams, I developed a comprehensive business model. Although not included in the design fiction itself, this document was crucial in presenting a compelling case to Google and other potential stakeholders.
Defining revenue streams and market positioning
.
What's Next?
We are currently in the process of presenting the design fiction to Google and other potential stakeholders. If given funding to move forward with the project, we will continue to test the platform and make further iterations, as well as add more engineers to the team to further develop the platform in Unity. We will also develop and implement our proposed AI/ML features, and see what is feasible with the technology we have available today.
.
What I'd Do Differently
💭
💭
💭
While my understanding of the subject matter may be limited, it is apparent that working in VR represents an entirely different realm of interaction, as it involves the utilization of the entire body. I would do a deeper dive into VR heuristics in order to uncover strategies for enhancing the user experience.
As there was a significant learning curve in the process of worldbuilding, it became apparent that further practice was necessary to successfully convey my design concepts to the developers and stakeholders.
While I opted for a simple field in consideration of user experience, I believe there is potential for incorporating more innovative navigation techniques in a virtual reality world. Exploration of vertical space and portals, for example, could add an exciting new dimension to the experience. I'm curious to explore other creative ways to navigate the space - perhaps utilizing teleportation or unique gestural inputs.
Accessibility in the VR space
Practice world building
Explore other 3D functionalities
What I Learned
💡
💡
💡
Limitless space is overwhelming, due to the pressure of feeling like you need to utilize all of it. Creating space in virtual reality is critical in guiding your users through what they need to do, and comforting for the user as it is difficult for the human brain to conceptualize empty space. It can even help in creating the narrative for your experience.
I often thought about disrupting the laws of physics throughout the design process, and how that ergonomically affected users. Floating while working is cool, but does it serve a purpose? Finding a balance between "cool" and "functional" really helped in developing features for this platform.
Technology is advancing at a more rapid rate than I thought. My team thought that creating images for a look book just by typing or saying a description would be a great feature to have that is easily developed in the next few years. Just a few weeks later, DALL-E was announced. This taught me to always push my creativity to the limit. If I'm ever content with a seemingly novel idea, I need to push that idea even further!
Navigating limitless space
Do we need physics?
Think big
