Objective – Share the story of discovering unknown parts of the ocean in the most enganging and fascinating ways

Position – Senior Product Designer

Contributions – Creative direction, STrategy, UX, UI, and Environments

Partners – James Cameron, Ocean X, BBC, National Geographic, the avatar alliance

Trailer

The Origin

The journey began in collaboration with the ambitious OceanX team, James Cameron, BBC, and National Geographic. Together, we launched a groundbreaking data ingestion pipeline that visualized diverse ocean data in mixed reality, specifically for use onboard the world’s most advanced research vessel, OceanXplorer. This allowed scientists and guests to plan missions and explore newly discovered data. My contributions included designing the environment for filming, transforming the tool prototype into a fully developed product, and shaping elements of the HoloLens storyline for the series.

Hololab

The initial task in the series was to create the primary 'character'—the environment, which became known as the Hololab. This was no ordinary environment; it required meticulous consideration of factors such as lighting, graphics, and spatial complexities to facilitate filming and integrate with the latest emerging technologies as shown below.

I designed the environment to seamlessly integrate with the existing ship, considering both stylistic components and theatrical elements for filming. The space required the inclusion of servers, cameras, dynamic lighting, and IMUs in order to make it production ready. I incorporated infrared lighting that could be hidden from human eye, yet light up the fiducial trackers to make a stable environment to track against. The final outcome resulted in a theatrical moody space with stable tracking for machine vision, and a dynamic environment that could reflect the emotions necessary to help tell a story within the space.

Show Runners, Scientist, and Us

With the environment operational on the ship, our next task was to create a comprehensive experience for the Hololab, addressing the needs of three major stakeholders. The showrunners aimed to deliver an inspiring narrative within the specified timeframe. The scientists required a mixed reality 3D visualization that integrated all their datasets into a single experience. Microsoft sought to demonstrate the capabilities of the HoloLens to enhance marketing efforts and expand business opportunities in related ventures.

Data Ingestion Pipeline to Mixed Reality

Considering the stakeholders' requirements, we embarked on developing a pioneering data ingestion pipeline for mixed reality. Our objective was to transform the diverse datasets from Mission Command into an interactive 3D visualization. These datasets encompassed bathymetry, sea temperatures, salinity levels, species populations, and more. We collaborated with various scientists to understand their specific needs, ranging from glaciologists to researchers studying Greenland shark migrations.

By observing scientists and prioritizing use cases, we identified the core functionalities required: navigating ocean topological data, filtering different data visualization layers, manipulating data to meet specific scientific needs, and creating custom features for unique requirements per episode. These core functionalities informed the design of the user interface, which was divided into two primary components relative to the user. The hand menu UI, activated when the user raised their hand, provided quick access to essential tools, while the Rail UI, attached to the table in front of the user, offered detailed data sets for in-depth analysis.

The Hand and Rail UI were essential for navigating the data, but a more interactive approach was necessary to fully engage with the datasets and environments. To achieve this, I investigated various interaction methods and ultimately adopted a dual approach: direct manipulation of the environment via contact with an external sphere for map navigation, and far-field interactions for precise pinpointing within the map data. The initial prototype, illustrated below, demonstrates the efficacy of these interaction techniques.

Direct Manipulation Example

Far Field Interaction Example

Beyond Function

With the core functionalities established, we proceeded to develop tailored experiences for each individual scientist. We created a suite of tools, including fish finders, salinity datasets, and even integrated some dolphin cameos. We also started to expand our team, we started as 3 and were now 14.

Telling a Story with Data

With fundamental interactions and tools for the scientists in place, it was time to weave a narrative with the components we built. Adopting a showrunner's perspective, I focused on crafting a compelling mixed reality story for our viewers at home, exploring ways to extend beyond the limited field of view provided by the HoloLens. It was crucial to inspire our audience with both the narrative and the technologies used to convey it.

In the example below, I created a story featuring a special monitoring device implanted in a glacial river to measure unique datasets as it traveled downstream. By utilizing Lidar mapping of the surrounding area and data from the measurement device, I transformed this data-driven story into a visually stunning experience. The narrative culminated in a dramatic moment, where the visuals enveloped the entire room, captivating viewers at home with an immersive, spectacular finale

Conclusion

Our journey resulted in the production of six episodes for OceanXplorers and its adventures. We pioneered the use of sensor fusion, data visualization, and mixed reality to tell stories in unprecedented ways about our ocean. OceanXplorers will be available on Disney plus August 18th 2024. - OCEANXPLORERS Trailer