Annotator: Anchoring AR Experiences In Bloomingdales

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form

Overview

iOS, B2B SaaS, Augmented Reality

Company
ContextGrid
Client
Bloomingdales
Date & Project Length
2017 - 5 Months
Role
Cofounder, UX Designer
Tools
Adobe XD, Post-it Notes, ARKit.2.0, Google Computing Cloud
Framework
Lean/Kanban

About

Bloomingdales customer base is primarily gen X customers (older folks). If that doesn't change in couple years, we will no longer have Bloomingdales.

Problem

How do we make Bloomingdales feel accessible and relevant to Gen Z kids?

Solution

In an effort to attract a younger customer base, we created away for Bloomingdales shoppers find deals and host snapchat-like store experiences at exact points in their departments using augmented reality called Holopop (now HoloGrid).

But, In order for these experiences to exist, Bloomingdales stores needed a tool that allows them to anchor holograms to exact points in their stores, such as a coach handbag. To do this, we made Annotator, an iOS app that uses computer vision and outside-in point clouds to recognize points in their stores.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form

Discovery

We started with a stakeholder workshop called Post Up. We then affinity mapped the problems into trends.

Then as team, we mapped out the Risk Using a Risk/Reward Matrix.


From the workshop, we came up with the following Epic. As an associate:

  1. I need to be able to log in to the Bloomingdales account so I can create a computer vision model for my store/microlocations.
  2. I need to be able to record a video and mark the points I want to annotate.
  3. My recordings needs to train the model in the background.
  4. I need to be able to view places, microlocations, and data added in WorldRegistry.
  5. I need to be able to attach my video to a places microlocation, and some data.
  6. I need to be able to Edit and view old recording information.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form

Ideation

We started out with sketching and a paper prototype, showing how the user would mark the object in the video recording to train the AI model.

Following the ContextGrid Branding

Using our style guide, I created the visual designs to match our web app.

Onboarding Screens.
View for assigning a modal to the ContextGrid Registry.
Place List View.
Initial Screen for your Modals Inventory, Structured by Place.
The Modals associated to a Place.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form

Prototyping

Using Adobe XD

We created a lo-fi prototype of annotator and tested it with Bloomingdales employees, which underwent 3 rounds.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form

Testing

Improving how users create anchors

After we had a version to test, we did ADHOC testing in the Bloomingdales stores with the associates. What we found was that the modal creating part of taking a video and marking it up, took too long. Store Associates have other tasks at hand, and they need to be able to create an anchor in under 1 minute to make it worth their time.


The other problem was there was a lack of feedback in regards to the success of creating an anchor. This was because it took 6 hours for the system to validate the creation of the anchor using machine learning with our system.

In short, we had an inefficient, non scalable solution.

ARKit2 Beta: Apple’s Point Cloud System

Luckily, Apple’s ARKit 2 Beta had an out-the-box solution for creating outside-in point clouds (this allowed us to also create an anchor for a single object). The creation process which gamified for the user, quick, and it creates the modal instantly. We made a test app and got feedback from the store associates and they found it to be a massively more productive and enjoyable experiencel. We quickly replaced our computer vision/machine learning tech with ARKit2’s Process.


Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form

Final Product

Results

The app was proved usable, easy to navigate, and solved Bloomingdale’s problem of not being able to annotate a store with data in quickly, at scale. Unfortunately, due to a change in leadership, the project was discontinued with talks of reconsidering the pilot stores in the Bay Area of California Q2 2019. More info coming soon!

Demo

View on the Appstore

Click here to view