Intuitive, easily operated bubble user interface for augmented reality

An augmented reality user interface that displays information as a series of nested bubbles in 3D virtual space.

Problem: 
Augmented reality (AR) is the process of superimposing graphics/sounds to what a person sees, typically by adding modifications to what is seen through a tablet or phone camera (e.g. social media filters, Pokémon GO). With widespread application from games to apps to enhanced museum experiences, it represents a rapidly growing field with a market size of 18 bn USD worldwide.

As its popularity increases, so does the need for more sophisticated ways of presenting information in AR. However, designing an effective AR user interface necessitates the unique challenge of translating traditional 2D methods of organizing information into 3D.  

Solution:
The inventors tackled the problem by taking inspiration from the directory tree organization of desktop files and folders. Rather than attempt to create an entirely new way of organizing information as others have done, they instead took advantage of a data structure with which many are already intimately familiar, resulting in a creative, user-friendly solution especially well-suited for AR.

Technology Overview: 
The inventors designed a user interface in which information is visualized as nested bubbles superimposed on what is seen on a virtual screen. For example, when a user views his/her surroundings through a smartphone camera, an app utilizing the interface would display bubbles on screen as part of the environment. The bubbles are organized into a hierarchal structure (called the “foam”) similar to folders in a typical file system. Users can navigate the foam by interacting with their screen and/or moving in real space. The information associated with each bubble is displayed on screen in response to the user’s actions.

Advantages:

• Unlike other AR interfaces, takes advantage of an intuitive structure already familiar to most users
• Designed to have widespread application in education, commerce, and beyond, instead of being limited to one AR application as other user interfaces are (e.g. an app)
• Offers additional features to enhance user engagement, including shadows for realism, popups, sound effects, and cues for the user to look around

Stage of Development: 

• Concept
• Proof of Concept
• Bench Prototype

Bubble User Interface. An example of how a nested hierarchy of information nodes may be organized abstractly as bubbles (drawn as circles) is depicted in (a). Note the hierarchy need not be as regular as pictured; bubbles can have different numbers of sub-bubbles. An example of how the interface may appear to users is shown in (b).

Intellectual Property: 

• US Patent Pending
• Research Materials

Reference Media: 
Lane, SH et al, SIGGRAPH ’20: ACM SIGGRAPH Appy Hour, 2020

Desired Partnerships: 

• License
• Co-development

Docket # 21-9466