Thesis – Physical AI as an interaction design material.

Work In Progress

This page is a living document of the master thesis I am currently working on – digging into what physical AI as a material means for the world of interaction design. It aims to explore the possibilities and opportunities that arise when a space understands, reasons, and interacts with the people and things inside of it. How our experiences with our environment change, and our relationship with technology transforms through physical AI.

As AI understands more of the physical world, it enables us to make sense of our surroundings in new and easier ways.

Partner

For this project I am collaborating with Archetype AI, a San-Francisco based startup born out of Google's ATAP department (project Soli). Archetype AI leads the way in developing a Large Behavior Model, which makes sense of the physical world around us by fusing sensor data with natural language. Check the video bellow for more info!

Project Info

6 months, spring 2024
Master Graduation Project
Umeå Institute of Design

Partner

Archetype AI

Explorations

Zero-Shot Object Detection

Hand overlapping with input prompts to look for "a photo of a pen".

MediaPipe and OwlViT (Python).

Accelerometer Analysis

LLM making sense of raw sensor data.

Adafruit Feather Sense with OpenAI's LLM (Python).

Haptic Glove

Giving feedback through haptics, incorrect is a quick double pulse, correct is a slower single pulse.

MediaPipe, YOLO v8 and Hapticlabs (Python).

Stethoscope

Understand the patient together with AI. Stay in the moment with the other person, while listening to the heart beat and getting insights from the AI.

A01 Summarize API, OpenAI's LLM/TTS, OpenCV, Librosa and Arduino (Python).

Binoculars

AI powered depth camera. Switch between color and depth view, and get insights from the AI about the main object in the viewport.

A01 Summarize API, iPhone's LiDAR and Adafruit Feather (Swift).

Sketch Assistant

What if you could sketch with AI together in the physical world? Using traditional tools with an AI layer projected on top.

Latent Consistency Model, Projector and Adafruit Feather (TouchDesigner).

Dynamic Actuators

An LLM deciding what interaction is suitable for the context and the prompt. It can communicate to the person through the desk lamp, the fan, the waving arm and/or the speaker.

A01 Summarize API, OpenAI's LLM and Arduino (Python).

AI as a 6th sense

What if the space understands your intention, and it could let you feel the relevant information of an object when you interact with it, like a 6th sense.

In this example, a person wants to know if the parcel might be damaged. The AI will pick the most relevant history of the sensor data, and show it through haptics to give the person an understanding of how the parcel was handled, and whether it might be damaged or not.

Hapticlabs, OpenAI's LLM and Adafruit Feather (Python).

Physical context as prompt

The doctor examines the patient's knee, the AI comprehends this intricate context and generates insights on demand based on all available data – for example medical records, history and scans.

The doctor can express its need for assistance by looking at the screen, indicating he wants the AI to share its insights through that actuator. Subtle facial expressions can be used to get new suggestions, or enlarge them.

MediaPipe Face Landmarker (Node.js).

More coming soon...