Project Aria Research Summit · Poster Companion

Toward Personal Egocentric AR Journaling

Evidence Capture · Edge Processing · Retrieval · Memory

Turning continuous first-person video into compact, agent-ready evidence — not just raw footage.

Future AR glasses may continuously capture what a person sees, reads, touches, and hears. But raw egocentric video is long, noisy, private, and difficult for AI agents to use directly. This project frames personal AR journaling as an evidence-lifecycle problem: how to capture meaningful moments, process them under user control, retrieve them efficiently, and serve them to personal agents when they are useful.

Completed Work Ongoing Pilot Early Exploration System Vision

Rutgers University / WINLAB · Systems & Multimedia Lab · Zihao Ding & Dr. Yao Liu

Poster PDF → NOSSDAV Demo → Papers →

This page synthesizes ongoing Project Aria exploration and prior collaborative research. The components are evaluated independently and point toward a future integrated system.

The Evidence Lifecycle

From first-person sensing to a grounded personal agent

Hover, tap, or focus a stage to read what it does. The five stages connect ongoing Project Aria work with prior systems research on privacy, retrieval, and context serving.

{{ cur.label }}

Jump to {{ cur.linkLabel }} →

{{ cur.source }}

⟳ Feedback to the wearer may return through phone UI, voice, or future AR display output. · Conceptual end-to-end flow; components are evaluated independently.

Module 01 · Constructing Egocentric Evidence

Ongoing Project Aria Pilot

Turning reading moments into timestamped, searchable evidence

Our ongoing offline prototype focuses on reading events. It starts from Project Aria video and eye-gaze data, detects likely reading intervals, extracts candidate frames, uses gaze to guide region selection, and applies cropping and OCR to produce timestamped text evidence.

This is an engineering prototype rather than a completed memory system: the current endpoint is evidence extraction, while long-term storage, retrieval integration, and systematic evaluation remain future work.

Route A · Modular CV Pipeline

Aria video + gaze→ reading-event detection→ candidate frames→ representative frame→ gaze-guided crop / SAM→ OCR→ evidence record

Route B · Unified Multimodal VLM (design direction)

video + gaze + audio→ multimodal VLM→ event interpretation→ structured evidence

Structured Evidence Record

event: Reading
time: 00:01:23 – 00:01:28
evidence: selected crop + OCR text
confidence: likely_read
source: Project Aria recording

Figure 1. Reading-event gating. The reading-recognition signal is smoothed and thresholded to identify candidate intervals.

Figure 2. Gaze-guided SAM-1 masks. Multi-scale SAM-1 candidate masks around the gaze point propose the likely read object.

Figure 3. Crop-to-text evidence. The selected crop is used for transcription / OCR / VLM text extraction and becomes timestamped evidence.

Pilot A · Reading Evidence

From likely reading moments to text evidence

Detect candidate reading intervals, use gaze-guided crops or SAM-1 masks, and preserve selected visual/text evidence with timestamps and source frames.

Pilot B · Hand–Object Evidence

From interaction cues to object evidence

Use hand, gaze, and spatial cues to identify candidate objects the wearer touched or used, while preserving surrounding context such as surfaces and nearby items.

Early Exploration

Module 02 · Privacy-Aware Edge Processing

Completed · ACM/IEEE SEC ’25

Distribute heavy vision without handing over the whole frame

{{ privCur.title }}

In an AR journaling pipeline, evidence construction may require heavy vision models for segmentation, detection, or multimodal preprocessing. This module asks how those models can be offloaded without exposing the complete egocentric frame to one external server.

Always-on wearable vision may need powerful models, but transmitting complete frames to one external server can expose bystanders, screens, documents, and private spaces. Our SEC ’25 work explores a privacy-enhanced distributed framework for ViT-based visual intelligence.

A trusted edge device partitions the input into windows, distributes window processing across independent external servers, and merges the returned embeddings locally for global attention and final prediction. The demonstrated case study adapts SAM-style ViT processing while keeping complete-image coordination on the trusted edge.

wearable capture→ trusted edge→ window partitioning→ external windows→ local merge→ prediction

No single external server receives the complete image.
Full-image aggregation and final prediction remain on the trusted edge.
Near-baseline SAM segmentation utility under the stated threat model.

Framed as privacy-enhanced: the method limits complete-image exposure and reduces reconstruction risk under the stated threat model — it does not claim to fully solve egocentric video privacy.

Module 03 · Token-Efficient Storyboard Retrieval

Completed · NOSSDAV ’26

Answer questions over hours of video without sending the whole archive

We study egocentric daily video QA as a retrieve-then-reason systems problem. A long personal archive becomes compact visual evidence units; only a small set of relevant evidence is sent to a VLM. The key design choice is evidence packaging: 3×3 storyboards pack multiple frames into one image, increasing temporal coverage per prompt token.

66.7%

@ 3.9k prompt tokens — best efficiency, 3×3 storyboards.

70.6%

@ 12.2k prompt tokens — highest accuracy, tile-level retrieval.

~85%

fewer prompt tokens than frame baselines at matched accuracy.

Presentation video

A short NOSSDAV ’26 presentation walks through the motivation, the storyboard retrieval pipeline, and the main accuracy–token trade-offs. Open on YouTube →

video archive→ shot / frame sampler→ 3×3 storyboard→ embed + index→ retrieve→ VLM answer

Retrieval results and controls

{{ retCur.title }}

Qualitative example · Grounded reasoning from a storyboard

A 3 by 3 storyboard of egocentric construction frames showing the person handling an angle grinder near a plastic dust sheet. — Sample 3×3 storyboard · Ego4D video 5014 · 1280px

Reasoning-guided selection

Q: Where are the red pencil and folding ruler when the person picks up the angle grinder?

Options

A. Scaffolding plank

B. Plastic dust sheet ✓

C. Electrical backbox

D. Behind metal track

Model reasoning

“…visible on the white plastic dust sheet on the ground, near the base of the wall…”

→ Answer: (B) ✓

The reasoning trace confirms the model is interpreting the retrieved storyboard evidence, not guessing from text alone.

Results · Ego4D pilot

Click a column to sort. Teal-marked rows are the efficiency and accuracy operating points.

{{ h.label }} {{ h.arrow }}
{{ r.rep }}	{{ r.config }}	{{ r.accD }}	{{ r.tokD }}	{{ r.note }}

Daily egocentric QA is visual-first. Storyboards increase temporal coverage per prompt token; retrieval granularity gives a tunable accuracy–cost knob. Evaluated on an Ego4D pilot — not on Project Aria data; OCR and lightweight metadata are natural future additions.

Module 04 · Situated Context Serving

System Vision · ContextStation Position Paper

Pair user-controlled memory with place-bound context

Personal agents need more than memory; situated questions also depend on current, place-bound facts. ContextStation reframes the local edge as a context-bearing serving tier, separating a regional Context Plane from an Inference Plane that decides whether a request is answered on a personal device, at a nearby station, or through controlled cloud fallback.

ContextStation architecture: a Personal Device Tier sends situated requests to a local edge serving tier with a Context Plane (agent-agnostic place memory) and an Inference Plane (serving placement and orchestration), with optional cloud-tier fallback. — Figure 4. ContextStation tiers. A bounded, read-only Context Plane serves regional evidence to an Inference Plane; cloud models may consume context without owning it.

Personal memory is user-specific. Regional context is place-bound and shared. Personal evidence stays user-controlled; nearby context — building hours, notices, events, device state — is served through bounded interfaces.

Scenario 01

“Is the office I’m looking at open now?”

Needs image grounding, location, current building hours, and temporary notices.

Scenario 02

“What did I read earlier about this device?”

Combines user-controlled reading evidence with nearby device or lab context.

Scenario 03

“Summarize today’s relevant events.”

Future AR journaling may combine selected personal evidence with place-bound context and user intent.

Context ownership ≠ model capacity.
Regional context can be reused across nearby authorized users.
Bounded read-only lookups keep models from owning the context store.
Cloud models can be consumers of context, not default owners.

Putting the Pieces Together

An evidence lifecycle, not a finished product

These components are not yet one completed AR Journal. They are building blocks toward a system in which egocentric sensing becomes structured evidence, evidence remains under user-aware control, retrieval respects token and privacy constraints, and personal agents answer with grounded context rather than raw footage.

01

Raw egocentric video is not yet useful personal memory.

02

Evidence construction requires event detection, selection, grounding, and provenance.

03

Long-term retrieval must balance accuracy, token cost, privacy, and compute.

04

Nearby context serving helps agents answer situated questions without making the cloud the default owner.

Open questions

?How do we robustly select the right moments from continuous egocentric streams?
?When should we use modular CV pipelines versus unified multimodal VLMs?
?What evidence should remain on a personal edge device, and what can be safely offloaded?
?How should personal memory and regional context be combined without confusing ownership?

Contact and Materials

Papers, source materials, and extended figures

This page accompanies the Project Aria Research Summit poster “Toward Personal Egocentric AR Journaling.” Links below collect the related publications and project materials.