#Tag · bonfire.cafe

People form ad hoc conventions, by establishing linguistic & gestural abstractions, and shift information across speech and gesture to communicate more efficiently over time.

In our upcoming #CHI2026 paper, we study how these multimodal communications evolve in repeated physical collaboration.

Led by Kiyosu Maeda in close collaboration with @jefan, @rdhawkins, and team: William McCarthy, Ching-Yi Tsai, Jeffrey Mu, and Haoliang Wang.

🧵👇 1/4

Top: Modality shift in block instruction: R1: Take the green block and put it on the left side of the grid. A hand is holding an imaginary piece toward the left column of a 2×2 grid; label reads Redundant position and orientation. R4: the green block pointing this way. A hand is pointing near the bottom left cell with an arrow showing movement toward the top left cell; label reads Complementary position and orientation. Target tower: a 3 block green and red C-shape tower on a 2x2 grid. Bottom: Modality shift in tower instruction: R1: they are going to form a C-shape. A c-shape hand pose with the index and thumb is shown far from the grid; the label reads No information about position or orientation. R4: Put the C on the left side, facing away from you. Right hand shows the C shape facing away, and left hand with the palm open indicates placement on the left side; labels read Redundant position and orientation.

Parastoo Abtahi

@parastoo@hci.social · last week

Using #AR, we carefully isolate speech and gestures, removing other cues (e.g., gaze, facial expressions). This allows us to analyze how partners coordinate on abstractions and how information shifts across these modalities over time.

We develop a computational model, extending the Rational Speech Act (RSA) framework to multimodal settings, and simulate the behaviors we observe.
2/4

Princeton HCI boosted

Parastoo Abtahi

@parastoo@hci.social · 2 weeks ago

Check out “Explainable OOHRI: Communicating Robot Capabilities and Limitations as AR Affordances” at #HRI2026 for more details!
🔗 Project page: xoohri.github.io 📄 Paper: arxiv.org/abs/2601.14587
Led by Lauren Wang, in collaboration with Mo Kari @hci and @princetoncs.
#HCI #AR #Robotics #HRI
4/4

Parastoo Abtahi

@parastoo@hci.social · 2 weeks ago

Beyond pick-and-place, X-OOHRI exposes abstract robot actions via a radial menu after selecting a real-world object. Users then manipulate virtual twins to specify missing spatial parameters.

This can also support remote teleoperation 🎮

3/4

GIF

Parastoo Abtahi

@parastoo@hci.social · 2 weeks ago

Parastoo Abtahi

@parastoo@hci.social · 3 months ago

📣 I’m recruiting 1–2 #HCI PhD students interested in spatial computing, #AR, and #HRI. More information: https://parastooabtahi.com/applicants

If you’re interested, apply to Princeton CS by Dec 15 and mention my name in your application.

Parastoo Abtahi

Princeton HCI Applicants — Parastoo Abtahi

sohyeon hwang boosted

Lei Zhang

@raynez@hci.social · 6 months ago

Most AI-driven AR for children casts them as consumers, not creators. That’s like being able to read but not write. How can we help them write with #AI and #AR? 🪄

In our #UIST2025 paper, we introduce Capybara, an authoring tool that empowers kids to build expressive, AI-enabled AR experiences.

🧵1/6

Lei Zhang

@raynez@hci.social · 6 months ago

Most AI-driven AR for children casts them as consumers, not creators. That’s like being able to read but not write. How can we help them write with #AI and #AR? 🪄

In our #UIST2025 paper, we introduce Capybara, an authoring tool that empowers kids to build expressive, AI-enabled AR experiences.

🧵1/6

Parastoo Abtahi

@parastoo@hci.social · 6 months ago

Even virtual agents’ actions can have physical effects, with motion paths that divert attention from the hidden robot. 🐝

3/5

Parastoo Abtahi

@parastoo@hci.social · 6 months ago

In #AR, using real-time on-device 3D Gaussian splatting, we create the illusion that physical changes occur instantaneously, while a hidden robot fulfills the “reality promise” moments later, updating the physical world to match what users already perceive visually. 🤖

4/5

Parastoo Abtahi

@parastoo@hci.social · 6 months ago

Check out Lauren Wang’s #UIST2025 poster on GhostObjects: life-size, world-aligned virtual twins for fast and precise robot instruction, with real-world lasso selection, multi-object manipulation, and snap-to-default placement.

This is the first piece in her ongoing work on #AR for #HRI 🤖👓 @hci

Pedro Lopes

@pedrolopes@hci.social · 7 months ago

Want to work with one of the best thinkers out there, with endless creativity and super supportive environment in futuristic Taipei? 😎
My former PhD student, (now Prof.) Shan-Yuan Teng just started the Dexterous Interaction Lab @ NTU https://lab.tengshanyuan.info/#hci #VR #AR #haptics #phd #AcademicChatter 😍