Introduction
Leveraging the strong communication capabilities of Large Language Models (LLMs) as foundation models, our framework implements a dual-module architecture with a DAG (Directed Acyclic Graph) structure: a Coordinator that provides high-level, low-frequency strategic guidance, and a Manager that delivers subtask-specific, high-frequency instructions. This design enables both passive and active interaction modes, allowing robotic agents to seamlessly transition between supportive and directive roles based on real-time assessment of human needs and task demands.
HRT-PR interface based on Overcooked AI environment
Environment
We built our system on top of the original Overcooked AI environment, which provides an ideal testbed for studying human-robot collaboration as it requires coordination, task division, and real-time communication—key elements of effective teaming.
Different game layouts used in our experiments
Agent Modes
Our framework supports three different agent interaction styles:
Overview of different agent modes and their characteristics
Default
- A greedy agent collaborates with humans without any communication
- No proactive or reactive feedback
- Baseline for comparison
Passive Leader-Follower Style
- Agent receives and acts upon instructions based on human preferences
- Reactive communication only
- Supportive role in collaboration
Active Peer-to-Peer Style
- Besides receiving and acting upon instructions, it also provides feedback based on human past behavior
- Both proactive and reactive communication
- More directive role in collaboration
Demo Video
The human player controls the blue hat agent, while the AI agent (red hat) adapts its communication frequency and strategy based on the HRT-PR framework.
Survey Design
Survey design and participant feedback collection methodology
Results
Experimental results showing the relationship between task complexity and optimal communication frequency
Detailed analysis of performance metrics across different agent modes and task complexities
Key Findings
Our results reveal a nuanced relationship between task complexity, human capabilities, and optimal robot communication strategies. As task complexity increases relative to human capabilities, human teammates demonstrate a stronger preference for robots that offer frequent, proactive support.
Critical Threshold
However, we identify a critical threshold: when task complexities exceed the LLM's capacity, superactive robotic agents can generate noisy and inaccurate feedback that hinders team performance.
Principles
Based on our experimental results, we identify four key cases that determine optimal robot communication strategies:
Four cases of task complexity (Th) vs. agent capability (Cl) and human capability (Ch)
Case 1: Low Task Complexity, High Capabilities
Th < Cl, Th < Ch
- Task is manageable for both human and agent
- Minimal communication needed
- Passive or default agent mode optimal
Case 2: High Task Complexity, High Capabilities
Th > Cl, Th < Ch
- Task exceeds agent capability but human can handle
- Agent should provide supportive communication
- Passive leader-follower style recommended
Case 3: Low Task Complexity, Low Capabilities
Th < Cl, Th > Ch
- Task exceeds human capability but agent can handle
- Agent should take more active role
- Active peer-to-peer style optimal
Case 4: High Task Complexity, Low Capabilities
Th > Cl, Th > Ch
- Task exceeds both human and agent capabilities
- Communication may become noisy and counterproductive
- Need to balance support without overwhelming