Virtual Sensor Data Generation Using LLM Agents in Simulated Home Environments
Smart-home activity recognition is starved for labelled data. AgentSense generates it: an LLM writes residents and their routines, embodied agents live out those routines in a simulated home full of ambient sensors, and the sensor stream pretrains real-world recognisers.
A day in a simulated smart home
Replay a generated day. The agent follows its LLM-written routine through the home, motion sensors fire when it is in range, doors and appliances report when they are used, and every step carries the activity labels of five real smart-home datasets. Click the timeline to jump to any moment.
From a persona to a sensor stream
Each stage is generated, not hand-written. Step through real outputs of the pipeline for one persona.
One persona, ten homes
The same Monday routine, executed in ten different VirtualHome layouts. The routine is grounded to whatever objects each home actually has, so every layout yields a different trajectory and sensor stream.
Eighteen generated personas
Ages, occupations, health conditions and lifestyles written by the LLM. Each persona's weekly schedule and daily routines follow from its description.
Pretraining on AgentSense data helps on every real dataset
A recogniser pretrained on virtual data and fine-tuned on real data, against the same model trained on real data only. Mean ± std over three folds.
Largest gains when real data is scarce
Performance as the amount of real training data grows (TDOST-Basic). Hover for values.
Every kind of diversity adds up
Aruba macro F1 with a fixed amount of virtual data, adding homes, days of the week and personas (TDOST-Basic).
Inside the extended VirtualHome
Agents executing real generated routines in the extended VirtualHome (Unity), rendered for this page. The motion sensors in the replay above read exactly these movements.
BibTeX
@inproceedings{leng2026agentsense,
title = {AgentSense: Virtual Sensor Data Generation Using LLM Agents
in Simulated Home Environments},
author = {Leng, Zikang and Thukral, Megha and Liu, Yaqi and
Rajasekhar, Hrudhai and Hiremath, Shruthi K. and He, Jiaman
and Pl{\"o}tz, Thomas},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
volume = {40}, number = {3}, pages = {1891--1899},
year = {2026}
}