We are excited to share that Minghui (Scott) Zhao presented our work, “FlexiFly: Interfacing the Physical World with Foundation Models Empowered by Reconfigurable Drone Systems,” at SenSys’25 during CPS-IoT Week 2025. This research was developed through a collaborative effort between Columbia University and Northwestern University, with contributions from Minghui (Scott) Zhao, Junxi Xia, Kaiyuan Hou, Yanchen Liu, and Professor Stephen Xia, Professor Xiaofan (Fred) Jiang.
In this work, we addressed a fundamental challenge in enabling foundation models to interact meaningfully with physical environments. While foundation models excel at digital tasks, they struggle when required to understand localized events in large physical spaces or perform real-world actions. Our contributions are summarized as follows:
(1) Novel Foundation Model Orchestration Framework: We introduce a comprehensive framework that integrates multiple large language models, vision-language models, and an open-set object detection model for dynamic, task-driven sensor/actuator selection.
(2) Advanced Image Segmentation Approach: We propose a new segmentation method that reliably pinpoints task-relevant areas even in large, cluttered environments, overcoming limitations of existing approaches.
(3) Custom Reconfigurable Drone Platform: We present a purpose-built drone system capable of autonomously adapting its sensors and actuators in response to commands from the FM orchestration framework, enabling seamless physical interaction and significantly improving task success rates in real-world deployments compared to static sensor approaches.
FlexiFly enables foundation models to “zoom in” on areas of interest using reconfigurable drones that can automatically swap between different sensing and actuation modules. Through extensive real-world evaluations, our system demonstrated up to 85% improvement in task success rates across diverse applications including object identification, environmental monitoring, surveillance, and physical item delivery.
This work represents a significant step toward truly intelligent physical environments that can adapt to users’ needs through foundation model-powered robotic systems. For more details, please refer to our paper here.
You must be logged in to post a comment.