Artificial Intelligence Towards Intelligent Environments

DIA: A Drone-based System for Intelligent and Autonomous Homes

Homes are becoming more intelligent due to the growth of smart sensors and devices found in typical homes. However, most of these sensors and devices function independently from one another, limiting the amount of utility and services a truly “smart” home can provide. We introduce two key ideas towards more intelligent homes. First, we explore the usage of mobile drones in the home environment. Second, we propose DIA, a system that seamlessly connects to the home environment and automatically discovers and jointly utilizes smart sensors and actuators around the home to provide services that are otherwise not possible.

[SenSys’21 Demo][Demo]
[Best Demo Award]

SoFIT: Self-Orienting Camera Network for Floor Mapping and Indoor Tracking

We present SoFIT, an easily-deployed and privacy-preserving camera network system for occupant tracking. Unlike traditional camera network-based systems, SoFIT does not require a person to calibrate the network or provide real-world references. This enables anyone, including non-professionals, to install SoFIT. Once installed, SoFIT automatically localizes cameras within the network and generates the floor map leveraging movements of people using the space in daily life, before using the floor map and camera locations to track occupants throughout the environment.

Artificial Intelligence for Mental Health

aiMSE: Towards and AI-based Online Mental Status Examination

Mental status examination (MSE) is an important tool used by mental health providers for assessing mental health. Currently, MSEs are conducted by licensed professionals, which is a barrier for patients in low-income and remote areas. We propose an AI-based Personal Online Mental Status Examination (aiMSE), the first interactive mental status examination platform, where users can self-administer MSEs at home through a web browser, using only a camera and microphone. aiMSE uses multimodal image, speech and natural language processing algorithms to detect signs of abnormalities in mental functioning and recommend them for further examination by a mental health specialist.

Embedded Artificial Intelligence for Acoustics

AvA: Adaptive Audio Filtering for Mobile, Embedded, and Cyber-Physical Systems

There are many applications that would benefit from a platform that allows users and developers to select which sounds to keep and enhance and sounds/noises to remove and filter out. Such a platform is difficult to realize because of the large number of different sounds, models, and signal representations that developers may use in their applications. We introduce, AvA, an acoustic selective filtering architecture that intelligently integrates the physics of sound waves with a wide range of data-driven models in an adaptive feedback architecture to filter and enhance sounds depending on the application.

[Project Page]

EV-interfaced Smart Grid with Human in the Loop

Optimal Power Flow Estimation of Microgrid Considering the Grid Services of EV Batteries

Demand for the grid state estimation with partial power network observation is growing rapidly with the increasing amount of distributed energy resources (DER) connected to the grid with incomplete measured information. The grid services that could be provided by these DER, such as electrical vehicles (EVs), have the potential to affect the resilience and efficiency of the grid. This project proposes a constrained optimization solver based on the AC power flows to recover the incomplete information of the grid. Further reactive power constraints following the specifications in IEEE 1547 are added to the solver to explore the effects of adding grid services to the steady-state microgrid.

Platforms for Continuous Fever Screening

SIFTER: Low-Cost In-Situ System for Continuous Multi-Person Fever Screening

With the recent societal impact of COVID-19, companies and government agencies alike have turned to thermal camera based skin temperature sensing technology to help screen for fever. However, the cost and deployment restrictions limit the wide use of these thermal sensing technologies. In this work, we present a low-cost system based on a RGB-thermal camera for continuous fever screening of multiple people. This system detects and tracks heads in the RGB and thermal domains and constructs thermal heat map models for each tracked person, and classifies people as having or not having fever.

[Project Page]

Building Energy Optimization

ePrints: Personal Energy Footprinting

With the rise of population and emergence of Smart Buildings and Smart Cities, conserving energy is of prime importance. While people are careful about their usage at home, energy wastage in public buildings/offices goes unaccounted for as users do not pay for it. We develop a system that can generate real-time energy footprint for all users and provide related data analytics to help conserve their energy and even implement advanced functionalities such as prediction and  modeling.

[Project Page][BuildSys’17][TOSN’18]

RecEnergy: An Energy Saving Recommender System

Example Energy Saving Recommendation, with energy metrics.

Recent research efforts have made significant progress in reducing commercial building energy consumption through a variety of methods, including optimizing building heating, ventilation, and air conditioning (HVAC), lighting, and personal electric devices. However, these works focus on reducing energy consuming resources while treating occupants as immovable objects separate from the building energy optimization problem.

RecEnergy is a recommender system for reducing energy consumption in commercial buildings with human-in-the-loop. We formulate the building energy optimization problem as a Markov Decision Process, show how deep reinforcement learning can be used to learn energy saving recommendations, and effectively engage occupants in energy-saving actions.

[Project Page][UMAP’18][IoTJ’20]

CityEnergy: City-Scale Energy Footprinting

Example energy and population estimates for different buildings at 8 am in Manhattan.

In urban cities such as New York City, buildings are responsible for up to 75% of total greenhouse gas emissions, and over 90% of total benchmarked energy consumption. A significant portion of the energy consumed directly services humans in retail, commercial, and residential buildings. As sustainability increasingly becomes an important factor in modern society, energy consumption in the built environment is one area where reduction can have a major impact.

CityEnergy is a scalable, real-time system for computing personal energy footprint estimates. As shown in the figure, CityEnergy estimates the energy consumption and population for each building in real-time; these estimates are used to calculate the per capita energy footprint at the building level. CityEnergy is able to provide personal energy footprints to people who provide their location data through a mobile application.

[Project Page][BuildSys’19]

Wearables for Emotion Sensing

SPIDERS: Wearable Emotion Sensing

We present a System for Processing In-situ Bio-signal Data for Emotion Recognition and Sensing (SPIDERS)– a low-cost, wireless, glasses-based platform for continuous in-situ monitoring of user’s facial expressions (apparent emotions) and real emotions. We present algorithms to provide four core functions, using the bio-signals acquired from three non-contact sensorsm, running continuously for up to 9 hours and costing less than $20.

[Project Page][IPSN’20 Demo][IoTDI’20]

Audio Wearables for Urban Safety

PAWS: Pedestrian and Biker Safety

PAWS (formerly SEUS) is a smart system that alerts pedestrians and bikers of any possible vehicle collision. Addressing multiple areas of study, PAWS uses sensor fusion and advanced recognition algorithms to locate incoming cars and effectively warns the user to prevent accidents, especially in urban environments.

[Project Page][SenSys’16 Demo][IoTDI’18][IoTJ’19][Github]

CSafe: Construction Worker Safety

Construction worker safety is one of the most challenging scenarios in urban safety because construction sounds are often orders of magnitude louder than vehicles. As such, we developed and integrated a novel adaptive noise filtering architecture into a construction helmet wearable that filters out construction sounds, improving vehicle detection and localization.

[Project Page][IPSN’21][Presentation]

Mobile Health

Smart Textiles for Continuous Perspiration Sensing

We present a novel wearable sensor system that measures an individual’s sweat level. The system consists of multiple cotton-covered conductive threads that are then braided into one sensor. The braided 3D structure allows robust perspiration level estimation despite the sensor distortion or the motion of the person.


SPINDLES: A Smartphone Platform for Intelligent Detection and Notification of Leg Shaking

For many people, the habit of leg shaking is an annoyance to themselves and people around them. By simply becoming aware of the motion, one is able to voluntarily stop shaking. SPINDLES is a low-power, low-latency, non-intrusive and high accuracy mobile application to detect leg shaking and notify the user in a timely manner. SPINDLES is robust to the placement of the phone and implements a signal processing pipeline that eliminates the need for training and calibration phases common in machine learning approaches.


Intelligent Privacy Preserving Pillbox

We present a novel privacy-preserving pillbox system to help patients improve medication adherence while maintaining privacy and security. Existing smart pillbox systems have been successful in notifying, tracking and reporting medication adherence, but do not provide sufficient security and privacy for the patients’ medication adherence records. In the proposed system, we protect data that needs to be reported to the doctor by ensuring that the data can only be decrypted by the doctor, authorized by a face-to-face key exchange process through out-of-band communication channels. Our system also gives patients fine-grained control over how their information is transmitted and shared, which has been a major privacy concern for many patients.


Zika Virus Monitoring

The goal of this project is to identify and map possible breeding grounds of Zika carrying mosquitoes, using geotagged images, temperature, humidity and wind velocity information from a drone. We aim to visualize heat maps encoding the probability of a location being a potential breeding ground of Zika carrying mosquitos and make these visualizations widely available to stop the spread of the disease.

[Project Page]

Previous Projects


PAM: Pervasive Air-Quaity Monitoring
We approach the challenging problem of accurate and affordable PM2.5 monitoring from a novel cloud-based data analytics perspective. By carefully designing and building our own PM2.5 monitors, we are able to obtain reasonably accurate PM2.5 measurement in real-time and at low cost. And by aggregating their data, plus other types of data at the cloud, we are able to learn and create model for particulate matter, which in turn helps us calibrate sensors, and infer PM2.5 concentrations.

News and media:  CCTVPeople’s DailyNikkei TechnologyChinese Computer WorldDragonTVNanfang WeekendBusiness Times
Publications: SenSys ’14, MobiSys ’13 


QiLoc: A Qi-Wireless Based Platform for Robust User-Initiated Indoor Location Services
QiLoc is a simple yet effective way to accurately locate and identify occupants inside buildings. By utilizing the Qi wireless charging protocol, a QiLoc-enabled desk is able to identify a mobile phone placed on it and therefore locate the user. The cloud-side QiLoc server maintains location information of occupants, and provides a set of APIs via standard web services, such as location, ad-hoc group membership, and authentication. We demonstrate a number of smart-building applications, such as an Android app to locate others in the same room, a Windows widget to popup notifications on colleagues’ entry/exit events, and a proof-of-concepts integrating precise and live location information with calendar, instant messaging, and email systems.


VeriCloud: Cloud-based Smartphone Geniune Verfications
VeriCloud is an IoT approach to tackling the counterfeit smartphone problem. An Android app is develop to generate unique “fingerprints” for Android smartphones, taking in account of both hardware specs and software information; the Android app also performs benchmarks that complete within 30 seconds. These data are used in combination with an online “signature database” to identify whether a smartphone is counterfeit of genuine. Several techniques to used to grow the database with crowd-source data while maintaining “signature” integrity.


LiveSynergy: Bridging the Gap between Virtual and Physical Worlds
LiveSynergy is a novel magnetic-based wireless proximity detection platform, with accuracy and consistency better than existing solutions such as BLE, WiFi, ZigBee, and long-range RFID. Building on top of this platform, we provide cloud-based APIs that enable real-time and rich interactions between humans and their physical environment. LiveSynergy enables a range of indoor applications such as location-based targeted advertising and presence detection.

News and media: [video]
Publications: IPSN ’12, MobileSense ’12, SenSys ’11


SEPTIMU / MusicalHeart / LifeX
In this project, we build software and hardware solutions that utilize and/or augment mobile phones to continuously monitor users’ wellness without changing their existing lifestyle. Our system consists of HW/SW, apps, Cloud and social networks and features a closed-loop design with both sensing and actuating capabilities. In particular, instead of solely passive monitoring, we further explore the actuation possibilities, i.e., seek to leverage the social networks to properly motivate the user towards improved health conditions.

News and media: The EconomistNew ScientistCNET,  Gizmodo
Publications: IPSN ’12, SenSys ’12, HotMobile ’13, MobiSys ’13


PhoneWeb: The Other Social Network
The PhoneWeb project seeks to enable – through the use of new technologies such as Low-Energy Bluetooth, GPS, Low-Power Wi-Fi, Wi-Fi direct, and etc – handheld devices to continuously and accurately discover all the people around it and to create and maintain a local neighborhood map. We also seek to implement new types of local/social applications based on the PhoneWeb infrastructure.

News and media: [video]


Green Building / ACme / sMAP
ACme is an open source hardware and software platform that enables wireless energy/power measurement and control of AC devices. The ACme node fills the gap between inexpensive LCD watt-meters (e.g. Kill-A-Watt) and expensive networked enterprise energy monitors. The ACme network (pictured on the right) is an IPv6 based mesh network that enables direct IP communication with individual ACme nodes.

News and media: GigaOMCITRISLBNLMoteware
Publications: SenSys ’09, SenSys ’10, BuildSys ’10, IPSN ’10, FIIW ’11


LoCal: Architecture for Localized Electrical Energy Reduction, Generation and Sharing
The LoCal Energy Network is a cyber overlay on the energy distribution system in its various physical manifestations, e.g., machine rooms, buildings, neighborhoods, isolated generation islands and regional grids. Pervasive information exchange will enable a more efficient scalable energy system with improved resilience and quality of delivered power. LoCal brings together (1) pervasive information about energy availability and use, (2) interactive load/supply negotiation protocols, (3) controllable loads and sources, and, (4) logically packetized energy, buffered and forwarded over a physical energy network.
Publications: HotEmNets ’10, Energy2030, etc


SPOT: Micro Power Meter for Energy Monitoring of Wireless Sensor Networks at Scale 

Scalable Power Observation Tool (SPOT) enables in situ measurement of nodal power and energy over a dynamic range exceeding four decades (as small as μA) or a temporal resolution of microseconds.

Publications: IPSN ’07


Prometheus: Perpetual Environmentally Powered Sensor Networks 
Prometheus is a multi-stage solar powered system architecture that utilizes sensor node’s microprocessor to intelligently and efficiently manage energy transfers between multiple storage elements, resulting in near perpetual operation. Prometheus is implemented in the Berkeley Trio motes.

Publications: IPSN ’05


Ultrasound Ranging / Distributed Localization
Ultrasonic Ranging is a Mica2/Mica2Dot device designed for range estimation. It is used by the Calamari project lead by Kamin Whitehouse for distributed localization as part of the DARPA-funded Network Embedded Systems Technology (NEST) project.

Publications: IPSN ’04, EmNetS-II