With the prevalence of smartphones, pedestrians and joggers today often walk or run while listening to music. Since they are deprived of their auditory senses that would have provided important cues to dangers, they are at a much greater risk of being hit by cars or other vehicles. PAWS/SEUS is a wearable system that uses multi-channel audio sensors embedded in a headset to help detect and locate cars from their honks, engine and tire noises, and warn pedestrians of imminent dangers of approaching cars. The system acts as a second pair of ears in situations where the user’s sense of hearing is greatly diminished, such as when the user is taking a phone call or listening to music, and warns users of imminent dangers well in advance, allowing users ample time to react and avoid traffic accidents.
System Implementation
We develop a three stage architecture consisting of an array of headset mounted audio sensors, an embedded front end that samples from the audio sensors and performs signal processing and feature extraction on the observed audio signals, and a smartphone application that performs machine learning based classification using the features extracted from the embedded front end to detect and localize oncoming vehicles.
The audio sensors mounted on the headset are sampled by the embedded front-end platform, also housed within the headset, to estimate the interaural time difference (ITD) and interaural level difference (ILD) between the microphones. Just like in human ears, we leverage the differences between the time in which sounds arrive at each microphone (ITD) and the differences in the loudness of the sounds at each microphone (ILD) to estimate the direction of the car with respect to the user. The ITD and ILD features extracted from the embedded front-end are transmitted to the smartphone application, where we employ novel machine learning classifiers to detect and localize vehicles. A diagram of the headset integrated with the audio sensors and embedded front-end is shown below.
To detect vehicles, we propose a new frequency domain feature, Non-Uniform Binned Integral Periodogram (NBIP), that outperforms standard Mel-frequency cepstral coefficients (MFCCs) when used in a machine learning classifier to detect vehicles. Unlike standard MFCCs, NBIP allows designers to emphasize or de-emphasize different parts of the spectrum depending on the application at hand. We tuned the parameters for computing NBIP features that capture the most important portions of the spectrum for detecting vehicles and show that we are able to achieve a greater separation between vehicle and non-vehicle sounds using NBIPs than using standard MFCCs.
We evaluate PAWS in three different urban environments (in a college town, near a highway, in a metropolitan area), showing that the system is able to provide early danger detection in real-time, from up to 80m distance, with up to 99% precision and 97% recall, and alert the user on time (about 6s in advance for a 30mph car).
Reducing Power Consumption
In order for the system to be practical, it needs to be long-lasting. To reduce the power consumption of the embedded front-end platform, we propose, implemented, and incorporated an application-specific integrated circuit (ASIC). The ASIC uses polarity coincidence correlation (PCC) to estimate the interaural time differences between microphones at nano-watt level power consumption. This reduced the power consumption of the embedded front-end platform by an order of magnitude, allowing the system to run continuously on two coin cell batteries for almost a full day.
Ongoing Work
Our vision is to create an audio safety platform that is easy to integrate almost anywhere, be it other wearables or smart city infrastructure, and is capable of detecting vehicles, localizing vehicles, and alerting users to imminent dangers in any urban scenario. However, there are many scenarios where the sound of an approaching vehicle could be entirely dwarfed by other extraneous sounds. A person could be speaking to someone else on his phone or a construction worker could be operating a power tool such as a jackhammer, making it very difficult to detect and localize oncoming vehicles. We are currently exploring sound source separation, adaptive filtering, and acoustic beamforming methods for enhancing vehicle sounds and filtering out loud or other unwanted noises commonly found in urban environments.
Contributors
Stephen Xia, Daniel De Godoy Peixoto, Bashima Islam, Md Tamzeed Islam, Shahriar Nirjon, Peter Kinget, Xiaofan Jiang
Code Repository
Front-End Code: https://github.com/Columbia-ICSL/PAWS-FrontEnd
Smartphone Application: https://github.com/Columbia-ICSL/PAWS-Smartphone
Publications
2019 | IEEE Internet of Things Journal
Improving Pedestrian Safety in Cities using Intelligent Wearable Systems
Stephen Xia, Daniel de Godoy, Bashima Islam, Md Tamzeed Islam, Shahriar Nirjon, Peter R. Kinget, Xiaofan Jiang
2018 | IEEE VNC ’18 / Taipei, Taiwan
PAWS+: A Smartphone Platform for Pedestrian Safety
Stephen Xia, Daniel de Godoy, Bashima Islam, Stephen Xia, Md Tamzeed Islam, Shahriar Nirjon, Peter Kinget, and Xiaofan Jiang
[Runner-Up Best App Award]
[Best Presentation Award]
2018 | ACM/IEEE IoTDI ’18 / Orlando, FL, USA
PAWS: A Wearable Acoustic System for Pedestrian Safety
Daniel de Godoy, Bashima Islam, Stephen Xia, Md Tamzeed Islam, Rishikanth Chandrasekaran, Yen-Chun Chen, Shahriar Nirjon, Peter Kinget, and Xiaofan Jiang
2018 | ACM/IEEE IoTDI ’18 / Orlando, FL, USA
An Ultra-Low-Power Custom Integrated Circuit based Sound-Source Localization System
Daniel de Godoy, Stephen Xia, Wendy Fernandez, Xiaofan Jiang, and Peter Kinget
[Best Demo Award]
2018 | IEEE CICC ’18 / San Diego, CA, USA
A 78.2nW 3-Channel Time-Delay-to-Digital Converter using Polarity Coincidence for Audio-based Object Localization
Daniel de Godoy, Xiaofan Jiang, and Peter Kinget
2016 | ACM SenSys ’16 / Palo Alto, CA, USA
SEUS: A Wearable Multi-Channel Acoustic Headset Platform to Improve Pedestrian Safety: Demo Abstract
Rishikanth Chandrasekaran, Daniel de Godoy, Stephen Xia, Md Tamzeed Islam, Bashima Islam, Shahriar Nirjon, Peter Kinget, and Xiaofan Jiang
[Best Demo – Runner Up Award]
You must be logged in to post a comment.