Seat Occupancy Detection
This project is an attempt to determine seat occupancy of a class room, hall, library etc.
Image processing of live feed data through a camera enables us to determine empty seats. The camera is interfaced to the Raspberry Pi 3 which uses OpenCV to process the images. The processed information is uploaded to a cloud server where further processing enables data visualization.
We came up this project idea when we realized that we spent way too much time in searching for study spaces in libraries and reading halls. This project aims at simplifying this process by determining the occupancy of a room through an embedded system with cloud computing, whose data can be made availalbe on an Android app and a web app.
The system development of the project can be categorized into the following phases.
1) Capture image frames at intervals through a camera which does live monitoring of a room.
2) Interface camera to a single board computer for image processing.
3) Uploading of processed data to the cloud for storage and analysis.
4) Data visualization for seat occupancy, individual seat status and coordinates.
5) Android Application that offers a unified experience with website for observing the current seat availability.
1. Image Processing- The technicalities and the dynamic of the image processing algorithms are discussed below. The processing algorithm utilizes Haar features for training accuracy and their robustness. In Haar like feature set, there are multiple features. Few of them are shown below.
The processing steps used result in three different cases of operation: a. When we have multiple seats at the fixed position: This is the most basic case, where the position of the chairs are assumed to be fixed. Then the detection if the seat is empty or not is based on face detection algorithm based on Haar-like cascade features. This isn’t a real-life scenario, as chairs aren’t generally fixed, but movable. We tackled this problem in part-b of this project. b. Single moving chair: In this case, the position of the single chair is determined position, and it is tracked using the color detection algorithm. Then the occupancy is determined based on the same face detection method as discussed above. Then upon movement of the chair, the frame of detection of the chair is changed from fixed to dynamic, which can be further explored for various applications. c. Multiple fixed chairs: This scenario is much closer to real-life scenario. In this case, positions of the chairs are fixed and differentiation between the positions is done, based on color based detection. Also, face detection algorithms are implemented to detect the occupancy of the chairs. This case can be further explored by combining it with case b, where we have multiple moving chairs that can be tracked based on color based tracking algorithm. This can be considered as very close to real-life scenarios.
2. Server- Flask App and MongoDB database: FLASK app is made based on the FLASK framework for python which helps to design a RESTful API to send and receive data through HTTP requests. The FLASK app specifies independent app routes for handling POST requests from the Raspberry Pi which sends in the processed data in JSON format. The data such as the coordinates of the chair, Occupancy status, timestamp (to indicate time instance at which we are taking the information) and seat number of the chairs are parsed and stored in a MongoDB database. The data stored can be visualized through a web application that uses HTTP GET requests for requesting the data stored in MongoDB to a HTML template for rendering dynamic HighCharts.
3. Android Application and Data Analytics: Android Application is one of frontend interface alongside the web app. It also includes the data visualization in the form of dynamic graphs similar to that offered in the web app. The android application also indicates seat occupancy status at given time, percentage occupancy and even the coordinates of the chair.
Raspberry Pi: The Raspberry Pi is a series of credit card-sized single-board computers developed in the United Kingdom by the Raspberry Pi Foundation. The Raspberry Pi 3 is the third-generation Raspberry Pi and it has better connectivity and much improved specs over its predecessors. Its hardware specifications include a 64-bit ARM based processor and 1 GB RAM. It can run a variety of desktop class OS and applications. One of the many supported operating systems is the Raspbian OS, a Linux distribution. It can be used in embedded applications through its versatile interfacing and connectivity . It has 40 GPIO pins, HDMI input, 4 USB inputs, Audio jack and Ethernet port. It supports the latest versions of both Bluetooth and Wi-Fi and it is therefore IoT ready.
USB Camera: The portable USB live webcam by Creative is used for capturing live feed of images. It supports recording at a frame rate of upto 30fps and 720p. it additionally has built in noise canceller for minimizing background noise and interference. It has install clip for mounting on a stable video frame or a pedestal.
AWS EC2: Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers by allowing users to rent virtual computers to run their own computer applications. EC2 encourages scalable deployment of applications by providing a web service through which a user can boot an Amazon Machine Image (AMI) to configure a virtual machine, which Amazon calls an "instance", containing any software desired. A user can create, launch, and terminate server-instances as needed, paying by the hour for active servers – hence the term "elastic". EC2 provides users with control over the geographical location of instances that allows for latency optimization and high levels of redundancy. An Ubuntu 14.04 AMI is used through AWS EC2 for creating and running the FLASK app, storing received data through MongoDb and hosting the web application.
In this Seat occupancy detection system, we basically started off with a simple prototype of seat occupancy detection for single seat at a fixed position. Then, after the successful detection of the occupancy of the single seat using face detection algorithm based on Haar-like cascade features, we then went on to work on the similar mechanism for multiple seats with fixed position.
Then the next case was to work on the movement of the seat and detecting its movement and position using the color based detection. In color-based detection, we put different colored sheets on multiple seats and then use that as a differentiating component to determine the position of the seats. Then we adjusted our frame of detection of chair accordingly, to cope up with the movement of the seat. This is basically our second case.The color of the sheets signifies the color of the seat. As this is our first prototype, we took the liberty of attaching, colored sheets in order to develop and test a tracking algorithm.
Then in third case, the first two cases were combined to make a system for more than a single seat and fort determining both the position and occupancy of the seats. From this system in the third part, we figured out that multiple colored tags will be needed to detect multiple seats in the library or auditorium, which will very likely be the biggest problem of our system, typically when it comes to scaling.
Hence, we concluded that since this system is a prototype model of a long term, more accurate and robust system that we want to build, we need to go and implement higher level image processing and machine learning techniques, such as Neural Networks. In Neural Networks, we can try to make the system more accurate, by working more on the classifiers and boosting the necessary and strong ones and taking out the weak ones (more like the refined version of Adaboost algorithm).
We present result for the 3 parts of developing our image processing algorithm below-
1)Seat occupancy detection of fixed chairs with fixed frame of references-
This is the 4 regions of interest that we fixed for 4 chairs
Here we see the face detection result for the 3rd chair
We see the corresponding result in our android application
Here we see the face detection result for the 2nd chair
We see the corresponding result in our android application
2)Tracking of a single moving chair and determining its occupancy
This is the result for locating the position of the chair in one frame
This is the result for face detection in the region of interest
This is the result for locating the new position of the chair in another frame
This is the result for face detection in the new region
3)Detection of multiple chairs and determining their occupancy
Here is the frame of reference obtained for 2 different chairs
Here is the seat occupancy determined for seat1
Here is the seat occupancy determined for seat2
We see that we get some false positives for detection in seat1. These are some errors we obtained routinely during the execution of the third step. This needs to be filtered out in future through suitable algorithms.
We see this error in the android application output as well. Both seat1 and seat2 are shown as occupied-
We can summarize our results as following-
1) We are able to detect the occupancy of a single moving chair.
2) We are able to identify 2 different chairs and determine their occupancy.
3) We are getting some false errors for face detection and eye detection for the case where we wish to process multiple seats
4) Our approach is based on the Haar Face detection cascade, which fails beyond a particular distance. Adding eye classifier adds to false positives, though it improves accuracy in face detection
5) Our tracking algorithm works well and it is able to determine the presence of chair for large distances. The next step of this algorithm is detecting specific shapes of colours, which are inherent in chairs
6) After performing extensive tests and calibrations, we have come to the conclusion that our approach has severe limitations in terms of distance and lightining conditions. For better accuracy a neural network based training algorithm should be used as it offers better results for detection and rectification
7) We have implemented 3 concepts which can further be improved using neural networks to develop a truly dynamic seat occupancy detection algorithm.