We say a
lot about...
everything

After writing the previous blog post about face recognition, we decided to build a real-world face recognition project in Ars Futura – a face recognition door lock. Smart Lock, as we call it, recognizes people based on their face and unlocks the door if the person works at Ars Futura.

As you can see in the video above, we installed a front door camera that unlocks the door based on who is outside the office. I can proudly say that we have had Smart Lock running for over a year. After some initial hiccups, we managed to get it working pretty well. Smart Lock successfully recognises our employees in >95%¹ of the cases, we practically got used to it, it’s the default way to enter the office now². 😊

Let’s dive into the details and explain how it works. Smart Lock consists of 4 parts: Door API, Camera, Face Recognition API, and Smart Lock Engine. In the rest of this post, I’ll explain each of these components and how they interact. Let’s get started!

DISCLAIMER: Before we dive into the details, it’s IMPORTANT to note that Smart Lock unlocks the door based on IMAGES, taken from a camera. This means that someone can just print a photo of your face on a piece of paper, show it to the camera, and get access to your office/home which makes Smart Lock NOT VERY SECURE. Ars Futura’s office, where we have Smart Lock installed, is located in a business building with security personnel at the entrance, checking the ID of anyone entering the building. Also, Smart Lock is enabled only during working hours.

Door API

I’ll start with Door API because it is a crucial part, your door needs to provide an API to unlock it programmatically. If you have a commercial smart lock installed, such as the August lock, chances are they provide the API for unlocking (you can use this nice python package for August). Unfortunately, our front door is not compatible with any of the existing smart locks with an open API, so we had to create our own solution.

Fortunately, our door has an electric strike which enables us to unlock it by applying electric current to it. But how do you do that programmatically? By using a Raspberry Pi and a 5V relay. What’s a relay? Yeah, I was wondering the same thing when a guy from the door maintenance company told me I need a relay. A relay is basically an electric switch that can be controlled with a small current. We connected the Raspberry Pi to the 5V relay using GPIO pins and connected the relay to the door’s strike electric circuit. The Raspberry Pi can generate a small current through GPIO pins which allows us to control the relay programmatically → we can control the door’s electric strike through a Raspberry Pi. Here is a nice tutorial on how to control the relay with a Raspberry Pi.

WARNING: If you are working with alternating current, be extremely cautious! Don’t try anything without the supervision of a trained professional!

I wrote a simple Flask API with the /unlock endpoint which activates relay through GPIO pins. When the relay is “activated”, it connects the electric strike circuit which unlocks the door. That’s how our door lock became “smart”, now we have an API for unlocking the door. Check api.py for full implementation.

If you decide to install the same setup on your electric strike-powered door, 👉 check out the instructions on how to run Door API.

The easiest way is to run the pre-built Docker image:

docker run --name smart-lock-door-api --privileged --restart=always -d -p 80:80 arsfutura/smart-lock-door-api:latest --pin GPIO_PIN_INDEX

GPIO_PIN_INDEX is the index of the Pi GPIO pin which controls the relay. Once you have Door API up and running, the following request should unlock the door:

POST http://DOOR_IP/unlock

Door API will return 202 if the unlock request is successfully accepted.

One of my colleagues created a simple Mac app that uses this API to unlock the door, another created the iOS and Apple watch app to unlock the door. We also created an iOS shortcut which enables us to unlock with Siri, by just saying “Siri unlock Ars Futura door” which will, in turn, unlock our office doors using this API. That’s cool!

Camera

Camera is a component that provides a video stream. Our only requirement was to easily connect to the stream programmatically.

We implemented this part with a Raspberry Pi and a Raspberry Pi Camera v1.3 because we had a few spare Pis in the office and a Pi camera we didn’t use, but any IP camera will do here. We found a case for the Pi and the camera, 3D printed it, glued it to the wall in front of the office, and drilled a hole through the wall for the power supply.

picamera library provides streaming server code which we used.

This is what it looks like³:

Yeah, I know, it’s not the prettiest camera you’ve seen, but it does its job. 🙂

If you decide to use a Pi with a camera, we prepared a Docker image you can run on Pi (make sure you configure the camera before using it):

docker run --name smart-lock-camera --privileged --restart=always -d -p 80:80 arsfutura/smart-lock-camera:latest

Once you have the Camera up and running, you can access the live camera stream on http://CAMERA_IP and the MJPG stream on http://CAMERA_IP/stream.mjpg.

Face Recognition API

Face Recognition API provides the /face-recognition endpoint for classifying people in images. Check out the face-recognition repository for instructions on how to generate the Face Recognition API.

This is the core part of the Smart Lock and it’s the hardest part to get right. I’ll come back to this later in the post.

face-recognition

A framework for creating and using a Face Recognition system.

• deep-learning
• face-recognition
• facenet
• mtcnn

python

Smart Lock Engine

Engine is a component that connects everything together and makes Smart Lock work. Engine connects to the Camera, does the classification of camera frames using Face Recognition API, and unlocks the door using the Door API if a known person is recognised on a camera frame. The actual engine algorithm is a little bit more complicated, check engine.py for the details.

Engine depends on Camera, Door API, and Face Recognition API but these three components don’t have to be the exact ones described here. Camera stream can really be any video stream that is supported by OpenCV VideoCapture. Door API can be any HTTP POST endpoint, Engine doesn’t care what it does. Engine does assume that Face Recognition API is from the face-recognition library, but again, it treats it as an HTTP POST endpoint. If you implement the same response as Face Recognition API, you can have anything behind that endpoint. Basically, instead of unlocking, you could set up Engine to send you a message when someone appears in the video stream.

Again, there is a Docker image published, so you can easily run Smart Lock Engine if you want:

docker run --name smart-lock-face-recognition-engine --restart=always -d arsfutura/smart-lock-engine:latest --threshold THRESHOLD --smart-lock-camera-url SMART_LOCK_CAMERA_URL --face-recognition-api-url FACE_RECOGNITION_API_URL --smart-lock-api-url SMART_LOCK_URL

Ars Futura Smart Lock setup

In Ars Futura, Smart Lock components are deployed as Docker images on two Raspberry Pi 3 Model B+ and a Mac Mini 2018. Door API is deployed on a Raspberry Pi which is connected to the door electric strike, Camera is deployed on a Raspberry Pi with a camera outside of the office, Face Recognition API and Smart Lock Engine are deployed on Mac Mini in our office.

You can access Smart Lock’s components only from our local network in the office, which means you cannot unlock the door without having access to our WiFi. Also, communication latency between components is very low since all components are on the same network.

Why deploy on three devices instead of one?

Door API and Camera could easily be deployed on a single Raspberry Pi, but connecting the door’s electric strike and camera (which needs to be outside of the office) to a single Raspberry was harder than having two Raspberries where one is connected to the door on the inside and the other is mounted on the wall outside the office with a camera.

Face Recognition API and Engine are relatively computationally and memory intensive, so we couldn’t deploy these components on a Raspberry Pi (I tried). Luckily, we have an always-on Mac Mini 2018 in the office which we use for CI that was perfect for deploying these components. A Mac Mini is probably an overkill for this, you could probably get away with something less powerful.

Since Smart Lock is not very secure and it can be fooled by a picture, we enable Smart Lock only on weekdays from 7:00–19:00. We have flexible working hours, this time period z`covers everyone, and we always have a backup keypad. We implemented this with cron on a Mac Mini where we start Face Recognition API and Engine at 07:00 and shut them down at 19:00.

How to get it right

Smart Lock works really well in Ars Futura office. It’s accurate, it will unlock the door correctly in >95% of the cases. It’s fast, it takes approximately 1–2 seconds to unlock the door. Unfortunately, this is not so easy to achieve, it takes more than the information I gave you in this blog post to get it right. Here are the main pain points we encountered while working on this.

Communication Latency

You need Smart Lock to work almost real-time. Communication between the components needs to be fast. Engine needs to grab video stream frames really fast, Face Recognition API needs to process images as fast as possible, and unlock requests sent to the Door API need to be instant. Deploying components on the same network is the best solution for reducing communication latency.

The real bottleneck here is the Face Recognition API. Performing the classification of images takes time, much more time than it takes to grab a frame from a video stream. This means that you cannot call Face Recognition API for every frame from Camera, because if you do → when someone appears in front of the door, Face Recognition API will still be processing frames from seconds ago which will make unlocking the door lag a lot. Engine has an fps parameter that controls how many frames per second are processed with Face Recognition API. The optimal value of this parameter depends on how fast the Face Recognition API is. If you set it too high or too low, unlocking the door will lag too much. You need to experiment with it a little bit and find the optimal value for your setup. For more powerful machines that run Face Recognition API, this value will be higher.

Face Recognition

There are really two parts of the Smart Lock: the Face Recognition part and everything else. This post is about “everything else”.

face-recognition library makes it very easy to generate face recognition API without much knowledge about face recognition. But, you do need to know a thing or two about face recognition and machine learning to be able to compile a good dataset of face images and train the classifier.

I spent most of my time on getting the Face Recognition part right, and since this is such an important part, I decided to dedicate two more blog posts to this topic. There I’ll share my experience with it and try to give you a guide on how to do it right. Stay tuned!

That’s all folks! You can find Smart Lock’s code on Github. I hope you enjoyed this post, let me know what you think. If you decide to give it a try with Smart Lock and install it, I’d love to hear from you! Please share your experience in the comments. 🙂

smart-lock

Face Recognition Door Lock

• IoT
• Machine Learning
• Deep Learning
• Smart Home
• Face Recognition
• facenet
• Smart Lock
• Smart Door

python

I want to thank Ivan Oštrić, Luka Bašek, and Ivan Božić for reading the draft and providing useful feedback.I want to thank Borjana Katić for creating visualizations for this post. Special thanks to Luka Drezga for the awesome animation! Last, but not least, I want to thank Lea Metličić for editing the post (many times 😀 ). Thank You! :)

[1] What I mean by >95% is that it will let you in the office in 95% of the cases. I didn’t really measure this number (it’s complicated), I just talked with people from the office who told me there were some rare occasions when Smart Lock didn’t let them in. But in the vast majority of cases, the door would be unlocked within a second or two. Based on this, I estimate that in 95% of cases Smart Lock works fine.

[2] I know what you are thinking right now “There is a COVID-19 pandemic out there dummy! Everyone is working from home, you might as well have a 100% success rate because nobody is coming to the office”. You are (partially) right because the majority of our employees have been working from home for the last year. The first version of Smart Lock was actually deployed ~2 months before lockdown, so it had exposure to all the employees for that period. After that, there were some lonely times in the office where there was only me and the CTO, but eventually, people started to return to the office, and at the time of writing this (February 2021) we have ~40% of the people working from the office. 😊

[3] Yes, we drilled a hole in the wall for RPi’s power supply, and yes, we accidentally made a huge hole on the outside and covered it with this ugly plastic square.