In Part 4 of this tutorial, we extend deep learning models to deal with a specific type of object detection: the localization and identification of faces.
We need to reload all of the Python modules we used in the Part 1.
%pylab inline import collections import numpy as np import scipy as sp import pandas as pd import importlib import os from os.path import join from matplotlib.colors import rgb_to_hsv
Populating the interactive namespace from numpy and matplotlib
import matplotlib.pyplot as plt import matplotlib.patches as patches plt.rcParams["figure.figsize"] = (8,8)
Let's load and look at the third corpus we have provided. It contains still images from two episodes of the sit-com Bewitched.
bewitched = pd.read_csv("meta/bewitched.csv") bewitched.head()
|0||bw-s03-e03-mother-its-the-middle-of-the.jpg||Mother, it's the middle of the night.||1199||40.4||Bewitched||3||Witches and Warlocks Are My Favorite Things||1966-09-29|
|1||bw-s03-e03-thats-the-proper-time-for-us.jpg||That's the proper time for us, you know.||1259||42.4||Bewitched||3||Witches and Warlocks Are My Favorite Things||1966-09-29|
|2||bw-s03-e03-well-what-is-it-i-thought.jpg||Well, what is it? I thought you should know...||1393||46.9||Bewitched||3||Witches and Warlocks Are My Favorite Things||1966-09-29|
|3||bw-s03-e03-that-tomorrow-has-been-chosen-as.jpg||that tomorrow has been chosen as the day of th...||1499||50.5||Bewitched||3||Witches and Warlocks Are My Favorite Things||1966-09-29|
|4||bw-s03-e03-coven-why-who.jpg||Coven? Why? Who?||1614||54.4||Bewitched||3||Witches and Warlocks Are My Favorite Things||1966-09-29|
To run the code in this notebook from scratch, you will also need the face_recognition module for working with neural networks. This are not included in the default Anaconda Python installation and need to be installed seperately. The code below checks if you have face_recognition installed. If you do, it will be loaded. Otherwise, a flag will be set so that the code below that requires the module will load the pre-loaded data.
if importlib.util.find_spec("face_recognition") is not None: import face_recognition as fr fr_flag = True else: fr_flag = False
If you are struggling with installing these, we are happy to assist. You'll be able to follow along with keras, but will not be able to apply the techniques you learned today to new datasets without it.
Our last set of steps concern the location and identification of faces in an image. While faces are particularly interesting, similar models exist for detecting the location of other objects using neural networks.
As an example of what this new corpus looks like, here is an image of Darrin and Samantha in their living room at the start of the episode "Witches and Warlocks Are my Favorite Things".
img_path = join('images', 'bewitched', bewitched.iloc['filename']) img = imread(img_path) plt.imshow(img)
<matplotlib.image.AxesImage at 0x11384dd68>
There are two faces in this image, which we can detect using the face_recognition algorithm.
if fr_flag: faces = fr.face_locations(img, 1, model="cnn") else: faces = [(172, 500, 254, 418), (81, 346, 179, 248)] print(faces)
[(172, 500, 254, 418), (81, 346, 179, 248)]
The output indicates that two faces are detected, and the numbers gives the coordinates for the faces known as bounding boxes. We can plot them in Python with the following snippet of code.
fig,ax = plt.subplots(1,1) plt.imshow(img) n, m, d = img.shape for face in faces: rect = plt.Rectangle((face, face), face - face, face - face, edgecolor='orange', linewidth=2, facecolor='none') ax.add_patch(rect)
And, as hoped, the detected faces line up with the two characters in the frame.
In addition to detected where a face is, we also want to determine who the face belongs to. In order to do this, we again make use of a pre-trained neural network that returns a sequence of numbers. Just as in Step 11 with image similarity, we assume that faces of the same person are identified with similar sequences of numbers.
To illustrate how this works, lets take a set of four faces from Bewitched. The first two are of the same character (Samantha) but the second two are of of Larry and Darrin, respectively.
plt.figure(figsize=(14, 14)) for id, index in enumerate([145, 300, 420, 707]): plt.subplots_adjust(left=0, right=1, bottom=0, top=1) plt.subplot(1, 4, id + 1) img_path = join('images', 'bewitched', bewitched.iloc[index]['filename']) img = imread(img_path) plt.imshow(img) plt.axis("off")
We can compute the 128-dimension number associated with each face using the function
fr.face_encodings applied to each face.
if fr_flag: embed =  for id, index in enumerate([300, 145, 420, 707]): img_path = join('images', 'bewitched', bewitched.iloc[index]['filename']) img = imread(img_path) f = fr.face_locations(img, 1, model="cnn") e = fr.face_encodings(img, known_face_locations=[f]) embed.append(e) else: embed = np.load(join('data', 'face_test.npy'))
Using the first image of Samantha as a baseline, look at how close each of the other three images are to it.
embed = np.array(embed) np.sum((embed - embed[0, :])**2, 1)
array([0. , 0.20154738, 0.70860072, 0.71007563])
The other image of Samantha is just 0.202 away but the images of the two male characters are 0.709 and 0.710 away. Using a cut-off (around 0.35 works well), we can identify images of Samantha with a reasonably high accuracy.
Finally, let's apply our face detection algorithm of the entire corpus of Bewitched
iamges. The face detect is fairly slow (it took about 30 minutes on my small MacBook
to do all of the images), so we recommend you set
False and load
the faces in from the save file.
process_new = False if process_new: embed = ; output =  corpus = pd.read_csv(join("meta", cn + ".csv")) for index, row in corpus.iterrows(): img_path = join('images', cn, row['filename']) img = sp.misc.imread(img_path) faces = fr.face_locations(img, 1, model="cnn") output.append(faces) enc = fr.face_encodings(img, known_face_locations=faces) embed.append(enc) else: faces = np.load(join("data", "bewitched_faces.npy")) embed = np.load(join("data", "bewitched_embed.npy"))
We will compare each of the embeddings to first image used in Step 13, storing the
distance in the array
samantha = embed samantha_dist = np.ones(bewitched.shape) for item in range(embed.shape): for em in embed[item]: samantha_dist[item] = np.sum((samantha - em)**2)
Using a cut-off of 0.35, how many frames contain an image of Samantha?
np.sum(samantha_dist < 0.35)
It looks like a total of 119 images show a definitive image of Samantha. Using a similar block of code to the similarity metrics used throughout these notes, what images contain the most similar Samantha faces to our prototype image?
plt.figure(figsize=(14, 24)) sam_index = np.argsort(samantha_dist).tolist() for ind, i in enumerate(sam_index[:24]): plt.subplots_adjust(left=0, right=1, bottom=0, top=1) plt.subplot(8, 3, ind + 1) img_path = join('images', 'bewitched', bewitched.iloc[i]['filename']) img = imread(img_path) plt.imshow(img) plt.axis("off")