1 Tattoo Detection for Soft Biometric De-Identification Based on Convolutional NeuralNetworks Tomislav Hrkać, Karla Brkić, Zoran Kalafatić Faculty of Electrical Engineering and Computing University of Zagreb, Croatia
Introduction Motivation o Omnipresence of surveillance cameras serios privacy risk for law abiding individuals o Aggravated by the development of various techniques for: video retrieval person re-identification o Juridical regulation for the protection of personal data o For video sequences: removing personally identifying features of recorded individuals in reversible fashion maintaining information on the action and its context 2
Introduction De-identification o removal of personally identifying features from data o Biometric features: fingerprints, iris, face,... o Soft biometric / non-biometric features: tattoos, birthmarks hairstyle clothing,... 3
Introduction De-identification o Proposed / comercially used approaches: face blurring soft and non-biometric features mostly ignored o Possible approaches to soft / non-biometric de-identification: de-identification of whole persons localization and de-identification of individual soft and nonbiometric identifiers 4
Related work o De-identification most work concerned with de-identifing biometric features, especially the face (e.g. Gross et al. 2009.) little to no work devoted to soft and non-biometric de-identification o Tattoo recognition studied in the context of forensic applications (Jain et al. 2008, 2013...) cropped images; content based image retrieval detection in the wild (Heflin et al. 2012) o Deep learning applied to wide variety of problems with competitive results convolutional neural networks can learn good features on their own (Krizhevsky et al. 2014, LeCun et al. 2015,...) 5
Proposed method Tattoos o Great variability of designs, skin color, lighting conditions, may resemble many different real objects... o Difficult to devise good hand-crafted features o Convolutional neural networks (CNNs) able to learn good features for many classification tasks 6
Proposed method Patch labeling using convolutional neural network (CNN) o CNNs: convolutional layers in charge of learning good features local receptive fields shared weights pooling layers reduce the feature space dimensionality fully connected layers perform the classification task contain majority of learned weights 7
8 Proposed method The proposed network architecture o inspired by VGGNet (Simonyan and Zisserman 2014)
Dataset o No readily available tattoo dataset we developed our own subset of ImageNet containing tattoos tattoos precisely annotated manually (890 images) 9
Dataset o Positive and negative samples: 22700 image patches generated by randomly sampling image patches (N*N) from the annotated images (2 classes: tattoo & background) 10
Training the network o Patches dataset (11359 P + 11341 N = 22700 total): training data: 2/3 of the dataset (7573 P + 7560 N = 15134 total) validation data: 1/6 of the dataset (1893 P + 1890 N = 7573 total) test data: 1/6 of the dataset (1893 P + 1890 N = 7573 total) patches extracted from the same image end: up in the same subset o Network trained on the training dataset optimizing mean sqared error loss stochastic gradient descent with momentum mini-batch: 32 samples learning rate 0.1 max 40 epochs; early stopping 11
Evaluation o Network trained and evaluated for different patch sizes: 8*8, 12*12, 16*16, 24*24, 32*32, 48*48 larger patches presumably provide more information about context smaller patches faster to train and test o Results (using test dataset) Patch size 8*8 12*12 16*16 24*24 32*32 48*48 # FN 152 229 187 213 248 290 FN (%) 8.03 % 12.10 % 9.88 % 11.25 % 13.10 % 15.32 % # FP 593 418 444 436 337 408 FP (%) 31.37 % 22.12 % 23.49 % 23.07 % 17.83 % 21.59 % Acc (%) 80.31 % 82.90 % 83.32 % 82.83 % 83.54 % 81.55 % 12
Tattoo detection - examples o Preliminary experiments in a sliding window setting original images labeled tattoo patches 13
Some observations o Larger patch sizes (up to 32*32) better accuracy less false positives more false negatives (!?) o Encurraging results on complete images relatively simple examples most tattoo patches correctly detected some misclassifications more difficult examples (more bg with textured objects) - #FP rises o Possible improvements to elliminate false positives combine with person detector (most FP are in the background) merge detections into blobs; disregard unconnected detections 14
Conclusions and outlook o Some conclusions: We addressed the challenging problem of tattoo detection for soft biometric de-identification Instead of hand-crafting the features, we applied deep learning Deep CNN trained using a dataset of positive and negative patches generated from a subset of annotated ImageNet tattoo images Using CNN to classify small image patches can be a reliable way to detect candidate tattoo regions Patch size should be kept up to 32*32 Further experiments needed o Possible improvements: Combining the tattoo detector with person detector Merging detections into blobs 15
16 Discussion o Thank you for your attention! o Questions?