Hybridization of Facial Features and Use of Multi Modal Information for 3D Face Recognition
Subject Areas : Image, Speech and Signal Processing
1 - Computer Technology, Priyadarshini college of Engineering,nagpur
Keywords:
Abstract :
1. Adler, A., M.E.J.I.T.o.S. Schuckers, Man,, and P.B. Cybernetics, Comparing human and automatic face recognition performance. 2007. 37(5): p. 1248-1255; Available from: https://sci2s.ugr.es/keel/pdf/specific/articulo/Comparing%20Human%20and%20Automatic%20Face%20Recognition%20Performance.pdf.
2. Lu, X. and A.K. Jain. Automatic feature extraction for multiview 3D face recognition. in 7th International Conference on Automatic Face and Gesture Recognition (FGR06). 2006. IEEE.
3. Belghini, N., A. Zarghili, and J.J.S.I.I.J.C.A.S.E.D.E.S. Kharroubi, 3D face recognition using Gaussian Hermite moments. 2012. 1: p. 1-4; Available from: https://www.researchgate.net/profile/Naouar_Belghini/publication/269105595_3D_Face_Recognition_using_Gaussian_Hermite_Moments/links/5480e9cc0cf20f081e726b20.pdf.
4. Gervei, O., et al., 3D face recognition using modified PCA methods. 2010. 39; Available from: https://pdfs.semanticscholar.org/670f/30de92077cb242e76375374530ebd300cda9.pdf.
5. Gottumukkal, R. and V.K.J.P.R.L. Asari, An improved face recognition technique based on modular PCA approach. 2004. 25(4): p. 429-436; Available from: https://www.sciencedirect.com/science/article/pii/S0167865503002654.
6. Cavalcanti, G.D., T.I. Ren, and J.F.J.E.S.w.A. Pereira, Weighted modular image principal component analysis for face recognition. 2013. 40(12): p. 4971-4977; Available from: https://www.sciencedirect.com/science/article/pii/S095741741300153X.
7. Tan, K. and S.J.N. Chen, Adaptively weighted sub-pattern PCA for face recognition. 2005. 64: p. 505-511; Available from: https://www.sciencedirect.com/science/article/pii/S0925231204005600.
8. Kumar, A.P., S. Das, and V. Kamakoti. Face recognition using weighted modular principle component analysis. in International Conference on Neural Information Processing. 2004. Springer.
9. Berretti, S., et al., 3D face recognition using isogeodesic stripes. 2010. 32(12): p. 2162-2177; Available from: https://ieeexplore.ieee.org/abstract/document/5432188/.
10. Han, X., et al., Face recognition in the presence of expressions. 2012. 5(05): p. 321; Available from: https://file.scirp.org/pdf/JSEA20120500002_68626424.pdf.
11. Smeets, D., et al. Fusion of an isometric deformation modeling approach using spectral decomposition and a region-based approach using ICP for expression-invariant 3D face recognition. in 2010 20th International Conference on Pattern Recognition. 2010. IEEE.
12. Wang, C., et al., A hybrid method to build a canonical face depth map. 2011. 5(5).
13. Xu, C., et al., Automatic 3D face recognition from depth and intensity Gabor features. 2009. 42(9): p. 1895-1905; Available from: https://www.sciencedirect.com/science/article/pii/S0031320309000089.
14. Assadi, A. and A. Behrad. A new method for human face recognition using texture and depth information. in 10th Symposium on Neural Network Applications in Electrical Engineering. 2010. IEEE.
15. Hasan, M.H.M., et al., 3-D Face Recognition Using Improved 3D Mixed Transform. 2012; Available from: http://www.cscjournals.org/download/issuearchive/IJBB/Volume6/IJBB_V6_I1.pdf#page=19.
16. Thakare, N.M. and V. Thakare, A Robust and Novel Framework for Subdivision of Face Image Based on the Concept of Golden Rule. Available from: https://pdfs.semanticscholar.org/776c/757c96b42070ec58de5f7be2062ed82811b5.pdf.
17. Meisner, G.B., Beauty in the Human Face and the Golden Ratio. 2014; Available from: https://www.goldennumber.net/beauty/.
18. Research, C.f.B.a.S., Note on CASIA 3D Face Database. 2005; Available from: http://www.cbsr.ia.ac.cn/english/3DFace%20Databases.asp.
3
Journal of Advances in Computer Engineering and Technology
Hybridization of Facial Features and Use of Multi Modal Information for 3D Face Recognition
Abstract— Despite of achieving good performance in controlled environment, the conventional 3D face recognition systems still encounter problems in handling the large variations in lighting conditions, facial expression and head pose. The humans use the hybrid approach to recognize faces and therefore in this proposed method the human face recognition ability is incorporated by combining global and local information to develop a robust face recognition system.
In this papers, it is proposed that hybridization of global and local facial features and combination of 2D and 3D modality helps in improving performance of face recognition system. The main issue of existing face recognition systems is the high false accept rate which is not desirable when security is the main concern. Most of the existing face recognition techniques overcome these problems with some constraints. However, the proposed methodology has achieved better results and succeeded in handling all the three issues. Also the use of 2.5D images (Depth Map) and dimensionality reduced data (Eigen faces) has shown that the system is computationally reasonable.
Index Terms— 3D Face Recognition, Hybridization, Feature Extraction, Depth Map.
I. INTRODUCTION
E
ven though face recognition is the most non-intrusive and socially acceptable biometric methodology , the error rates are too high. The change in the lighting condition, facial orientation and facial expression increases both false rejection rates (FRR) and false acceptance rates (FAR), and therefore face recognition is considered as one of the most challenging biometrics. But still in most of the cases the computerized face recognition system outperforms human results because human’s face recognition capability depends on the external factors like motivation, fatigues, training and speed [1]. The performance of 2D face recognition is adversely affected by the factors like variations caused by pose, expressions, illumination, age, face-marks and the accessories used by the subject therefore this paper proposes use of combination of 3D and 2.5 D data.
1. Motivation
After studying the various 3D face recognition algorithms it is found that implementation of an efficient and robust face recognition system is still a research issue. Among the different biometric techniques facial recognition may not be the most reliable and efficient but it has several advantages over the others: it is natural, easy to use and does not require aid from the test subject. Despite the successes of many systems, many issues remain to be addressed. Among those issues, the following are prominent for most systems; the illumination effect, the varied poses, partial occlusion, scaling variability, low quality images, etc.
Use of 3D facial images for face recognition handles most of the above mentioned problem but the three vital problems that mostly degrade the recognition performance are variation in head pose, change in lighting conditions and different facial expressions. Because of these three major problems, the same face appears differently causing the face recognition systems to misclassify the identity of the Inter-personal as well as intra-personal face images. The researchers are still working on these three factors.
Motivation of this paper is to implement the 3D face recognition methodology based on hybrid approach by combining features from salient regions of face and use of both the modalities; 2.5 D and 3D information.
II. Literature Overview
The overview of literature on face recognition reveals that the performance of face recognition gets affected by numerous issues; the extensive work on face normalization, feature extraction and face recognition is still in consideration, and can be addressed by improvements during the various stages. The work proposed in this paper is based on following benchmarks.
1. Face alignment and Normalization
In pre-processing and normalization phase, the main aim is to normalize the face images so that the comparison of probe image with the images in dataset is done on common platform. In [2][3][4] proper face alignment is done by localizing the nose tip which help in normalizing the face images.
2. Face subdivision and Region Based Feature Extraction
While handling varied expressions, the conventional face recognition systems based on only global features encounter problems. The face recognition approaches based on contribution of various face regions work on the proper subdivision of a face image. Research works presented by [5][6][7][8][9][10][11] have proved that the region based face recognition or the face recognition based on local features work well on varied facial expressions.
3. Face recognition using Depth Map
The depth maps are used as 2.5 D representation of 3D image, containing the depth information of each point in the 3D image. It is simply an image with depth information having advantage over 3D images [12]. The depth maps are robust to the change of illumination and color because the value on each point represents the depth value which does not depend on illumination or color. A lot of work has been done on depth maps of 3D face models and as mentioned in [12][13][14][15], it is widely accepted by the face recognition research community that the depth maps handle the varied lighting effects efficiently. Therefore use of information in two forms is proposed in this paper; Mesh Images (3D) and Depth Map images (2.5 D).
III. Proposed Methodology
This paper aims at developing the 3D face recognition methodology which is robust against illumination effect, changing expressions and varied poses of a face image. After analyzing the current state-of-the-art techniques, it has been found that the reasons for poor recognition result are; improper pre-processing and normalization framework, selection of irrelevant features, and inefficient recognition technique. The objective of proposed methodology is to perform the following tasks:
i. Implementation of precise pre-processing and normalization framework by sub-dividing facial image using Leo-Nardo- Da-Vinci’s Golden Rule for face structure [16][17]. It is used to extract global/ holistic features and relevant local features.
ii. Building a robust face recognition model with an ability to combine features from depth Maps (2.5. D) and Mesh images (3D)
1. Objective
The main objective of proposed methodology is to develop an efficient 3D face recognition methodology using hybrid approach. Thus implementation can be achieved by following technique.
The conventional approach like holistic feature extraction does not work significantly so combination /fusion of holistic and local features should be adapted. Information from local regions handles the problems of varied expressions therefore the local regions like forehead, eyes, nose and mouth should be precisely separated from the face image. Therefore proper implementation of automatic face split-up framework is needed to improve performance of face recognition systems. The resemblance to human face recognition capability should be implemented by combining all the prominently contributing features. Use of 3D images, depth map and shape information help in implementing fusion of information.
2. An Approach for Proposed Technique
The proposed methodology uses the hybrid approach at different levels; image representation, feature extraction as well as at decision level. Also for recognition purpose a fuzzy logic tool ANFIS (adaptive neuro-fuzzy inference system) is used.
1) 3D Face Dataset
The algorithms of the proposed system are tested on two datasets. i.e. 3D databases;
§ CASIA 3D face Database [18]
§ GavaDB 3D Face Database [19]
Both the databases contain images in VRML format with extention (.wrl) with variations in pose, expression and lighting conditions. As shown in figure 1.
Fig 1. CASIA Dataset
2) Outline of the proposed work
The proposed methodology claims that fusion of holistic and local feature based methodology outperforms the traditional approach. The fusion of information is proposed to implement the face recognition system which ensembles FR capability of the human-being.
The architecture of proposed methodology depicting the hybrid approach for face recognition is shown in the figure 2.
Fig 2. The hybrid approach for proposed methodology |
It consists of the following steps for implementation of the proposed face recognition system as shown in table I.
Table I : Algorithm: Hybrid Approach for 3D Face Recognition
GIVEN : A 3D probe image in ‘wrl’ format |
OUTPUT : A class of the probe face image
|
STEP1 : Read 3D face image in ‘wrl’ format
|
STEP2 : Preprocessing and Registration Localization of the nose tip Apply ICP for pose alignment (50-60 degree to frontal) Apply preprocessing on face image (Spike removal and Hole filling) Generate depth maps Crop and resize the image Apply coarse alignment using ICP
|
STEP 3 : Subdivide the image using Golden rule
|
STEP 4 : Feature Extraction Apply PCA on whole face as well as sub regions (PCA,MPCA,ASPCA) Generate Moment Invariant features |
STEP 5 : Recognize face using FNN Decide the similarity score
|
STEP 6: Fusion of adaptively selected classifiers (global and local Experts) using fuzzy Integrals |
STEP 7: Declare Results |
As mentioned in the proposed approach , Face images are converted into depth maps. Depth maps are considered as 2.5 D images, they are used to make the system cost effective. To incorporate human being’s inherent ability of combining global and local information of the face, a face image was divided into main four regions. For effective subdivision of face regions , based on the Leonardo da Vinci’s golden rule , Face- subdivision algorithm is applied as shown in Table II.
Further global and local features are extracted from whole face and four regions using PCA respectively. In addition to 2.5D information, Moment invariant features were extracted from mesh image of a face.
Table II: Algorithm : Face sub-division
GIVEN : A Depth map containing a whole face image |
OUTPUT : Four images containing fourface parts : forehead,eyes ,nose and mouth |
STEP1 : Resize the face image Resize the image based on the Dr. Jefferson’s theory based on a golden ration proportion for an ideal human face structure. Thus fibonnaci number is used to resize face image with Height=262 and width 162 units
|
STEP2 : Subdivide the image Refer the vertical proportion and subdivide and image horizontally in the proportion ; 0.87:0.44:0.44:0.87 |
STEP 4 : Save each region separately
|
ANFIS (Adaptive Neuro-Fuzzy Inference System) is used to construct a fuzzy inference system as shown in table III. The membership function parameters are adjusted using a back-propagation algorithm.
Table III : Algorithm : Classification using Fuzzy Neural Network
Let : Total Classes : C The feature vectors to be recognized : F The Fuzzy set membership functions : m1,m2,m3…mC
|
STEP 1 : Input the input-vectors : x1,x2,….xn containing PCA of depth map representations and moment invariants of mesh models of face images
|
STEP 2: If there are ‘n’ test images Generate ‘n’ sets of fuzzy rules
|
STEP 3 : Calculate weights using the difference between the desired output and actual output
|
STEP 4 : Defuzzifiction : Use threshold value to convert the membership value to either “0” or “1”
|
STEP 5 : declare the class with highest membership value as The class containing the test image |
Hybridization of local and global features as well as use of 2.5D and 3D data has contributed significantly to improve the performance of face recognition.
IV. Results and Discussion
1. Experiments and Results
Various sets of experiments were carried out to test the performance of the proposed methodology and to achieve the desired results. In training and testing dataset images with all type of variations are included. Experiments are carried out on two 3D face datasets; the CASIA dataset and GavabDB dataset. The CASIA dataset contains face images of 123 persons with 33 image samples per person. GavabDB contains the images of 61 individuals. For each individual 9 images are used to present variations in face poses and expressions.
The proposed system reads 3D face images from the face dataset. The images are in VRML format. The Z buffer algorithm converts 3D image into 2.5D image format (Depth Map Image. The Depth maps are the 2.5 dimension representations that contains depth information of 3D image. The depth map representation of a 3D image is shown in figure 3. In depth map each pixel shows the distance of the corresponding point from the camera. The closer point has the brightest pixel value.
Fig 3 Depth Map Representation |
The normalization process is implemented to align all the images in frontal position. ICP algorithm is used to align the face images; Nose tip is located as a reference point for alignment purpose. Then all rotated frontal images are cropped using OTSU algorithm as shown in figure 4. The cropped images are then are re-sized according to the proportion specified by the Leo Nardo da vinci’s Golden rule.
Fig 4 Face Image after Normalization |
The results are generated by carrying out following Experiments:
Experiment 1 : Face subdivision using Golden rule:,
This experiment was carried out to sub-divide a face precisely into four distinct regions and it was observed that by applying golden rule, face image can be subdivided in to four sub-parts as forehead, eyes, nose and mouth.
Table IV ::Performance of face subdivision algorithm using golden rule
Expt. | No. of persons | No. of images/person | I/p Image | Subdivision Performance | ||||
CASIA 3D | GavaDB | Dataset1 CASIA3D | Dataset2 GAVA DB | Rotated/ Frontal | Correctly Cropped | Incorrectly Cropped | % | |
1 | 123 | 61 | 18 | 5 | Front | 2499 | 20 | 99.2 |
2 | 123 | 61 | 4 | 2 | <300 | 609 | 6 | 99.1 |
3 | 123 | 61 | 4 | 2 | <=600) | 611 | 3 | 99.5 |
The performance of face subdivision algorithm using golden rule is shown in table IV. From the results obtained it is found that the images are properly cropped with the average performance of 99.29% for the images with up to 600 rotations. These cropped images are later used for extraction of local features.
Experiment 2: Neutral Vs Neutral Global features :
This experiment was carried out by extracting holistic features from neutral face images. This experiment has shown the performance of face recognition on face images without expressions.
Experiment 3 : Non-Neutral Vs Non-Neutral Global features :
In this experiment holistic features were extracted from the face images having different facial expressions. This experiment has demonstrated the efficient face recognition using global features of non-neutral faces.
Experiment 4: Non-neutral Vs Non-neutral Local features:
This experiment worked on Local features of four face regions (Forehead, Eyes Nose and Mouth). These features were extracted from Non-neutral face images. This experiment was carried out to test performance of face recognition using face regions with varied expressions.
Experiment 5 : Non-Neutral Vs Non-Neutral Global + Local features .
In this experiment, the performance was tested on hybrid features extracted from faces with varied expressions. This was the experiment demonstrating effectiveness of the proposed system having capability of handling variations in expressions with combination of global as well as local features.
Thus from experiment-2 to experiment-5, Performance of PCA is tested on Neutral and non-neutral face images and it has been observed that variations in expressions affects the result adversely when only global features are used. But inclusion of local features helps in improving the recognition rate . In addition it has been observed that non-neutral images, expression variation in mouth region lowers down the performance of the system. So Experiment 6 was carried out.
Experiment 6 : Non-Neutral Vs Non-Neutral Global + Local features excluding mouth Region.
This experiment has selected the mouth region adaptively, features of mouth region were included only when they contributed in improving the performance, and it has been observed that this adaptively selected PCA approach has considerably improved the recognition rate.
The recognition rate was further improved by implementing the hybrid approach; this proposed approach has used the moment invariants as the rotation invariant features. The final results obtained by implementing hybrid 3D face recognition are shown in table V.
Table V: . Recognition performance of individual face region.
Approach | Feature vectors/class | ||||
20 | 40 | 60 | 80 | 100 | |
WF_Expert+ FP_Experts | 81.29 | 83.19 | 85.70 | 87.85 | 89.5 |
WF _Expert + FP_Expert + MT_Expert | 85.10 | 85.08 | 87.15 | 89.31 | 92.95 |
WF_Expert+ FP_Experts + MT_Expert +MInv_Expert | 92.45 | 95.55 | 96.95 | 97.51 | 98.05 |
*WF – Whole Face, FP- Face Parts, MT- Mouth MInv – Moment Invariants
Experiments were carried out on different approaches proposed in this paper and after comparative study final results were generated as shown in table VI. The recognition rate was improved when mouth region, with significant facial expressions was omitted from the face image.
The face regions are selected adaptively during the region based feature selection process. The proposed experimental setup has shown an efficient performance with minimum FAR 0.02 and maximum FAR 0.09 and the recognition rate was 99.85 % which is considerably good as compared to other benchmark face recognition approaches.
Table VI. The Overall Recognition Performance of proposed Methodology
Database | Face parts | Recognition Rate |
GAVAbDB | Forehead | 71.56 |
Eyes | 70.88 | |
Nose | 72.33 | |
Mouth | 54.23 | |
WF + all Face Parts | 90.41 | |
WF + all Face parts excluding Mouth | 95.85 | |
CASIA3D | Forehead | 72.11 |
Eyes | 79.12 | |
Nose | 73.34 | |
Mouth | 52.71 | |
WF + all Face Parts | 92.82 | |
WF + all Face Parts excluding Mouth | 96.87 |
V. Conclusion and future work
By implementing the face alignment, face subdivision and the face recognition, we have built an efficient 3D face recognition system. Each task of these three stages can be completed automatically. The most important aspect of our system is the highly reliable face subdivision technique which can extract the face parts from face images with 600 of head rotation. The robustness to the minute pose variation of this method has been achieved by using moment invariant features. The proposed face recognition method recognizes faces more effectively and produces a relatively high performance by ignoring computationally inefficient features and by selecting most contributing face parts. But still, 3D face databases used in this proposed system has a limited range of head pose variations, only up to 600 of the head orientation is considered which may not be enough to prove the pose-invariant ability of our approach. To make the application more realistic the images with large pose variations would be used.
Acknowledgment
Portions of the research in this paper use CASIA 3D Face database collected by institute of Automation, Chinese Academy of Sciences.
References
1. Andy Adler and Michal E. Schuckers, Comparing Human and Automatic Face recognition Performance, IEEE Tranactionon Systems, Man and Cybernetics, Vol.37 No. 5, October 2007.
2. Xiaoguang Lu and Anil K. Jain,Automatic Feature Extraction for Multiview 3D Face Recognition, Proc. International Conference on Automatic Face and Gesture Recognition, Southsampton, UK, April 2006
3. Naouar Belghini , 3D Face Recognition using Gaussian Hermite Moments ,Special Issue of International Journal of Computer Applications, 0975 – 8887 September 2012
4. Omid Gervei et all, 3D Face Recognition Using Modified PCA Methods, World Academy of Science, Engineering and Technology 63, 2010
5. R. Gottumukkal, V.K. Asari, An improved face recognition technique based on modular PCA
approach, Pattern Recognition Letters 25,
429 436,2014
6. George D.C. Cavalcanti et al, Weighted Modular Image Principal Component Analysis for face recognition, Expert Systems with Applications Volume 40, Issue 12,, 4971–4977 ,Sept 2013
http://dx.doi.org/10.1016/j.eswa.2013.03.003
7. K. Tan and J. Yang, Adaptively weighted sub-pattern PCA for face Recognition, Neurocomputing, Vol.64,. 505-511, 2005
8. A. Pavan kumar et al, Face Recognition Using Weighted Modular Principle Component Analysis,11th International Conference, ICONIP 2004, Calcutta, India, 362-367, November 22-25, 2004
9. Stefano Berretti et al, 3D Face Recognition using iso-Geodesic Stripes, IEEE transaction on Pattern Recognition and Machine Learning, (vol. 32 no. 12) . 2162-2177 , December 2010
10. Xia Han et all, Face Recognition in the Presence of Expressions, Journal of Software Engineering and Applications, , 5, 321-329 , 2012
11. D. Smeets, T. Fabry, J. Hermans, D. Vandermeulen and P. Suetens, Fusion of an Isometric Deformation Modeling Approach Using Spectral Decomposition and a Region- Based Approach Using ICP for Expression-Invariant 3D Face Recognition, The 20th International Conference on Pattern Recognition, Istanbul,. 1172-1175 , 23-26 August 2010,
12. Chuanjun Wang, Xiamu Niu,Wenwei Lu, Jianren Gong A Hybrid Method to Build a Canonical Face Depth Map , International Journal of Digital Content Technology and its Applications. Volume 5, Number 5, May 2011
13. Automatic 3D face recognition from depth and intensity Gabor features ,C. Xu et al. / Pattern Recognition 42 ) 1895 – 1905 ,2009
14. Atefe Assadi and Alireza Behrad, A New Method for Human Face Recognition using Texture and Depth Information,10TH Symposium on Neural Network Applications in Electrical Engineering, NEUREL , 978-1-4244-8820-9/10 , 201-205,2010
15. Mr. Hamid M. Hasan et al, 3-D Face Recognition Using Improved 3D Mixed Transform, International Journal of Biometrics and Bioinformatics (IJBB), Volume (6) : Issue (1) , pp 278-290,2012
16. Nita M. Thakare, Dr.V.M. Thakare ,A Robust and Novel Framework for Subdivision of Face Image Based on the Concept of Golden Rule, IOSR Journal of Computer Engineering (IOSRJCE) ISSN: 2278-0661, ISBN: 2278-Volume 8, Issue 3, 33-38, Feb. 2013
17. https://www.goldennumber.net/beauty/
18. http://www.cbsr.ia.ac.cn/english/3DFace%20Databases.asp
19. http://gavab.escet.urjc.es
