My Final Analysis

...for emotions detection. A while back (I don't have a lot of time to work on this, so it takes a while to get from one conclusion to another) I said that the C++ program I found on the Internet for smile detection was comparing the data it got from an image with the data in the haarcascade xml file. I then went through the code and looked up the C++ definitions of the OpenCV functions it was using; I was wrong in my cursory apraisal, and added a note that what the program was doing was comparing the number of hits it was getting on the xml file to the first time it got a smile hit, and using the ratio to state how good of a smile the image was ("is" for the real-time capture off of a webcam) displaying. To get to this depth of understanding I had to read some more documentation:

CascadeClassifier::detectMultiScale

Detects objects of different sizes in the input image. The detected objects are returned as a list of rectangles.

C++: void CascadeClassifier::detectMultiScale(const Mat& image, vector<Rect>& objects, double scaleFactor=1.1, int minNeighbors=3, int flags=0, Size minSize=Size(), Size maxSize=Size())

Python: cv2.CascadeClassifier.detectMultiScale(image[, scaleFactor[, minNeighbors[, flags[, minSize[, maxSize]]]]]) → objects¶

Python: cv2.CascadeClassifier.detectMultiScale(image, rejectLevels, levelWeights[, scaleFactor[, minNeighbors[, flags[, minSize[, maxSize[, outputRejectLevels]]]]]]) → objects

C: CvSeq* cvHaarDetectObjects(const CvArr* image, CvHaarClassifierCascade* cascade, CvMemStorage* storage, double scale_factor=1.1, int min_neighbors=3, int flags=0, CvSize min_size=cvSize(0,0), CvSize max_size=cvSize(0,0) )

Python: cv.HaarDetectObjects(image, cascade, storage, scale_factor=1.1, min_neighbors=3, flags=0, min_size=(0, 0)) → detectedObjects

Parameters:

Parameters:	cascade – Haar classifier cascade (OpenCV 1.x API only). It can be loaded from XML or YAML file using `Load()`. When the cascade is not needed anymore, release it using`cvReleaseHaarClassifierCascade(&cascade)`. image – Matrix of the type `CV_8U` containing an image where objects are detected. objects – Vector of rectangles where each rectangle contains the detected object. scaleFactor – Parameter specifying how much the image size is reduced at each image scale. minNeighbors – Parameter specifying how many neighbors each candidate rectangle should have to retain it. flags – Parameter with the same meaning for an old cascade as in the function `cvHaarDetectObjects`. It is not used for a new cascade. minSize – Minimum possible object size. Objects smaller than that are ignored. maxSize – Maximum possible object size. Objects larger than that are ignored.

cascade – Haar classifier cascade (OpenCV 1.x API only). It can be loaded from XML or YAML file using Load(). When the cascade is not needed anymore, release it usingcvReleaseHaarClassifierCascade(&cascade).
image – Matrix of the type CV_8U containing an image where objects are detected.
objects – Vector of rectangles where each rectangle contains the detected object.
scaleFactor – Parameter specifying how much the image size is reduced at each image scale.
minNeighbors – Parameter specifying how many neighbors each candidate rectangle should have to retain it.
flags – Parameter with the same meaning for an old cascade as in the function cvHaarDetectObjects. It is not used for a new cascade.
minSize – Minimum possible object size. Objects smaller than that are ignored.
maxSize – Maximum possible object size. Objects larger than that are ignored.

If you look at the code for creating these marked up videos (click on the link below the videos) you'll see that I didn't require any minNeighbors; i.e., any hit on smiling data in haarcascade_smile.xml will bring up that blue rectangle on the left side of the video. ...but, all those green rectangles you see in the video are hits (they have to be in the face region to get the blue rectangle moving) and some of them land in hair, the eyes, etc. I let the hits occur anywhere in the image, and hits occured in the sky,etc. When I first tried detecting smiles I restricted the hit area to the mouth; worked when I smiled in my videos (and those weren't even real smiles) but it missed smiles in the video of smiling celebrities. The celebrity smiles just weren't in haarcascade_smile.xml What I could do is restrict the smile hits to the face region and require minNeighbors to be 6 (or whatever; I need to play with this). ...or, I could check each hit if it occurs in the eye region, nose region, and the mouth region (those are the only regions my Python program looks for). I'm going to try the face region first, and set minNeighbors to 6; my next blog post will follow that coding.