Physics Maths Engineering
Convolutional Neural Network-Based Automated System for Dog Tracking and Emotion Recognition in Video Surveillance
Huan -Yu Chen,
Huan -Yu Chen
Department of Computer Science and Information Engineering, National Taichung University of Science and Technology
Chuen-Horng Lin,
Chuen-Horng Lin
Department of Computer Science and Information Engineering, National Taichung University of Science and Technology
Jyun-Wei Lai,
Jyun-Wei Lai
Department of Computer Science and Information Engineering, National Taichung University of Science and Technology
Yung-Kuan Chan
Yung-Kuan Chan
Department of Management Information Systems, National Chung Hsing University
Abstract
This paper proposes a multi–convolutional neural network (CNN)-based system for the detection, tracking, and recognition of the emotions of dogs in surveillance videos. This system detects dogs in each frame of a video, tracks the dogs in the video, and recognizes the dogs’ emotions. The system uses a YOLOv3 model for dog detection. The dogs are tracked in real time with a deep association metric model (DeepDogTrack), which uses a Kalman filter combined with a CNN for processing. Thereafter, the dogs’ emotional behaviors are categorized into three types—angry (or aggressive), happy (or excited), and neutral (or general) behaviors—on the basis of manual judgments made by veterinary experts and custom dog breeders. The system extracts sub-images from videos of dogs, determines whether the images are sufficient to recognize the dogs’ emotions, and uses the long short-term deep features of dog memory networks model (LDFDMN) to identify the dog’s emotions. The dog detection experiments were conducted using two image datasets to verify the model’s effectiveness, and the detection accuracy rates were 97.59% and 94.62%, respectively. Detection errors occurred when the dog’s facial features were obscured, when the dog was of a special breed, when the dog’s body was covered, or when the dog region was incomplete. The dog-tracking experiments were conducted using three video datasets, each containing one or more dogs. The highest tracking accuracy rate (93.02%) was achieved when only one dog was in the video, and the highest tracking rate achieved for a video containing multiple dogs was 86.45%. Tracking errors occurred when the region covered by a dog’s body increased as the dog entered or left the screen, resulting in tracking loss. The dog emotion recognition experiments were conducted using two video datasets. The emotion recognition accuracy rates were 81.73% and 76.02%, respectively. Recognition errors occurred when the background of the image was removed, resulting in the dog region being unclear and the incorrect emotion being recognized. Of the three emotions, anger was the most prominently represented; therefore, the recognition rates for angry emotions were higher than those for happy or neutral emotions. Emotion recognition errors occurred when the dog’s movements were too subtle or too fast, the image was blurred, the shooting angle was suboptimal, or the video resolution was too low. Nevertheless, the current experiments revealed that the proposed system can correctly recognize the emotions of dogs in videos. The accuracy of the proposed system can be dramatically increased by using more images and videos for training the detection, tracking, and emotional recognition models. The system can then be applied in real-world situations to assist in the early identification of dogs that may exhibit aggressive behavior.
Key Questions
How does the system detect dogs in videos?
The system uses a YOLOv3 model, a state-of-the-art object detection algorithm, to identify dogs in each frame of a video. It achieves high accuracy rates of 97.59% and 94.62% on two different datasets, though errors can occur when a dog’s facial features are obscured, the dog is of a rare breed, or the dog’s body is partially covered.
How does the system track dogs in real time?
The system employs a deep association metric model called DeepDogTrack, which combines a Kalman filter with a convolutional neural network (CNN). This allows it to track dogs accurately, achieving a 93.02% accuracy rate for single-dog videos and 86.45% for videos with multiple dogs. Tracking errors occur when dogs enter or exit the frame, causing partial visibility.
How does the system recognize dog emotions?
The system categorizes dog emotions into three types: angry (or aggressive), happy (or excited), and neutral (or general). It uses a long short-term deep features of dog memory networks model (LDFDMN) to analyze sub-images extracted from videos. Emotion recognition accuracy rates are 81.73% and 76.02% on two datasets, with anger being the most accurately recognized emotion.
What are the main challenges in detecting dog emotions?
Challenges include subtle or fast movements, blurred images, suboptimal camera angles, and low video resolution. Additionally, removing the background can make the dog region unclear, leading to incorrect emotion recognition.
How accurate is the system in detecting dog emotions?
The system achieves emotion recognition accuracy rates of 81.73% and 76.02% on two datasets. Anger is the most accurately recognized emotion, while happy and neutral emotions are slightly harder to identify due to subtler behavioral cues.
What datasets were used to test the system?
The system was tested on two image datasets for dog detection and three video datasets for tracking. Emotion recognition experiments were conducted using two additional video datasets. These datasets included a variety of dog breeds, behaviors, and environmental conditions.
What are the practical applications of this system?
The system can be used in real-world scenarios such as:
- Early identification of aggressive dog behavior to prevent incidents.
- Monitoring dogs in shelters or homes to ensure their well-being.
- Assisting veterinarians and dog breeders in understanding dog behavior.
How can the system’s accuracy be improved?
The system’s accuracy can be enhanced by:
- Training the models with larger and more diverse datasets.
- Improving video resolution and camera angles during data collection.
- Refining the emotion recognition model to better capture subtle behaviors.
What are the limitations of the system?
Limitations include:
- Difficulty in tracking dogs when they enter or exit the frame.
- Lower accuracy in recognizing emotions for dogs with subtle or fast movements.
- Challenges in handling rare breeds or dogs with obscured features.
What makes this system unique compared to previous approaches?
This system integrates multiple advanced techniques, including YOLOv3 for detection, DeepDogTrack for real-time tracking, and LDFDMN for emotion recognition. It also uses expert-annotated datasets to ensure accurate emotion categorization, making it a comprehensive solution for dog behavior analysis.
What are the future directions for this research?
Future research could focus on:
- Expanding the system to recognize more nuanced emotions and behaviors.
- Improving robustness in challenging environments, such as low-light conditions.
- Developing real-time applications for pet owners, veterinarians, and animal shelters.
