EmotionAI

Real-time facial emotion recognition with a custom SE-attention CNN and a live Grad-CAM dashboard

THE PROBLEM

Automated emotion recognition has applications in accessibility tools, driver monitoring, and behavioral research. Building it to run in real time, on standard CPU hardware, without a cloud dependency requires deliberate tradeoffs between model accuracy and inference latency.

SYSTEM DESIGN

ENGINEERING DECISIONS

Two trained models, switchable live

A custom 3-block CNN with SE attention and a fine-tuned MobileNetV2 are both trained on FER2013 and saved independently. The dashboard exposes a live switch between them mid-session, so the same webcam feed can be compared against a from-scratch architecture and a transfer-learning one without restarting anything, useful both for demoing and for seeing where the two actually disagree.

SE (Squeeze-and-Excitation) attention on the residual blocks

A plain CNN weighs every channel equally regardless of how useful it is for the current input. SE blocks learn per-channel attention weights, squeeze, excite, scale, so the network can emphasize the channels that actually carry expression-relevant signal on a given face. Added to the residual blocks rather than replacing them, since the residual connections were already doing the gradient-flow job well; SE attention answers a different question, which channels matter, not whether the block can train.

MediaPipe for detection, Haar cascade only as a fallback

The primary detector is MediaPipe's BlazeFace model, downloaded on first run; it's more accurate across angles and lighting than a classical detector. OpenCV's Haar cascade sits behind it as a fallback for when the model file isn't available, so the pipeline degrades gracefully instead of failing outright if the download never completes.

Grad-CAM as a live, toggleable overlay, not a one-off analysis notebook

Interpretability tools usually live in a separate notebook you run after the fact. The dashboard exposes Grad-CAM as a real-time overlay you can switch on during a live session, generated from the last suitable convolutional layer of whichever model is currently active. Seeing which pixels drove a specific prediction, live, is a different kind of debugging tool than a static heatmap generated afterward.

Persistent face identity across sessions, not just per-frame detection

Detecting a face and recognizing whose face it is are different problems. A SQLite-backed registry stores a dlib face embedding per person; when an unknown face holds steady for a few seconds, the dashboard prompts to name it, and matches it by Euclidean distance against stored embeddings on every future session, not just within the current one.

Softmax output instead of sigmoid

Emotions are mutually exclusive in the way this model frames them. A face expresses one dominant emotion, not several independent ones in parallel. Softmax produces a probability distribution that sums to 1, matching that constraint and making the dominant class legible from the output. Sigmoid treats each class independently and would have obscured the primary signal.

OUTCOMES

01Two trained models, a custom SE-attention CNN and a fine-tuned MobileNetV2, switchable live in the dashboard
02Persistent face identity: unknown faces get registered once and recognized across future sessions
03Live Flask + SocketIO dashboard with a toggleable real-time Grad-CAM overlay
04PDF and CSV session export with per-emotion distribution and timeline charts

STACK

Python · TensorFlow · Keras · OpenCV · MediaPipe · Flask · SocketIO · SQLite · dlib