ML Training Results

PetFinder AdoptionSpeed prediction — comparing Baseline ResNet, Two-Stage ResNet, and LightGBM pipeline across accuracy, QWK, feature attribution, and ablation.

EDA Results

Preliminary Results — Model Leaderboard

Best Accuracy
36.3%
LightGBM Pipeline
Best QWK
0.2616
LightGBM Pipeline
Models Evaluated
7
ResNet · LightGBM · XGBoost · CatBoost · DT
Val Set Size
2 931
20% stratified split
RankModelType Val AccuracyQWKNotes

Validation accuracy and QWK for each model. LightGBM leads both metrics.

Multi-axis comparison scaled 0–1. Accuracy (×100), QWK (×10 for visibility).

Key Findings

  • LightGBM leads all 7 models on both accuracy (36.3%) and QWK (0.2616); CatBoost is 2nd by accuracy (35.69%) but 3rd by QWK, while TwoStage ResNet + GradNorm is 2nd by QWK (0.243).
  • XGBoost achieves the best Stage-1 Breed1 accuracy (63.66%) but its Stage-2 QWK (0.2108) is weaker — likely because its class probability distributions are less calibrated as Stage-2 inputs.
  • Decision Tree (27.94%, QWK 0.123) performs on par with Baseline ResNet (28.32%, 0.114), confirming tree stumps are too weak to leverage the 860-dim feature space.
  • All models struggle with Speed 0 (same-day adoptions) — a highly imbalanced class with only 83 val samples.

Confusion Matrix — AdoptionSpeed

Raw prediction counts per (true class, predicted class).

Row-normalised recall matrix — each row sums to 1.0. Diagonal = per-class recall.

Pattern Analysis

  • Speed 4 (No adoption) has the highest recall across all models — the most common and distinct class.
  • Speed 0 (Same day) is systematically misclassified as Speed 2 or Speed 4 by ResNet models. LightGBM handles it marginally better.
  • ResNet models collapse most predictions into Speed 2 and Speed 4, missing Speed 3 almost entirely.
  • LightGBM distributes predictions more evenly across classes, explaining the higher QWK.

Stage 1 — Feature Prediction Accuracy

Stage 1 trains each model to predict tabular attributes from the image alone. High accuracy means the visual signal contains enough information to infer the attribute.

Feature # Classes ResNet (No Norm) ResNet (GradNorm) LightGBM XGBoost CatBoost DT Best

Per-feature validation accuracy across all Stage-1 configurations. Type and Health exceed 90% in every model; Breed1 is hardest for LightGBM/CatBoost but XGBoost (63.7%) matches CNN accuracy.

Insights

  • Type (Dog/Cat) and Health are visually obvious — all models exceed 92%, with XGBoost reaching 97.2%.
  • Breed1: XGBoost (63.7%) and ResNet GradNorm (61.0%) are the top two; LightGBM (31.2%) is weakest — it lacks the fine-grained visual feature learning of the CNN backbone during multi-task training.
  • Gender is the weakest non-breed feature across all models (43–52%); Decision Tree is worst (43.8%), confirming gender is poorly discriminated by visual appearance alone.
  • XGBoost leads on 5 of 10 features, LightGBM leads on 4, suggesting boosting with deeper per-tree splits better exploits the 512-dim embedding space.

Ablation Analysis — Feature Group Importance

Combinatorial ablation on the LightGBM Stage-2 classifier (298 subsets, max combo size 3). Each group is zeroed out; ΔQWK measures the impact relative to the full baseline (QWK = 0.2616).

Single-Feature QWK Impact

ΔQWK when each feature group is individually zeroed. Negative = harmful to remove; positive = redundant.

All 298 ablation combinations plotted by combo size. Each point is one subset; colour encodes ΔQWK magnitude.

Most Harmful Combinations (Worst 5)

Features AblatednAccuracyQWKΔACCΔQWK

Most Redundant Combinations (Best 5 — removing helps or barely hurts)

Features AblatednAccuracyQWKΔACCΔQWK

Ablation Insights

  • Image embeddings are by far the most critical feature — removing them alone drops QWK by 0.26 (to near-random, ≈ −0.001).
  • Sterilized is the most informative tabular head (ΔQWK = −0.074 when removed), more than PhotoAmt (−0.027).
  • Color1 is the most redundant feature — removing it actually slightly improves QWK (+0.025), suggesting it adds noise rather than signal.
  • Gender + Color1 together improve QWK by +0.033 when removed, making them a clean candidate for pruning in the next experiment.

Grad-CAM — What the Model Sees

Gradient-weighted Class Activation Maps (Grad-CAM) back-propagate through the last ResNet block (backbone.layer4[-1]) to highlight the image regions that drove the AdoptionSpeed prediction. Warmer colours (red) indicate higher influence; cooler colours (blue/green) are less attended. Overlays are from the TwoStage ResNet + GradNorm Stage-2 head on 16 validation samples.

3 Correct 13 Incorrect 16 Total Samples

Observations

  • Correct predictions tend to focus on the pet's body and face — semantically relevant regions for assessing condition and appeal.
  • Speed 1 → Speed 2 errors are common (5 of 13 misses): the model over-predicts slower adoption, possibly due to class imbalance during Stage-2 training.
  • Speed 0 (Same day) is predicted as Speed 4 — the model sees no distinguishing visual cue for urgent adoption, which aligns with Speed 0 having zero recall in the confusion matrix.
  • Several incorrect predictions still attend to the pet rather than background, suggesting the error originates in the decision boundary rather than feature extraction.

Conclusion — Best Model by Category

Best Overall
LightGBM Pipeline
Accuracy 36.3% · QWK 0.2616
Best ResNet
TwoStage ResNet + GradNorm
Accuracy 35.4% · QWK 0.243
Best No-Norm ResNet
TwoStage ResNet (No Norm)
Accuracy 34.66% · QWK 0.2148
Best Breed Prediction (Stage 1)
TwoStage ResNet (either variant)
Breed1 Accuracy ~60–61% vs LightGBM 31%
Most Critical Feature
Image Embeddings
Removing drops QWK by −0.262 to near-random
Recommended Prune
Color1 + Gender heads
Removing improves QWK by +0.033

Supplementary — Experiment Setup