1. Objective of the Guide
The objective of this guide is to approach the entire process of training an object detection model using a modern YOLO architecture through a scientific and measurable method—moving beyond merely running a simple training command. We will cover everything from analyzing the dataset and model selection to establishing a baseline, conducting hyperparameter optimization, and finally achieving real-time deployment on an edge device.
Our core approach is: first, understand the problem and the data; then, establish a reliable baseline; interpret the results; conduct controlled experiments on single or small groups of parameters; and define the next step based on the outcome of each experiment.
2. Core Working Philosophy
Core Principle: Randomly changing hyperparameters is not optimization. Every modification must be based on a measurable hypothesis.
Define the problem and the target device.
Analyze the dataset and label quality.
Select the appropriate model variant.
Establish a reliable baseline using standard settings.
Analyze Loss, mAP, precision, recall, and F1 results.
Identify signs of overfitting or underfitting.
Design targeted experiments.
Apply Hyperparameter Optimization (HPO) in a controlled manner.
Validate the best model using independent test data.
Measure latency and FPS on the actual target device.
3. Project Analysis: What You Need to Know Before Training
3.1 Project and Task Definition
The objects and classes to be detected must be clearly defined.
How each class is visually distinguished must be determined.
The approximate sizes of the objects within the images should be evaluated.
If small object detection is required, image size and model selection must be adjusted accordingly.
The camera may be fixed or moving; perspective and angle variations heavily influence the training strategy.
For real-time systems, the target FPS and acceptable end-to-end latency must be defined in advance.
The edge device where the model will be deployed must be known prior to the training process.
3.2 Dataset Analysis
Extract the total number of images and the number of instances per class.
Examine the train / validation / test splits.
Check for class imbalance.
Analyze image resolutions and the distribution of aspect ratios.
Verify the accuracy of the bounding box labels.
Identify identical (duplicate) or highly similar (near-duplicate) images. Note that these are not the same: a duplicate is a completely identical frame, while a near-duplicate refers to images that are almost identical but not exactly the same (such as consecutive video frames).
Prevent data leakage, such as the same scene or the same object leaking from the training set into the validation set.
Ensure that the validation and test data represent real-world usage conditions as closely as possible.
3.3 Hardware Analysis
GPU model and VRAM capacity
CPU model
Amount of RAM
Disk type and capacity
Versions of CUDA, PyTorch, and Ultralytics
The machine used for training
The target edge device for inference
4. Training Strategy Based on Dataset Size
Data Size | Recommended Approach |
< 1,000 images | Transfer learning, strong but context-appropriate augmentation, early stopping, K-fold cross-validation if necessary. |
1,000 – 10,000 | Transfer learning, medium-level augmentation, controlled Hyperparameter Optimization (HPO), and a balanced validation set. |
10,000 – 100,000 | Larger batch size, longer training duration, controlled reduction of augmentation, narrowed-down HPO. |
> 100,000 | From-scratch training (training the model from scratch with randomly initialized weights without using pre-trained weights) can be considered; data quality, I/O bottlenecks, and distributed training become critical. |
Important: Image count alone is not a sufficient metric. Thousands of similar images from the exact same scene may actually constitute a very small dataset in terms of genuine visual diversity.
5. Model Selection
When targeting edge devices, a model should not be selected based solely on accuracy. Nano or small variants can be considered great starting points for most real-time applications. However, for tasks requiring the detection of small objects or processing high-resolution inputs, the advantages of larger models must be evaluated experimentally.
Criterion | Why is it Important? |
mAP50 | To evaluate overall detection success. |
mAP50-95 | To observe localization quality under stricter IoU thresholds. |
Precision | To understand the level of false positives. |
Recall | To evaluate how many actual objects are missed (false negatives). |
F1 Score | To see the balance between precision and recall in a single metric. |
Latency / FPS | To determine suitability for real-time deployment. |
VRAM / Model Size | To ensure the model fits within the target hardware limits. |
See you in Part 2 of the guide! Stay tuned to our blog so you don't miss our upcoming posts and technical content.
