Welcome to the second part of our object detection guide using the YOLO architecture. In our first post, we laid a solid foundation by defining our project, analyzing our dataset, and selecting the right model. Now, it's time to put theory into practice. In this article, we will roll up our sleeves and step-by-step explore how to train a reliable baseline model, interpret the results from a scientific perspective, conduct controlled hyperparameter optimization, and finally prepare our trained model for deployment on the target edge device.
6. Baseline Training
Before diving into hyperparameter optimization, a standard and highly reliable baseline training must be conducted. The goal of the baseline is not to find the absolute best model, but to establish a reference point against which all subsequent experiments can be compared.
Check and clean the data.
Create the train / validation / test splits.
Select the appropriate YOLO26 variant.
Verify your installed Ultralytics version.
Run the baseline with default or recommended training settings.
Record the results: loss, mAP50, mAP50-95, precision, recall, and F1.
Review the confusion matrix and Precision-Recall (PR) curve.
Log the baseline results in your experiment journal.
6.1 Example: Interpreting a Baseline Result
The graph examples mentioned in this section are purely representative and randomly generated; they do not depict an actual training output. The aim is to teach you how to interpret loss, precision-recall, F1-confidence, and confusion matrix graphs in conjunction with one another. In our example scenario, there are three classes (A, B, and C); what matters here is not the numerical values, but how to read the shapes of the graphs and the differences between classes.
Loss Graph: A steady decrease in train loss as epochs progress indicates that learning is occurring. If the validation loss initially decreases and then flattens out, the model's generalization performance may have reached saturation. If the validation loss starts to rise, the possibility of overfitting must be investigated.
Precision-Recall Curve: We want this curve to remain in the upper-right region as much as possible. A rapid drop in precision at high recall levels may indicate that the model is starting to produce false positives in its attempt to capture more objects. On the F1-Confidence Curve, the confidence level that yields the highest F1 score can be considered a starting point for threshold selection, depending on the application.
Confusion Matrix: This shows where class-level errors are concentrated. High values on the diagonal represent correct classifications, while off-diagonal values indicate misclassifications between classes. For instance, if a significant portion of Class B is predicted as Class C, the visually similar features of these two classes should be examined separately.
Conclusion: Decisions should never be made based on a single metric. You must evaluate them together: loss graphs show learning behavior, the precision-recall curve shows the error balance, the F1-confidence curve highlights threshold behavior, and the confusion matrix reveals class-specific errors. A high mAP alone is not enough.
7. Optimizing the Initial Training as Much as Possible
Baseline training is mostly done not to find the highest possible score, but to establish a reliable reference point for all future experiments. However, setting up the initial training as healthily as possible saves time and ensures that subsequent interpretations are based on a solid foundation. The points below form a practical checklist for optimizing the first training run.
Start with pre-trained weights: Compared to randomly initialized weights, transfer learning almost always ensures faster and more stable convergence.
Choose the right image size: The image size (
imgsz) should be determined based on the actual size of the objects in the dataset and the target device's processing power. An unnecessarily large size prolongs training time, while a too-small size can reduce the visibility of small objects.Adjust batch size according to hardware: The batch size should be kept at the maximum value that runs stably within your graphics card's memory (VRAM). Overly small batches can result in noisy gradients.
Start without changing the learning rate: Leave the learning rate and warmup settings at their recommended default values. Not tinkering with these in the first run makes it easier to understand why results change later on.
Leave early stopping enabled: Keep the epoch count sufficiently high and leave the early stopping mechanism on. This answers the "how many epochs are enough?" question through measurement rather than assumption.
Fix the random seed: When the seed is fixed, comparisons with subsequent experiments become fair and reproducible.
Consider mixed precision training: If your hardware supports it, keeping mixed precision enabled both shortens training time and reduces VRAM usage.
Maintain regular validation and saving frequency: The best model should always be saved automatically based on the validation metric so you don't lose the best intermediate checkpoint.
Leave augmentation settings unchanged for the first run: Keeping augmentation at recommended defaults makes it easier to distinguish which specific change affected performance during later experiments.
Run a short dry-run before full training: A quick test run on a small subset of the dataset allows you to catch data loading, label mismatch, or coding errors before committing to hours of training.
Log hardware and software versions: Noting down the GPU, CPU, and software versions (CUDA, PyTorch, Ultralytics) allows you to rule out version discrepancies in future comparisons.
Objective: The goal of the first training is not to achieve the highest score, but to create a reliable, reproducible reference point to build upon. Without this reference, you cannot tell if subsequent hyperparameter changes actually provided an improvement.
8. Reading Loss Graphs
Loss graphs are crucial for understanding the model's learning behavior; however, model quality should not be determined solely by looking at the loss value.
Observation | Possible Interpretation |
Train loss ↓ and validation loss ↓ | Generally healthy learning. |
Train loss ↓ and validation loss ↑ | High probability of overfitting. |
Train & validation loss high and flat | Underfitting, insufficient capacity, or inappropriate training settings. |
Highly fluctuating loss | Learning rate, batch size, data quality, or augmentation should be investigated. |
Loss dropping but mAP plateauing | Saturation may have occurred between the optimization objective and actual performance. |
Rule: A decrease in loss alone does not mean a better model. It must be evaluated alongside mAP, precision, recall, F1, and actual test performance.
9. Overfitting and Underfitting
9.1 Overfitting Symptoms
Train performance consistently rises while validation performance degrades.
Train loss drops while validation loss rises.
Train mAP is very high, while validation mAP remains noticeably low.
The model performs exceptionally well under specific image conditions but noticeably poorly under different ones.
9.2 Possible Solutions for Overfitting
Collect more diverse data.
Use context-appropriate data augmentation.
Apply early stopping.
Try a smaller model variant.
Reduce training duration.
Cautiously explore weight decay and other regularization options.
Double-check the train/validation split and the risk of data leakage.
9.3 Underfitting Symptoms
Both train and validation performances are low.
Loss values are not improving sufficiently.
The model fails to learn the task adequately, even on the training data.
Approach: It is incorrect to scale up the model or increase augmentation without first determining the root cause of the problem.
10. Data Augmentation
The purpose of augmentation is not merely to artificially inflate the dataset size, but to make the model more robust to real-world variations.
Mosaic: Combines different images to create object and context diversity.
MixUp: Blends images to provide regularization; however, it is not suitable for every environment.
HSV: Increases robustness against lighting and color variations.
Scale: Provides robustness against object size variations.
Translate: Simulates changes in the object's position within the image.
Rotation: Should be used at a level appropriate to angle changes that might occur in the actual system.
Flip: Should only be used if right-left symmetry is physically plausible in the real-world use case.
Close Mosaic: Can be considered in the final phase of training to bring the model closer to the true image distribution.
Context Match: Aggressively applying transformations that will never be encountered in the real system can widen the gap between the model and the true data distribution.
10.1 Core Augmentation Techniques and Recommended Ranges
The following table summarizes the most commonly used augmentation techniques, their typical value ranges, and their purposes. The specified ranges are not strict rules but commonly preferred starting points; exact values should always be validated through controlled experiments based on your dataset.
Technique | Recommended Value / Range | Purpose |
Mosaic | Probability ~1.0; on for most of training | Increases object and context diversity by combining multiple images into a single frame. |
Close Mosaic | Recommended to turn off in the last 10-15 epochs | Helps the model adapt more closely to the actual image distribution at the end of training. |
MixUp | Probability ~0.0 - 0.2 | Provides a slight regularization effect by blending two images; not recommended for tasks requiring precise class distinction. |
Copy-Paste | Probability ~0.0 - 0.3 | Increases diversity for minority classes by pasting objects onto different backgrounds. |
HSV - Hue | ~0.0 - 0.02 | Provides robustness against minor color shifts. |
HSV - Saturation | ~0.4 - 0.9 | Simulates color saturation changes under different lighting and camera conditions. |
HSV - Value (Brightness) | ~0.2 - 0.5 | Increases robustness against changes in lighting intensity. |
Rotation | 0 - 15 degrees for general use | Simulates minor angle changes of the camera or object; keep near 0 for fixed-angle systems. |
Translate | ~0.05 - 0.2 of image size | Simulates shifts in the object's position within the frame. |
Scale | ~0.3 - 0.9 | Simulates size changes depending on the object's distance from the camera. |
Shear | Usually close to 0 | Can be slightly increased for scenes with prominent perspective distortion. |
Perspective | Usually 0.0 | Should only be tested with small values in systems where the camera angle changes significantly. |
Flip Left-Right (Fliplr) | Usually 0.5 probability | Should only be applied if right-left symmetry is physically possible in the real-world scenario. |
Flip Up-Down (Flipud) | Usually 0.0; off | Flipping upside-down is not physically meaningful in most real-world scenes. |
10.2 Augmentation Intensity Based on Dataset Size
Augmentation intensity must be adjusted according to the dataset size. Stronger augmentation is generally beneficial for small datasets, whereas excessive augmentation on large, already diverse datasets can slow down learning and become counterproductive.
Dataset Size | Augmentation Intensity | Description |
Very small (< 1,000) | Strong | Mosaic, HSV, and scale can be kept near the upper limits; MixUp and Copy-Paste can also be considered. |
Small - Medium (1,000 - 10,000) | Medium | Values near the midpoint of the recommended ranges are a reasonable starting point. |
Large (10,000 - 100,000) | Light - Medium | Since the data is already diverse, aggressive augmentation is usually unnecessary; excessive augmentation can slow learning. |
Very Large (> 100,000) | Light | Augmentation can be limited to focus solely on minor variations expected in the real world. |
10.3 Context-Specific Selection Principles
Do not add a transformation to your training that will never be encountered in the real system. For instance, applying wide-angle rotation or perspective shifts for a fixed, perpendicular camera system will distance the model from the actual data distribution.
When adding a new augmentation, make only one change at a time and compare the validation metrics with the previous experiment. If multiple augmentations are changed simultaneously, it is impossible to tell which one caused the effect.
Gradually increase augmentation strength. Starting from low values and stepping up while monitoring validation performance makes it clear at what point excessive augmentation starts harming the learning process.
Visually inspect a sample of the augmented images. Ensure that labels transform correctly alongside the image and that objects don't spill out of the frame completely.
Augmentation is a technique applied only to the training data. Validation and test data must reflect real-world conditions exactly as they are and remain completely free from augmentation.
Rule of thumb: When adjusting augmentation settings, change only one parameter at a time and compare the result via validation metrics; otherwise, you won't be able to distinguish which augmentation caused the shift in performance.
11. Hyperparameter Optimization (HPO)
Hyperparameter optimization is the controlled search for parameters that affect model performance, rather than randomly changing many parameters at once.
11.1 Priority Order
Learning rate and scheduler
Image size and batch size
Augmentation strength
Optimizer and related parameters
Weight decay / regularization
Warmup settings
Other low-impact parameters
11.2 HPO Methods
Method | Suitable Use Case |
Manual / Grid Search | For quick comparison of a few parameters. |
Random Search | Simple and effective starting point for medium-sized search spaces. |
Ultralytics Tune | For a fast and integrated start. |
Optuna | For efficient use of trial counts and Bayesian/TPE-based searching. |
Ray Tune | For multi-GPU or parallel experiment environments. |
Recommendation: For small and medium datasets, conduct a few meaningful manual experiments first; broad automated searches should only be applied after the baseline and problem behavior are well understood.
12. Experiment Design and Logging (Experiment Log)
Every training experiment must be recorded.
Experiment | Change | Goal | Next Step |
Baseline | Default settings | Establish a reference, record metrics | Determine the biggest issue |
Exp-01 | LR modification | Improve convergence | Compare with Baseline (Proceed / Revert) |
Exp-02 | Augmentation change | Generalization | Review Val performance (Proceed / Revert) |
Exp-03 |
| Small object performance | Check mAP + latency (Accuracy/FPS trade-off) |
Experiment Rule: Change as few variables as possible in a single experiment. This allows you to understand exactly which factor caused the change in performance.
We have trained our model, learned how to properly read the metrics, and optimized the hyperparameters! But how do we transfer this high-performing model to real hardware? Stay tuned for Part 3, where we will bring our project to the field with model export and hardware acceleration steps.
