How models actually learn, from vanilla gradient descent to Adam
Hyperparameters are training choices set outside the learned parameters: learning rate, batch size, weight decay, warmup length, dropout, model width, and many others.
Hyperparameter search is the process of trying configurations without fooling yourself. The goal is not to find a lucky run. The goal is to find a setting that works reliably on validation data.
Paint swatch cards make the search idea concrete. You do not repaint the whole room for every possible color. You test a structured set of swatches, narrow the range, then try the most promising shades under the right light. Hyperparameter search narrows training choices the same way. The figure below shows the machinery that scores each swatch: rotating validation folds, so every candidate setting is judged on data it never trained on. That honest scoring is what separates a search from a lucky run.