PYTHON / MACHINE LEARNING WITH PYTHON
Cross-validation and hyperparameter search
Score models with k-fold cross-validation and pick hyperparameters with GridSearchCV or RandomizedSearchCV without leaking your test set.
What you will learn
- Build KFold and StratifiedKFold splits and inspect what lands in each fold
- Run GridSearchCV and read best_params_, best_score_ and best_estimator_
- Count model fits so you can predict how long a search will take
- Keep a held-out test set because best_score_ is optimistically biased
Understanding Cross-validation and hyperparameter search
A single train/test split gives you one number, and that number depends on which rows happened to land in the test half. Swap the split seed and an accuracy of 0.91 can become 0.86 without anything about the model changing. k-fold cross-validation removes that dependence: it cuts the data into k blocks, trains k times, and each block is held out exactly once. You end up with k scores instead of one, and their mean is a lower-variance estimate of how the model generalises, while their spread tells you how much you should trust that mean.
Hyperparameters such as a tree's max_depth, k in k-nearest neighbours, or Ridge's alpha cannot be learned by fit, because the training loss almost always prefers the most flexible setting. So you score candidate settings on data the model did not train on. GridSearchCV automates this: give it an estimator and a dictionary mapping parameter names to lists of values, and it evaluates every combination with cross-validation. The cost is multiplicative, which is why the fit count is candidates times folds plus one final refit on all the data, and why RandomizedSearchCV, which samples a fixed n_iter of settings, is the better choice once the grid grows past a few hundred combinations.
The subtle part is that once you choose a setting because it scored best, that score stops being an honest estimate. Picking the maximum of many noisy numbers systematically picks the ones that got lucky, so best_score_ is biased upward. That is why you split off a test set first, run the whole search inside the training portion, and only touch the test set once at the end. The same reasoning applies inside a fold: any preprocessing that learns from data, such as a scaler's mean or a feature selector, must be wrapped in a Pipeline so it is refit on each training fold instead of on rows that the fold is about to be scored on.
import numpy as np
from sklearn.model_selection import KFold
X = np.arange(12).reshape(-1, 1)
kf = KFold(n_splits=4)
seen = []
for fold, (train_idx, test_idx) in enumerate(kf.split(X)):
print("fold", fold, "train size", len(train_idx), "held out", test_idx)
seen.extend(test_idx.tolist())
print("every row held out exactly once:", sorted(seen) == list(range(12)))Cross-validation turns one noisy split into an averaged estimate, and any choice made by looking at that estimate needs a fresh test set to be measured honestly.
Worked examples
Searching Ridge's alpha
A grid search over regularisation strength on data that is exactly linear, where weaker shrinkage must win.
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV, KFold
X = np.arange(60).reshape(-1, 1) / 10.0
y = 3.0 * X[:, 0] + 1.0
grid = {"alpha": [0.001, 1.0, 100.0, 10000.0]}
search = GridSearchCV(Ridge(), grid, cv=KFold(n_splits=4), scoring="r2")
search.fit(X, y)
print(search.best_params_)
print(len(search.cv_results_["params"]), "candidates evaluated")Example explained
Line 1y is a noiseless linear function of X, so any shrinkage of the slope can only hurt held-out R squared.
Line 2cv=KFold(n_splits=4) passes an explicit splitter instead of an integer, which makes the fold scheme visible in the code.
Line 3best_params_ is the dictionary that scored highest averaged over the four folds, here the smallest alpha.
Line 4cv_results_['params'] holds one entry per candidate, so its length is the size of the grid.
Why StratifiedKFold matters
Plain KFold on label-sorted data produces folds with no positive examples at all, while stratification keeps the class ratio.
import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold
y = np.array([0] * 12 + [1] * 8)
X = np.arange(20).reshape(-1, 1)
print("KFold:")
for train_idx, test_idx in KFold(n_splits=4).split(X, y):
print(" ", np.bincount(y[test_idx], minlength=2))
print("StratifiedKFold:")
for train_idx, test_idx in StratifiedKFold(n_splits=4).split(X, y):
print(" ", np.bincount(y[test_idx], minlength=2))Example explained
Line 1KFold with shuffle=False takes contiguous blocks, so sorted labels give folds of a single class.
Line 2np.bincount(..., minlength=2) forces a two-element count so an empty class shows as 0 instead of vanishing.
Line 3A fold with zero positives makes recall or ROC AUC undefined or meaningless for that fold.
Line 4StratifiedKFold splits each class separately, so every fold holds 3 negatives and 2 positives.
Budgeting the search
Counting candidates before fitting anything, to compare an exhaustive grid against a random sample of the same space.
from sklearn.model_selection import ParameterGrid
grid = {
"n_estimators": [100, 200, 400],
"max_depth": [3, 5, 8, None],
"min_samples_leaf": [1, 2, 5],
}
n_candidates = len(ParameterGrid(grid))
folds = 5
print("grid candidates:", n_candidates)
print("fits for GridSearchCV:", n_candidates * folds + 1)
print("fits for RandomizedSearchCV(n_iter=20):", 20 * folds + 1)Example explained
Line 1ParameterGrid expands the dictionary into the cartesian product, so 3 * 4 * 3 gives 36 settings.
Line 2Each candidate is fitted once per fold, hence the multiplication by folds.
Line 3The plus one is the final refit on the full training data that produces best_estimator_.
Line 4RandomizedSearchCV's cost depends only on n_iter, so adding a fourth parameter to the grid does not change it.
Important notes
Passing an integer to cv uses StratifiedKFold for classifiers and plain KFold for regressors; KFold does not shuffle unless you set shuffle=True, and shuffle needs random_state to be reproducible.
For time-ordered data use TimeSeriesSplit instead, because shuffling lets the model train on the future and score on the past.
Common mistakes
Fitting a StandardScaler or SelectKBest on the whole dataset before cross-validating: every fold's held-out rows influenced the transform, so the reported scores are optimistic and do not survive contact with new data.
Quoting best_score_ as the model's accuracy: it is the maximum over many noisy fold averages, so it is biased upward by the act of selection and is not an estimate of test performance.
Using cv=KFold on data sorted by label or grouped by subject: folds end up class-imbalanced or split the same subject across train and test, which either breaks metrics or inflates them.
Try it yourself
Change, predict, then run
Load the iris data, build a Pipeline of StandardScaler and KNeighborsClassifier, and grid-search n_neighbors over [1, 3, 5, 11, 21] with 5-fold cross-validation. Print best_params_, best_score_, and the score on a test set you split off before searching, then note the gap between the last two numbers.
Open the Python workspaceCheck your understanding
You grid-search 40 candidate settings with 5-fold CV and the best mean accuracy is 0.94. Why is 0.94 usually a poor estimate of accuracy on genuinely new data?
- Taking the maximum over 40 noisy fold averages tends to select settings that were lucky on those particular folds
- Each fold trains on only 80 percent of the data, so cross-validated scores are always too low
- Averaging five scores discards the standard deviation, and accuracy is only valid on a single split
- GridSearchCV refits on the full training set at the end, and refitting always raises the reported score
Show answer
The 40 mean scores each carry sampling noise, and picking the largest one preferentially picks upward noise, so the winner's score is biased high; only untouched data can measure it honestly. Training on 80 percent of the rows does make each fold model slightly weaker, but that pushes the estimate down rather than up, so it cannot explain why 0.94 is optimistic.