📌 Definition
Hyperparameters are the external configuration settings of a machine learning model that are set before training begins. They are not learned from the data directly (unlike model parameters such as weights and biases).
Instead, hyperparameters control the learning process and determine how the model is trained.
🔑 Examples of Hyperparameters
- Learning-related hyperparameters
- Learning Rate (η\etaη) → size of each step in Gradient Descent.
- Batch Size → number of training examples used in one iteration.
- Number of Epochs → how many times the entire dataset is passed through the model.
- Model-related hyperparameters
- Number of Layers & Neurons in a Neural Network.
- Activation Functions (ReLU, sigmoid, tanh).
- Dropout Rate (for regularization).
- Kernel size / filters in CNNs.
- Algorithm-specific hyperparameters
- K in K-Nearest Neighbors (KNN).
- Depth of trees in Decision Trees / Random Forests.
- Regularization strength (λ, C) in Logistic Regression or SVM.
⚖️ Difference Between Parameters and Hyperparameters
| Aspect | Parameters | Hyperparameters |
|---|---|---|
| Learned from Data? | ✅ Yes (during training) | ❌ No (set before training) |
| Examples | Weights, Biases | Learning Rate, Batch Size, # of Layers |
| Adjusted By | Training process | Human or automated tuning |
| Impact | Defines model’s prediction | Defines how model is trained |
🎯 Why Hyperparameters Matter?
- They control model performance (accuracy, speed, generalization).
- Poor choices → overfitting, underfitting, or very slow training.
- Correct tuning → balanced, efficient, high-performing model.
🔍 Hyperparameter Tuning Methods
- Manual Search → trial & error, expert intuition.
- Grid Search → try all combinations from a predefined set (exhaustive but costly).
- Random Search → randomly pick values from ranges (faster & often effective).
- Bayesian Optimization → uses probability to guide search for best values.
- AutoML Tools (like Google AutoML, Optuna, Hyperopt) → automate the process.
💡 Analogy:
Training a model is like baking a cake 🍰:
- Ingredients (flour, sugar) = training data.
- Final taste = trained model performance.
- Parameters = how much sugar the cake actually contains after mixing.
- Hyperparameters = oven temperature, baking time, mixing speed → you set these before baking, and they affect the final outcome.