From Linear to Polynomial Regression: The Art of Not Overlearning

 Last week you learned linear regression for prediction (how much will this house cost?). This week: polynomial regression and the critical problem of overlearning.

Same dataset. Different challenges. Same tidymodels framework.

The Overlearning Problem: When Models Know Too Much

Question: Can a model be too good at learning the training data? Answer: Absolutely, and it's called overlearning.

Overlearning happens when your model performs amazingly well on training data but falls flat when facing new data. It's like memorizing test answers without understanding the concepts - you'll ace that specific test but fail when the questions change slightly.

This isn't just a minor issue - overlearning is one of the most pressing and still not fully solved problems in machine learning.

When Does Overlearning Happen?

Three main scenarios make overlearning more likely:

1. Small training dataset: Not enough examples to generalize properly
2. Too many variables/parameters: The model has too many knobs to adjust
3. Highly non-linear models: The model is so flexible it can twist and turn to fit any pattern

To demonstrate this, we'll deliberately create conditions that trigger overlearning - using only 0.1% of our housing data (just 20 observations) for training!

Enter: Polynomial Regression

Linear regression gives us:

Price = β₁ × Sqft + β₂

But what if the relationship isn't a straight line? Polynomial regression lets us fit curves by adding powers of our variable:

Price = β₁ × Sqft + β₂ × Sqft² + β₃ × Sqft³ + β₄ × Sqft⁴ + β₅ × Sqft⁵ + β₆

We create Sqft², Sqft³, etc., as new variables and treat them like separate variables in multivariate regression. This makes our model linear in variables but non-linear in data.

The cool part? We can still use ordinary least squares (OLS) to find the optimal parameters.

The Spectacular Failure of High-Degree Polynomials

What happens when we fit a degree-5 polynomial to our tiny training dataset?

1. On the training data: Perfect fit! The curve weaves precisely through every training point.
2. On testing data: Complete disaster! The prediction line makes wild swings, predicting negative prices and ridiculous values.

Going to degree-10 makes it even worse. The metrics tell the story:

OLS (straight line) on testing data:

• RMSE: 265,889
• R²: 0.49

Polynomial (degree 5) on testing data:

• RMSE: 99,940,240
• R²:  0.0215

Polynomial (degree 10) on testing data:

• RMSE: 143,737,800,000
• R²: 0.0036

Finding the Sweet Spot: Hyper-Parameter Tuning

So how do we find the right polynomial degree? Enter hyper-parameter tuning.

Hyper-parameters are settings we choose before training, like k in k-NN or the polynomial degree. They can't be optimized directly by the training algorithm.

Two main approaches for tuning:

1. Validation Set Approach

Split your training data into:

• Actual training data (to learn parameters)
• Validation data (to test hyper-parameters)

Use the validation set to pick the best hyper-parameters, then evaluate final performance on your untouched testing data.

2. Cross-Validation Approach

Instead of one validation set, create multiple folds:

1. Split training data into 4 parts
2. Train on 3 parts, validate on 1
3. Rotate which part you use for validation
4. Average the results

This uses all your training data efficiently and reduces the risk of an unlucky validation split.

The 10-Step Recipe for Proper Modeling

Here's the workflow for any machine learning task with hyper-parameter tuning:

1. Split data into training and testing sets
2. Create a recipe (preprocessing steps)
3. Create a model design with tunable parameters
4. Build a workflow combining recipe and model
5. Create a grid of hyper-parameter values to try
6. Set up cross-validation folds
7. Run the tuning process across all combinations
8. Extract the best hyper-parameter values
9. Finalize the model with those values
10. Evaluate on the testing set

The Polynomial Sweet Spot

When we follow this process with our housing data, guess what degree works best? Degree: 6

A simple quadratic function outperforms both the straight line and the wiggly high-degree polynomials. When evaluated on testing data:

• RMSE: 240,706
• R²: 0.59
• MAE: 164,987

The model explains about 60% of house price variation, a significant improvement over our linear model.

Project Time: Tuning k in k-Nearest Neighbors

Remember our penguin species classifier from Lecture 4? We arbitrarily set k=4, but what's the optimal value?

Your task: Follow the 10-step process to tune k from 1 to 15, using 5-fold cross-validation.

The best k value for maximizing specificity? k=6!

Wait, k=6 Performed Worse Than k=4?

Interestingly, when we compare the confusion matrices, the tuned k=6 model might appear to perform slightly worse on the testing data than our original k=4 model. This raises an important question: If cross-validation found k=6 to be optimal, why doesn't it outperform k=4 on the testing data?

This apparent contradiction teaches us several valuable lessons:

1. Optimization target matters: We specifically optimized for specificity, not overall accuracy. The k=6 model might have better specificity (correctly identifying negatives) while having slightly lower overall accuracy.
2. Random variation in data splits: The specific random split between training and testing data can impact results. The k=6 model might perform better "on average" across many possible splits, even if it doesn't win on this particular testing set.
3. Cross-validation vs. single test set: Cross-validation uses multiple folds to find the most robust model across different data subsets. A single test set evaluation can sometimes be misleading due to the particular characteristics of that test set.
4. Different metrics tell different stories: The k=6 model might excel at some metrics (like specificity) while the k=4 model might excel at others (like overall accuracy). No model is best at everything.

This is a perfect real-world example of why model evaluation is nuanced. Just because a model is "optimal" in one context doesn't mean it will always outperform simpler models on every metric and every dataset. The goal of cross-validation is to find the model that performs best on average across different data splits, not necessarily the one that performs best on a specific test set.

Key Takeaways

• Overlearning happens when models fit training data too closely, failing on new data
• Complex models (high-degree polynomials, many variables) are more prone to overlearning
• Never use your testing data for tuning - that's what validation or cross-validation is for
• The tidymodels workflow is consistent: recipe → model → workflow → tune → finalize → evaluate
• Cross-validation gives more reliable hyper-parameter tuning than a single validation split
• Metrics alone can be misleading - visual inspection of models is equally important
• Often the "best" model isn't necessarily the simplest, but one that balances flexibility with restraint
• The "optimal" model found through cross-validation may not always outperform simpler models on every metric or every test set

What's Next

You now understand both the power and the dangers of flexible models. Next lecture: We'll explore how to regularize models to prevent overlearning while maintaining their predictive power.

Resources:

• Download today's slides: Lecture 6, or download the pdf here 
• Course textbook: Free at https://ai.lange-analytics.com/htmlbook/index.html
• Practice challenge: Try adding more predictors to the polynomial model. Does it improve performance or make overlearning worse?

You've now learned how to avoid one of machine learning's biggest pitfalls. Keep this wisdom with you for all your future models!