When Straight Lines Aren’t Enough: Intro to Polynomial Regression

Learn how to model curvy relationships in your data with this powerful technique.

When Straight Lines Aren’t Enough: Intro to Polynomial Regression

We’ve seen how Simple and Multiple Linear Regression try to fit a straight line (or a flat plane) through our data points. But what happens when the relationship between our input (X) and output (Y) isn’t a straight line? What if it curves?

Think about things like: the relationship between experience level and salary (often curves upwards faster later), the path of a thrown ball, or how temperature affects crop yield. These often show non-linear patterns.

This is where Polynomial Regression comes to the rescue! It’s a type of regression analysis that allows us to model these curved relationships by using polynomial terms (like x², x³, etc.) in our equation.

Main Technical Concept: Polynomial Regression models the relationship between the independent variable (X) and dependent variable (Y) as an nth degree polynomial. It’s used when data shows a curved pattern where linear regression fails.

How Does it Create Curves?

Adding Powers of X

Remember the simple linear equation: y = b₀ + b₁x. This can only draw straight lines.

Polynomial Regression extends this by adding higher powers of the independent variable x:

y = b₀ + b₁x + b₂x² + b₃x³ + ... + bnxn

Where:

• y, x, b₀, b₁ are the same as in linear regression.
• x², x³, …, xn are the higher power terms of the original input x.
• b₂, b₃, …, bn are the new coefficients for these higher power terms.
• The highest power, n, is called the degree of the polynomial.

By adding terms like x² (which creates a parabola), x³ (which creates an S-shape), etc., the model can create much more flexible curves that fit non-linear data better.

Think of it like this: a degree 1 polynomial is a straight ruler. A degree 2 polynomial is like bending the ruler once. A degree 3 polynomial is like bending it twice, and so on. More bends allow it to fit more complex shapes.

It’s Still “Linear”?

Interestingly, even though the relationship between X and Y is curved (non-linear), it’s still considered a type of linear model in a statistical sense. Why? Because the equation is linear with respect to the coefficients (b₀, b₁, b₂, ...). We are still just finding the best weights for each term, even if those terms involve powers of X. This means we can still use the same LinearRegression techniques from libraries like Scikit-learn!

How to Implement Polynomial Regression

The key trick is to transform our original independent variable(s) into polynomial features before fitting a standard linear regression model.

Steps

1. Load & Prepare Data: Import your data and separate features (X) from target (y).
2. Create Polynomial Features: Use Scikit-learn’s PolynomialFeatures class with a specified degree.
3. Train a Linear Regression Model: Fit on the polynomial features.
4. Make Predictions: Transform new data using the same PolynomialFeatures object, then predict.
5. Evaluate & Visualize: Check metrics and visualize the fit.

Python Code Example

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

# 1. Load dataset
dataset = pd.read_csv('Position_Salaries.csv')
X = dataset.iloc[:, 1:2].values  # Level - keep as 2D array
y = dataset.iloc[:, -1].values   # Salary

# 2. Create Polynomial Features (degree 4)
poly_reg = PolynomialFeatures(degree=4)
X_poly = poly_reg.fit_transform(X)

# 3. Train Linear Regression on Polynomial Features
poly_lin_reg = LinearRegression()
poly_lin_reg.fit(X_poly, y)

# 4. Make predictions
new_level = [[6.5]]
new_level_poly = poly_reg.transform(new_level)
predicted_salary = poly_lin_reg.predict(new_level_poly)
print(f"Predicted salary for Level 6.5: ${predicted_salary[0]:,.2f}")

# 5. Visualize
X_grid = np.arange(min(X), max(X), 0.1).reshape(-1, 1)
X_grid_poly = poly_reg.transform(X_grid)

plt.figure(figsize=(10, 6))
plt.scatter(X, y, color='#ec4899', label='Actual Salary')
plt.plot(X_grid, poly_lin_reg.predict(X_grid_poly), color='#14b8a6', label='Polynomial Fit (Degree 4)')
plt.title('Salary vs Level (Polynomial Regression)')
plt.xlabel('Position Level')
plt.ylabel('Salary')
plt.legend()
plt.grid(True, linestyle='--', alpha=0.6)

Choosing the Right Degree: A Balancing Act

A key decision in Polynomial Regression is choosing the degree of the polynomial.

• Too Low Degree (e.g., 1): If the data is truly curved, a low degree will act like linear regression and underfit (high bias). It won’t capture the pattern.
• Too High Degree (e.g., 10): The curve can become extremely wiggly and fit the training data perfectly, including the noise. This model will overfit (high variance) and perform poorly on new data.
• Just Right Degree: Captures the underlying trend without fitting the noise, balancing bias and variance.

How to Find the Right Degree?

• Visualization: Plot fits for different degrees and see which looks most reasonable.
• Evaluation Metrics: For larger datasets with a test set, check metrics like MSE or R² on the test set for different degrees.
• Cross-Validation: Use k-fold cross-validation to find the optimal degree that generalizes well.

Generally, start with degree 2, then try 3 or 4, and compare results. Very high degrees are often a sign of overfitting.

Common Problems & Solutions

Issue	Solution	Prevention
Linear regression fits poorly on visibly curved data	Switch to Polynomial Regression. Start with degree=2 and increase if needed.	Always visualize your data first!
Model fits training perfectly but performs terribly on test	The polynomial degree is too high (overfitting).	Reduce the degree. Use cross-validation or check test performance.
Code throws errors about input shapes	Ensure X is a 2D array. Use .values.reshape(-1, 1) or iloc[:, 1:2].	Inspect data shapes (.shape) and types (.dtypes).
Model predicts nonsensical values	High degree overfitting or polynomial doesn’t make sense outside observed range.	Choose reasonable degree. Don’t extrapolate far beyond training data.

Key Takeaways: Polynomial Regression

• Used when the relationship between one input (X) and one output (Y) is curved (non-linear).
• Works by adding powers of X (like X², X³, etc.) to the linear equation.
• The highest power used is the degree of the polynomial.
• Implemented by creating polynomial features from X using PolynomialFeatures and then fitting a standard LinearRegression model.
• Choosing the right degree is crucial to avoid underfitting (too simple) or overfitting (too complex).
• Visualize the fit to help choose the degree and assess performance.