Feature Scaling & Interactions

Let x be a single feature value, and x' be its scaled value.

1. Standardization (Z-score Normalization)

• Mathematical Basis:

  x' = (x - μ) / σ

  Where μ is the mean of the feature and σ is its standard deviation.
• Result: The transformed feature will have a mean of 0 and a standard deviation of 1. It centers the data around the origin and scales it by the standard deviation.
• When/Why Preferred:
  • When the feature is assumed to be (approximately) Gaussian distributed.
  • When algorithms assume features are centered around zero and have similar variance (e.g., PCA, regularized linear models like Ridge/Lasso, Neural Networks with certain weight initializations).
  • It's less affected by outliers than min-max scaling if the outliers don't drastically skew the mean and standard deviation too much (though it's not completely robust).

2. Normalization (Min-Max Scaling)

• Mathematical Basis:

  x' = (x - min(x_col)) / (max(x_col) - min(x_col))

  Where min(x_col) and max(x_col) are the minimum and maximum values of the feature column.
• Result: The transformed feature will be scaled to a specific range, typically [0, 1]. If scaling to [-1, 1] is desired, the formula is:

  x' = 2 * [(x - min(x_col)) / (max(x_col) - min(x_col))] - 1

• When/Why Preferred:
  • When an algorithm requires features to be within a specific bounded range (e.g., some neural network activation functions like sigmoid/tanh might perform better with inputs in [0,1] or [-1,1], though batch normalization often handles this internally now).
  • When the distribution of the feature is not Gaussian or is unknown, and you simply want to bring values into a common scale.
  • Useful for image processing where pixel intensities are often scaled to [0,1].
  • Caution: It is very sensitive to outliers. A single very large or very small outlier can drastically shrink the range of the majority of the data.

3. Robust Scaling

• Mathematical Basis: Uses statistics that are robust to outliers. A common method uses the median and Interquartile Range (IQR).

  x' = (x - median(x_col)) / (IQR(x_col))

  Where IQR = Q3 - Q1 (75th percentile - 25th percentile).
• Result: It centers the data around the median and scales it according to the spread of the bulk of the data, making it less influenced by extreme values. The resulting range is not fixed.
• When/Why Preferred:
  • When the dataset contains significant outliers that would adversely affect standardization or min-max normalization.
  • It provides a more robust way to scale features without being heavily skewed by a few extreme points.

How Scaling Affects Different Algorithms

• Distance-Based Algorithms (e.g., KNN, K-Means, SVM with RBF kernel, PCA):
  These algorithms are highly sensitive to feature scales because they rely on distance calculations (e.g., Euclidean distance). Features with larger ranges/variances can dominate the distance calculation, effectively making features with smaller ranges less important. Scaling ensures all features contribute more equally.
  Example: In KNN, if one feature ranges from 0-1000 and another from 0-1, the first feature will dominate distance calculations.
• Gradient-Based Algorithms (e.g., Linear/Logistic Regression, Neural Networks, SVM with linear kernel):
  Scaling can help these algorithms converge faster. If features are on vastly different scales, the loss surface can be elongated and ill-conditioned, leading to slow convergence or oscillations during gradient descent. Scaling can make the loss surface more spherical.
  Also, for regularized models (L1/L2 penalties), scaling ensures that the penalty is applied fairly to all coefficients, as the magnitude of coefficients depends on the scale of their corresponding features.
• Tree-Based Algorithms (e.g., Decision Trees, Random Forests, Gradient Boosting Trees):
  These are generally insensitive to the scale of the features. They make splits based on thresholds within individual features, so the relative ordering of values within a feature matters, not their absolute scale. Scaling typically does not improve (or harm) their performance significantly.

Important Note: Scaling parameters (mean, std dev, min, max, median, IQR) must be learned from the training data only and then applied to the training, validation, and test sets to avoid data leakage.

Handling Feature Interactions Mathematically

A feature interaction occurs when the effect of one feature on the target variable depends on the value of another feature (or features). If two features X₁ and X₂ interact, their combined effect is not simply additive.

Mathematically, we can introduce interaction terms into a model. For example, in a linear model:

y = β0 + β1X1 + β2X2 + β3(X1X2) + ε

Here, the term β₃(X₁X₂) captures the interaction. The effect of X₁ on y (i.e., ∂y/∂X₁) is now β₁ + β₃X₂, which depends on the value of X₂.

Polynomial Features vs. Neural Network Approaches for Interactions

1. Polynomial Features:

• Mechanism: This involves explicitly creating new features that are products or powers of the original features.
  • For two features X₁, X₂, a degree-2 polynomial expansion could include: X₁, X₂, X₁², X₂², X₁X₂.
  • The X₁X₂ term is a direct interaction term. Terms like X₁² capture non-linear effects of individual features.
• Mathematical Basis: Extends a linear model (or other models) by adding these engineered features. The model itself might still be linear in these new features. For example, Linear Regression with polynomial features can fit non-linear relationships.

  y = β0 + β1X1 + β2X2 + β3X12 + β4X22 + β5X1X2 + ε

• When to Choose:
  • When you have domain knowledge suggesting specific types of interactions or non-linearities (e.g., area = length × width).
  • When the number of original features is relatively small, as the number of polynomial features can grow very rapidly (combinatorially) with the degree and number of original features. For d features and degree p, the number of terms can be (d+p choose p).
  • When interpretability of specific interaction terms is important.
  • Can be used with any model that accepts these new features as input.
• Drawbacks:
  • Can lead to a very high-dimensional feature space (curse of dimensionality).
  • Prone to overfitting if too many polynomial features are added without enough data or regularization.
  • Requires manual specification of the degree and which interactions to include (though tools can generate all up to a certain degree).
  • May not capture very complex or subtle interactions that are not simple products or powers.

2. Neural Network Approaches:

• Mechanism: Neural networks, especially deep ones, can learn feature interactions automatically and implicitly through their layered architecture and non-linear activation functions.
  • In a hidden layer, each neuron computes a weighted sum of inputs from the previous layer, followed by a non-linear activation. For example, if neuron j in layer L takes inputs a_i^(L-1) from neurons i in layer L-1:

    zj(L) = Σi wji(L) ai(L-1) + bj(L)
    aj(L) = activation(zj(L))

  • The non-linear activation functions (ReLU, sigmoid, tanh) are crucial. Without them, a multi-layer network would just be a complex linear model.
  • Subsequent layers combine these activated outputs, allowing the network to learn hierarchical and complex combinations of features, which naturally includes interactions. For instance, a neuron in a deeper layer might activate strongly only when a specific combination of activations from neurons in the preceding layer occurs.
• Mathematical Basis: Universal Approximation Theorem suggests that a neural network with at least one hidden layer and a suitable non-linear activation function can approximate any continuous function to an arbitrary degree of accuracy. This includes functions with complex interactions. The network learns the weights (w_ji) and biases (b_j) through backpropagation to best represent these interactions as needed to minimize the loss function.
• When to Choose:
  • When dealing with high-dimensional data where manually specifying interactions is infeasible.
  • When the nature of interactions is unknown or expected to be highly complex and non-linear.
  • When large amounts of data are available to train the network and learn these interactions without severe overfitting.
  • For tasks like image recognition or natural language processing where interactions between local features (pixels, words) are fundamental.
• Drawbacks:
  • Less interpretable ("black box"): It's hard to pinpoint exactly which specific interactions the network has learned.
  • Requires more data and computational resources than simpler models with polynomial features.
  • More complex to train and tune (architecture choices, hyperparameters, optimization challenges).

In essence, polynomial features are an explicit, engineered way to introduce specific interactions, while neural networks learn interactions implicitly and can capture more complex ones, but at the cost of interpretability and potentially higher data/compute requirements.