Dimensionality Reduction: PCA vs. t-SNE vs. UMAP

Analogy: Making a 2D Map of a 3D Globe

• PCA (Principal Component Analysis) is like casting a shadow of the globe onto a flat wall. It's a linear projection that tries to capture the maximum possible variance (the overall shape and spread of the continents). Distances between far-apart cities are roughly preserved, but local details and the curvature of the earth are lost.
• t-SNE (t-Distributed Stochastic Neighbor Embedding) is like trying to draw a map of a crowded city center. Its only goal is to make sure that buildings that are close neighbors in reality are also close neighbors on the map. It will happily distort the distance between the city center and the suburbs to achieve this local fidelity.
• UMAP (Uniform Manifold Approximation and Projection) is like a modern digital cartographer's map. It tries to find a balance, preserving local neighborhood structures like t-SNE, but also doing a better job of preserving the large-scale, global structure, like the relative positions of different continents.

1. PCA (Principal Component Analysis)

• Mechanism: A linear algebra technique that finds a new set of orthogonal axes, called principal components, that align with the directions of maximum variance in the data. It then projects the data onto a lower-dimensional subspace formed by the top `k` principal components.
• Goal: To preserve the global variance of the data.
• Use Cases:
  • Data Preprocessing: It's an excellent, fast method for reducing dimensionality before feeding data into a machine learning model, as it helps combat the curse of dimensionality and can de-correlate features.
  • Data Compression: Storing the data in its lower-dimensional PCA representation.
• Limitations: Being linear, it cannot capture complex, non-linear structures (manifolds) in the data. It assumes the most interesting information lies in the directions of highest variance, which may not always be true.

2. t-SNE (t-Distributed Stochastic Neighbor Embedding)

• Mechanism: A probabilistic, non-linear technique. For each data point, it constructs a probability distribution of its neighbors in the high-dimensional space. It then tries to create a low-dimensional embedding where a similar probability distribution is reproduced.
• Goal: To preserve the local structure of the data. It focuses on keeping close points close in the low-dimensional map.
• Use Cases:
  • Data Visualization: This is its primary and most famous use case. It is exceptional at creating beautiful, well-separated clusters in 2D or 3D for visual exploration of high-dimensional data, like embeddings from a neural network.
• Limitations:
  • The global structure is not preserved. The distance and size of clusters in a t-SNE plot are not meaningful.
  • It is computationally very expensive, with a complexity of roughly O(n²), making it slow for large datasets.
  • It is a visualization technique, not a preprocessing step.

3. UMAP (Uniform Manifold Approximation and Projection)

• Mechanism: A newer, non-linear technique grounded in manifold theory and topological data analysis. Like t-SNE, it focuses on preserving local structure, but it also tries to maintain the global data structure.
• Goal: To create a balanced representation that preserves both local and global structure.
• Use Cases:
  • Data Visualization: It often produces visualizations with better separation of global clusters than t-SNE.
  • General-purpose Dimensionality Reduction: Unlike t-SNE, UMAP can be used as a pre-processing step for downstream machine learning tasks because it preserves more of the global topology.
• Limitations: It's still a relatively new algorithm, though it has gained immense popularity. Like t-SNE, its output can be sensitive to its hyperparameters (like `n_neighbors` and `min_dist`).

The Problem with t-SNE for Preprocessing

1. Focus on Local, Not Global, Structure: The primary goal of t-SNE is to create a visually appealing map by ensuring that points that are neighbors in high-dimensional space remain neighbors in the low-dimensional map. It makes no guarantees about preserving the distances or relationships between points that are far apart. In fact, it often significantly distorts these large-scale distances to better separate the local clusters. A machine learning model trained on this distorted representation would be learning from false global relationships.
2. No Meaningful Transformation for New Data: t-SNE is not a "transformation" in the same way PCA is. PCA learns a fixed set of linear projections (the principal components) that can be applied to new, unseen data points to project them into the same low-dimensional space. t-SNE, on the other hand, is a non-parametric method that optimizes the positions of the training data points themselves. It does not learn an explicit function to map new points from the high-dimensional space to the low-dimensional one. While some libraries have a `transform` method, it's an approximation and not its intended use.
3. Stochastic Nature: Running t-SNE multiple times on the same data can produce slightly different-looking plots. This lack of a stable, deterministic mapping makes it unsuitable for a reproducible preprocessing pipeline.

In contrast, PCA is perfect for preprocessing because it learns a deterministic, linear mapping that preserves as much of the global variance as possible. This global information is exactly what most machine learning algorithms (like clustering or classification models) rely on to find decision boundaries or separate groups.