Logistic Regression: Predicting Yes or No

Learn how this fundamental algorithm classifies data like ‘Yes/No’ or ‘Spam/Not Spam’.

Logistic Regression: Predicting Yes or No

Machine learning helps us make predictions. Sometimes we predict numbers (like house prices - that’s Regression). Other times, we want to predict categories or groups (like ‘Spam’ or ‘Not Spam’, ‘Cancer’ or ‘No Cancer’, ‘Yes’ or ‘No’ - that’s Classification).

Logistic Regression is one of the most fundamental and widely used algorithms specifically for Classification problems, especially when there are only two possible outcomes (Binary Classification). Despite its name containing “Regression,” its main job is to classify data!

Let’s explore how it works and how to use it.

Why Not Use Linear Regression for Classification?

You might wonder, “Can’t we just use the straight line from Linear Regression?” For classification, usually not. Here’s why:

• Output Isn’t Probability: Linear Regression predicts continuous numbers that can go below 0 or above 1. For classification, we want a probability between 0 and 1 (the chance of belonging to a specific class).
• Sensitivity to Outliers: Linear Regression lines can be heavily influenced by outliers, potentially shifting the decision point incorrectly.

We need a way to take the output of a linear-like equation and squash it neatly into the 0-to-1 probability range. That’s where the magic happens!

The Magic Ingredient: The Sigmoid Function

Squashing Values into Probabilities

Logistic Regression takes the familiar linear combination of inputs (just like in linear regression) but then passes the result through a special function called the Sigmoid Function (or Logistic Function).

First, calculate a value ‘z’ using a linear equation:

z = b₀ + b₁x₁ + b₂x₂ + ... + bnxn

(Where b’s are coefficients/weights and x’s are input features)

Then, plug this ‘z’ into the Sigmoid function, usually denoted by σ(z):

Sigmoid (Logistic) Function
σ(z) = 1 / (1 + e^-z)

e is Euler’s number (approx 2.718). No matter what value ‘z’ has (large positive, large negative, or zero), this function always outputs a value between 0 and 1.

This output, σ(z), is interpreted as the probability that the data point belongs to the positive class (usually labeled as ‘1’).

The S-Curve

The Sigmoid function creates a characteristic “S” shape:

• As ‘z’ gets very large (positive), σ(z) gets very close to 1.
• As ‘z’ gets very large (negative), σ(z) gets very close to 0.
• When ‘z’ is 0, σ(z) is exactly 0.5.

Making the Decision: The Boundary

From Probability to Class

The model outputs a probability (e.g., 0.7, 0.2, 0.5). But usually, we need a definite class label (e.g., ‘Yes’ or ‘No’, 1 or 0). How do we decide?

We use a Decision Boundary (or threshold). The most common threshold is 0.5:

• If the predicted probability σ(z) is ≥ 0.5, we classify the instance as Class 1 (Positive).
• If the predicted probability σ(z) is < 0.5, we classify the instance as Class 0 (Negative).

This threshold corresponds to the point where the linear part z = b₀ + b₁x₁ + ... equals zero. In geometric terms, this often creates a linear boundary (a line, plane, or hyperplane) separating the classes in the feature space.

Adjusting the Threshold

While 0.5 is common, you can adjust this threshold depending on your specific needs:

• In medical diagnosis (like cancer detection), you might lower the threshold (e.g., to 0.3). This makes the model more likely to predict ‘Cancer’ (Class 1), increasing Recall (finding more true cases) but potentially increasing False Positives. You prioritize not missing actual cases.
• In spam filtering, you might raise the threshold (e.g., to 0.8). This makes the model more confident before marking an email as ‘Spam’ (Class 1), increasing Precision (fewer important emails marked as spam) but potentially increasing False Negatives (letting more spam through).

Types of Logistic Regression

While the core idea is the same, Logistic Regression can handle different scenarios:

1. Binary Logistic Regression: The most common type, used when there are only two possible outcome categories (e.g., Yes/No, Spam/Not Spam, Pass/Fail, 0/1).
2. Multinomial Logistic Regression: Used when there are three or more categories that have no natural order (e.g., classifying flower species: Setosa/Versicolor/Virginica; classifying image types: Cat/Dog/Bird).
3. Ordinal Logistic Regression: Used when there are three or more categories that do have a natural order or ranking (e.g., customer satisfaction: Very Unsatisfied/Unsatisfied/Neutral/Satisfied/Very Satisfied; education level: High School/Bachelor’s/Master’s/PhD).

Scikit-learn’s LogisticRegression can handle Binary and Multinomial cases automatically in many situations.

Building a Logistic Regression Model (Python/Sklearn)

Here’s a standard workflow:

1. Load & Prepare Data: Import data using Pandas. Handle any missing values. Separate features (X) and the target variable (y). Ensure ‘y’ contains your categorical labels (e.g., 0 and 1).
2. Split Data: Divide into training and testing sets using train_test_split.
3. Feature Scaling: Very important for Logistic Regression, especially if regularization is used or if features have different scales. Use StandardScaler to scale X_train and X_test. Fit the scaler ONLY on X_train.
4. Train the Model:
   • Import LogisticRegression from sklearn.linear_model.
   • Create an instance: model = LogisticRegression(random_state=0) (setting random_state ensures reproducibility).
   • Fit the model to the scaled training data: model.fit(X_train_scaled, y_train).
5. Make Predictions: Predict probabilities (predict_proba) or class labels (predict) on the scaled test data (X_test_scaled).
6. Evaluate the Model: Assess performance using metrics appropriate for classification:
   • Confusion Matrix: Shows TP, TN, FP, FN. Use confusion_matrix(y_test, y_pred).
   • Accuracy Score: Overall percentage correct. Use accuracy_score(y_test, y_pred).
   • Precision, Recall, F1-Score: Especially important for imbalanced data. Use classification_report(y_test, y_pred).

Logistic Regression: Key Takeaways

• Logistic Regression is a fundamental algorithm for Classification tasks (predicting categories), especially binary (0/1) outcomes.
• It uses the Sigmoid function to convert a linear combination of inputs into a probability between 0 and 1.
• A Decision Boundary (threshold, often 0.5) is used to convert the probability into a final class prediction.
• Types include Binary, Multinomial (3+ unordered categories), and Ordinal (3+ ordered categories).
• Feature Scaling is important before training.
• Evaluation relies on the Confusion Matrix and metrics like Accuracy, Precision, Recall, and F1-Score, especially for imbalanced data.