Aha Subscription Churn Analysis

A key challenge will be comparing groups with inherently different commitment lengths and observation windows. For monthly subscribers, churn is typically defined as non-renewal at the end of their 30-day cycle. For yearly subscribers, "financial churn" only occurs at the 12-month renewal point. However, we also need to consider "engagement churn" for yearly users – signs of disengagement long before their renewal date.

Core Churn & Retention Metrics:

• Monthly Subscriber Churn Rate: (Number of monthly subs who did not renew in period X) / (Total monthly subs up for renewal in period X). Tracked monthly.
• Yearly Subscriber Renewal Rate: (Number of yearly subs who renewed at end of 12 months) / (Total yearly subs whose 12-month term ended). This is an annual lagging indicator.
• Cohort Retention Curves: For monthly subs, track % retained month-over-month. For yearly subs, track % still active/engaged month-over-month within their 12-month term, and then the renewal rate. This helps visualize loyalty decay.
• Time-to-Churn Distribution: For monthly subs, how many months do they typically stay before churning?

Leading Indicators for "Engagement Churn" (especially for Yearly Subs):

• Decline in Active Days per Month: Track trends in (Number of days with >X mins watch time) / (Days in month). A consistent drop for a yearly sub, say from 15 active days to 2 over consecutive months, is a red flag.
• Drop in Watch Time per Active Day/Session.
• Reduced Content Diversity: User stops exploring new genres or shows, sticks to only one type or rewatches old content (could be comfort or boredom).
• Shift in Content Preference (Sudden): E.g., a user who watched diverse Telugu movies/series suddenly only watches news or short-form content, indicating a change in how they value the platform.
• Decrease in Feature Usage: Drop in usage of watchlist, downloads, search functionality.
• Increase in Customer Support Tickets (Negative Sentiment): Especially if related to content availability or technical issues.
• Last Login Recency: While they are paid up, a yearly sub not logging in for 30-60 days is a strong signal of disengagement.

For yearly subscribers, I'd develop an "Engagement Health Score" based on a weighted combination of these leading indicators, tracked monthly. A significant, sustained drop in this score would signify engagement churn, even if financial churn is months away.

1. Normalized Time Windows & Survival Analysis:
   • The most robust approach is Survival Analysis. We can plot Kaplan-Meier survival curves for both groups. For monthly users, the "event" is non-renewal each month. For yearly users, the primary "event" is non-renewal at 12 months, but we also observe their engagement status throughout the year.
   • This naturally handles the different time scales and censoring (yearly users who haven't reached their renewal point yet are "censored"). We can compare median survival times or survival probabilities at specific points (e.g., P(Surviving > 12 months)).
2. Conditional Retention for Yearly Subscribers:
   • Within the yearly subscriber cohort, we can look at "conditional" monthly retention. For example, of those yearly subscribers who were active in month 3, what percentage were still active in month 4? This helps understand engagement decay within the annual cycle.
3. "Equivalent Lifecycle" Churn View (More Conceptual):
   • One could try to map the monthly churn over 12 months to an "equivalent annual churn." If monthly churn is `m`, then annual retention is `(1-m)^12`, so equivalent annual churn is `1 - (1-m)^12`. This allows a conceptual comparison of what the yearly churn would be if it behaved like 12 independent monthly decisions. However, this assumes constant monthly churn and independence, which is rarely true. Survival analysis is better.
4. Focus on Behavioral Leading Indicators:
   • Instead of just financial churn, compare the trends of leading engagement indicators (active days, watch time) over, say, the first 3-6 months for both groups. Do yearly subscribers show significantly more stable or higher engagement early on than monthly subscribers who eventually churn within that same timeframe?

The primary statistical challenge is immortal time bias for yearly subscribers – they cannot financially churn before month 12. Survival analysis inherently handles this by considering the time-to-event. Comparing engagement metrics over aligned early periods (e.g., first 3 months of subscription for both groups) can also provide valuable insights into initial stickiness.

• Financial Churn: This is the actual cessation of revenue. For monthly, it's immediate. For yearly, it's delayed until the renewal point.
• Engagement Churn (or Disengagement): This is a behavioral precursor. For yearly subscribers, high disengagement mid-term (e.g., at month 6) is a very strong predictor of financial churn at month 12.

Impact on Metrics & Business Importance:

1. Metrics for Yearly Subscribers Need Layering:
   • Lagging Metric: Year-end renewal rate (financial outcome).
   • Leading Metric: Monthly "Engagement Health Score" or "Risk of Non-Renewal Score." This score would be based on the leading indicators I mentioned (active days, watch time decay, etc.). A user dropping into a "High Risk of Non-Renewal" category at month 6 is a critical event, even if they've paid.
2. Business Importance of Engagement Churn for Yearly Subs:
   • Lost Renewal Opportunity: The primary concern is that they won't renew. The upfront payment for the current year is sunk revenue; the future LTV is at risk.
   • Reduced Word-of-Mouth/Advocacy: A disengaged user, even if paid up, is unlikely to recommend Aha.
   • Missed Upsell/Cross-sell Opportunities: If Aha introduces new features, content bundles, or merchandise, a disengaged user won't see or act on them.
   • Platform Health: High numbers of disengaged (but paid) users might skew overall platform engagement metrics if not properly segmented, potentially masking issues.
   • Intervention Window: Detecting engagement churn early for yearly subs gives us a longer window to intervene (e.g., targeted content recommendations, special feature access, surveys to understand dissatisfaction) compared to a monthly sub who might churn before we can react.

So, for yearly subscribers, while we've secured the current year's revenue, their mid-term engagement churn is a critical leading indicator for future revenue (LTV) and overall platform health. My metrics would reflect this by having a "Churn Risk Score" for yearly subs that is monitored throughout their term, triggering retention efforts if it crosses a threshold.

Modeling Time-to-Churn:

For Monthly Subscribers:

• The event (churn/non-renewal) can happen at the end of any month.
• Model Choice: A discrete-time survival model or a continuous-time model like Cox Proportional Hazards or Accelerated Failure Time (AFT) models could work. Given the monthly cadence, a discrete-time logistic regression model predicting churn probability each month, or a Cox model treating each month as a potential event point, would be suitable.
• Distribution Assumptions: If using parametric AFT models, we could test distributions like Weibull (which allows for increasing or decreasing hazard over time) or Log-Logistic. The hazard rate for monthly subscribers might initially decrease (as less committed users churn early) and then perhaps stabilize or slightly increase.
• Features: Would heavily rely on recent engagement (last 7-30 days), content watched, payment success/failures, interaction with new features, etc.

For Yearly Subscribers (Predicting Non-Renewal at 12 Months):

• The primary financial churn event is at the 12-month mark.
• Approach 1: Binary Classification at Renewal Point:
  • Treat it as a binary classification problem: will the user renew (1) or not (0) at month 12?
  • Model Choice: Logistic Regression, Random Forest, Gradient Boosting.
  • Features: Aggregate engagement over the entire year (e.g., average active days/month, total watch time, content diversity, engagement trend in months 9-11), usage of key features, customer support interactions, price sensitivity (if they used a discount to sign up).
• Approach 2: Survival Analysis for Engagement Decay (Leading Indicator):
  • Define an "engagement churn" event mid-term (e.g., user's Engagement Health Score drops below a threshold for X consecutive weeks, or no login for 60 days).
  • Model Choice: Cox Proportional Hazards model to predict the "time to severe disengagement."
  • Features: Dynamic features tracking engagement trends month-over-month.
  • The output of this (e.g., high hazard of disengagement by month 6) would then be a strong feature into the binary classification model for year-end renewal.
• Approach 3: Piecewise Hazard Model (Advanced):
  • One could model the hazard rate as very low for months 1-11 (as financial churn is impossible) and then have a step increase or a different hazard function for the renewal decision period (months 11-12). This is more complex but can be powerful if we want a unified survival framework. The hazard function `h(t)` would be defined differently for `t < 12 months` vs. `t = 12 months`.
  • `h(t) = h_engagement(t)` for `t < 12` (reflecting disengagement risk)
  • `h(t) = h_renewal_decision(features_at_t11)` for `t = 12` (reflecting renewal choice)

The core idea is that for yearly subscribers, predicting the final renewal decision often relies on summarizing their behavior and engagement over the preceding year, while also monitoring for drastic drops in engagement mid-term as strong warning signs.

Testing Proportional Hazards Assumption:

1. Visual Inspection of Log-Log Survival Plots: For a categorical predictor, plot `log(-log(S(t)))` vs `log(t)` for each category. If hazards are proportional, the curves should be parallel.
2. Schoenfeld Residuals: These residuals should not show a systematic pattern when plotted against time. A non-zero slope in a regression of Schoenfeld residuals against time for a covariate indicates a violation. Statistical tests based on these residuals (e.g., Grambsch and Therneau test) can formally test the assumption.
3. Time-Varying Covariates: Include interaction terms between covariates and a function of time (e.g., `covariate * log(time)`). If the interaction term is statistically significant, the assumption is violated for that covariate.

Handling Violations:

1. Stratification: If a categorical covariate (like `plan_type` if we were trying a unified model, or perhaps `acquisition_channel`) violates the assumption, we can stratify the model by that variable. This allows each stratum to have a different baseline hazard function, but the covariate effects within each stratum are assumed proportional.
2. Time-Varying Coefficients: Introduce explicit time-dependent coefficients for the problematic covariate. This means the effect of the covariate `β(t)X` changes over time. This is more complex to implement and interpret.
3. Time-Varying Covariates (as mentioned above): Create interaction terms like `feature * time` or `feature * log(time)`. This explicitly models how the feature's effect changes.
4. Alternative Models: If violations are widespread and severe, consider models that don't rely on the PH assumption, like some parametric AFT models (e.g., Log-logistic can model non-proportional hazards) or more flexible non-parametric approaches, though these might be harder to interpret.
5. Binning Time: For some analyses, one could split the time period into segments and fit separate Cox models or analyze hazard ratios within those segments where PH might hold locally.

For a feature like "average watch time per week," if its impact diminishes over time (e.g., high early watch time is very predictive, but high watch time in month 10 for a yearly sub is less impactful on the month 12 renewal than month 2 watch time was on month 4 churn for a monthly sub), then an interaction with time or a time-varying coefficient would be necessary for that feature.

Explaining the Flaw to a Non-Technical Executive:

"Imagine we're looking at two groups of people starting a new fitness program.

• Group A (Yearly Subscribers): They've committed and paid for a full year. We only check at the end of the year if they sign up for another year. If 15 out of 100 don't renew, that's a 15% churn for that specific point of decision.
• Group B (Monthly Subscribers): They decide every single month whether to continue. If their monthly churn is, say, 5.5%, then over a year, it might compound to look like 45% of them are gone.

The problem is, we're comparing apples and oranges because:

1. Different Commitment & Decision Points: Yearly users make one big decision after 12 months. Monthly users make 12 smaller decisions. The 45% annualized figure for monthly users tells us how many would have left if they were forced to make 12 sequential stay/leave decisions. But many monthly users who churn might do so in month 1, 2, or 3. They never even make it to month 12 to be comparable to the yearly cohort's decision point.
2. Survivor Bias in Yearly: By definition, all yearly subscribers in the 15% churn calculation survived the full 12 months to even have a renewal event. The monthly subscribers in the 45% annualized figure include many who left much earlier.
3. "Locked-in" Effect: The yearly plan has a "lock-in" effect. Even if a yearly user is disengaged at month 3, they don't "churn" financially until month 12. A monthly user in the same state of disengagement would have already churned financially.

So, to the executive, I'd say: "The 15% for yearly and 45% annualized for monthly aren't directly comparable because they measure different things over different commitment structures. The yearly figure looks better partly because those users are locked in for a year. A fairer comparison might be to look at how many of our new monthly subscribers are still with us after 12 full months, versus how many new yearly subscribers renew for a second year. Or, we can look at the LTV generated by an average monthly subscriber versus an average yearly subscriber over, say, an 18-24 month period to see their total value."

I'd emphasize using cohort retention curves plotted on the same timeline (e.g., months since subscription start) to visually show how each group decays over time. The yearly curve will be flat at 100% (financially) until month 12 then drop, while the monthly will decay from month 1.

Rationale for Separate Models:

1. Different Churn Mechanisms & Timelines:
   • Monthly: Churn is a high-frequency event, potentially driven by short-term factors like recent content drought, payment failure, a competitor's new offer, or immediate dissatisfaction. Prediction needs to be sensitive to very recent behavior (e.g., last 7-14 days).
   • Yearly: Financial churn is a low-frequency event (once a year). Predictors are likely longer-term engagement trends, perceived value over many months, and specific interactions around the renewal period. "Engagement churn" mid-year is a different phenomenon to model.
2. Feature Importance & Engineering Will Differ:
   • For monthly, features like "days since last session," "watch time in last 7 days," or "failed payment attempt this month" would be highly predictive.
   • For yearly (renewal prediction), features like "average monthly active days over last 6 months," "number of blockbuster movies watched in the year," "engagement trend in quarter leading to renewal," or "price sensitivity to renewal offer" would be more relevant. Using the same feature set might dilute their predictive power for each group.
3. Different Prediction Horizons & Actions:
   • Monthly churn prediction informs immediate, short-cycle interventions (e.g., "We miss you!" email after 2 weeks of inactivity).
   • Yearly non-renewal prediction informs longer-term interventions starting 1-3 months before renewal (e.g., renewal discount offers, content highlighting for the next year).
4. Model Complexity & Interpretability:
   • A unified model trying to capture these diverse patterns might become overly complex, harder to interpret, and potentially less accurate for specific subgroups than specialized models. It might try to find an "average" effect that fits neither group perfectly.

Trade-offs:

• Pros of Separate Models: Potentially higher accuracy for each segment, better feature relevance, tailored actionability.
• Cons of Separate Models: More development effort, separate maintenance and monitoring pipelines (potential technical debt), doesn't easily handle users switching plans (though this is a separate modeling problem).

A unified model with `plan_type` as a feature (and many interaction terms with `plan_type`) could theoretically learn these differences, but it might require a very large dataset and careful engineering to prevent one group's patterns from dominating or a "compromise" fit. I'd start with separate models for clarity and targeted accuracy, then potentially explore a unified architecture if there are significant shared underlying behaviors or resource constraints demand it.

Hybrid/Multi-Task Learning Architecture:

This approach aims to get the best of both worlds: shared learning for common patterns and specialized learning for plan-specific behaviors.

1. Shared Feature Engineering & Embedding Layer:
   • Develop a comprehensive set of features capturing user behavior, content consumption, demographics, etc., applicable to all users regardless of plan.
   • Input these features into a shared neural network (or gradient boosting) base that learns a common, rich representation (embedding) of each user's state and behavior. This base captures fundamental engagement patterns.
2. Task-Specific "Heads" (Output Layers):
   • From this shared embedding layer, branch out into separate "heads" or smaller models:
     • One head specifically trained to predict next-month churn for users currently on a monthly plan.
     • Another head trained to predict year-end renewal for users currently on a yearly plan (or their mid-term engagement churn risk).
   • These heads can have different architectures or even be different model types (e.g., a recurrent layer for monthly patterns, a simpler feed-forward layer for yearly renewal based on aggregated yearly behavior).
3. Handling Plan Switchers:
   • When a user switches plans (e.g., monthly to yearly), their learned user embedding from the shared base layer is preserved. This embedding, which captures their historical behavior, becomes a powerful input for the "yearly plan" prediction head.
   • The model for their new plan type can then leverage this rich history immediately, rather than treating them as a completely new user with a cold start problem for that plan type.
   • We would also need a feature indicating "time since last plan switch" and "previous plan type" as these could be highly predictive.
4. Joint Training:
   • The entire network (shared base + specific heads) can be trained jointly, allowing the shared layers to learn representations beneficial for all tasks, while the specific heads fine-tune for their particular prediction. Loss functions would be combined.

Addressing Overhead:

• Reduced Redundancy: Feature engineering is largely unified. Monitoring the health of the shared base is centralized.
• Focused Specialization: While there's still complexity in the heads, it's more targeted than fully independent pipelines.

This hybrid approach mitigates the technical debt by sharing common components and gracefully handles plan switchers by transferring learned knowledge through the shared user representation. It's more complex to build initially but more robust and efficient in the long run.

Tier 1: Executive Dashboard (Strategic Overview)

• Key Churn KPIs: Overall monthly churn, yearly renewal rates, trends over time (vs. targets).
• Cohort Performance: Visualizations of monthly vs. yearly cohort retention curves.
• Engagement Health: Trend of average engagement scores for yearly subscribers.
• Financial Impact: Estimated revenue at risk due to predicted churn, LTV of monthly vs. yearly segments.

Tier 2: Marketing & Retention Teams Dashboard (Tactical & Operational)

• Daily/Weekly Churn Risk List: Segmented lists of users with high churn probability (e.g., top 10% riskiest monthly users, yearly users with declining engagement score in months 7-9).
• Intervention Triggers: Alerts when a high-value user enters a "high risk" state.
• Recommended Actions: Based on churn drivers for specific segments (e.g., "User X shows declining watch time in preferred genre Y – suggest new content in genre Y or offer a targeted promotion").
• A/B Test Performance Tracker: For ongoing retention campaigns.

Tier 3: Data Science & Engineering Dashboard (Model Health & Performance)

• Model Accuracy Metrics: Precision, Recall, F1, AUC-ROC/PR for churn models, tracked over time.
• Feature Drift Monitoring: PSI for key predictive features. Alert if drift exceeds thresholds.
• Prediction Distribution Monitoring: Check for unexpected shifts in churn probability distributions.
• Data Pipeline Health: Latency, data quality checks.

Operational Processes:

• Automated Triggers: For certain risk thresholds, automatically trigger pre-defined interventions (e.g., a personalized "We miss you" email with content recommendations for a monthly user inactive for 15 days).
• Manual Review & Personalized Outreach: For very high-LTV users at risk, flag them for manual review by a retention specialist who might make a personalized call or offer.
• Feedback Loop: Regularly review effectiveness of interventions and feed learnings back into model refinement and strategy.

Detection Strategy for Model Staleness:

1. Population Stability Index (PSI):
   • Regularly (e.g., weekly/monthly) calculate PSI for all key input features. Compare the distribution of features in the current scoring data against the distribution in the training data. A PSI > 0.25 for critical features is a strong warning.
2. Prediction Distribution Drift:
   • Monitor the distribution of predicted churn probabilities. If the model suddenly starts predicting everyone as low risk, or everyone as high risk, something is likely wrong or the underlying user behavior has fundamentally shifted. Compare current prediction distributions to those from the validation set using metrics like KL divergence or Chi-squared tests.
3. Actual vs. Predicted Monitoring:
   • Continuously track actual churn rates against predicted churn rates for different probability buckets. If users predicted at 10% risk are actually churning at 30%, the model is miscalibrated or stale. Plot calibration curves.
4. Business Metric Correlation:
   • If overall platform churn rates (a business KPI) start diverging significantly from what the model's aggregate predictions would suggest, it indicates the model is out of sync with reality.
   • Monitor external event logs (new major content releases on Aha or competitor, start/end of cricket season) and see if model performance dips correlate.

Retraining Strategy:

• Scheduled Retraining:
  • Monthly Models: Likely need frequent retraining, e.g., weekly or bi-weekly, using a sliding window of the most recent data (e.g., last 3-6 months) to capture recent trends.
  • Yearly Renewal Models: Can be retrained less frequently, e.g., quarterly or semi-annually, as the event is less frequent. However, the "engagement churn" model for yearly subs might need more frequent updates.
• Trigger-Based Retraining:
  • If PSI or prediction drift alerts fire, or if model accuracy drops below a predefined threshold, trigger an ad-hoc retraining.
• Champion/Challenger Framework:
  • Always test a newly retrained model (challenger) against the current production model (champion) on a holdout set before deploying to ensure it actually performs better.
• Online Learning (Partial, for some components):
  • For some rapidly changing features or for the very short-term monthly churn prediction, components of the model could incorporate elements of online learning where weights are updated more frequently with new data streams, though full online retraining of complex models can be risky.

Cold Start Churn Prediction Strategy (Day 1-7):

1. Pre-Signup & Onboarding Data (if available):
   • Acquisition Source: Did they come from an organic search, a specific ad campaign (e.g., one promoting a blockbuster movie vs. a discount offer), a referral, or an app store feature? Different sources often have different intent levels and churn profiles.
   • Device Type & OS: Users on high-end devices might behave differently or have different expectations than those on older/budget devices.
   • Geographic Location (Region/City Tier): Can be a proxy for socio-economic factors or content preferences.
   • Subscription Plan Chosen Immediately: Did they opt for monthly or go straight for yearly (if that's an option at signup)? A yearly commitment upfront is a strong positive signal.
   • Profile Completion: Did they fill out optional profile information, select genre preferences during onboarding (if asked)? Higher engagement with onboarding might correlate with lower churn.
2. Very Early Engagement Signals (First few sessions / Day 1-7):
   • Time to First Play: How quickly did they start watching content after signup?
   • Content Discovery Behavior: Did they browse extensively, use search, or just play the first thing on the homepage?
   • Completion of First Few Key Titles: If they watch a full movie or a few episodes of a series early on, it's a good sign.
   • Session Length & Frequency (even if limited): Any early patterns.
   • Interaction with Core Features: Did they create a watchlist, try downloads, explore settings?
3. Lookalike Modeling / Collaborative Filtering (User-User):
   • Based on the sparse data we do have (e.g., acquisition source, device, first few titles browsed/watched), find "similar" established users whose churn outcomes are known. The new user's churn risk can be initially estimated based on the average churn risk of their nearest neighbors. This needs careful implementation to avoid sparsity issues.
4. Simpler Heuristic Models Initially:
   • A simpler rule-based model or a logistic regression with fewer features might be more robust for very new users than a complex GBM that relies on rich behavioral history.
5. Progressive Prediction:
   • The churn prediction for a new user should be updated frequently as more data comes in. The Day 1 prediction will be less certain; by Day 7, with some initial engagement data, it becomes more reliable. The model used might even change (e.g., from lookalike/heuristic to a behavioral model).

The goal for cold-start users is to get a reasonable initial risk assessment and then rapidly refine it as their behavior on the platform generates more data signals.

Handling External Shocks (e.g., COVID-like Lockdown):

1. Immediate Detection & Monitoring:
   • The feature drift and prediction drift monitoring systems I mentioned earlier should flag this. We'd see sudden, massive shifts in engagement metrics (watch time, active days likely up) and possibly changes in churn probability distributions.
   • Monitor external news and public health announcements to contextualize these data shifts.
2. Acknowledge Model Limitations:
   • The existing pre-shock models are likely to become unreliable. Their predictions need to be treated with extreme caution.
3. Adaptive Modeling Strategy:
   • Short-Term Response (Triage):
     • Potentially pause automated interventions heavily reliant on pre-shock model scores if they start behaving erratically.
     • Switch to simpler, more robust heuristics or rule-based triggers for immediate churn management if complex models are failing.
     • Increase the weight of very recent data in any short-term forecasting components.
   • Model Adaptation / Retraining:
     • Rapidly collect data from the "shock period."
     • Build a "Shock Period Model" (or Regime-Specific Model): Train a new model (or retrain the existing one) specifically on data from the lockdown period. This model would learn the new relationships between features and churn under these unique conditions.
     • Feature Engineering for Shock Context: Introduce new features like `is_lockdown_period`, `days_since_lockdown_start`, `regional_case_count_proxy`, `work_from_home_indicator_proxy`.
     • Ensemble/Hybrid Approach: Once we have a "shock period model," we might run an ensemble of the pre-shock model and the shock-period model, potentially weighting their outputs based on how "shock-like" the current conditions are.
   • Analyze Shifts in Churn Drivers:
     • During a lockdown, factors like "content library depth" or "new exclusive releases" might become even more critical churn drivers than usual, as users have more time to consume. Price sensitivity might also increase. Feature importance from the "shock period model" would reveal this.
4. Adjust Churn Management Strategy:
   • The reasons for churn might change. Pre-shock, it might be content discovery; during shock, it might be affordability or sheer content exhaustion. Retention campaigns need to adapt.
   • Proactive communication about new content, extended free trials (if business allows), or flexible payment options might become more important.
5. Post-Shock Transition:
   • As the shock recedes (e.g., lockdown ends), we'd need to manage the transition back. This might involve another period of model adaptation or a gradual shift back towards the pre-shock model, or a new "post-shock normal" model.

Essentially, it's about having systems to detect anomalies quickly, being prepared to build/adapt models for the new reality, and understanding that churn drivers themselves can shift dramatically during such events.

During an event like the initial COVID lockdown, there's a paradox:

• Increased Engagement (Positive Signal): People are home, screen time increases, content consumption naturally goes up. This might temporarily suppress some types of engagement-driven churn.
• Increased Economic Uncertainty & Price Sensitivity (Negative Signal): Job losses, pay cuts, general anxiety can make users scrutinize discretionary spending like OTT subscriptions more closely, potentially increasing price-driven churn.
• Increased Competition for Attention: While OTT usage is up, so is usage of gaming, social media, e-learning.

Balancing Model Predictions with Business Strategy During Anomaly:

My approach would be:

1. Don't Necessarily Relax Efforts Based on Gross Engagement Metrics:
   • While overall watch time might be up, this could mask underlying issues. The type of engagement matters. Are they exploring new content, or just re-watching comfort shows? Is content discovery still healthy?
   • The churn model, if adapting correctly to "shock period" data, should still identify users whose engagement is low relative to the new, higher baseline or who exhibit other risk factors (like payment issues).
2. Segment-Specific Strategy Adjustment:
   • For Price-Sensitive Segments (Identified via demographics, past discount usage, or model features): INTENSIFY sensitive retention efforts. Offer flexible payment options, temporary discounts if they try to cancel, or highlight value-for-money content bundles. Economic hardship could be a major churn driver here.
   • For Content-Driven Segments (High historical engagement, diverse tastes): Focus on content-based retention. Ensure they are aware of the full library, new releases, and personalized recommendations. Their risk might be content exhaustion or boredom if new content isn't flowing.
   • For Newly Acquired "Lockdown Users": These users joined during the anomalous period. Their long-term loyalty is unproven. Onboard them well and monitor their engagement closely as normalcy returns, as they might be at high risk of post-lockdown churn.
3. Shift Intervention Types:
   • Perhaps fewer aggressive "win-back discounts" are needed if organic engagement is high, but more focus on "value reinforcement" messaging, highlighting the breadth of the Aha library, new features, or family sharing options.
   • Proactive customer support to address any technical issues quickly becomes even more critical when users are relying heavily on the service.
4. Monitor "Value Churn" not just "User Churn":
   • Are users downgrading from yearly to monthly (if that's an option)? Are they shifting to lower-tier plans? This is a form of churn that needs tracking.
5. Prepare for Post-Shock Re-Normalization:
   • Anticipate that as the shock recedes (e.g., lockdowns lift), there might be a "correction" in engagement and potentially a spike in churn as people resume other activities. The strategy needs to plan for this transition.

So, I wouldn't broadly "relax" churn prevention. I would adapt it. The model's role is to identify who is at risk given the current (anomalous) conditions. The business strategy then decides how to act on that information, considering the unique context. For instance, a user whose engagement drops even during a lockdown is a very strong churn signal.

High-Level Infrastructure Design & Cost Optimization:

1. Data Ingestion & Storage:

• Raw Data: User activity logs, subscription events, content metadata ingested via a streaming pipeline (e.g., Kafka) into a data lake (e.g., AWS S3, Google Cloud Storage). Cost-effective for large volumes.
• Processed Data & Feature Store:
  • Aggregated features (e.g., rolling 7-day watch time, monthly active days) computed via batch jobs (e.g., Spark, Airflow DAGs) and stored in a more structured data warehouse (e.g., Redshift, BigQuery, Snowflake) or a dedicated Feature Store.
  • The Feature Store allows for efficient retrieval for both training and real-time inference, reducing redundant computation. Store frequently accessed features in faster, more expensive storage, and less frequent ones in cheaper tiers.

2. Model Training Infrastructure:

• Monthly Model (Daily/Weekly Retraining):
  • Compute: Cloud-based managed ML platforms (e.g., SageMaker, Vertex AI, Azure ML) using appropriately sized instances. For gradient boosting, CPU instances are usually sufficient.
  • Orchestration: Airflow or a similar workflow manager to schedule data preprocessing, training, evaluation, and deployment.
  • Cost Optimization:
    • Use spot instances or preemptible VMs for training jobs where possible (can be ~70% cheaper), with checkpoints to handle interruptions.
    • Optimize training data size – perhaps sample intelligently if full data isn't needed for daily incremental updates.
    • Efficient hyperparameter tuning (e.g., Bayesian optimization, early stopping).
• Yearly Model (Weekly/Quarterly Retraining):
  • Can use larger batch jobs, potentially more powerful instances if using deep learning components for the hybrid model. Similar cost optimization strategies apply.

3. Model Deployment & Inference:

• Monthly Churn Scores (Daily Batch Scoring):
  • A scheduled batch inference job can score all 1 million monthly users daily. Results (churn probability, risk segment) are written to a database accessible by marketing/retention tools. This is generally more cost-effective than real-time inference for all users if daily scores are sufficient.
• Yearly Engagement/Renewal Risk Scores (Weekly Batch Scoring):
  • Similar batch scoring process for the 200k yearly users.
• (Optional) Real-time Inference for Critical Triggers:
  • If certain immediate actions are needed (e.g., an offer upon cancellation attempt), a lightweight version of the model could be deployed on a scalable serverless endpoint (e.g., AWS Lambda + API Gateway, Google Cloud Functions) for specific critical user interactions. This would be for a small subset of events, not scoring all users in real-time.
• Model Serving: Use managed model serving solutions (e.g., SageMaker Endpoints, Kubeflow on Kubernetes) if real-time components are needed. Ensure auto-scaling to handle load and manage costs.

4. Monitoring & Cost Management:

• Implement comprehensive logging and monitoring for all pipeline components (data, training, inference).
• Use cloud cost management tools to track spend by service/tag. Set up budget alerts.
• Regularly review resource utilization and downsize/optimize instances where possible.
• Consider model compression/quantization for deployed models to reduce inference cost/latency if applicable.
• Archive old model versions and data artifacts to cheaper storage.

The key is a modular architecture, leveraging managed cloud services where sensible, optimizing batch vs. real-time based on business need, and aggressively using cost-saving features like spot instances for non-critical compute.

1. Shift in Customer Mix / Value Dilution (Most Likely):
   • Our retention efforts might have been more successful at retaining lower-value yearly subscribers while we still lost a similar or even higher number of high-value yearly subscribers (those who might have had higher engagement, potentially were on a slightly more premium tier if one existed, or were less discount-sensitive).
   • Essentially, we improved the count of retained users, but the average revenue per retained user decreased.
   • Action: Segment the renewal analysis by user LTV or historical spend. Did high-LTV users renew at a lower rate even if overall renewal improved?
2. Impact of Retention Offers & Discounts:
   • To achieve the 3% point reduction in churn (from 15% to 12%), we might have used aggressive discounts or offers for users at risk of non-renewal. While they renewed, they did so at a lower effective price point, thus reducing total revenue from that cohort.
   • Action: Calculate the effective net price paid by renewing subscribers. Quantify the cost of retention discounts.
3. Cannibalization from Monthly to Discounted Yearly:
   • If we offered attractive "upgrade to yearly at a discount" paths for monthly users as a retention tactic, some users who might have paid the full yearly price (or stayed longer on monthly at full price) could have shifted to a discounted yearly plan, lowering average revenue per yearly subscriber.
4. Changes in New Yearly Subscriber Acquisition:
   • The new yearly subscribers acquired during this six-month period (who haven't hit their renewal yet but contribute to current revenue) might be coming in at lower price points (e.g., more aggressive acquisition promotions for yearly plans) or might be a lower-value segment overall compared to previous yearly cohorts. This affects the total revenue pool.
5. External Factors Affecting ARPU:
   • Were there any platform-wide price adjustments or a shift towards more budget-friendly yearly plan variants during this period?
6. Misattribution or Flawed Projection:
   • Were the revenue projections too optimistic? Is the 5% drop relative to a baseline that was perhaps unachievable anyway?

Explaining to the CEO:

"While we successfully reduced the headline yearly churn rate by 3 percentage points, which means more users are choosing to stay with Aha for another year, our total revenue from this group is down 5%. This suggests that either the users we retained are, on average, contributing less revenue than those we retained previously, or the cost of retaining them (e.g., through discounts) has offset the benefit of higher numbers.

We need to immediately investigate:

1. The value profile of the users we retained vs. those we still lost.
2. The financial impact of the retention offers used.
3. The profile of newly acquired yearly subscribers.

Our goal should be to optimize for revenue churn or LTV-weighted retention, not just the raw user churn count. This means focusing efforts on retaining our most valuable subscribers, even if it means the overall churn count doesn't look as improved. We need to ensure our retention strategies are profitable."

This situation highlights the importance of not looking at metrics in isolation. A seemingly positive result on one metric can be coupled with negative results on others if the underlying dynamics aren't fully understood.

Firstly, regarding the collaboration between Data Science, Product, and Engineering for a project like this churn analysis: could you describe the typical interaction model and decision-making process at Aha? How are priorities set when analytical insights suggest a product change or a new marketing intervention?

And secondly, considering the dynamic nature of the Telugu OTT market with strong local players, what does Aha see as its primary competitive differentiator, and how does data science contribute to strengthening that differentiation beyond just content recommendations?