Sri Krishna Pearls: Biryani A/B Test Interpretation

Problem Statement

You work as a data analyst for "Sri Krishna Pearls," a popular biryani chain that started in Hyderabad and now has 75+ outlets across Telangana and Andhra Pradesh. The marketing team ran an A/B test on their Swiggy/Zomato listing, changing the featured image from their signature "Hyderabadi Dum Biryani" to "Special Vijayawada Biryani." The test shows a statistically significant increase in order rate (p = 0.001). However, the actual increase in orders is only from 2.00% to 2.01% of page views.

Statistical vs. Practical Significance

MODERATE

Explain how a result can be statistically significant but not practically important in this biryani marketing context.

Solution

Sri Krishna Pearls, the popular biryani chain from Hyderabad, tested a new picture ("Special Vijayawada Biryani") on Swiggy/Zomato against their usual "Hyderabadi Dum Biryani" picture. The new picture got slightly more orders – an increase from 2.00% of people viewing the page to 2.01%. The math (p=0.001) says this tiny increase is almost certainly real and not just luck.

Statistically Significant (Math says it's real): The very low p-value (0.001 is much less than 0.05) means we are very confident that the "Special Vijayawada Biryani" picture does genuinely lead to a slightly higher order rate. It's not a random fluke.

Practically Important? (Does it matter much for the business?): Maybe not! The increase is from 2.00% to 2.01%. That's an increase of only 0.01 percentage points. Imagine 10,000 people view the page:

Old picture: 2.00% * 10,000 = 200 orders.
New picture: 2.01% * 10,000 = 201 orders.

That's just one extra order for every 10,000 views.

So, for Sri Krishna Pearls (with 75+ outlets in Telangana and Andhra Pradesh): While the math proves the new picture is technically better, the improvement is so tiny (0.01%) that it might not be worth the effort to change the picture on all their Swiggy/Zomato listings. The business impact is very small. This is the difference: "statistically significant" means the effect is likely real, but "practically important" means the effect is big enough to make a real difference to the business. Here, it's real but tiny.

A result can be statistically significant but not practically important (or practically significant) in this biryani marketing context for Sri Krishna Pearls. Here's how:

Statistical Significance (p = 0.001):
- A p-value of 0.001 means there is a very low probability (0.1%) of observing an increase in order rate from 2.00% to 2.01% (or larger) if, in reality, the "Special Vijayawada Biryani" image had no different effect on orders compared to the "Hyderabadi Dum Biryani" image.
- Because this p-value is well below the common significance level (e.g., α = 0.05), we conclude that the observed increase is unlikely to be due to random chance alone. We are statistically confident that the new image (Special Vijayawada Biryani) genuinely performs differently (in this case, slightly better) than the old image.
Practical Importance (Effect Size: 2.00% to 2.01%):
- Practical importance refers to the magnitude or size of the difference and whether this difference is meaningful or valuable from a business perspective for Sri Krishna Pearls, which has 75+ outlets across Telangana and Andhra Pradesh.
- In this case, the order rate increased from 2.00% to 2.01%. This is an absolute increase of 0.01 percentage points, or a relative increase of (0.01 / 2.00) * 100% = 0.5%.
- For every 10,000 page views, this change would result in only one additional order (10,000 * 0.0200 = 200 orders vs. 10,000 * 0.0201 = 201 orders).
- While this effect is "real" in a statistical sense, the actual impact on the number of biryani orders and subsequently on revenue might be very small, potentially negligible for a large chain like Sri Krishna Pearls.
Why the Discrepancy?
- Statistical significance is heavily influenced by sample size. With a very large number of page views (common for popular food delivery app listings, especially during festival seasons like Sankranti and Bathukamma), even tiny, trivial differences can become statistically significant. The test has enough power to detect very small effects.
- The p-value tells you about the reliability of detecting an effect, not necessarily the size or importance of that effect.
Context for Sri Krishna Pearls:
- The marketing team needs to consider if the effort, cost, or potential brand implications of changing the featured image across all Swiggy/Zomato listings for their 75+ outlets (from Hyderabad to smaller towns) is justified by such a minuscule increase in order rate.
- If changing the image is a complex process, involves new design costs, or risks alienating a segment of customers loyal to the "Hyderabadi Dum Biryani" image, then a 0.01 percentage point increase might not be practically worthwhile, despite its statistical significance.

Therefore, while the data strongly suggests the "Special Vijayawada Biryani" image does indeed lead to a slightly higher order rate than the "Hyderabadi Dum Biryani" image, the actual improvement is so small that it may not have a meaningful impact on overall business performance, making it statistically significant but not necessarily practically important.

Factors Beyond p-value for Decision

MODERATE

What other factors or metrics would you consider before recommending Sri Krishna Pearls implement this menu display change based solely on the p-value?

Solution

The math (p-value) says the new "Special Vijayawada Biryani" picture is slightly better for Sri Krishna Pearls. But before we tell all 75+ outlets from Hyderabad to other Andhra Pradesh/Telangana cities to change their Swiggy/Zomato image, we need to look at more than just that one number.

Other important things to check:

How big is the improvement, really? (Effect Size): We know it's 0.01%. Is this tiny increase worth it? If it means just 10 extra orders a day across all 75 stores, maybe not. If it's 1000 extra orders, then yes!
What's the range of possible improvement? (Confidence Interval): The 0.01% is an estimate. The true improvement might be a bit lower or a bit higher. If the range includes zero or a negative number, then we're not even sure it's an improvement.
Cost vs. Benefit: How much time and money will it take to update the image for all outlets? Does the tiny sales increase cover this cost?
Did it hurt anything else? Maybe more people ordered the Vijayawada Biryani, but fewer ordered their famous Hyderabadi Dum Biryani, so overall sales didn't change much. Or maybe the new image confused some customers.
Brand Image: Sri Krishna Pearls is known for Hyderabadi Biryani. Does changing the main image to Vijayawada Biryani affect their brand identity?
Long-Term Effects: Will this small improvement last, or will people get used to the new image and the effect wears off?
Did it work for everyone? Did the image perform well in Hyderabad (where Hyderabadi biryani is king) as well as in Vijayawada?

Just because a change is "statistically significant" doesn't automatically mean it's a good business decision. We need the full picture.

Relying solely on a p-value (even a highly significant one like p=0.001) to recommend Sri Krishna Pearls implement the menu display change (featuring "Special Vijayawada Biryani" over "Hyderabadi Dum Biryani") would be insufficient. Several other factors and metrics should be considered:

1. Effect Size (Magnitude of the Difference):
- As established, the increase is from 2.00% to 2.01%, an absolute difference of 0.01 percentage points or a relative lift of 0.5%. This is a very small effect.
- The key question is whether this small lift translates into a meaningful business impact (e.g., substantial increase in revenue or profit) across their 75+ outlets in Telangana and Andhra Pradesh.
2. Confidence Interval of the Difference:
- Calculate and examine the 95% confidence interval for the difference in order rates between the two images.
- Does the entire interval lie above zero? How wide is it? If the interval is, for example, [0.005%, 0.015%], it means we are 95% confident the true improvement is within this (still very small) range. If the lower bound is very close to zero, the practical benefit is even more questionable.
3. Cost of Implementation vs. Potential Gain:
- What is the cost and effort involved in changing the featured image across all Swiggy/Zomato listings for 75+ outlets? This includes design time, coordination, and any platform fees.
- Compare this cost to the projected increase in revenue from the 0.01% lift in order rate. A Return on Investment (ROI) calculation would be beneficial. If the cost outweighs the marginal gain, the change isn't practical.
4. Impact on Other Key Metrics:
- Average Order Value (AOV): Did the new image attract customers who order lower or higher value biryanis or overall orders? A higher order rate with a lower AOV might not be beneficial.
- Sales of "Hyderabadi Dum Biryani": Did featuring "Special Vijayawada Biryani" cannibalize sales of their signature "Hyderabadi Dum Biryani" or other items? The net effect on total sales is important.
- Customer Complaints/Feedback: Any qualitative feedback suggesting confusion or preference related to the image change.
5. Brand Perception and Strategy:
- Sri Krishna Pearls is known for "Hyderabadi Dum Biryani." Does changing the primary featured image to "Special Vijayawada Biryani" align with their overall brand strategy and positioning in diverse markets from Hyderabad to smaller towns?
- Could it confuse customers or dilute their brand identity as a premier Hyderabadi biryani destination?
6. Long-Term Effects & Novelty:
- Is the observed lift due to a novelty effect of seeing a new image, which might wear off over time? Running the test for a longer period or monitoring post-change could clarify this.
7. Segmentation Analysis:
- Did the "Special Vijayawada Biryani" image perform differently across user segments? For example, did it perform much better in Vijayawada but poorly in Hyderabad, with the overall 0.01% lift being an average?
- Understanding if the effect varies by city (Hyderabad, Vijayawada, Tirupati etc.), customer type, or time of day is crucial.
8. Business Goals and Thresholds:
- What was the minimum improvement the marketing team was hoping for? Does this 0.01% lift meet any pre-defined success criteria for A/B tests at Sri Krishna Pearls?

A recommendation to implement the change should only come after considering these factors. A statistically significant result simply means the observed difference is unlikely due to chance; it doesn't automatically mean it's a strategically sound or impactful business decision, especially for a well-established chain.

Sample Size and Small Effect Sizes

ADVANCED

How does sample size play a role in achieving statistical significance for very small effect sizes, especially considering the high daily traffic their listings receive during festival seasons like Sankranti and Bathukamma?

Solution

Sri Krishna Pearls' Swiggy/Zomato listings get tons of views, especially during festivals like Sankranti and Bathukamma when everyone in Hyderabad and other Telugu cities is ordering food. How does this high traffic (large sample size) make it possible to find a "statistically significant" tiny improvement like 0.01%?

Think of it like trying to hear a very faint whisper in a room:

Small Sample (Few Views, Quiet Room): If only a few people visit the biryani page, a tiny 0.01% difference in orders is like a super faint whisper. It's hard to tell if you really heard it or if it was just background noise (random chance). You wouldn't be confident.
Large Sample (Lots of Views, Very Quiet Room): If millions of people view the page (like during Sankranti for their 75+ outlets), our "hearing" (statistical test) becomes super sensitive. Even a tiny, tiny whisper (the 0.01% difference) can be clearly detected above the (now relatively smaller) background noise. With lots of data, the random variations average out, making even small, consistent differences stand out as "real."

So, a very large sample size (like high traffic during festivals) gives our A/B test very high statistical power. This means it's very good at detecting even extremely small true differences between the "Hyderabadi Dum Biryani" image and the "Special Vijayawada Biryani" image.

This is why we can get a p-value of 0.001 (highly significant, meaning the difference is likely real) for a change from 2.00% to 2.01% (a tiny effect). The sheer volume of data during peak times for Sri Krishna Pearls allows us to be very confident that this tiny difference isn't just a fluke. The challenge then becomes deciding if this tiny-but-real difference is worth acting on.

Sample size plays a crucial role in achieving statistical significance, especially for very small effect sizes like the 0.01 percentage point increase (from 2.00% to 2.01%) in order rate observed for Sri Krishna Pearls' biryani listings. This is particularly relevant given the high daily traffic their listings receive during festival seasons like Sankranti and Bathukamma across their 75+ outlets in Telangana and Andhra Pradesh.

1. Increased Statistical Power:
- Larger sample sizes lead to higher statistical power. Power is the probability of correctly detecting a true effect if one actually exists (i.e., correctly rejecting a false null hypothesis).
- With high traffic (many page views for Swiggy/Zomato listings, especially during festivals), the A/B test for Sri Krishna Pearls has a very large sample size for both the "Hyderabadi Dum Biryani" image (control) and the "Special Vijayawada Biryani" image (variant).
2. Reduced Standard Error:
- The standard error of the difference between two proportions (or means) decreases as the sample size increases (it's inversely proportional to the square root of n).
- A smaller standard error means that our estimate of the difference in order rates (0.01%) is more precise. The "noise" or random variability in the estimate is reduced.
3. Easier Detection of Small True Effects:
- When the standard error is small (due to large sample size), even a very small true difference between the two groups (a small effect size) can be distinguished from random chance.
- The test statistic (like a z-score or t-score) is typically calculated as (Observed Difference - Hypothesized Difference (0)) / Standard Error. With a very small standard error in the denominator, even a small Observed Difference (like 0.01%) can result in a large test statistic.
- A large test statistic, in turn, leads to a very small p-value (like the p=0.001 observed), indicating that the observed small difference is unlikely to have occurred by chance if there were no true difference.
4. The "Law of Large Numbers" in Play:
- As sample sizes get very large, the sample proportions (order rates for each image) will converge very closely to their true underlying population proportions. Therefore, even a tiny true difference in these underlying proportions will become apparent and statistically distinguishable with enough data.
Implication for Sri Krishna Pearls during Sankranti/Bathukamma:
- The high traffic volumes during these festival periods provide an excellent opportunity to run A/B tests with high statistical power. This means the marketing team can be very confident in detecting even minute differences in performance between different marketing assets (like images).
- This is precisely why a tiny effect size (0.01% change in order rate) can still yield a highly significant p-value (0.001). The large volume of data from users across Hyderabad, Vijayawada, and other cities allows the test to confidently say that the small difference is "real" and not just noise.

In essence, large sample sizes act like a powerful magnifying glass, making it possible to reliably detect very small true differences. The challenge for Sri Krishna Pearls is then not whether the difference is "real" (statistical significance addresses this), but whether that tiny, real difference is large enough to be "meaningful" or "important" from a business or practical standpoint (practical significance), as discussed in Q1.

Your Biryani Business Call!

What are your thoughts on these scenarios? Try answering the questions yourself and share your insights or alternative approaches in the comments section below!

Back to Inferential Stats

Problem Statement

Statistical vs. Practical Significance

Related Concepts

Hint

Solution

Factors Beyond p-value for Decision

Related Concepts

Hint

Solution

Sample Size and Small Effect Sizes

Related Concepts

Hint

Solution

Your Biryani Business Call!