Leverage Graphs to Analyze Your Campaign Report

#Overview

Graphs in Wingify are pictorial representations of your campaign data, capable of delivering the entire picture concisely. This enables you to grab information faster. Indeed, a glance is all it takes to understand the chances of your versions becoming the winner or the smart decision. However, reading values from graphs can be challenging at times. But don’t you worry yet. Just gear up - we’re gonna decipher them now.

The concepts explained in this article mainly concern Web Experimentation campaigns. However, they’re also applicable to Web Personalization campaigns.

Note: This article covers SmartStats, Wingify’s Bayesian-powered statistics engine, which applies only to:

• Accounts on the non-Data Platform framework, and
• Campaigns created before an account was migrated from non-Data Platform to Data Platform.

For new campaigns (created after migration) or all campaigns in accounts created directly on the Data Platform framework, refer to Campaign Reports in Data Platform, Wingify’s upgraded statistical engine that offers deeper insights and improved control over statistics.

#Types of Graphs

Based on your campaign report data, Wingify provides the following graphical formats:

#Date Range

The Date Range graph gives you the real-time status of the conversions, revenue, conversion rate, and the number of visitors that have arrived at your campaign URLs. The following are the Key Performance Indicators (KPIs) represented in the graph:

#Probability Density Curve

The Probability Density (PD) curve is also referred to as the Bell curve. It depicts the prediction for your campaign, giving you the probable scope of the conversion rate. It only features two parameters for this - the Percentage of Conversion Rate on the X-axis and the Probability Density on the Y-axis.

Let’s look at the following example of a PD curve of the control version in a campaign.

The curve progresses upwards from around 39% and reaches the peak at 44.12%. It finally touches the ground at 49%. This means that the conversion rate range for your campaign is likely to fall between 39% and 49%, with 44.12% being the highest probability.

This range is generated by accumulating data points, which will keep the curve shrinking towards the centre with the incoming visitors. This means the nearness to certainty keeps increasing with more data.

In the below graph, the orange curve represents the control version, and the blue curve represents the variation. Here, the orange curve (control) is narrower than the blue curve (variation), indicating that the orange curve (control) has had more visitors as compared to the blue one (variation).

Note: The PD curve keeps changing with the accumulation of data as the conversion rate is susceptible to change with it.

For a campaign that has several variations, you will be able to see the PD graph with a curve for the control and each of the variations. This brings the following scenarios based on the overlap of the curves.

#No Overlap

When the curves appear so far apart from each other that there is no overlap between them, then the curve at the right would be the one with the higher conversion rate. This means that the one on the right side has a greater scope of becoming a winner.

The following graph shows the control (orange curve) and a variation (blue curve), which do not overlap each other, with both the control and the variation sharing almost the same level of probability density.

However, the control has a conversion rate of just 44.12%, while that of the variation is at 54.71%. In this case, the variation is a clear winner.

#With Overlap

Quite often, you might encounter scenarios where these curves overlap. Overlap of the curves means that the probability of conversion rate is being shared between the variations. The area of overlap plays a significant role in deciding the winner.

Case 1:

If the majority of the curves fall within the overlapping area, the winning chances are equally spread between the two. This leaves no room for a conclusion. The following graph features two variations - blue and brown.

Here, the median conversion rates are much closer to each other, and so are the ranges of probabilities. In such cases, declaring a winner is almost impossible.

Case 2:

If the overlapping area is smaller, the chances of declaring a winner certainly falls to the party with the higher probability density and conversion rate. However, it doesn’t always become the case where both these parameters belong to the same candidate. Sometimes, the curve on the left has more certainty and a lesser range than the one on the right. This would more probably lead to a situation where declaring a winner could be really complex, and you would rather settle down to declare a variant as the smart decision.

Case 3:

Chances are more likely that you will run your campaign with more variations than just one. Overlaps, here, can provide a good indication of which variants are superior to others. You can choose to select only those variations whose trends you want to read for a detailed comparison.

Note: The concept of overlap with probability to beat the baseline is only applicable when comparing a variation to the baseline.

In the following graph, the orange curve indicates control, the pink curve indicates variation 1, and the blue curve indicates variation 2.

As you can see, the blue curve has no overlap with the orange or pink curves, you can declare the blue curve (variation 2) as the winner, as it outperforms the others.

#Box Plot

Note: Box plot graphs are not applicable for campaigns that use the Enhanced Statistical Engine with Enhanced Reports. They are applicable only for Bayesian campaigns that use the Classic Statistical Engine in both account types, Data Platform and non-Data Platform. For more information on any of these features, contact Wingify Support.

A Box Plot graph is a combination of the Date Range graph and the PD graph. It gives you both the real-time values (the count of visitors and the conversions) and the predictive analysis in one place.

Here, the control and the variations are represented as bars on the X-axis, spaced at regular intervals with no specific measure.

The Y-axis, by default, features the Conversion Rate(%). You can also view the graph for Unique Conversions or Visitors by selecting the appropriate option from the KPI dropdown.

Note: If the campaign involves a revenue goal, you will haveRevenue Per Visitorin place of Conversion Rate(%), and in the case of metrics, you will have Expected Value per visitor.

For Unique Conversions and Visitors, the heights of the bars correspond to the relevant numbers.

While viewing Conversion Rate(%), Expected Value per visitor(for metrics), orRevenue Per Visitor(for goals), the box plot becomes analogous to the aerial view of a PD graph. The bars feature the best and worst case values along with the most likely value almost in the middle of each of them. The best and worst case values form the range of certainty of the version becoming a winner or getting selected as the smart decision. Like the PD curve, whose range of probability shrinks (in other words, certainty sharpens), the range of each of the bars shrinks with the incoming visitors. It also features the overlap zone between the baseline and individual versions.

Note: The best case, worst case and the overlap area for a version are always displayed with respect to the baseline. To view them, you need to hover over the corresponding version. However, hovering over the baseline version will render no detail.

#With Overlap

If you see significant vertical overlap in the box plot for the two versions, you can infer that they do not yet clearly outperform each other. If this is the case, you should allow more data to be collected, as with more data, there will be an appreciable change in the height of the bars to witness a clear gap between the two versions.

#Without Overlap

If there is no vertical overlap, you can be confident that the higher variant is the winner and can safely deploy it.

If you look at the graph below, the orange bar represents the control version, and the blue bar represents variation. As you can see that the worst case of the blue variant (variation) is better than the best case of the orange variant (control) given plot, you can be confident that the blue variant will outperform the orange variant even in the worst case, and you can safely deploy it.

#Need more help?

For further assistance or more information, contact Wingify Support.