Challenge

Cardlytics Incremental Lift Study

Cardlytics is a pay-for-performance program in which customers receive offers ads within their banking portal or email, and DoorDash pays only for users acquired. Historically this channel has been a strong acquisition lever and has constantly generated a CPA 50% below the benchmark. Because of banking privacy issues, we are unable to use our internal MTA to understand the true CPA or incrementally of the platform.

Before investing incremental budgets in this channel we were tasked with understanding the incrementality of this program.

  • We planned on focusing on the 3 areas

    • Are our ads driving actual value, or just claiming credit for an action that would naturally occur without our ad

    • How many incremental new customers/orders are being generated?

    • What is the incremental Cost Per Acquisition and Cost Per Order?

Solution


  • Because we cannot pass back sales data to our internal MTA, we decided to work with our partners at Cardlytics to help design an A/B utilizing their data. By creating an A/B test we would be able to see the impact of our ads by comparing the sales data of users who were exposed to our ad versus those not exposed to our ads.

  • Test Setup

    • Test mechanism: A/B

    • Country: US only

    • Campaign Duration: usually one month

    • Control/Treatment Split: 5% control (not exposed to an ad) / 95% treatment (exposed to an ad)

    • Experience: both Doordash and Caviar

    • Target Audience: 

      • Users without Doordash or Caviar orders in the past 36 months 

      • Users with orders from competitors in restaurants or grocery deliveries verticals in the past 12 months

  • Due to the fact that Cardlytics receive its sales data from banking partners, we were confident that the order data was 100% deterministic and that every order can be attributed to a click on our ad.


Results

  • We discovered that Cardlytics was able to generate 16,414 new users at an 8.2% lift and $18.39 iCPA at a tatistical significance

  • Due to these findings, we were able to increase the budget to the platform the following quarter.