Boost organic results with A/B testing on Google Play and the App Store

Here’s a step-by-step guide on how to configure and run a Google store listing experiment:

Step 1: setting up the experiment

• Select store listing experiments: Log in to your Google Play Console, navigate to the “Store listing experiments” tab, and click “Create experiment.” Choose between a global test (which runs in one language for all users) or a localized test (which allows testing in up to five languages). Localized tests are ideal for targeting specific regions.
• Name your experiment: Assign a descriptive name that reflects the attribute and localization being tested, such as "Icon_de" for app icons in Germany.
• Choose the target metric: Select between "retained first-time installers," which measures users who keep the app for at least one day, and "first-time installers," which tracks all new users. For more meaningful results, retained first-time installers are recommended.

Step 2: configuring experiment parameters

• Define variants: Test up to three variants alongside your current store listing, focusing on one specific element (e.g., app icon, screenshots, or description) to ensure clear results.
• Set the experiment audience: Determine the percentage of store visitors who will see the test variants. For example, if you allocate 50% of your audience to the experiment with two variants, each will be seen by 25%.
• Tailor minimum detectable effect (MDE): Set the smallest change in conversion rates you want to detect. A lower MDE requires a larger sample size, while a higher MDE allows for quicker tests.
• Select the confidence level: Choose a confidence level (90%, 95%, 98%, or 99%) for your results. A higher confidence level reduces the risk of false positives but requires more data.

Step 3: running the experiment

• Upload and label variants: Configure your test variants (e.g., alternate app icons or descriptions), focusing on one attribute at a time.
• Start the experiment: Click “Start experiment” to distribute traffic according to your audience split and begin data collection.
• Monitor the experiment: Allow the experiment to run for at least seven days to account for variations in daily and weekend traffic. Google Play will provide metrics like first-time installers, retained first-time installers, and conversion rates.

Step 4: analyzing results and applying changes

• Review metrics: Analyze conversion rate improvements and retention rates. Google Play provides insights on whether to apply a variant or continue collecting data.
• Consider external factors: Take into account any seasonal trends, traffic anomalies, or paid campaigns that may influence your results. If discrepancies arise, consider follow-up tests like A/B/B or B/A tests for validation.
• Apply the winning variant: If a variant shows significant improvement, implement it in your store listing and monitor its long-term performance.

Limitations of Google Play’s store listing experiments

Google Play’s store listing experiments is a powerful and free tool for optimizing app listings, but it comes with limitations. Some key challenges that app developers and marketers should consider include the lack of traffic source segmentation, limited post-install engagement data (beyond retention), and an absence of monetization metrics. It can be challenging, for example, to determine how specific user segments respond to changes, limiting the ability to tailor optimizations for different acquisition channels. 

To engage in these types of tests, use of Google’s custom store listings feature will be most impactful. This is, however, for unique URLs from paid campaigns, as opposed to split testing on organic Play Store search results. 

Read more about Google’s custom store listings and premium growth tools.

Like store listing experiments, PPO offers powerful customization options, including localization, enabling developers to tailor experiments for specific regions and maximize relevance. With audience allocation, you can control the percentage of users exposed to each test version, starting with small-scale trials before expanding to a broader audience. Treatments gain wide visibility, appearing in prominent App Store sections like the Today and Apps tabs, as well as in search results.

To enhance the user experience, PPO ensures that users consistently see the same test version across the App Store throughout the experiment. This consistency builds user familiarity and provides reliable performance metrics.

How Apple’s product page optimization works

Here's a step-by-step guide on how to set up and manage a PPO test effectively.

Step 1: configuring your test

• Log in and select your app: Log into App Store Connect, navigate to the “My Apps” section, and select the app you want to optimize.
• Create a new test: In the product page optimization section, click “Create Test” or the (+) icon. Assign a descriptive name (up to 64 characters), such as “Summer 2024 icon and screenshots test”.
• Define the number of treatments: Choose up to three treatments (variations of your product page) to test against your default page.
• Set traffic allocation: Allocate a percentage of your audience to the test; for instance, if you allocate 30% and have two treatments, each will receive 15%.
• Select localizations: Choose the eligible regions and languages for your test based on your current app version.
• Estimate test duration: Utilize the “Estimate Your Test Duration” feature to predict how long the test might take. Tests can run for up to 90 days and may be stopped manually.

Step 2: launching your test

1. Submit test treatments: After configuring your test, submit your creative assets (app icons, screenshots, preview videos) for App Review; no new app version is needed unless testing alternate app icons.
2. Start the test: After approval, click “Start Test.” The test will go live, and users will be randomly assigned to view either the default product page or one of the test treatments.

Step 3: monitoring and analyzing test performance

• Track metrics in app analytics: Monitor performance metrics, such as impressions and conversion rates, in the App Analytics section to compare each treatment against the default product page.
• Monitor progress: Track progress through App Analytics, which will indicate if more data is needed or if a treatment is outperforming the baseline.

Step 4: applying the winning treatment

• Select and apply a treatment: Based on test results, apply a winning treatment to your default product page. Ensure you’ve achieved at least 90% confidence before making changes. If testing alternate app icons, remember to include the winning icon in your next app update.

Limitations of Apple’s PPO

While Apple’s PPO provides a robust native A/B testing solution for app marketers on iOS, it also has a few limitations that are important to keep in mind. These include that you can only run one test at a time, that metadata testing isn’t possible, that insights into individual region results are not available, and that high sample sizes are required. This means, for example, that you can only actively run one test per app at any given time and that tests are restricted to visual elements (as opposed to things like app subtitles)—testing app icons is also complex. It’s also worth noting that PPO’s confidence level is capped at 90%, which is lower than the 95% standard typically used throughout the industry. 

For more granular, detailed testing, you can look at Apple’s custom product pages feature. Like Google’s custom store listings, this is for paid campaign/unique URL testing, as opposed to App Store organic split testing.