Push A/B Testing
Last updated
Was this helpful?
Last updated
Was this helpful?
A/B testing for mobile and web push notifications allows you to compare different components and content to determine which variation drives better engagement. By testing different elements, you can optimize for higher open rates, conversions, and user interaction.
🚀 You can create A/B tests for Mobile and Web Push Notifications to refine your communication strategy and maximize impact.
Messages > Campaigns > Start A/B Testing
In the Setup step of the Create New Campaign process:
Enter a campaign name in the Campaign Name field.
Choose the campaign type: Push Notification (for mobile) or Web Push.
Choose the Push Notification type (text, media, banner, etc.) to proceed with customization.
View the Estimated Reach on specific platforms, showing the potential number of users and devices your message can reach.
In this step, you define the first variation (Group A) of your push notification. This serves as one of the test versions, which will later be compared against Group B to measure effectiveness.
Key Elements to Configure in Group A
Category: Assign a category to better organize and analyze campaign performance.
Notification Message: Craft the main content of your notification. Ensure it is clear, engaging, and action-driven.
Title: Define the push notification's headline. This should be attention-grabbing and relevant to your audience.
Subtext: Optionally, include additional information to provide more context.
Media (Images, Video, or Sound): Attach media assets like images, banners, or videos to enhance engagement.
Platform-Specific Customization: Define different elements for iOS and Android notifications, such as thumbnails, push icons, or media URLs.
For additional information on the elements, please refer to Define Message Content: What.
This step involves designing the second test variation (Group B) to compare against Group A. The key here is to modify specific elements while keeping the rest of the structure the same.
Choosing Elements to Test in Group B
Headline Variation: Try different wording, emojis, or formatting in the title.
Message Copy: Adjust the wording, tone, or length to see which resonates more.
Media Experimentation: Swap the image, use different visuals, or test between static and animated content.
Call-to-Action (CTA): Experiment with different CTAs, such as “Read Now” vs. “Stay Informed.”
Personalization: Compare a generic message against one using dynamic fields like user names or preferences.
💡 Example A/B Test:
Group A: “🔴 Breaking News: Stay Updated! A major event just happened. Tap to get the full story now!” (Image A > A Text Image: Breaking News)
Group B: “🔴 Breaking News: Stay Updated! A major event just happened. Tap to get the full story now!” (Image B > An Actual Newsroom Image)
In this step, you specify the audience segments that will receive your push notification test. You can tailor your campaign based on various targeting options:
Send All: Broadcast your message to all available users.
Select Users: Choose specific users based on filters like demographics, behavioral data, or custom attributes.
Advanced: Employ more detailed criteria or combinations of filters to create highly focused segments.
Distribution List: Target a pre-defined or uploaded list of users.
Group A/B Percentage: Choose what percentage of your total target group receives each test variant. Each group can range from 1% to 30% of your total audience, so the combined A/B test can involve between 2% and 60% of your total audience.
Number of Devices: Displays the total number of devices that will receive each group’s notification based on your chosen percentage.
Number of iOS and Android Devices: Shows a breakdown of how many iOS and Android devices will be included in each test group.
Post-Test Audience Targeting Once you determine the winning variant, you can send that version to the remaining portion of your target group. For instance, if your audience is 29,000 and you choose to test Group A with 10% and Group B with 10%, your testing group will be 20% of 29,000 users, which equals 5,800. In the end, you can target the remaining 23,200 users to send your winning result.
Here, you determine when and how your notifications are delivered.
Start Sending Messages:
Now: Dispatch the campaign immediately.
On a Specific Time: Schedule the campaign for a particular date and time.
Send on Best Time for Each User Between: Use personalized delivery times within a specified time window to increase engagement based on user behavior.
Message Expiry When Not Reached to Client:
Never: The notification will not expire from the user's device.
Until a Specific Time: If the user’s device does not receive the notification before the chosen expiry time, the notification is removed from the queue and will not appear later in the notification center.
Delivery Speed:
Send Fast: Deliver all notifications simultaneously, ideal for time-sensitive announcements.
Send in Packages: Spread out delivery in batches to manage user traffic and avoid potential server overload.
For additional information on the elements, please refer to Define Campaign Schedule: When.
On this final page:
Verify Your Settings: Double-check all configurations, from targeting and scheduling to notification content.
Preview Notifications: See how each variation appears on different devices (iOS and Android).
Confirm and Launch: Once satisfied with the setup, click the launch button to begin sending your A/B test campaigns.
This screen displays the results of your A/B test, allowing you to compare the performance of both message variants. Once you identify the better-performing campaign, you can send the winning variant to the remaining members of your target audience by clicking "Send message X to the rest of users!"
Target Audience: The total number of users included in each test group.
Sent: The number of notifications that were sent out.
Success: The number of notifications successfully delivered to users' devices.
Clicked: The number and percentage of users who clicked on the notification.
Conversion: The number of users who completed the desired action after clicking.
Revenue: Any revenue generated from the campaign, if applicable.
A conversion event must be set during campaign creation in Step: What-A and What-B to track conversions in your campaigns.
For additional information on the elements, please refer to .
