A/B testing infrastructure

A/B testing infrastructure is a system for testing hypotheses on real traffic: you make two (or more) variants of an element — a headline, a button, a price, a block, a whole page — and the system shows them to different visitors and counts which one brings more leads, sales or clicks. This way decisions are made based on data, not on a hunch. We choose the tool for the task and budget: there are paid cloud ones (VWO, Optimizely, AB Tasty, Convert) with a ready editor and analytics, and free open-source ones (GrowthBook, PostHog) installed on your server; Google Optimize is shut down, so we build the setup on current solutions together with GA4. It is easier with an example than the difference from feature flags (see the “Feature flags infrastructure” service): feature flags are a switch “show the new cart to 10% of users” (who to show it to); an A/B test is “compare the old and new cart and find out which one sells more” (what is better). Honestly about traffic: for the result to be reliable, you need enough visitors and conversions — as a rough guide, at least a few thousand unique visitors a month and a noticeable number of target actions; on less traffic the test will run for a long time or give a random result, and we will say so in advance. About money: the base price is for our setup, and if you choose a paid service (VWO, Optimizely), its subscription is separate and recurring — we warn in advance. Picture this: you argue which button is better — “Buy” or “Add to cart”; instead of guessing, we show half one and half the other, and after a while we see which actually sells. The base price starts from 30,000 ₽ for setting up the infrastructure; it depends on the tool and the number of tests.

• You change the site “by eye” — sometimes it turns out worse than before, and without a comparison you cannot tell.
• The team argues which variant is better — there is no data to decide.
• You are afraid to roll out a full redesign — what if conversion drops, and rolling back is expensive.
• Ads are expensive, but what actually converts on the site is unknown.

• Choosing the tool for the task and budget (VWO / Optimizely / AB Tasty or open-source GrowthBook / PostHog)
• Connecting to the site and integrating with analytics (GA4, goals, events)
• Setting up traffic split — correct division of visitors between variants
• Help with the first test: the hypothesis, the variants, the success metric
• Correct statistical significance calculation — so as not to take chance for a result
• No flicker: the variant is shown right away (flicker is when the old variant briefly flashes before the new one)
• Speed and SEO in mind — tests should not harm rankings
• Personal data processing under the law, consent for cookies and analytics

• Decisions based on data — you roll out only what has statistically proven an effect
• If there is no significant gain — the change is not rolled out (which is also valuable: not making it worse)
• Less risk from changes — you test on part of the traffic before a full rollout
• The team argues with facts, not opinions — there are numbers

• You leave a request and describe what you want to test
• We choose the tool, connect it, set up the split and analytics
• We launch the first test, calculate significance and help with conclusions

• How much traffic is needed for an A/B test to work?

  The less traffic and conversions, the longer the test and the less reliable the result. A rough guide is at least a few thousand unique visitors a month and a noticeable number of target actions; on less, a reliable result can take months or never come. Then it is more honest to improve the site by other methods (analytics, usability) than to guess with A/B. Before the start we assess your traffic and say whether it makes sense.

• Which tool to choose and how much does it cost afterwards?

  Paid cloud ones (VWO, Optimizely, AB Tasty, Convert) are convenient but charge a subscription, sometimes a noticeable one — a recurring cost on top of our setup. There are free open-source ones (GrowthBook, PostHog) — we install them on your server, without a subscription, but they need support. Google Optimize is shut down, so we use current solutions together with GA4. We choose one for the budget and tell you about the recurring costs in advance.

• How are A/B tests different from feature flags?

  Feature flags (see the “Feature flags infrastructure” service) are feature switches: turn a feature on or off, roll it out to part of the users (managing who to show it to). An A/B test is about comparing variants and measuring what converts better. Technically they are close and often work together (a feature flag rolls out a variant, A/B measures the effect), but the goal is different.