diff --git a/docs/docs/experimentation-ab-testing.md b/docs/docs/experimentation-ab-testing.md index 95055fc8a9fa..646fa0fb6870 100644 --- a/docs/docs/experimentation-ab-testing.md +++ b/docs/docs/experimentation-ab-testing.md @@ -4,76 +4,111 @@ sidebar_label: Experimentation (A/B Testing) sidebar_position: 4 --- -A/B testing enables you to experiment with design and functionality variants of your application. The data generated will allow you to make modifications to your app, safe in the knowledge that it will have a net positive effect. +:::tip -You can use Flagsmith to perform A/B tests. Using a combination of [multivariate flags](/managing-flags/core-management) and a 3rd party analytics tool like [Amplitude](https://amplitude.com/) or [Mixpanel](https://mixpanel.com/), you can easily perform complex A/B tests that will help improve your product. +Flagsmith now runs experiments natively, with a managed warehouse and built-in statistics: see the +[Experimentation overview](/experimentation/). This guide covers the do-it-yourself approach with a third-party +analytics platform. -Running A/B tests require two main components: a bucketing engine and an analytics platform. The bucketing engine is used to put users into a particular A/B testing bucket. These buckets will control the specific user experience that is being tested. The analytics platform will receive a stream of event data derived from the behaviour of the user. Combining these two concepts allows you to deliver seamless A/B test. +::: -We have [integrations](/third-party-integrations/analytics/segment) with a number of analytics platforms. If we don't integrate with the platform you are using, you can still manually send the test data to the downstream platform manually. +A/B testing enables you to experiment with design and functionality variants of your application. The data generated +will allow you to make modifications to your app, safe in the knowledge that it will have a net positive effect. + +You can use Flagsmith to perform A/B tests. Using a combination of [multivariate flags](/managing-flags/core-management) +and a 3rd party analytics tool like [Amplitude](https://amplitude.com/) or [Mixpanel](https://mixpanel.com/), you can +easily perform complex A/B tests that will help improve your product. + +Running A/B tests require two main components: a bucketing engine and an analytics platform. The bucketing engine is +used to put users into a particular A/B testing bucket. These buckets will control the specific user experience that is +being tested. The analytics platform will receive a stream of event data derived from the behaviour of the user. +Combining these two concepts allows you to deliver seamless A/B test. + +We have [integrations](/third-party-integrations/analytics/segment) with a number of analytics platforms. If we don't +integrate with the platform you are using, you can still manually send the test data to the downstream platform +manually. By the end of this tutorial, you will be able to: -- Set up a multivariate flag in Flagsmith for A/B testing. -- Implement logic in your application to bucket users and display variants. -- Send A/B test data to an analytics platform. -- Understand how to use anonymous identities for A/B testing on unknown users. +- Set up a multivariate flag in Flagsmith for A/B testing. +- Implement logic in your application to bucket users and display variants. +- Send A/B test data to an analytics platform. +- Understand how to use anonymous identities for A/B testing on unknown users. ## Before you begin To follow this tutorial, you will need: -- A basic understanding of [multivariate flags](/managing-flags/core-management) in Flagsmith. Remember: multivariate bucketing only works when users are identified (use real identities or persistent anonymous GUIDs). -- Access to a third-party analytics platform (e.g., Amplitude, Mixpanel) where you can send custom events. You can explore Flagsmith [integrations](/third-party-integrations/analytics/segment) for this purpose. +- A basic understanding of [multivariate flags](/managing-flags/core-management) in Flagsmith. Remember: multivariate + bucketing only works when users are identified (use real identities or persistent anonymous GUIDs). +- Access to a third-party analytics platform (e.g., Amplitude, Mixpanel) where you can send custom events. You can + explore Flagsmith [integrations](/third-party-integrations/analytics/segment) for this purpose. - A development environment for your application where you can implement changes and integrate the Flagsmith SDK. ## Scenario - Testing a new Paypal button -For this example, lets assume we have an app that currently accepts credit card payments only. We have a hunch that we are losing out on potential customers that would like to pay with PayPal. We're going to test whether adding PayPal to the payment options increases our checkout rate. +For this example, lets assume we have an app that currently accepts credit card payments only. We have a hunch that we +are losing out on potential customers that would like to pay with PayPal. We're going to test whether adding PayPal to +the payment options increases our checkout rate. -We have a lot of users on our platform, so we don't want to run this test against our entire user-base. We want 90% of our users to be excluded from the test. Then for our test, 5% of our users will see the new Paypal button, and the remaining 5% will not see it. So we will have 3 buckets: +We have a lot of users on our platform, so we don't want to run this test against our entire user-base. We want 90% of +our users to be excluded from the test. Then for our test, 5% of our users will see the new Paypal button, and the +remaining 5% will not see it. So we will have 3 buckets: 1. Excluded (Control) Users 2. Paypal test button users 3. Test users that don't see the Paypal button -Because Flagsmith flags can contain both boolean states as well as multivariate flag values, we can make use of both. We will use the boolean flag state to control whether to run the test. Then, if the flag is `enabled`, check the multivariate value. In this example, we will only show the PayPal button if the value is set to `show`. +Because Flagsmith flags can contain both boolean states as well as multivariate flag values, we can make use of both. We +will use the boolean flag state to control whether to run the test. Then, if the flag is `enabled`, check the +multivariate value. In this example, we will only show the PayPal button if the value is set to `show`. ## Steps -1. Create a new [multivariate flag](/managing-flags/core-management) that will control which of the 3 buckets the user is put into. We'll call this flag `paypal_button_test`. We will provide 3 variate options: +1. Create a new [multivariate flag](/managing-flags/core-management) that will control which of the 3 buckets the user + is put into. We'll call this flag `paypal_button_test`. We will provide 3 variate options: 1. Control - 90% of users 2. Paypal button - 5% of users 3. Test users that don't see the Paypal button - 5% of users -2. In our app, we want to [identify](/flagsmith-concepts/identities) each user before they start the checkout process. All Flagsmith multivariate flags need us to identify the user, so we can bucket them in a reproducible manner. -3. When we get to the checkout page, check the `value` of the `paypal_button_test` flag for that user. If it evaluates to `show`, show the PayPal payment button. Otherwise, don't show the button. -4. Send an event message to the analytics platform, adding the name/value pair of `paypal_button_test` and the value of the flag; in this case it would be one of either `control`, `show` or `hide`. +2. In our app, we want to [identify](/flagsmith-concepts/identities) each user before they start the checkout process. + All Flagsmith multivariate flags need us to identify the user, so we can bucket them in a reproducible manner. +3. When we get to the checkout page, check the `value` of the `paypal_button_test` flag for that user. If it evaluates + to `show`, show the PayPal payment button. Otherwise, don't show the button. +4. Send an event message to the analytics platform, adding the name/value pair of `paypal_button_test` and the value of + the flag; in this case it would be one of either `control`, `show` or `hide`. 5. Deploy our app, enable the flag and watch the data come in to your analytics platform. Here is what creating the flag would look like. ![Image](/img/ab-test-paypal-example.png) -Once the test is set up, and the flag has been enabled, data will start streaming into the analytics platform. We can now evaluate the results of the tests based on the behavioral changes that the new button has created. +Once the test is set up, and the flag has been enabled, data will start streaming into the analytics platform. We can +now evaluate the results of the tests based on the behavioral changes that the new button has created. ## Handling Anonymous/Unknown Identities -To do A/B testing you need to use identities. Without an identity to key from, it's impossible for the platform to serve a consistent experience to your users. +To do A/B testing you need to use identities. Without an identity to key from, it's impossible for the platform to serve +a consistent experience to your users. -What if you want to run an A/B test in an area of your application where you don't know who your users are? For example on the homepage of your website? In this instance, you need to generate _anonymous identities_ values for your users. In this case we will generate a GUID for each user. +What if you want to run an A/B test in an area of your application where you don't know who your users are? For example +on the homepage of your website? In this instance, you need to generate _anonymous identities_ values for your users. In +this case we will generate a GUID for each user. -A GUID value is just a random string that has an extremely high likelihood of being unique. There's more info about generating GUID values [on Stack Overflow](https://stackoverflow.com/a/2117523). +A GUID value is just a random string that has an extremely high likelihood of being unique. There's more info about +generating GUID values [on Stack Overflow](https://stackoverflow.com/a/2117523). The general flow would be: 1. A new browser visits your website homepage for the first time. 2. You see that this is an anonymous user, so you generate a random GUID for that user and assign it to them. 3. You send that GUID along with an identify call to Flagsmith. This will then segment that visitor. -4. You add a cookie to the browser and store the GUID. That way, if the user returns to your page, they will still be in the same segment. +4. You add a cookie to the browser and store the GUID. That way, if the user returns to your page, they will still be in + the same segment. -These techniques will be slightly different depending on what platform you are developing for, but the general concept will remain the same. +These techniques will be slightly different depending on what platform you are developing for, but the general concept +will remain the same. ## Next steps diff --git a/docs/docs/experimentation/_category_.json b/docs/docs/experimentation/_category_.json new file mode 100644 index 000000000000..72e54fcb50c5 --- /dev/null +++ b/docs/docs/experimentation/_category_.json @@ -0,0 +1,5 @@ +{ + "label": "Experimentation (Beta)", + "position": 5, + "collapsed": true +} diff --git a/docs/docs/experimentation/analyse-an-experiment.md b/docs/docs/experimentation/analyse-an-experiment.md new file mode 100644 index 000000000000..acccc526a182 --- /dev/null +++ b/docs/docs/experimentation/analyse-an-experiment.md @@ -0,0 +1,57 @@ +--- +title: Analyse an Experiment +sidebar_label: Analyse an Experiment +sidebar_position: 6 +description: Read the results page (lift, credible intervals, win probability) and decide when to roll out a winner. +--- + +:::info Enterprise beta + +Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join. + +::: + +The experiment's detail page shows who entered the experiment, how each variation performed, and whether you can trust +the numbers. + +## Results + +Results are computed on demand: click **Refresh results** (at most once every 5 minutes). The **As of** timestamp shows +how fresh the figures are. + +The summary scorecards show **Users enrolled**, the current **Winning variation**, its **Chance to be best** and its +**Lift vs control**. See the [Statistics Glossary](/experimentation/statistics) for what each term means. + +When a variation reaches a 95% chance of beating control, a recommendation banner suggests rolling it out. + +![An experiment page with a winner recommendation banner](/img/experimentation/experiment-results-overview.png) + +The analysis table breaks each variation down against control: + +- **Exposures**: identities that entered on this variation. +- The metric value (rate, count, sum or mean, depending on the metric). +- **Delta**: how much better or worse the variation did than control, as a percentage. +- **Credible Interval (95%)**: the range the true delta sits in with 95% certainty. If it crosses zero, the result is + inconclusive. +- **Win Probability**: the chance the variation beats control. + +![Results scorecards and the analysis table with credible intervals](/img/experimentation/experiment-results-analysis.png) + +While a variation has fewer than 50 identities (or fewer than 5 conversions, for occurrence metrics), it shows +**Collecting data**. Keep the experiment running. + +If a **sample ratio mismatch** warning appears, traffic didn't split the way you configured. Don't act on the results. + +## Exposures + +The **Exposures** panel charts enrolment over time, per variation, with each variation's share of the total. + +Identities that were served more than one variation are quarantined and shown as excluded. A small number is normal; a +growing one means users are slipping between variations. + +## Ending the experiment + +When you have a conclusive result, click **End Experiment**. Results are frozen as final and the flag is unlocked. + +To roll out the winner, update the flag so every user gets the winning variation. Once it's permanent, clean up the flag +and remove the losing variations. diff --git a/docs/docs/experimentation/connect-a-warehouse.md b/docs/docs/experimentation/connect-a-warehouse.md new file mode 100644 index 000000000000..29b5a91fe013 --- /dev/null +++ b/docs/docs/experimentation/connect-a-warehouse.md @@ -0,0 +1,49 @@ +--- +title: Connect a Warehouse +sidebar_label: Connect a Warehouse +sidebar_position: 2 +description: Enable the managed Flagsmith warehouse to store and query experiment events. +--- + +:::info Enterprise beta + +Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join. + +::: + +Experiment data (exposures and conversion events) is stored and queried in a data warehouse. Each environment has one +warehouse connection, and it must be in place before you can create experiments. + +## Connect the Flagsmith warehouse + +The **Flagsmith** warehouse is managed and hosted by Flagsmith. There is nothing to provision or configure. + +1. Go to **Environment Settings > Warehouse**. +2. Select **Flagsmith** and click **Enable**, then confirm. + +![The Warehouse tab with the Flagsmith managed warehouse selected](/img/experimentation/warehouse-tab.png) + +The connection is created immediately. To verify that events can flow, click **Send your first event** and Flagsmith +sends a test event on your behalf. The connection shows **Pending Connection** while it waits for the first event, and +switches to **Connected** as soon as the environment's first event arrives; an event sent from your own application +counts too, not just the test event. Processing the first event can take up to a few hours, and the status refreshes +about once a minute while you keep the Warehouse tab open. + +Once connected, the warehouse card shows the total number of events received and the number of unique event names, which +is useful for checking that your instrumentation is arriving. + + + +### Connection statuses + +The current status is shown on the warehouse card in **Environment Settings > Warehouse**. + +- **Created**: the connection exists but no events have been received yet. +- **Pending Connection**: waiting for the first event to arrive. +- **Connected**: events have been received; you are ready to run experiments. +- **Errored**: something went wrong; contact support. + +## Coming soon + +Bring your own warehouse: connections for **Snowflake**, **BigQuery** and **Databricks**, so experiment results are +computed directly in your own data platform. diff --git a/docs/docs/experimentation/create-an-experiment.md b/docs/docs/experimentation/create-an-experiment.md new file mode 100644 index 000000000000..68f6bdd05272 --- /dev/null +++ b/docs/docs/experimentation/create-an-experiment.md @@ -0,0 +1,66 @@ +--- +title: Create an Experiment +sidebar_label: Create an Experiment +sidebar_position: 4 +description: Set up an experiment on a multivariate flag. Name it, configure the rollout, and pick a primary metric. +--- + +:::info Enterprise beta + +Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join. + +::: + +An experiment serves the variations of a [multivariate flag](/managing-flags/core-management) to a share of your users +and measures the impact on a metric. + +## 0. Pre-requisites + +- A [connected warehouse](/experimentation/connect-a-warehouse). +- A [**multivariate flag**](/managing-flags/core-management) with a variation for each treatment you want to test. The + flag's current value acts as the **control**. A flag can only be in one active experiment at a time. +- A [metric](/experimentation/create-metrics). You can also create one from inside the wizard. + +On the **Experiments** page, click **Create Experiment**. The wizard has four steps. + +## 1. Setup + +Give the experiment a name and a hypothesis, and select the flag to experiment on. A good hypothesis names the change, +the metric, the expected magnitude and the timeframe. For example: _"Redesigning the checkout button with a clearer CTA +will increase conversion rates by at least 15% within 30 days."_ + +![The Setup step: experiment name, hypothesis and feature flag](/img/experimentation/create-experiment-setup.png) + +## 2. Rollout configuration + +Two settings control who sees what: + +- **Rollout**: the percentage of identities that enter the experiment at all. Identities outside the rollout keep the + flag's normal behaviour and are not counted in the results. +- **Variation split**: how identities _inside_ the rollout are distributed across control and the variations. Weights + must sum to 100, and control takes whatever the variations don't. Use **Split evenly** for an equal distribution. + +For example, with a rollout of 20% and a 50/50 split, 10% of your identities see the control, 10% see the variation, and +the remaining 80% are untouched. + +Bucketing is deterministic on the identity key, so the same identity always lands in the same variation. + +![The Rollout configuration step: rollout percentage, variation split and distribution bar](/img/experimentation/create-experiment-rollout.png) + +## 3. Measurement + +Select the **primary metric** (the metric the experiment is judged on) and set the **expected direction**: increase, +decrease, should not increase, or should not decrease. If the metric doesn't exist yet, click **Create Metric**. + +## 4. Review & Launch + +Review the configuration and click **Create Experiment**. The experiment starts immediately. + +:::caution + +While an experiment is running, its flag configuration and variations are locked. The rollout percentage can still be +edited from the experiment page. + +::: + +A running experiment can be **ended** at any time. Its results are frozen as final, and this cannot be undone. diff --git a/docs/docs/experimentation/create-metrics.md b/docs/docs/experimentation/create-metrics.md new file mode 100644 index 000000000000..cfa0b5a69b6b --- /dev/null +++ b/docs/docs/experimentation/create-metrics.md @@ -0,0 +1,51 @@ +--- +title: Create Metrics +sidebar_label: Create Metrics +sidebar_position: 3 +description: Define the outcomes your experiments measure, computed from the events your application sends. +--- + +:::info Enterprise beta + +Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join. + +::: + +A metric defines an outcome you measure, computed from the events your application sends to the warehouse. Metrics are +defined per environment and can be reused across experiments. + +To create one, go to the **Metrics** page and click **Create Metric**, or click **Create Metric** in the experiment +wizard's Measurement step. + + + +## Fields + +**Name** and an optional description, e.g. "Signup Completion Rate". + +**What do you want to measure?** sets how the metric aggregates events, per identity: + +| Aggregation | What it measures | Example | +| -------------- | ---------------------------------------- | ------------------- | +| **Occurrence** | Whether an event is seen at least once | Signup completion | +| **Count** | Number of times an event occurred | Number of purchases | +| **Sum** | Total of a numeric value across events | Total revenue | +| **Mean** | Average of a numeric value across events | Average order value | + +:::important + +Sum and Mean aggregate the numeric `value` sent with each event, so your application must include it: + +```javascript +flagsmith.trackEvent('purchase', { value: 99.5 }); +``` + +::: + +**Direction** sets whether higher is better, lower is better, or the metric is informational only. This is the metric's +inherent polarity; it pre-fills the **expected direction** you choose when attaching the metric to an experiment. + +**Event name** is the event this metric aggregates, e.g. `checkout_completed`. It must exactly match the event name your +application sends (see [Run an Experiment](/experimentation/run-an-experiment)). + +A metric attached to an active experiment cannot be deleted. diff --git a/docs/docs/experimentation/example-paypal-button.md b/docs/docs/experimentation/example-paypal-button.md new file mode 100644 index 000000000000..2cadf32cd192 --- /dev/null +++ b/docs/docs/experimentation/example-paypal-button.md @@ -0,0 +1,84 @@ +--- +title: 'Example: Testing a PayPal Button' +sidebar_label: 'Example: PayPal Button' +sidebar_position: 8 +description: An end-to-end walkthrough of a real experiment, from flag to rollout. +--- + +:::info Enterprise beta + +Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join. + +::: + +Our app currently accepts credit card payments only. We suspect offering PayPal will increase completed checkouts, and +we want to prove it on a small slice of traffic before rolling it out. This page walks through the whole experiment. + +## 1. Create the flag + +Create a [multivariate flag](/managing-flags/core-management) called `paypal_button`. The flag's current behaviour (no +PayPal) is the **control**; add one variation named `show-paypal`. + +Your code will branch on the variation _name_, not the flag value, so keep names short and stable. + +## 2. Create the metric + +With a [warehouse connected](/experimentation/connect-a-warehouse), [create a metric](/experimentation/create-metrics): + +- **Name**: Checkout Completion +- **Measure**: Occurrence +- **Direction**: Higher is better +- **Event name**: `checkout_completed` + +## 3. Create the experiment + +[Create an experiment](/experimentation/create-an-experiment) on `paypal_button`: + +- **Hypothesis**: _"Offering PayPal at checkout will increase checkout completion by at least 10% within 30 days."_ +- **Rollout**: 10%, split evenly. 5% of identities see PayPal, 5% act as control, and the remaining 90% stay out of the + experiment entirely. +- **Primary metric**: Checkout Completion, expected to **increase**. + +Launch it from the Review step. + +## 4. Instrument the checkout + +On the checkout page, resolve the flag (recording the exposure) and branch on the variation name. When a checkout +completes, send the conversion event. The SDK must be initialised with `enableEvents: true` and the user identified; see +[Run an Experiment](/experimentation/run-an-experiment). + +```tsx +import flagsmith from '@flagsmith/flagsmith'; +import { useExperiment } from '@flagsmith/flagsmith/react'; + +const CheckoutPage: React.FC = () => { + // Evaluates the flag and records the exposure + const flag = useExperiment('paypal_button'); + const showPaypal = flag?.enabled && flag?.variant === 'show-paypal'; + + const onOrderConfirmed = () => { + // Record the conversion + flagsmith.trackEvent('checkout_completed'); + }; + + return ( + <> + + {showPaypal ? : null} + + ); +}; +``` + +Outside React, `flagsmith.getExperimentFlag('paypal_button')` does the same job as the hook. The same flow works in +Python; see [Run an Experiment](/experimentation/run-an-experiment) for both SDKs. + +If checkout is available to visitors who aren't logged in, identify them with a persistent anonymous GUID so bucketing +and events stay consistent; see [Identities](/flagsmith-concepts/identities). + +## 5. Read the results and roll out + +On the experiment page, check the **Exposures** panel: enrolment should track about 10% of your identities, split 50/50. +Then watch the results as conversions arrive; see [Analyse an Experiment](/experimentation/analyse-an-experiment). + +When the result is conclusive, end the experiment, roll the PayPal button out to everyone, and clean up the flag. diff --git a/docs/docs/experimentation/index.md b/docs/docs/experimentation/index.md new file mode 100644 index 000000000000..02e4553bddc6 --- /dev/null +++ b/docs/docs/experimentation/index.md @@ -0,0 +1,41 @@ +--- +title: Experimentation +description: + 'Run A/B tests natively in Flagsmith: serve variations, collect events, and analyse results with built-in Bayesian + statistics.' +--- + +:::info Enterprise beta + +Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join. + +::: + +Flagsmith Experimentation lets you run A/B tests end to end on the platform: serve variations of a feature with a +[multivariate flag](/managing-flags/core-management), collect events from your application into a managed data +warehouse, and read the results with a built-in Bayesian statistics engine. + +Experiments are scoped to an **environment**: the warehouse connection, metrics and experiments all live at the +environment level. + +## Set up and run an experiment + +1. **[Connect a warehouse](/experimentation/connect-a-warehouse)**: enable the managed Flagsmith warehouse for your + environment. Done once, in a few clicks. +2. **[Create metrics](/experimentation/create-metrics)**: define the outcomes you measure, computed from the events your + application sends. +3. **[Create an experiment](/experimentation/create-an-experiment)**: pick a multivariate flag, decide how much traffic + enters the test and how it is split, and choose a primary metric. +4. **[Run the experiment](/experimentation/run-an-experiment)**: instrument your application to record exposures and + conversion events with the Flagsmith SDKs. +5. **[Analyse the results](/experimentation/analyse-an-experiment)**: read lift, credible intervals and win probability + on the experiment's results page, then roll out the winner. + +Prefer to see it end to end? Follow the [PayPal button example](/experimentation/example-paypal-button). + +Unfamiliar with a term? The [Statistics Glossary](/experimentation/statistics) explains every concept in plain language. + +## A/B testing without the Experimentation feature + +You can also run A/B tests with multivariate flags and your own analytics platform. See +[Experimentation (A/B Testing)](/experimentation-ab-testing). diff --git a/docs/docs/experimentation/run-an-experiment.mdx b/docs/docs/experimentation/run-an-experiment.mdx new file mode 100644 index 000000000000..6447db1fa748 --- /dev/null +++ b/docs/docs/experimentation/run-an-experiment.mdx @@ -0,0 +1,101 @@ +--- +title: Run an Experiment +sidebar_label: Run an Experiment +sidebar_position: 5 +description: Instrument your application to record exposures and conversion events while the experiment runs. +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +:::info Enterprise beta + +Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join. + +::: + +Once an experiment is running, Flagsmith serves the variations automatically through the flag. Your application has two +jobs: + +1. **Record exposures**: tell Flagsmith when an identity actually experienced a variation. +2. **Record conversion events**: send the events your metrics aggregate. + +## Instrument your application + +Event collection is currently available in the **JavaScript** and **Python** SDKs, with more to follow. Enable it with +the `enableEvents` / `enable_events` option. + +Users must be [identified](/flagsmith-concepts/identities). Exposures and conversion events are joined per identity, so +use the same identifier for flags and events. + + + + +```javascript +import flagsmith from '@flagsmith/flagsmith'; + +await flagsmith.init({ + environmentID: 'YOUR_CLIENT_SIDE_ENVIRONMENT_KEY', + enableEvents: true, +}); +await flagsmith.identify('user_42'); + +// Evaluate the flag and record an exposure in one call +const flag = flagsmith.getExperimentFlag('checkout_button'); +// ...render based on flag.value + +// Record a conversion event; the name must match your metric's event name +flagsmith.trackEvent('purchase', { value: 99.5 }); +``` + +In React, the `useExperiment` hook resolves the flag and records the exposure: + +```javascript +const flag = useExperiment('checkout_button'); +``` + + + + +```python +from flagsmith import Flagsmith + +flagsmith = Flagsmith( + environment_key="YOUR_ENVIRONMENT_KEY", + enable_events=True, +) + +# Evaluate the flag and record an exposure in one call +flag = flagsmith.get_experiment_flag("checkout_button", identifier="user_42") + +# Record a conversion event; the name must match your metric's event name +flagsmith.track_event("purchase", identifier="user_42", value=99.5) +``` + + + + +### Exposures + +An exposure is the moment an identity is served a variation; it is what enrols them in the results. `getExperimentFlag` +/ `get_experiment_flag` evaluates the flag and records the exposure in one call. + +Record the exposure where the user actually experiences the change; if you evaluate in one place and render in another, +call `trackExposureEvent` / `track_exposure_event` at the point of display instead. + +Exposures are deduplicated per identity: each identity is counted once, in the variation it saw first, so recording an +exposure more than once is safe. + +### Conversion events + +`trackEvent` / `track_event` sends a named event, optionally with a numeric `value`, traits and metadata. + +Any event whose name matches a metric's event name feeds that metric. Conversions are attributed post-exposure: an +identity's events only count after its first exposure. + +## While it runs + +- The flag is locked while the experiment is running. +- Monitor enrolment on the experiment page's **Exposures** panel to confirm data is arriving. +- You can edit the rollout from the experiment page, but changing it mid-experiment can affect the statistical + validity of your results. diff --git a/docs/docs/experiment-statistics.md b/docs/docs/experimentation/statistics.md similarity index 96% rename from docs/docs/experiment-statistics.md rename to docs/docs/experimentation/statistics.md index c9093941584e..f31cc08b1e8e 100644 --- a/docs/docs/experiment-statistics.md +++ b/docs/docs/experimentation/statistics.md @@ -1,13 +1,9 @@ --- -title: Experiment Statistics -sidebar_label: Experiment Statistics -sidebar_position: 5 +title: Statistics Glossary +sidebar_label: Statistics Glossary +sidebar_position: 7 --- -Flagsmith's statistics engine answers three questions about an experiment: **Am I winning?** (is the variant better than -control), **by how much?** (the lift), and **can I trust it?** (is the difference real, and was traffic split fairly). -This page explains the terms you'll see, in plain language — no statistics background needed. - :::info Coming soon — Enterprise beta Experiment statistics aren't generally available yet. They are launching as a beta on **Enterprise** plans — to join, @@ -16,6 +12,10 @@ release; it previews what the feature will do. ::: +Flagsmith's statistics engine answers three questions about an experiment: **Am I winning?** (is the variant better than +control), **by how much?** (the lift), and **can I trust it?** (is the difference real, and was traffic split fairly). +This page explains the terms you'll see, in plain language — no statistics background needed. + ## Terms you'll see **Experiment** — A controlled comparison: show different versions of a feature to different people, measure which @@ -143,5 +143,5 @@ Nothing here is generally available yet — the table shows what the upcoming En | Risk / decision banner / trend chart | Not currently planned | | Frequentist engine | Deferred | -For experiment setup — multivariate flags, bucketing, identities — see -[Experimentation (A/B Testing)](/experimentation-ab-testing) and [managing identities](/flagsmith-concepts/identities). +For experiment setup, see [Create an Experiment](/experimentation/create-an-experiment) and +[managing identities](/flagsmith-concepts/identities). diff --git a/docs/static/img/experimentation/create-experiment-rollout.png b/docs/static/img/experimentation/create-experiment-rollout.png new file mode 100644 index 000000000000..526ab33b5b0b Binary files /dev/null and b/docs/static/img/experimentation/create-experiment-rollout.png differ diff --git a/docs/static/img/experimentation/create-experiment-setup.png b/docs/static/img/experimentation/create-experiment-setup.png new file mode 100644 index 000000000000..ea9b9df7638a Binary files /dev/null and b/docs/static/img/experimentation/create-experiment-setup.png differ diff --git a/docs/static/img/experimentation/experiment-results-analysis.png b/docs/static/img/experimentation/experiment-results-analysis.png new file mode 100644 index 000000000000..fd7f9c334bc9 Binary files /dev/null and b/docs/static/img/experimentation/experiment-results-analysis.png differ diff --git a/docs/static/img/experimentation/experiment-results-overview.png b/docs/static/img/experimentation/experiment-results-overview.png new file mode 100644 index 000000000000..aa578b9bd5fd Binary files /dev/null and b/docs/static/img/experimentation/experiment-results-overview.png differ diff --git a/docs/static/img/experimentation/warehouse-tab.png b/docs/static/img/experimentation/warehouse-tab.png new file mode 100644 index 000000000000..e72456374384 Binary files /dev/null and b/docs/static/img/experimentation/warehouse-tab.png differ diff --git a/docs/vercel.json b/docs/vercel.json index fb9721bdc8d7..0f64bc33a629 100644 --- a/docs/vercel.json +++ b/docs/vercel.json @@ -144,6 +144,14 @@ "source": "/ab-testing", "destination": "/experimentation-ab-testing" }, + { + "source": "/experimentation/ab-testing", + "destination": "/experimentation-ab-testing" + }, + { + "source": "/experiment-statistics", + "destination": "/experimentation/statistics" + }, { "source": "/permissions", "destination": "/administration-and-security/access-control/rbac" @@ -168,10 +176,6 @@ "source": "/flag-analytics", "destination": "/managing-flags/flag-analytics" }, - { - "source": "/experimentation/ab-testing", - "destination": "/experimentation-ab-testing" - }, { "source": "/experimentation/flag-analytics", "destination": "/managing-flags/flag-analytics" @@ -730,4 +734,4 @@ } ], "trailingSlash": false -} \ No newline at end of file +}