diff --git a/README.md b/README.md index d60e376..f65b172 100644 --- a/README.md +++ b/README.md @@ -3,18 +3,38 @@ The FeatBit documentation site, built with Next.js and Fumadocs. Migrated documentation lives under `content/docs` and is served from `/docs`. -Run development server: +Install dependencies with pnpm 10.33.4 using the committed lockfile, then start +the development server. `npx` runs the required pnpm version even if your global +installation is older: ```bash +npx --yes pnpm@10.33.4 install --frozen-lockfile npm run dev -# or -pnpm dev -# or -yarn dev ``` Open http://localhost:3000 with your browser to see the result. +If you previously installed the Nextra version of this site, your `node_modules` +may still contain Next.js 13 and React 18. Errors mentioning those packages +alongside Fumadocs indicate that the old installation needs to be replaced. +From the repository root in PowerShell: + +```powershell +Remove-Item -LiteralPath node_modules -Recurse -Force +if (Test-Path -LiteralPath .next) { + Remove-Item -LiteralPath .next -Recurse -Force +} +npx --yes pnpm@10.33.4 install --frozen-lockfile +npm run dev +``` + +Keep `pnpm-lock.yaml` so the reinstall uses the project's recorded dependency +versions. + +If Next.js reports that `pages` and `app` must be under the same folder, check +for an empty root-level `pages` directory left by the migration and remove it. +The current app lives in `src/app`. + ## Explore In the project, you can see: diff --git a/content/docs/experimentation/assets/experimentations/bayesian-analysis-results.jpg b/content/docs/experimentation/assets/experimentations/bayesian-analysis-results.jpg new file mode 100644 index 0000000..6daa730 Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/bayesian-analysis-results.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/coding-agent-recommendation.jpg b/content/docs/experimentation/assets/experimentations/coding-agent-recommendation.jpg new file mode 100644 index 0000000..baa72f2 Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/coding-agent-recommendation.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/coding-agent-setup.jpg b/content/docs/experimentation/assets/experimentations/coding-agent-setup.jpg new file mode 100644 index 0000000..3caa759 Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/coding-agent-setup.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/configure-primary-and-guardrail.jpg b/content/docs/experimentation/assets/experimentations/configure-primary-and-guardrail.jpg new file mode 100644 index 0000000..45192e2 Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/configure-primary-and-guardrail.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/copy-coding-agent-prompt.jpg b/content/docs/experimentation/assets/experimentations/copy-coding-agent-prompt.jpg new file mode 100644 index 0000000..a8365ac Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/copy-coding-agent-prompt.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/create-bayesian-run.jpg b/content/docs/experimentation/assets/experimentations/create-bayesian-run.jpg new file mode 100644 index 0000000..68a80e9 Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/create-bayesian-run.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/create-checkout-experiment.jpg b/content/docs/experimentation/assets/experimentations/create-checkout-experiment.jpg new file mode 100644 index 0000000..939a050 Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/create-checkout-experiment.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/create-simplified-checkout-flag.jpg b/content/docs/experimentation/assets/experimentations/create-simplified-checkout-flag.jpg new file mode 100644 index 0000000..3a4ad5c Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/create-simplified-checkout-flag.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/decision-and-learning.jpg b/content/docs/experimentation/assets/experimentations/decision-and-learning.jpg new file mode 100644 index 0000000..14d00de Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/decision-and-learning.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/flag-targeting-50-50.jpg b/content/docs/experimentation/assets/experimentations/flag-targeting-50-50.jpg new file mode 100644 index 0000000..f305aa0 Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/flag-targeting-50-50.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/intent-and-hypothesis.jpg b/content/docs/experimentation/assets/experimentations/intent-and-hypothesis.jpg new file mode 100644 index 0000000..715ff49 Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/intent-and-hypothesis.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/observation-window.jpg b/content/docs/experimentation/assets/experimentations/observation-window.jpg new file mode 100644 index 0000000..feef974 Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/observation-window.jpg differ diff --git a/content/docs/experimentation/assets/experimentations/saved-exposure-and-metrics.jpg b/content/docs/experimentation/assets/experimentations/saved-exposure-and-metrics.jpg new file mode 100644 index 0000000..dff770b Binary files /dev/null and b/content/docs/experimentation/assets/experimentations/saved-exposure-and-metrics.jpg differ diff --git a/content/docs/experimentation/assets/layers/create-checkout-layer.jpg b/content/docs/experimentation/assets/layers/create-checkout-layer.jpg new file mode 100644 index 0000000..024e4bb Binary files /dev/null and b/content/docs/experimentation/assets/layers/create-checkout-layer.jpg differ diff --git a/content/docs/experimentation/assets/layers/edit-checkout-layer.jpg b/content/docs/experimentation/assets/layers/edit-checkout-layer.jpg new file mode 100644 index 0000000..6983107 Binary files /dev/null and b/content/docs/experimentation/assets/layers/edit-checkout-layer.jpg differ diff --git a/content/docs/experimentation/assets/layers/layer-after-window-ends.jpg b/content/docs/experimentation/assets/layers/layer-after-window-ends.jpg new file mode 100644 index 0000000..9c50250 Binary files /dev/null and b/content/docs/experimentation/assets/layers/layer-after-window-ends.jpg differ diff --git a/content/docs/experimentation/assets/layers/layer-run-allocation.jpg b/content/docs/experimentation/assets/layers/layer-run-allocation.jpg new file mode 100644 index 0000000..1a03c9d Binary files /dev/null and b/content/docs/experimentation/assets/layers/layer-run-allocation.jpg differ diff --git a/content/docs/experimentation/assets/layers/run-layer-assignment.jpg b/content/docs/experimentation/assets/layers/run-layer-assignment.jpg new file mode 100644 index 0000000..a7dff36 Binary files /dev/null and b/content/docs/experimentation/assets/layers/run-layer-assignment.jpg differ diff --git a/content/docs/experimentation/assets/metrics/create-checkout-metric.jpg b/content/docs/experimentation/assets/metrics/create-checkout-metric.jpg new file mode 100644 index 0000000..a1dd3fa Binary files /dev/null and b/content/docs/experimentation/assets/metrics/create-checkout-metric.jpg differ diff --git a/content/docs/experimentation/assets/metrics/edit-checkout-metric.jpg b/content/docs/experimentation/assets/metrics/edit-checkout-metric.jpg new file mode 100644 index 0000000..095cd6b Binary files /dev/null and b/content/docs/experimentation/assets/metrics/edit-checkout-metric.jpg differ diff --git a/content/docs/experimentation/assets/metrics/filter-by-experiment.jpg b/content/docs/experimentation/assets/metrics/filter-by-experiment.jpg new file mode 100644 index 0000000..815e6f3 Binary files /dev/null and b/content/docs/experimentation/assets/metrics/filter-by-experiment.jpg differ diff --git a/content/docs/experimentation/assets/metrics/filter-by-metric-key.jpg b/content/docs/experimentation/assets/metrics/filter-by-metric-key.jpg new file mode 100644 index 0000000..651bbf5 Binary files /dev/null and b/content/docs/experimentation/assets/metrics/filter-by-metric-key.jpg differ diff --git a/content/docs/experimentation/assets/metrics/numeric-aggregation-options.jpg b/content/docs/experimentation/assets/metrics/numeric-aggregation-options.jpg new file mode 100644 index 0000000..82b203e Binary files /dev/null and b/content/docs/experimentation/assets/metrics/numeric-aggregation-options.jpg differ diff --git a/content/docs/experimentation/assets/metrics/saved-checkout-metrics.jpg b/content/docs/experimentation/assets/metrics/saved-checkout-metrics.jpg new file mode 100644 index 0000000..73ad954 Binary files /dev/null and b/content/docs/experimentation/assets/metrics/saved-checkout-metrics.jpg differ diff --git a/content/docs/experimentation/experimentations.mdx b/content/docs/experimentation/experimentations.mdx new file mode 100644 index 0000000..b95c75f --- /dev/null +++ b/content/docs/experimentation/experimentations.mdx @@ -0,0 +1,268 @@ +--- +title: Experiments +description: Run a Bayesian checkout experiment and use a coding agent with FeatBit MCP to interpret results, record a decision, and capture learning. +--- + +import { ImageZoom } from 'fumadocs-ui/components/image-zoom'; +import createCheckoutExperiment from './assets/experimentations/create-checkout-experiment.jpg'; +import intentAndHypothesis from './assets/experimentations/intent-and-hypothesis.jpg'; +import createSimplifiedCheckoutFlag from './assets/experimentations/create-simplified-checkout-flag.jpg'; +import flagTargeting5050 from './assets/experimentations/flag-targeting-50-50.jpg'; +import configurePrimaryAndGuardrail from './assets/experimentations/configure-primary-and-guardrail.jpg'; +import savedExposureAndMetrics from './assets/experimentations/saved-exposure-and-metrics.jpg'; +import createBayesianRun from './assets/experimentations/create-bayesian-run.jpg'; +import runLayerAssignment from './assets/layers/run-layer-assignment.jpg'; +import observationWindow from './assets/experimentations/observation-window.jpg'; +import bayesianAnalysisResults from './assets/experimentations/bayesian-analysis-results.jpg'; +import decisionAndLearning from './assets/experimentations/decision-and-learning.jpg'; +import codingAgentSetup from './assets/experimentations/coding-agent-setup.jpg'; +import copyCodingAgentPrompt from './assets/experimentations/copy-coding-agent-prompt.jpg'; +import codingAgentRecommendation from './assets/experimentations/coding-agent-recommendation.jpg'; + +## Overview + +This walkthrough answers a concrete question: **Does a shorter checkout improve completion without increasing checkout errors?** You will save a hypothesis, bind a feature flag and metrics, configure a Bayesian Run, collect events, and inspect the evidence. Then you will connect a coding agent through FeatBit MCP to interpret the results, record a recommendation, and capture learning. + +The screenshots use **demonstration data** from the same **First Project / Dev** environment throughout. They illustrate the workflow and do not establish that a real checkout change improves customer outcomes. + +## Before you begin + +Prepare the following resources in the same project environment: + +| Resource | Name or setting | +| --- | --- | +| Primary metric | **Checkout completed** / `checkout-completed`: Binary conversion, Once per user | +| Guardrail | **Checkout errors** / `checkout-errors`: Numeric value, Count all | +| Layer | **Checkout experience** / `checkout-experience`: `user.keyId` | +| Application instrumentation | An official SDK connected to the matching environment and Evaluation Server | + +Follow [Metrics](./metrics.mdx) and [Layers](./layers.mdx) to create these resources. You will create the flag below. The application needs to evaluate that flag when a user encounters checkout and send outcome events afterward, using the same stable user ID. + +## Create the experiment and write the hypothesis + +1. Confirm the project and environment in the header, then open **Experiments** under **Release Decision**. +2. Click **New experiment**. Enter **Name**: `Simplify checkout`. +3. Enter **Description**: `Learn whether fewer checkout steps improve completion without increasing checkout errors.` +4. Click **Create experiment**. The experiment opens in **Intent & Hypothesis**. + + + +5. Click **Edit details** and fill in the following fields, then click **Save**: + +| Field | Example | +| --- | --- | +| Goal | Increase checkout completion without increasing checkout errors. | +| Intent | Reduce abandonment caused by unnecessary checkout steps. | +| Hypothesis | For users who start checkout, fewer form steps will increase the share who complete checkout because there is less effort before payment. | +| Change | Use simplified-checkout: Control (false) keeps the current flow; Treatment (true) uses fewer steps. | +| Constraints | Local demonstration data only. Keep a 50/50 flag split; use checkout-errors as a guardrail. Turning the flag off returns Control. | + +6. Refresh the page and confirm the saved details. + + + +For a real experiment, also agree on a useful improvement, a representative observation period, and acceptable guardrail behavior before collecting data. A testable hypothesis connects the proposed change to a user behavior and explains why that behavior might change. + +## Prepare the feature flag + +1. Open **Feature Flags** in the same environment and click **New flag**. +2. Enter **Name**: `Simplified checkout` and **Key**: `simplified-checkout`. +3. Enter **Description**: `Compare a shorter checkout with the current experience.` +4. Keep **Variation type** as **BOOLEAN**. Rename the `true` variation to **Treatment** and the `false` variation to **Control**. +5. Keep **Serve when off** as **Control**. Leave **Turn on after creation** off while preparing the rollout, then click **Create flag**. + + + +6. On the new flag's **Targeting** tab, under **Default rule → When flag is ON**, select **Rollout percentage**. +7. Set **Treatment** to `50%` and **Control** to `50%`, with **Dispatch by** set to `keyId`. Click **Apply**. +8. Click **Review & save**, review the change, and click **Save changes**. +9. Turn **Flag status** on and confirm. Refresh to verify **ON**, the saved 50/50 rollout, **No users added** under individual targeting, and **No rules yet** under targeting rules. + + + +The new flag uses default experiment-exposure collection. Keep SDK event collection enabled and verify that exposure samples appear when you analyze the Run. The current **Targeting** UI does not expose a separate experiment-collection switch. + +The flag controls application behavior. Selecting Control and Treatment in a Run later only chooses how to compare those variants; it does not configure flag rollout. + +## Bind the flag and metrics + +1. Return to **Simplify checkout** and click **Continue to Exposure**. +2. Click **Select flag**, choose **Simplified checkout**, and click **Select flag** to save. +3. Inspect the saved variation table. Confirm **Control = false** and **Treatment = true**, and use the displayed variation IDs to distinguish the actual variants. +4. Click **Add metrics**. +5. Choose **Checkout completed (checkout-completed)** as the **Primary metric** and keep **Direction → Higher is better**. +6. Click **Add guardrail** and choose **Checkout errors (checkout-errors)**. Set **Alert if → Increases**. +7. Click **Save metrics**, refresh, and confirm the saved configuration. + + + + + +The saved guardrail label is **Increase is bad**; analysis displays the equivalent preferred direction as **Lower is better**. Directions belong to this experiment's metric bindings, while the underlying metric definitions remain reusable. + +Your variation IDs will differ from the screenshot. Select by the confirmed name and value, not by list position. + +## Create a Bayesian Run + +1. Click **Continue to Measuring**, then **New run**. +2. Keep **Analysis method → Bayesian A/B/n**. +3. Under **Control**, explicitly select the variation named **Control** (`false`). Do not accept the initially selected first variation without checking it. +4. Under **Treatments**, check the variation named **Treatment** (`true`). +5. Set **Start date and time** to the beginning of your collection period. +6. Keep **End → No fixed end** while collecting the demonstration batch. +7. Set **Minimum sample per variant** to `500` for this example, then click **Create run**. + + + +The new `run-1` appears under **Measuring**. The minimum is checked for each variant, not against the combined total. It is a demonstration threshold, not a universal sample-size recommendation. Passing it does not guarantee a useful or conclusive effect. + +Before analysis, **Full analysis** may show **No evaluable data** and **Window: Not recorded**. Inspect **Observation window** to verify the saved Run configuration; the analysis metadata is populated when analysis runs. + +## Configure Layer eligibility and sampling + +1. In `run-1`, click **Edit assignment** under **Experiment traffic assignment**. +2. Verify the Control and Treatment selections again. +3. Select **Layer → Checkout experience (checkout-experience)**. +4. Keep **Assignment unit** as `user.keyId`. Set **Bucket start** to `0` and **Bucket end** to `50`. +5. Keep both variants' **Analysis sampling** at `100%`, and leave **Audience filters** empty. +6. Click **Save changes**. Refresh and reopen the editor to check that all settings persisted. + + + +This selects the first half of the Layer for analysis. The flag still serves its own 50/50 split across users. Analysis sampling retains all eligible users in each variant. See [Layers](./layers.mdx#distinguish-eligibility-serving-and-sampling) for the distinction and the saved reservation display. + +## Collect events and analyze + +1. Use the [.NET Server SDK example](./metrics.mdx#report-metric-events), or another [official SDK](../sdk/overview.md), in the same environment. +2. For each user who encounters checkout, evaluate `simplified-checkout` with a stable user ID and serve the corresponding flow. +3. After an actual completion, send `checkout-completed` for that user. For each actual checkout error, send `checkout-errors` with value `1`. Users with no outcome event still contribute to the exposed-user denominator. +4. Allow queued SDK events to flush and the server to process them. +5. In **Measuring → run-1**, click **Analyze latest data**. Wait for analysis to finish and inspect **Data as of**, the analysis **Window**, sample counts, and metrics. + +The screenshots below came from **4,000 demonstration users** evaluated by the official .NET SDK, without prefiltering by Layer. Simulated outcomes were generated after each actual flag evaluation. The Layer retained **1,982** users: **994 Control** and **988 Treatment**. Your counts and results will depend on your users, instrumentation, and observation window. + +### Finish the demonstration window + +For a reproducible snapshot of a finished collection period: + +1. Click **Observation window** on `run-1`. +2. Select **End → On a specific date** and set an end after your last intended event. +3. Click **Save window**, then **Analyze latest data** again. +4. Refresh and confirm that the saved analysis **Window** matches the Run's configured window. + + + +Date and time inputs use your browser's local time. Choose a window that includes your own events. + +After changing a window, old results can remain visible with a notice that they use the saved analysis window. Analyze again before interpreting them. Ending a Run's observation window does not turn off its flag. + +## Read the evidence + + + +The actual results for this demonstration were: + +| Result | Control | Treatment | +| --- | ---: | ---: | +| Samples (n) | 994 | 988 | +| Checkout completions | 318 | 408 | +| Completion rate | 32.0% | 41.3% | +| Checkout errors per user | 0.0503 | 0.0283 | + +Read the page in this order: + +1. **Window and Data as of:** Confirm that the analysis covers the intended events and was recomputed after configuration changes. +2. **SRM:** The page reports the sample ratio mismatch check. Here it shows **p=0.8928 · ok**. A mismatch deserves investigation before trusting the effect; an `ok` result is not proof that all instrumentation is correct. +3. **Sample check:** Both variants pass the configured minimum of `500`. Read the per-variant counts rather than the combined total. +4. **Primary metric:** Treatment's observed completion rate is 41.3%, compared with 32.0% for Control. The displayed **relative lift** is **+29.1%**, with a **95% credible interval of 13.9%–44.2%** and **P(win) greater than 99.9%**. +5. **Guardrail:** Errors fall from 0.0503 to 0.0283 per user. The page shows **−43.7% relative lift**, a **95% credible interval of −69.3% to −18.1%**, **P(harm) below 0.1%**, and **guardrail clear**. + +Relative lift compares the change with the Control baseline. Here the absolute conversion-rate difference is about **9.3 percentage points**; that is different from a **29.1% relative lift**. The credible interval and probabilities summarize uncertainty under the analysis model and its displayed prior. The posterior chart is labeled as a **Normal approximation**. + +The primary metric reports a strong signal in this dataset, and the guardrail supports an improvement in the preferred direction. Because these outcomes are simulated, the result validates the demonstration workflow, not the business hypothesis for real customers. + +If you see **No evaluable data**, check environment, flag key, exposure collection, user IDs, metric keys, window, Layer range, and sampling before collecting more data. If samples are below the minimum or evidence remains uncertain, record that limitation instead of declaring a winner. + +## Set up Coding Agent Mode + +FeatBit calculates the experiment statistics. A coding agent, such as Codex, uses the **FeatBit experimentation skill** and **FeatBit MCP** to review those statistics in context and save recommendations and learning back to the experiment. + +1. Click **Coding Agent Mode** in the experiment's upper-right corner. +2. Under **Install release-decision skill**, copy and run the installation command once at the user or project level: + +```sh +npx skills add featbit/featbit-skills --skill featbit-experimentation +``` + +3. Under **Connect FeatBit MCP**, click **Create MCP token** if no token exists or the saved token has expired. If **Token ready** is already shown, reuse that token. +4. Click **Copy setup prompt** and paste it into your coding agent. Ask it to apply the connection configuration and verify access by listing the FeatBit tools and reading this experiment. Complete any restart or reload the agent requires before continuing. + + + +The setup prompt contains your token; keep it private and out of source control. Setup is shared across experiments your account can access in the organization. When the token expires, create a new one and repeat the connection step. **Token ready** confirms that a token exists; the agent's successful experiment read confirms the connection works. + +## Record the decision and learning + +Use the connected coding agent for the following steps. **Analyze latest data** produces statistical results; the agent interprets the evidence and writes the Run's recommendation and learning through MCP. + +### Ask for a recommendation + +1. Return to **Measuring** and select `run-1`. +2. Click **Copy coding agent prompt** next to the analysis. +3. Paste the copied prompt unchanged into the coding agent where you configured FeatBit MCP. It identifies the experiment and selected Run, and asks the agent to record a recommendation, summary, reason, and the analysis **Window** and **Data as of**. + + + +The agent should read the saved experiment, review the hypothesis, sample health, SRM, primary metric, guardrails, and observation window, then explain its recommendation. For this walkthrough, make clear that the outcomes are **demonstration data**. You can ask follow-up questions before accepting the reasoning. + +4. After the agent confirms that it saved the recommendation, refresh FeatBit and return to **Measuring → run-1**. +5. Check **Coding Agent recommendation**, its decision label and **Summary**, then expand **Reason** to review the evidence and limitations. + + + +In this walkthrough, a follow-up clarified that the recommendation concerns the recorded demonstration data. Codex then saved **CONTINUE**: Treatment improves checkout completion, the error guardrail is clear, both samples pass the minimum, and SRM passes. This supports the example's next step; real customer benefit still needs validation with representative traffic. + +| Decision | Recommended next step | +| --- | --- | +| CONTINUE | Move toward Treatment when the evidence and rollout conditions support it. | +| PAUSE | Hold the current rollout while investigating a concern. | +| ROLLBACK | Return to the safer experience when the evidence shows harm. | +| INCONCLUSIVE | Collect better evidence before choosing a rollout direction. | + +The recommendation is advice for human review. Saving it does not execute a feature-flag change. If you decide to change rollout or targeting, explicitly request and review that separate action. + +The current recommendation panel may label its separate timestamp fields **Not recorded**. Check the **Window** and **Data as of** included in the agent's reason against **Full analysis**. After changing data or configuration, refresh the analysis and ask the agent to review the Run again. + +### Capture learning in the same conversation + +Continue in the same coding-agent conversation. For example: + +```text +Capture the learning for run-1: what changed, what happened, what was +confirmed or refuted, why, and the next hypothesis. Distinguish the +demonstration results from claims about real customers. Save the learning +to this Run through FeatBit MCP and update the experiment's Key learning +summary. Keep the original hypothesis for comparison. +``` + +1. Discuss the agent's interpretation and refine the next hypothesis as needed. +2. Ask it to save the agreed learning through FeatBit MCP. A response in chat alone does not update FeatBit. +3. Refresh FeatBit and open **Learning**. Check **Key learning** for the experiment summary and **Experiment run learnings → run-1** for the Run's decision summary and learning fields. + + + +In this example, the learning records the higher completion rate and clear error guardrail that support **CONTINUE** within the walkthrough. It preserves the original hypothesis and proposes testing it with representative real-customer traffic in a new Run. + +**Edit learning** can update the experiment-level note manually. The coding-agent workflow also fills the structured Run learning shown below it, keeping the interpretation attached to the evidence from that Run. + +## Keep discussing the experiment with your agent + +You can use the same conversation throughout the experiment, from intent and hypothesis to exposure, measurement, decisions, and the next iteration. Ask it to read the current FeatBit state when answering questions such as: + +- “Is our hypothesis specific enough to test, and what result would challenge it?” +- “Explain how the flag's 50/50 rollout, the Layer range, and analysis sampling affect this Run.” +- “Which results support this recommendation, and what limitations should I review?” +- “What would change the recommendation, and what should we measure next?” +- “Review the whole experiment and point out any missing steps before the next Run.” + +Ask for explanations when exploring an idea, and explicitly ask the agent to save agreed changes when you want FeatBit updated. Reopen **Coding Agent Mode** to copy the experiment's **Start using it** prompt when starting a new conversation. diff --git a/content/docs/experimentation/layers.mdx b/content/docs/experimentation/layers.mdx new file mode 100644 index 0000000..9a53b93 --- /dev/null +++ b/content/docs/experimentation/layers.mdx @@ -0,0 +1,109 @@ +--- +title: Layers +description: Create a Layer, assign a Run to a bucket range, and understand current traffic reservations. +--- + +import { ImageZoom } from 'fumadocs-ui/components/image-zoom'; +import createCheckoutLayer from './assets/layers/create-checkout-layer.jpg'; +import editCheckoutLayer from './assets/layers/edit-checkout-layer.jpg'; +import runLayerAssignment from './assets/layers/run-layer-assignment.jpg'; +import layerRunAllocation from './assets/layers/layer-run-allocation.jpg'; +import layerAfterWindowEnds from './assets/layers/layer-after-window-ends.jpg'; + +## Overview + +A Layer is a shared bucket space used to decide which users are eligible for a Run's analysis. Experiments can use separate ranges in that space to coordinate mutually exclusive analysis populations. + +This tutorial creates **Checkout experience** with key `checkout-experience`, then assigns **Simplify checkout / run-1** to the first half of the Layer. It uses the same **First Project / Dev** environment as [Metrics](./metrics.mdx) and [Experiments](./experimentations.mdx). + +You can create the Layer before the experiment exists. To complete the association steps, prepare the experiment and its Bayesian Run using [Create a Bayesian Run](./experimentations.mdx#create-a-bayesian-run). + +## Create the Layer + +1. Confirm the project and environment in the header. +2. Open **Layers** under **Release Decision** and click **New layer**. +3. Enter **Name**: `Checkout experience` and **Key**: `checkout-experience`. +4. Check **Assignment unit**. In the current UI it is read-only and fixed to `user.keyId`. +5. Enter **Description**: `Shared analysis space for checkout experiments.` +6. Click **Create layer**. + + + +The Layer appears as **Active**. Until a Run reserves a range during a current observation window, the allocation bar shows **No allocation**. + +**Assignment unit** identifies the user consistently for bucketing. In the .NET SDK example, `FbUser.Builder("customer-1234")` supplies the user key that corresponds to `user.keyId`. Reuse that ID for exposure and outcome events. All Runs in the Layer use the same assignment unit. + +## Edit and verify the Layer + +1. In the Layer row, click **Edit**. +2. Confirm the saved name and assignment unit. The key cannot be changed after creation. +3. Change **Description** to `Shared analysis space for checkout experiments, assigned by stable user ID.` +4. Click **Save changes**, wait for the editor to close, and refresh. +5. Reopen **Edit** to confirm the description persisted, then close it with **Cancel**. + + + +Changing this description does not allocate traffic. The bucket range belongs to a Run. + +## Associate a Run with the Layer + +1. Open **Experiments → Simplify checkout → Measuring** and select `run-1`. +2. Under **Experiment traffic assignment**, click **Edit assignment**. Creating the Run alone does not save a Layer association. +3. Confirm the variation roles: **Control** is the flag variation named **Control** (`false`), and the checked **Treatment** is the variation named **Treatment** (`true`). +4. Under **Layer eligibility**, select **Checkout experience (checkout-experience)**. +5. Keep **Assignment unit** as `user.keyId`. Set **Bucket start** to `0` and **Bucket end** to `50`. +6. Under **Analysis sampling**, keep **Control** and **Treatment** at `100%`. Leave **Audience filters** empty for this example. +7. Click **Save changes**. Refresh the experiment and reopen **Edit assignment** to verify the saved values. + + + +The saved summary shows **Active range 0%–50% · Width 50%**. The range includes its start and excludes its end: `[0, 50)`. + +## Distinguish eligibility, serving, and sampling + +These settings answer different questions: + +| Setting | What it controls | This example | +| --- | --- | --- | +| Layer range | Which users are eligible for the Run's analysis, based on stable bucketing. | First 50% of the `checkout-experience` Layer. | +| Feature flag variation rollout | Which variation the application actually serves when it evaluates the flag. | `simplified-checkout` serves Control and Treatment at 50/50. | +| Analysis sampling | How much of each eligible variation's traffic is retained for analysis. | 100% for both variants. | + +Assigning a Run to a Layer does not change the flag's rollout. Users outside this Run's Layer range can still receive either flag variation; their data is excluded from this Run's analysis. Do not prefilter SDK users yourself to imitate the Layer. + +For example, this walkthrough evaluated the flag for **4,000 demonstration users**. The analysis included **1,982** users after Layer eligibility: **994 Control** and **988 Treatment**. A 50% range does not guarantee exactly half of a finite user population or identical group sizes. + +## Check current reservations + +1. Return to **Layers** and search for `checkout-experience` using **Filter by name or key**. +2. Check **Experiment runs** for **Simplify checkout**, `run-1`, and `0–50%`. +3. While the Run's observation window covers the current time, inspect the allocation bar and **Allocation status**. + +While `run-1` was collecting events with **No fixed end**, the Layer showed **50% reserved · 50% free** and **No conflicts**: + + + +Interpret the labels as follows: + +- **Reserved** is the part of the Layer occupied by Runs whose observation windows cover the current time. +- **Free** is the remaining unreserved space. +- **No conflicts** is the allocation status reported for this configuration. Here, only the tutorial Run is associated with this Layer. +- **No allocation** means no range is currently reserved; it does not mean all historical Run associations were removed. + +For additional experiments, choose non-overlapping ranges and inspect **Allocation status** before using them. This walkthrough does not create a conflicting Run. + +## Understand what happens when the window ends + +After event collection, set an end time as shown in [Experiments](./experimentations.mdx#finish-the-demonstration-window). Once the observation window has ended: + +1. Refresh **Layers** and search for `checkout-experience` again. +2. Confirm that **Simplify checkout / run-1** remains associated with `0–50%`. +3. Confirm that the current allocation bar now shows **No allocation**, **0% reserved · 100% free**, and **No conflicts**. + + + +The finished Run still uses its saved Layer range when analyzing events in its observation window. The allocation bar describes current reservations, not the number of users previously analyzed. Ending the window also does not turn off the feature flag. + +## Next step + +Continue with [collecting and analyzing experiment data](./experimentations.mdx#collect-events-and-analyze). Keep the Layer range, flag rollout, and analysis sampling settings consistent throughout the observation window. diff --git a/content/docs/experimentation/meta.json b/content/docs/experimentation/meta.json new file mode 100644 index 0000000..6bde69d --- /dev/null +++ b/content/docs/experimentation/meta.json @@ -0,0 +1,9 @@ +{ + "title": "Experimentation", + "pages": [ + "metrics", + "layers", + "experimentations", + "native-data-warehouse" + ] +} diff --git a/content/docs/experimentation/metrics.mdx b/content/docs/experimentation/metrics.mdx new file mode 100644 index 0000000..dde9ce8 --- /dev/null +++ b/content/docs/experimentation/metrics.mdx @@ -0,0 +1,133 @@ +--- +title: Metrics +description: Create reusable metrics, find their experiment runs, and report checkout events with SDKs and APIs. +--- + +import { ImageZoom } from 'fumadocs-ui/components/image-zoom'; +import createCheckoutMetric from './assets/metrics/create-checkout-metric.jpg'; +import editCheckoutMetric from './assets/metrics/edit-checkout-metric.jpg'; +import numericAggregationOptions from './assets/metrics/numeric-aggregation-options.jpg'; +import savedCheckoutMetrics from './assets/metrics/saved-checkout-metrics.jpg'; +import filterByMetricKey from './assets/metrics/filter-by-metric-key.jpg'; +import filterByExperiment from './assets/metrics/filter-by-experiment.jpg'; + +## Overview + +A metric defines the behavior or numeric value you want to measure. Experiments reference these reusable definitions as a **Primary** metric or a **Guardrail**. Creating a definition does not start collecting events: your application must also report the matching metric key. + +This tutorial creates two metrics for a checkout experiment: + +| Name | Key | Type | Aggregation | Purpose | +| --- | --- | --- | --- | --- | +| Checkout completed | `checkout-completed` | Binary conversion | Once per user | Measure whether an exposed user completes checkout. | +| Checkout errors | `checkout-errors` | Numeric value | Count all | Measure how many checkout errors an exposed user encounters. | + +You need access to **Metrics** in a project environment. The screenshots use **First Project / Dev**. Keep the same environment when creating the feature flag, metrics, Layer, and experiment. + +## Create a conversion metric + +1. Confirm the project and environment in the page header, then open **Metrics** under **Release Decision**. +2. Click **New metric**. +3. Enter **Name**: `Checkout completed` and **Key**: `checkout-completed`. The key is the event name your application will send. +4. Choose **Type**: **Binary conversion**. **Aggregation** is fixed to **Once per user** for this type. +5. Enter **Description**: `Users who complete checkout.` +6. Click **Create metric**. + + + +The new metric appears as **Active** in the list. Before an experiment references it, **Experiment runs** shows **No experiment runs**. + +## Edit and verify the definition + +1. Click **Edit** in the **Checkout completed** row. +2. Confirm that the saved name, type, and aggregation are correct. The **Key** is read-only after creation. +3. Change **Description** to `Users who complete checkout after seeing the checkout experience.` +4. Click **Save changes** and wait for the editor to close. +5. Refresh the page and reopen **Edit**. Confirm that the new description remains. + + + +Use **Cancel** to close the editor without further changes. Keep an event key's meaning consistent while collecting experiment data. + +## Add the numeric guardrail + +1. Click **New metric** again. +2. Enter **Name**: `Checkout errors` and **Key**: `checkout-errors`. +3. Choose **Type**: **Numeric value** and **Aggregation**: **Count all**. +4. Enter **Description**: `Number of checkout errors after exposure.` +5. Click **Create metric**, then refresh the list to verify both definitions. + +Numeric metrics expose the following aggregation choices: + + + + + +## Choose the right type and aggregation + +Analysis first calculates each eligible exposed user's contribution, then compares the variants. An exposed user with no matching metric event contributes zero. + +| Type | Supported aggregation | What one user contributes | Example | +| --- | --- | --- | --- | +| Binary conversion | Once per user only | 1 if at least one matching event occurred; otherwise 0. | Did the user complete checkout? | +| Numeric value | Once per user | 1 if at least one matching event occurred; otherwise 0. It does not take the first event's numeric value. | Did the user encounter any checkout error? | +| Numeric value | Count all | The number of matching events. | Three error events contribute 3. | +| Numeric value | Sum values | The sum of the events' numeric values. | Orders worth 10 and 20 contribute 30. | +| Numeric value | Average values | The mean of that user's event values. | Checkout durations of 10 and 20 seconds contribute 15 seconds. | + +**Average values** gives each exposed user a contribution; it is not a pooled average of all events. For example, if one user reports values 10 and 20 and a second exposed user reports no events, their contributions are 15 and 0. The variant's mean is 7.5. + +For **Checkout errors**, send one event for each actual error. Sending `checkout-errors` with a value of `0` still creates an event and therefore increases **Count all**. If no error occurred, do not send that error event. + +## Find metrics and their experiment runs + +1. In **Metrics**, use **Filter by name, key, or experiment**. Enter `checkout-errors` to find the guardrail by its key. +2. Clear the field to restore the list. You can also search by metric name, such as `Checkout completed`. +3. Complete the metric binding and Run creation in [Experiments](./experimentations.mdx#bind-the-flag-and-metrics). +4. Return to **Metrics** and enter `Simplify checkout` in the filter. +5. Check **Experiment runs**: **Checkout completed** is **Primary**, **Checkout errors** is **Guardrail**, and both reference `run-1`. + + + + + +Associations appear after they have been saved in the experiment. A matching name alone does not establish a relationship. + +## Set direction in the experiment + +The definition and its experiment bindings have different jobs: + +| Setting | Where to edit it | Checkout example | +| --- | --- | --- | +| Name, key, description, type, aggregation | **Metrics** | `checkout-completed`, Binary conversion, Once per user | +| Primary metric improvement direction | Experiment **Exposure → Edit metrics → Direction** | **Higher is better** for checkout completion | +| Guardrail warning direction | Experiment **Exposure → Edit metrics → Alert if** | **Increases** for checkout errors; the saved binding shows **Increase is bad** | + +For a guardrail that should increase, **Alert if → Decreases** produces **Decrease is bad**. Choose the direction for that experiment; editing a shared metric definition is not how you change one experiment's direction. + +## Report metric events + +All FeatBit SDKs can report metric events. You can also use APIs such as the [Track Insights API](../api-guides/track-insights-api.md) to send events directly. See the [SDK overview](../sdk/overview.md) for the available SDKs. + +Here is a [.NET example](https://github.com/featbit/featbit-dotnet-sdk) using an initialized `client`. The `orderCompleted` and `checkoutError` conditions represent outcomes from your checkout flow. + +```csharp +var user = FbUser.Builder("customer-1234").Build(); + +// Evaluate the flag when the user enters checkout. +var simplifiedCheckout = client.BoolVariation("simplified-checkout", user, false); + +// When the order is completed: +if (orderCompleted) + client.Track(user, "checkout-completed"); + +// For each checkout error: +if (checkoutError) + client.Track(user, "checkout-errors", 1.0); +``` + +Use the metric keys defined above, the same user ID and environment for evaluation and events, and timestamps within the Run's observation window. After events are processed, click **Analyze latest data** in the Run to see the results. + +## Next step + +Create the [Checkout experience Layer](./layers.mdx), then use these two metrics in the [complete experiment walkthrough](./experimentations.mdx). diff --git a/content/docs/experimentation/native-data-warehouse.mdx b/content/docs/experimentation/native-data-warehouse.mdx new file mode 100644 index 0000000..eb4169f --- /dev/null +++ b/content/docs/experimentation/native-data-warehouse.mdx @@ -0,0 +1,8 @@ +--- +title: Native Data Warehouse +description: Contact FeatBit about using your own data source for experimentation. +--- + +If you would like to use your own data source instead of having FeatBit record experiment data, contact FeatBit. FeatBit will provide a solution based on your requirements. + +Email [contact@featbit.co](mailto:contact@featbit.co), or open the [FeatBit website](https://www.featbit.co/) and choose **Support** to find the available contact channels. diff --git a/content/docs/meta.json b/content/docs/meta.json index 42a6a86..6aedde5 100644 --- a/content/docs/meta.json +++ b/content/docs/meta.json @@ -4,6 +4,7 @@ "getting-started", "installation", "feature-flags", + "experimentation", "sdk", "api-guides", "integrations", diff --git a/package.json b/package.json index dc5c54f..474d831 100644 --- a/package.json +++ b/package.json @@ -2,6 +2,7 @@ "name": "featbit-docs", "version": "0.0.17", "private": true, + "packageManager": "pnpm@10.33.4", "scripts": { "build": "next build", "dev": "next dev",