Skip to main content

A/B testing + conversion goals

Per-campaign N variants with deterministic per-player assignment + conversion attribution against any canonical event.

Variants​

Each campaign has 1..N rows in campaign_variants:

{
  "name": "Variant A",
  "label": "A",                  // short badge label
  "template_id": "...",          // operator clones template, tweaks copy
  "weight": 50,                  // relative weight 0..10000
  "sort_order": 0
}

Weights need not sum to 100 — assignment normalises by total. Two variants at weight=1 each = 50/50 split. One at 70, one at 30 = 70/30.

Deterministic assignment​

bucket = FNV1a32(`${campaign_id}|${player_id}`) mod sum_weights
variant = first variant where cumulative_weight > bucket

Same player on the same campaign always lands in the same variant — even across re-sends. Prevents variant-drift attribution mud.

Same player on different campaigns can land in different variants (the campaign_id varies the hash input).

Conversion goals​

// On the campaign:
{
  "goal_event": "deposit_confirmed",     // canonical event name
  "goal_attribution_days": 7,            // 1..90
  "goal_value_property": "amount_base"   // jsonpath for revenue attribution
}

GET /v1/campaigns/:id/conversion-stats joins message_sends (PG) with raw_events (CH):

  • For each player who received a message in (sent | delivered | opened | clicked) status,
  • Find the earliest goal_event in [sent_at, sent_at + attribution_days * 24h)
  • Group by variant_id → count + value sum

Earliest because a player might do the goal multiple times — we credit ONE conversion per player per campaign.

Stats response​

{
  "campaign_id": "...",
  "goal_event": "deposit_confirmed",
  "goal_attribution_days": 7,
  "total_recipients": 3085,
  "total_conversions": 151,
  "variants": [
    {
      "variant_id": "...",
      "variant_name": "Variant A",
      "variant_label": "A",
      "recipients": 1547,
      "conversions": 87,
      "conversion_rate": 0.0563,
      "conversion_value": 870000  // sum in tenant base currency cents
    },
    {
      "variant_id": "...",
      "variant_name": "Variant B",
      "variant_label": "B",
      "recipients": 1538,
      "conversions": 64,
      "conversion_rate": 0.0416,
      "conversion_value": 640000
    }
  ],
  "winner_variant_id": "...A's id"
}

Winner picker​

A variant is declared the winner when:

  1. It has the highest conversion rate among all variants
  2. Its recipients count is ≥ 30 (avoids declaring winners on noise)
  3. Tie-break: highest recipient count (more confidence) → first by sort_order

Below 30 sample size, winner_variant_id is null until more sends accumulate.

Mid-campaign weight changes​

Operators can edit weights mid-campaign — only new sends respect the new split. Already-enqueued message_sends keep their assigned variant_id so conversion attribution stays clean.