A simple A/B testing setup with Simple Analytics

I recently tweeted this and people seems to find it interesting. In this post, I'm going to share my Super Simple Setup™ for A/B Testing with React (Next.JS) & Simple Analytics

How does it work?

So basically we need to:

1. Randomly show 2 versions of the website to different users
2. Track the conversion of each version
3. A report dashboard to check the result

Let talk about each step.

1. Show 2 versions of the website to different users

We want to show each visitor a version of the website. One version is the "control" group, or the version already in use. The second version changes a single element.

How these versions behave is up to you. For example, below I was running a headline test. Variant A (left) and variant B (right).

You can run other tests for calls to action text, pop ups, featured images etc. but the method is the same.

Since I'm already using Next.JS, I figured it would be faster to just store these variant in cookie and use a simple conditional rendering to show the content of each version.

Note: I don't use Server-Side Rendering so it might render an empty h1 before it shows the correct content.

import Cookies from 'js-cookie'
import { useEffect, useState } from 'react'

function Page() {
  const [variant, setVariant] = useState(null)

  useEffect(() => {
    let cookieValue = Cookies.get('variant')
    if (!cookieValue) {
      cookieValue = Math.random() < 0.5 ? 'a' : 'b'
      Cookies.set('variant', cookieValue, {
        expires: 30,
      })
    }

    setVariant(cookieValue)
  }, [])

  return (
    <h1>
      {variant === 'a' && <span>Current headline</span>}
      {variant === 'b' && <span>New headline</span>}
      {variant === null && <span>&nbsp;</span>}
    </h1>
  )
}

2. Track the conversion of each version

Now we got the variant stored in cookie. We need to track the conversion event of each version.

I use Simple Analytics for this, but you can use any other tools.

Tracking these conversion events with Simple Analytics is, well, simple. For BoltAI, I want to track the number of downloads.

So it looks something like this:

function Hero() {
  const handleBeforeDownload = () => {
    const variant = Cookies.get('variant') || null
    if variant {
      window.sa_event && window.sa_event('click_download', { variant })
    } else {
      window.sa_event && window.sa_event('click_download')
    }
  }

  return (
    <Button onClick={handleBeforeDownload}>
      <span className="mx-2">Download</span>
    </Button>
  )
}

3. A report dashboard to check the result

In Simple Analytics website dashboard, open tab "Goals", click "Add goal".

• Pick the conversion event name. It's click_download in my case
• Make sure to filter unique visits & with the right variant
• Set a friendly name.

Do the same for variant b.

Now you should be able to see the conversion stats on the dashboard.

There is one minor issue here is that you will need to track the start date of the experiment manually. But hey, did I mention this is a stupid implementation 😄

That's all

Hope you find it useful. Find me on Twitter 👋

If you are new here, BoltAI is a native macOS app that allows you to access ChatGPT inside any app. Download now.