Skip to main content
Michael de Beer
Menu

Python for analytics and A/B testing

Python for merging and cleaning analytics data, running Bayesian A/B tests and calculating sample sizes before a test launches.

Python takes web analytics and A/B testing past the limits of standard dashboards. It handles very large datasets, automates complex statistical tests, and supports custom attribution models that GA4 can’t build out of the box.

Data acquisition and cleaning

Python notebook merging GA4 sessions with CRM sales data

Python merges data from different sources, such as GA4 session data with internal CRM sales data. It handles missing values, filters out bot traffic and aligns timestamps so funnel analysis is accurate.

Bayesian A/B testing

Bayesian A/B test showing the probability that variant B beats variant A

Standard (frequentist) A/B testing gives you a p-value, which is hard to interpret. A Bayesian test gives you the number people actually want: the probability that variant B is better than variant A.

Sample size and power calculation

Most people guess how long a test should run. It can be calculated exactly.

I built Python scripts for power analysis that set the minimum sample size and minimum detectable effect (MDE) before launch. That optimizes traffic allocation and prevents underpowered tests.

Statistical & Experimentation Methodologies

  • Hypothesis Testing: Frequentist (two-proportion z-tests, t-tests, chi-square) and Bayesian experimentation frameworks.
  • Experimental Design: Power analysis, sample size estimation, Minimum Detectable Effect (MDE) calculation, and test duration forecasting.
  • Variance Reduction & Advanced Techniques: CUPED (Controlled-experiment Using Pre-Experiment Data), stratified sampling, and multi-armed bandits (MAB).
  • Bias & Error Mitigation: Sample Ratio Mismatch (SRM) detection, sequential testing / peeking corrections (e.g., alpha spending, always-valid p-values), and multiple testing corrections (Bonferroni, Benjamini-Hochberg).
  • Alternative Causal Inference: Difference-in-Differences (DiD), uplift modeling, and propensity score matching (for when randomized controlled trials aren’t feasible).

Python & Data Science Stack

  • Core Data Manipulation: Pandas, NumPy, Polars (for large-scale event log processing).
  • Statistical Modeling: SciPy (scipy.stats), Statsmodels, Pingouin.
  • Data Visualization & Storytelling: Matplotlib, Seaborn, Plotly (building automated readout charts and confidence interval plots).
  • Data Extraction & Warehousing: SQL (BigQuery, Snowflake, Redshift), Python-to-warehouse connectors (google-cloud-bigquery, SQLAlchemy).
  • Workflow & Reproducibility: Jupyter Notebooks, Git/GitHub, automated Python reporting pipelines.

Analytics, Tagging & Experimentation Infrastructure

  • Testing Platforms: Optimizely, VWO, Adobe Target, Statsig, LaunchDarkly, or custom feature-flagging architectures.
  • Telemetry & Event Tracking: Google Analytics 4 (GA4), Adobe Analytics, Google Tag Manager (GTM), Segment, raw event-stream validation.
  • Data Governance: Designing standardized tracking schemas, ensuring clean assignment logging, and auditing pre-test telemetry.

Leadership & Program Strategy

  • Experimentation Roadmap Management: Prioritization frameworks (ICE, PIE, RICE), intake workflows, and velocity scaling across product and marketing teams.
  • Cross-Functional Alignment: Partnering with engineering, UX/UI, product management, and executive stakeholders to translate business questions into testable hypotheses.
  • Governance & Culture: Establishing experimentation Centers of Excellence (CoE), standardizing readout templates, and reducing false-positive deployments.
  • Business Impact: Connecting micro-conversions to macro-revenue metrics, holdout group validation, and incremental lift reporting.

See how this fits into a wider experimentation program.

Stack
  • Python
  • Bayesian statistics
  • GA4