experiments | Chen Xing

Synthetic Controls for Experimental Design (Abadie and Zhao 2025)

Tue, 28 Oct 2025 00:00:00 +0000

TL;DR

Abadie and Zhao propose using synthetic control methods to design experiments when only one or a few large aggregate units (e.g., markets, cities) can be treated. Rather than randomizing treatment assignment—which can produce large post-randomization biases with small samples—their method jointly selects which units receive treatment and which serve as controls by matching pre-treatment characteristics. Simulations and an empirical application using Walmart sales data show that synthetic control designs substantially outperform randomized designs in settings with few treated units, reducing bias and improving statistical power while allowing for valid inference through novel permutation-based methods.

What is this paper about?

This paper addresses a fundamental challenge in experimental design: how to evaluate interventions when the experimental units are large aggregate entities (like local markets or cities) and only one or a small number can be exposed to treatment. In such settings, standard randomization can produce substantial post-randomization bias because the treated unit(s) may differ markedly from control units in baseline characteristics that affect outcomes. The motivating example involves a ride-sharing company testing a new driver compensation plan in one city—randomizing drivers within a city creates equity concerns and spillover effects, while randomizing across cities with few units risks poor balance. The authors propose adapting synthetic control methods (traditionally used in observational studies) to experimental design, where both the choice of treated units and control units are optimized jointly to reproduce aggregate counterfactuals of interest.

What do the authors do?

The authors develop several variants of synthetic control experimental designs that select units for treatment and control by matching weighted averages of pre-treatment predictors (including lagged outcomes and covariates) to population-level or treated-unit averages. Their baseline “unconstrained” design minimizes discrepancies between synthetic treated units and the population average, and between synthetic control units and the population average, subject to non-negativity and sum-to-one constraints on weights. They extend this to:

“constrained” designs (limiting the number of treated units for cost reasons)
“weakly-targeted” designs (targeting either average treatment effects or treatment effects on the treated)
“unit-level” designs (fitting separate synthetic controls for each treated unit)
clustered designs (accounting for natural groupings like regions)

They derive formal bias bounds under a linear factor model, showing that bias depends on the number of fitting periods, the scale of idiosyncratic shocks, and the number of unobserved factors. For inference, they propose novel permutation tests that rearrange estimated treatment effects across “blank periods” (pre-treatment periods not used for fitting) and post-treatment periods, proving the test is exact when factor loadings are exchangeable and approximately valid more generally. They also construct confidence intervals using split conformal prediction. The methods are validated through Monte Carlo simulations under both linear and nonlinear data-generating processes, and through a placebo test using weekly sales data from 45 Walmart stores, comparing performance against randomization, stratified randomization, regression adjustment, and nearest-neighbor matching.

Why is this important?

This work fills a critical gap in experimental methodology for settings where large-scale randomization is infeasible or undesirable—situations increasingly common in tech companies, policy evaluation, and corporate decision-making. When only a few aggregate units can be treated, randomization provides unbiased estimates ex ante (before randomization) but can yield large biases ex post (after randomization) if treated and control groups differ substantially at baseline. The synthetic control design directly addresses this by selecting units that jointly minimize imbalance, making the realized experiment more credible. The paper’s theoretical contributions—bias bounds that clarify the role of fitting periods, idiosyncratic noise, and unobserved factors, plus new inferential methods that allow for time-series dependence and non-stationarity—strengthen the foundation for applied work. The empirical validation demonstrates practical feasibility: in the Walmart data, synthetic control designs achieve root mean squared errors 3-10 times smaller than randomized alternatives. This makes rigorous causal inference possible in settings where researchers previously either accepted high bias or abandoned the project when pre-trends looked poor.

Who should care?

Applied researchers and practitioners conducting experiments with aggregate units should care, particularly in settings where intervention at the micro-unit level is impractical (e.g., testing market-level policies, district-level education reforms, or state-level regulations). This includes data scientists in technology companies designing market experiments, policy analysts evaluating place-based programs, economists studying geographically-clustered interventions, and corporate strategists testing operational changes across stores or regions. Methodologists working on causal inference, experimental design, and synthetic controls will find the paper’s extensions—especially the bias-variance trade-offs across designs, the treatment of time-series dependence in inference, and the integration of design selection with estimation—valuable for advancing the field. Anyone frustrated by the limitations of randomization in small-sample settings with aggregate units now has a principled alternative.

Do we have code?

Yes. The authors state that replication codes are available on GitHub (linked in the paper’s first page footnote). The online appendix provides detailed implementation guidance, including how to solve the optimization problems using both enumeration methods (for constrained designs with small numbers of treated units) and quadratic programming (for unconstrained and penalized designs). They implemented the synthetic control problems using the “lsei” function from the “limSolve” package in R 4.0.2, and the quadratic programs using Gurobi 9.0.2 in R. The appendix also documents the computational approach for each design variant, making the methods reproducible and accessible for practitioners.

In summary, this paper transforms synthetic controls from an observational tool into an experimental design strategy, showing both theoretically and empirically that careful unit selection can dramatically reduce bias compared to randomization when few aggregate units are available for treatment. By jointly optimizing the choice of treated and control units while preserving valid inference, it expands the frontier of rigorous experimental evaluation in settings previously considered methodologically challenging.

Reference

Abadie, Alberto and Jinglong Zhao (2025), “Synthetic controls for experimental design.” https://doi.org/10.48550/arXiv.2108.02196

The Experimental Selection Correction Estimator

Mon, 06 Oct 2025 00:00:00 +0000

TL;DR

Athey, Chetty, and Imbens develop a new method to estimate treatment effects on primary outcomes (like graduation rates) by combining experimental data where treatment is randomized but only secondary outcomes are observed (like test scores) with observational data where both primary and secondary outcomes are measured but treatment is not randomized. Their Experimental Selection Correction (ESC) estimator uses differences in secondary outcomes between the two samples to correct for selection bias, relying on a new “latent unconfoundedness” assumption that requires the same unobserved confounders to affect both primary and secondary outcomes. Applied to class size effects, the method reveals that a 25% reduction in third grade class size increases high school graduation rates by 0.7 percentage points, while standard observational methods yield implausible negative estimates.

What is this paper about?

This paper addresses a common challenge in causal inference where researchers have access to two complementary but incomplete datasets: experimental data with randomized treatment and secondary outcomes (like test scores) but missing primary outcomes of interest (like graduation rates), and observational data with both primary and secondary outcomes but non-random treatment assignment plagued by selection bias. The motivating example involves estimating how third grade class size affects high school graduation when Project STAR provides experimental evidence on test score effects but lacks graduation data, while NYC school district records contain graduation information but suffer from confounded class size variation. The authors aim to leverage the internal validity of experiments to purge selection bias from observational estimates without relying on the strong surrogacy assumptions that typically require treatment to affect primary outcomes only through secondary outcomes.

What do the authors do?

The authors develop both the theoretical foundations and practical implementation of the ESC estimator through identification results and multiple estimation approaches. Theoretically, they prove that the average treatment effect on the primary outcome is point-identified under three key assumptions: random assignment in the experimental sample, conditional external validity (requiring treatment effects to generalize across samples conditional on covariates), and their novel latent unconfoundedness condition (requiring that unobserved confounders affecting the primary outcome are the same as those affecting the secondary outcome). They show this approach strictly weakens standard surrogate assumptions by permitting direct effects of treatment on primary outcomes and allowing for unobserved confounding in the observational data. For estimation, they present four equivalent approaches—control function, imputation, weighting, and influence function methods—with the control function approach being most straightforward: estimate the experimental treatment effect on the secondary outcome, calculate residuals in the observational sample as the difference between actual and predicted secondary outcomes, then regress the primary outcome on treatment while controlling for these residuals. They validate their method empirically by applying it to estimate class size effects, demonstrating that ESC estimates on holdout outcomes (test scores in grades 4-8) closely match experimental benchmarks and capture well-known fadeout patterns, while standard OLS estimates remain severely biased even after controlling for rich demographic covariates.

Why is this important?

This method addresses a pervasive limitation in policy evaluation where long-term outcomes are too costly or time-consuming to measure in experiments but are routinely captured in administrative observational data that suffer from selection bias. The ESC estimator provides a principled way to harness experimental internal validity for correcting observational estimates without requiring the restrictive surrogacy assumption that treatment operates exclusively through the secondary outcome—an assumption frequently violated in practice (as evidenced by the education literature showing that early interventions affect long-term outcomes through pathways beyond test scores, such as non-cognitive skills). The latent unconfoundedness assumption, while novel and untestable in isolation, is substantially weaker than assuming either surrogacy or unconfoundedness in the observational sample, making it applicable in settings where conventional methods fail. The paper also formalizes a common empirical heuristic: when observational and experimental estimates align on secondary outcomes, researchers often proceed to estimate effects on primary outcomes using the observational data, and the authors clarify precisely when this practice is justified. Methodologically, the connection to control function methods and missing data theory provides a familiar statistical framework, while the demonstration that standard covariate adjustment fails where ESC succeeds highlights the method’s ability to address selection on dimensions typically unobserved in administrative data.

Who should care?

Applied researchers conducting policy evaluations in education, labor economics, health, and other fields where experiments measure short-term proxies but administrative data track long-term outcomes should pay close attention to this method. It is particularly relevant for analysts working with combined experimental and observational datasets who currently rely on surrogate approaches but suspect violations of the exclusion restriction (that treatment affects primary outcomes only through secondary outcomes). Econometricians and statisticians interested in causal inference methodology will find value in the identification results, especially the latent unconfoundedness framework and its connections to control function methods, missing data assumptions, and semiparametric efficiency. Policy analysts and government agencies that commission experiments but need evidence on outcomes with long observation lags (like earnings, mortality, or recidivism) can use this approach to accelerate evidence generation. Education researchers studying intervention effects will find the class size application instructive, as it provides rare causal estimates of elementary school inputs on high school completion and demonstrates how to validate identifying assumptions using holdout outcomes.

Do we have code?

Yes, replication code is available at the GitHub repository: https://github.com/OpportunityInsights/Experimental-Selection-Correction-Replication-Code.git. The paper also provides straightforward Stata code in Appendix C for implementing the control function version of the ESC estimator in three steps (estimate experimental treatment effect on secondary outcome, construct residuals in observational sample, regress primary outcome on treatment controlling for residuals), though proper inference requires bootstrapping rather than conventional standard errors. The appendix additionally provides R code and details alternative estimation approaches including imputation, weighting, and influence function methods for researchers who prefer different implementations.

In summary, this paper introduces a method that strategically combines experimental and observational data to estimate treatment effects on primary outcomes that cannot be measured in experiments. By using experimental estimates on secondary outcomes to construct a selection correction term, the ESC estimator removes biases in observational data under the novel latent unconfoundedness assumption that the same confounders affect both primary and secondary outcomes. This assumption is substantially weaker than surrogacy (which prohibits direct treatment effects on primary outcomes) and weaker than standard unconfoundedness (which requires no unmeasured confounding). The application to class size effects demonstrates the method’s ability to recover experimental benchmarks on holdout test scores while revealing that class size reductions meaningfully increase graduation rates—a finding obscured by severe selection bias in standard observational estimates that persists even after rich covariate adjustment. The method opens new possibilities for leveraging experiments to accelerate policy learning about long-term outcomes routinely captured in administrative data.

Reference

Athey, S., Chetty, R., & Imbens, G. (2025). The experimental selection correction estimator: Using experiments to remove biases in observational estimates (Working Paper No. 33817; Working Paper Series). National Bureau of Economic Research. https://doi.org/10.3386/w33817

Enhancing Statistical Power In Marketing Experiments — A Practical Implementation Guide

Mon, 05 May 2025 00:00:00 +0000

1. Introduction

This guide provides practical techniques for increasing the statistical power of marketing experiments without relying solely on large sample sizes. Based on Meyvis and van Osselaer’s work (2018), these methods enable researchers to detect subtle marketing effects with feasible sample sizes by increasing observed effect sizes through proper design and analysis decisions.

2. Essential Tools And Resources

Required Tools:

Statistical software (R, SPSS, Stata)
Survey platforms (Qualtrics, SurveyMonkey)
Pre-registration platforms (OSF, AsPredicted.org)
Data visualization tools

Required Planning Elements:

Clearly defined hypotheses
Detailed experimental designs
Predetermined analysis plans
Transparency protocols

3. Pre-Study Protocol

Pre-Registration Process:

Define your research question with specificity
Develop theory-based, testable hypotheses
Determine all analyses before data collection
Establish participant exclusion criteria
Justify sample size using power analysis
Document all decisions on a pre-registration platform

Key Implementation Step: Create a comprehensive pre-registration document that includes all exclusion criteria, covariates, and analysis plans before collecting any data.

4. Experimental Design Techniques

4.1 Within-Subjects Design Implementation:

Have each participant experience all experimental conditions
Counterbalance condition order systematically
Include buffer tasks between conditions to reduce carryover
Vary stimuli on multiple dimensions to reduce demand effects
Include checks for hypothesis guessing when manipulation is obvious

Application Criteria: Most effective when sample availability is limited and individual differences are substantial.

4.2 Covariate Implementation:

Identify variables with strong expected correlation to your DV (r > .2)
Measure covariates before introducing your manipulation
Use different measurement scales for covariates and DVs
Test for treatment-by-covariate interactions
Only include covariates that meet all statistical assumptions

Critical Requirements:

Manipulation must not affect the covariate
Covariate must not interact with the treatment
Measurement of covariate should not influence DV response

4.3 Manipulation Optimization:

Design direct rather than indirect manipulations
Create clean manipulations that avoid confounds
Use pre-tests to calibrate manipulation strength
Include manipulation checks in study design
Select manipulation levels where marginal effects are strongest

Implementation Note: Balance manipulation strength against potential demand effects.

5. Participant Management

5.1 Quality Control Procedures:

Implement Instructional Manipulation Checks (IMCs)
Monitor response times for unusually fast completion
Apply consistent exclusion criteria across all studies
Document all exclusions transparently
Complete all exclusions before hypothesis testing

Application Protocol: Define exclusion criteria explicitly in pre-registration and never deviate based on results.

5.2 Participant Selection Optimization:

Define relevant participant characteristics
Screen participants before the main study
Create more homogeneous participant groups
Consider targeted recruitment for higher relevance
Balance specificity against generalizability concerns

Implementation Strategy: Target participants for whom stimuli are relevant but avoid introducing selection biases.

6. Analytical Techniques

6.1 Planned Contrast Implementation:

Specify expected pattern of means before data collection
Develop contrast codes that directly test hypotheses
Use focused tests instead of omnibus tests
Test residual variance to confirm pattern specificity
Apply consistent analysis approaches across studies

Application Benefit: Increases power by testing only the specific pattern of interest.

6.2 Interaction Analysis Protocol:

Select moderators based on theoretical mechanisms
Avoid “meaningless moderation” that creates ceiling effects
Increase sample size appropriately for interaction tests (4x main effect sample)
Report simple effects to clarify interaction patterns
Interpret moderation in relation to underlying theory

Implementation Warning: Never test moderators post-hoc without theoretical justification.

7. Avoiding Methodological Pitfalls

Common Malpractice Warning Signs:

Analyzing data before collection is complete
Testing multiple exclusion criteria selectively
Adding covariates post-hoc based on results
Optional stopping when results become significant

Prevention Protocol:

Establish all analytical decisions before data collection
Apply criteria consistently across all studies
Report all analyses conducted, significant or not
Maintain a detailed research log for all decisions
Conduct confirmatory replications for important findings

8. Power Enhancement Techniques Summary

Measurement Optimization:

Implement multi-item scales rather than single items
Verify reliability (α > .8) before deployment
Select measures with appropriate sensitivity

Error Variance Reduction:

Standardize experimental environment
Create consistent procedural instructions
Implement computerized timing when possible

Stimuli Optimization:

Select stimuli with sufficient room for movement (avoid floor/ceiling)
Match stimuli appropriately to participant demographics
Conduct pilot tests to assess malleability

Data Quality Control:

Remove problematic data points using predetermined criteria
Apply appropriate transformations for skewed distributions
Document all data processing steps transparently

9. Replication Protocol

Implementation Steps:

Reproduce original study with minimal modifications
Apply identical exclusion criteria and analyses
Compare effect sizes between original and replication
If confirmed, extend with additional conditions
If unsuccessful, systematically examine methodological differences

Strategic Application: Use replications to validate effects and build cumulative knowledge.

10. Effect Size Reference Guide

Effect Size Type	Small	Medium	Large
Cohen’s d	0.2	0.5	0.8
η²	.01	.06	.15
R²	.01	.09	.25

Sample Size Required for 80% Power (Two-tailed, $ \alpha = .05$)

Effect Size (d)	Between-Subjects	Within-Subjects (r=.5)
0.2 (Small)	394 per group	199 total
0.5 (Medium)	64 per group	33 total
0.8 (Large)	26 per group	14 total

Conclusion

Effective marketing experiments require both scientific rigor and practical feasibility. By implementing these techniques systematically, researchers can increase statistical power without relying solely on massive samples. Remember that the goal is not simply statistical significance but accurately measuring marketing phenomena with precision and integrity.

Reference

Meyvis, T., & Van Osselaer, S. M. J. (2018). Increasing the Power of Your Study by Increasing the Effect Size. Journal of Consumer Research, 44(5), 1157–1173. https://doi.org/10.1093/jcr/ucx110