Subsections of How-To Guides

Prepare a Tracker

The modelling workflow starts from a long-format tracker. Use one row per extracted effect estimate.

Required columns

client,evidence_type,metric,value,unit,period,source_type,source_status,source_file_or_link,owner_or_contact,analysis_ready,notes

Optional uncertainty columns

study_id,estimand_note,standard_error,ci_lower,ci_upper,ci_level,ci_type,p_value,p_value_sidedness,uncertainty_scenario

Supported pooled evidence

Evidence Default treatment
geo_test Pool comparable effects on log_relative scale.
bls Pool comparable effects on percentage_point scale.
mmm, pa2, ROI, CPA, CPO, iROAS Treat as triangulation unless harmonised upstream.

Validate the tracker

marketbayesmeta-check-tracker path/to/tracker.csv
marketbayesmeta-check-tracker path/to/tracker.csv --audit --include-partial

Resolve or explicitly accept:

  • duplicate (evidence_type, client, metric, period) candidates;
  • missing source uncertainty;
  • interval rows missing ci_type or ci_level;
  • p-values without p_value_sidedness=two_sided;
  • rows marked analysis-ready but using a unit incompatible with the requested scale.

Estimand judgement

The tracker and config enforce metric names and units, but they cannot prove that studies answer the same business question. Before pooling, confirm that population, measurement window, study design, and estimand are comparable enough for a pooled effect to mean something.

Run an Analysis

Use a YAML config when an analysis should be repeatable and reviewable.

marketbayesmeta-check-config examples/config.yaml
marketbayesmeta-run examples/config.yaml

For local repository use:

python runme.py examples/config.yaml

Conservative default gates

Config runs stop on directional and not_recommended readiness by default, and failed sampler diagnostics are blocking by default.

diagnostics:
  allow_directional: false
  allow_not_recommended: false
  allow_duplicates: false
  fail_on_diagnostic_failure: true
  minimum_studies_ready: 4
  minimum_studies_directional: 3

Override these only for explicit exploratory work. Record the reason in project.notes or the downstream analysis review.

Paths

data.tracker and outputs.directory are resolved relative to the config file when they are not absolute paths.

Reportability

Successful execution does not mean the result is reportable. Always review run_status.reportable and the supporting artefacts before using pooled summaries.

Review Outputs

Config runs write a compact artefact set intended for analyst review.

First files to open

File Review question
analysis_report.md What is the run status, readiness, and headline result?
run_status.json Is reportable true? Why did the run complete, warn, block, or fail?
readiness.csv How many comparable rows were eligible, and are they enough?
diagnostics.csv Did sampler diagnostics pass configured thresholds?
prior_diagnostics.csv Do priors imply warnings or implausible prior-predictive ranges?
effect_preparation.csv Which rows were included, transformed, excluded, or scenario-based?
ppc.csv Are individual studies out of line with in-sample posterior predictive checks?

Status meanings

run_status.json may report:

  • completed
  • completed_with_warnings
  • blocked
  • failed

A completed run is not automatically reportable. Treat reportable: false as a hard prompt for analyst review before any downstream presentation.

Common reasons a run is not reportable

  • Readiness is directional or not_recommended.
  • Sampler diagnostics failed or were downgraded to warnings.
  • The tracker contains blocking duplicate rows.
  • Prior diagnostics or sensitivity indicate material prior dependence.
  • One or more rows rely on scenario uncertainty.

Run Sensitivity

Small-sample Bayesian meta-analysis can be prior-sensitive and uncertainty-sensitive. For serious reporting, sensitivity is part of the result.

Enable prior sensitivity

sensitivity:
  prior: true

When prior_specs are omitted, the package uses a scale-aware grid:

  • log_relative: regularising, default, weak
  • percentage_point: regularising, default, weak

Review prior_sensitivity.csv for pooled mean, probability positive, interval width, diagnostics, and deltas versus the default prior.

Enable uncertainty sensitivity

sensitivity:
  uncertainty: true

Rows with source-derived uncertainty keep that uncertainty. Rows with uncertainty_scenario use the configured low, medium, or high model-scale standard error.

Scenario assumptions live in:

uncertainty:
  scenario_standard_errors:
    log_relative:
      low: 0.03
      medium: 0.08
      high: 0.15
    percentage_point:
      low: 1.0
      medium: 3.0
      high: 5.0

Report when reasonable sensitivity settings change the substantive conclusion.