Methods · validation

How well do FAERS disproportionality signals work?

Tested against 485 drug–event pairs whose answer is known · OMOP 2010 + EU-ADR reference sets · run 2026-10-09

Why this page exists. A signal-detection method is only useful if it flags real drug–event associations more often than false ones. Published reference sets list pairs that are established adverse drug reactions (positive controls) and pairs with good evidence of no association (negative controls). Running OpenPV's statistics on them measures how often the site's signals are right — and how they are biased.

1. Discrimination: can the statistics tell true from false associations?

AUC is the probability that a randomly chosen true association scores higher than a randomly chosen false one (0.5 = coin flip, 1.0 = perfect). 95% bootstrap intervals in grey. "Coverage" is the share of pairs that appear in OpenPV's FAERS data at all; missing pairs score as "no signal".

Reference set	Pairs	Coverage	PRR	ROR lower	EBGM	EB05	IC025
all pairs (deduplicated)	485 (201 positive)	60%	0.710.66–0.75	0.710.66–0.75	0.710.66–0.76	0.710.66–0.75	0.710.66–0.75
OMOP 2010	399 (165 positive)	54%	0.700.65–0.75	0.700.65–0.75	0.700.65–0.75	0.700.65–0.75	0.700.65–0.75
EU-ADR	86 (36 positive)	85%	0.770.65–0.87	0.760.63–0.87	0.770.65–0.87	0.760.63–0.87	0.770.64–0.88
pairs present in FAERS cache	289 (143 positive)	100%	0.780.73–0.83	0.800.74–0.84	0.790.73–0.84	0.800.74–0.84	0.800.75–0.85

All five statistics discriminate about equally well (AUC ≈ 0.71 overall, ≈ 0.80 when the pair is present in the data), consistent with published FAERS benchmarks. The Bayesian shrinkage estimators (EBGM, IC) do not rank pairs much better than PRR; their advantage is in stability for small counts, which is why OpenPV marks PRR-only signals.

2. Thresholds: sensitivity / specificity of the usual signal rules

Rule	All pairs	OMOP 2010	EU-ADR
PRR ≥ 2, χ² ≥ 4, n ≥ 3 (Evans)	62% / 76%PPV 65%	59% / 78%PPV 65%	78% / 70%PPV 65%
EB05 ≥ 2	56% / 81%PPV 68%	53% / 82%PPV 68%	72% / 76%PPV 68%
IC025 > 0	67% / 65%PPV 58%	63% / 69%PPV 59%	83% / 46%PPV 53%
ROR lower CI > 1	67% / 63%PPV 56%	63% / 68%PPV 58%	83% / 44%PPV 52%

Each cell: sensitivity / specificity, with positive predictive value below. The classic PRR rule catches about 62% of true associations and wrongly flags about 24% of false ones. EB05 ≥ 2 is stricter (fewer false alarms, more misses); IC025 > 0 and the ROR rule are looser.

3. By outcome

Outcome	Pairs	AUC (EB05)	AUC (PRR)
acute kidney injury	96 (27+)	0.75	0.74
acute liver injury	125 (83+)	0.76	0.75
acute myocardial infarction	111 (40+)	0.54	0.56
anaphylaxis	10 (5+)	0.84	0.80
aplastic anaemia	10 (5+)	0.78	0.82
bullous eruptions sjs ten	10 (5+)	0.96	0.92
neutropenia agranulocytosis	10 (5+)	0.84	0.84
rhabdomyolysis	9 (4+)	1.00	1.00
upper gi bleeding	99 (27+)	0.74	0.74

4. Systematic bias and empirical calibration

If FAERS statistics were unbiased, negative-control pairs (drugs known not to cause the event) would scatter around ROR = 1 and only ~5% would be "significant". They do not:

2.1

median ROR among 146 negative controls

78%

negatives "significant" at p < 0.05 (nominal)

after empirical calibration (target 5%)

17%

of 586,667 PRR-flagged pairs stay significant after calibration + FDR

Method (Schuemie et al., Stat Med 2014): the negative controls' log-RORs and standard errors are used to fit the distribution of systematic error, N(μ = 0.74, τ = 0.73) on the log scale — i.e. a "null" ROR anywhere between 0.5 and 8.8. A calibrated p-value then asks whether a pair's ROR is unusual relative to that null, not relative to ROR = 1. Applied to every pair OpenPV flags with the classic rule, only 98,075 of 586,667 remain significant after Benjamini–Hochberg correction. This is why the drug pages report calibrated p-values instead of the meaningless "p < 1e-300".

Sources of the bias include confounding by indication, notoriety and stimulated reporting, and the max-over-terms scoring used here (each outcome maps to several MedDRA terms, which inflates negatives somewhat; for single-term pairs on the site the calibration is therefore conservative). The negatives cover only 10 outcomes, so transferring this null to every reaction is an assumption.

5. What this means for reading OpenPV

A FAERS signal is a screening result with roughly 60–90% sensitivity and 55–80% specificity depending on the rule — useful for prioritising, not for concluding.
Most of the site's 586k classic signals are not distinguishable from systematic reporting bias. Prefer pairs that are significant after calibration, supported by EBGM/IC, stable over time (no stimulated-reporting spike) and consistent in serious-only and stratified analyses.
Unlabeled, calibrated, multi-method signals are the queue for the Bradford Hill assessments.

Sources & reproducibility

Ryan PB, Schuemie MJ, Welebob E, et al. Defining a reference set to support methodological research in drug safety. Drug Saf 2013;36 Suppl 1:S33–47.
Coloma PM, Avillach P, Salvo F, et al. A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases. Drug Saf 2013;36:13–23.
Schuemie MJ, Ryan PB, DuMouchel W, Suchard MA, Madigan D. Interpreting observational studies: why empirical calibration is needed to correct p-values. Stat Med 2014;33:209–218.
Reference pairs parsed from the OHDSI MethodEvaluation R package (omopReferenceSet, euadrReferenceSet). Code: openpv/validate_methods.py; results: method_validation.json.