Introduction: Primary dysmenorrhea (PD) is a common gynecological disorder that usually begins in adolescence, and affects patients’ daily activities and quality of life. Non-steroidal anti-inflammatory drugs (NSAIDS) are considered the first-line treatment, and hormonal contraceptives are also recommended for PD, but both are prone to side-effects. The Chinese herbal formula Siwutang (SWT) and its derivative formulas are a common treatment for PD in China. This review assessed the efficacy and safety of SWT for the treatment of PD. Methods: PubMed, EmBase, Cochrane CENTRAL, CNKI, Wanfang and CBM were searched. We included randomized controlled trials (RCTs) that investigated SWT for PD, compared with no intervention, placebo, or conventional Western medicine. The outcome measurements included pain intensity measured by visual analogue scale (VAS) or other validated scales, the Cox Menstrual Symptom Scale (CMSS), quality of life, response rate and adverse events. The Cochrane Collaboration's tool was used to assess the risk of bias. RevMan V.5.3 was used for data synthesis and meta-analysis. Risk ratio (RR) with 95 % confidence intervals (CIs) or mean difference (MD) with 95 % CIs was calculated for dichotomous data or continuous data, respectively. Heterogeneity among studies was evaluated using both a chi-square test and an I2 test. Results: A total of 38 RCTs involving 3982 participants were identified. The methodological quality of the included trials was generally poor. Moreover, the results for SWT compared with placebo were unclear, as there was only 1 RCT. SWT improved pain intensity measured by VAS (3 RCTs, n = 220, MD:-2.61, 95 % CI:-3.72 to -1.51) when compared with conventional medicine, and these results were statistically significant. The meta-analysis showed the superior effect of SWT (including derivative formulas) on response rate (35 RCTs, n = 3,695, RR: 1.28, 95 % CI: 1.22–1.34) with medium heterogeneity (I2 = 48 %). Both original SWT and its derivative formula XFSWT had a higher response rate than conventional medicine (23 RCTs, n = 2,493, RR: 1.28, 95 % CI: 1.23–1.33) (11 RCTs, n = 1,076, RR: 1.36, 95 % CI: 1.20–1.53). These results were statistically significant. No trial reported on quality of life or CMSS. Adverse events were reported by 5 studies, and meta-analysis showed SWT may be safer than conventional medicine in terms of the incidence of adverse events (3 RCTs, n = 236, RR: 0.17, 95 % CI: 0.07−0.38, I2 = 0%). Conclusion: In conclusion, the included trials showed favorable effects of SWT for treating primary dysmenorrhea when compared with conventional medicine. SWT may be safer than conventional medicine, but insufficient data was reported. The level of evidence is low because of the high risk of bias. Thus, further well-designed clinical trials with large sample sizes are warranted. Registration number: CRD42019136230 in PROSPERO 2019. © 2020 The Author(s)