This study tests the feasibility of applying the pragmatic-explanatory continuum indicator summary (version "PRECIS-2") tool to randomized controlled trials of Chinese herbal medicine. A search was conducted to identify potentially eligible randomized controlled trials. Using the PRECIS-2 tool, assessment of trials was performed independently by 2 evaluators using a scale of 1-5 for each criterion (1 = maximal efficacy, 5 = maximal effectiveness). A total of 7,166 reports were retrieved from databases and 159 were included in the full text. Though PRECIS-2 describes quantitative scoring in detail, evaluators were uncertain about several specific operationalizations and found high evaluator variation in the first independent ratings. After discussion and reaching consensus, inter-evaluator reliability improved. For PRECIS-2 ratings over time, there was no evidence that the design and performance of RCTs of CHM paid more attention to "efficacy" criteria after the implementation of PRECIS (all P > 0.05). More research is needed to establish the easiest and most useful tool to distinguish between effectiveness and efficacy results.