Retinal dehydrogenase 5 (RDH5) is an important enzyme in the visual cycle. Several studies have reported that the RDH family may play crucial roles in tumor prognosis. However, the role of RDH5 in tumor prognosis is still unclear. We examined the mRNA level of RDH5 by using q-PCR in hepatocellular carcinoma (HCC) and adjacent non-cancerous tissues. The proliferation rate of HCC cells was detected by MTS assay, and the invasive ability was examined by transwell and scratch wound assays. The YAP protein localization and expression were visualized by immunofluorescence in two different cell lines. CpG islands in the promoter region were predicted by using the methprimer database. Clinical characteristics of a patient cohort data came from The Cancer Genome Atlas database. RDH5 was significantly downregulated in hepatocellular carcinoma tissues, and low RDH5 expression was associated with metastasis and poor patient prognosis. Functional assays revealed that the RDH5 promoter is methylated in HCC cell lines. Moreover, overexpressing RDH5 can suppress metastasis by reversing the epithelial-mesenchymal transition (EMT) process, and RDH5 also inhibits cell proliferation in HCC cell lines. Furthermore, suppressing RDH5 can activate the Hippo/YAP signaling pathway and promote the nuclear translocation of YAP. Clinical data demonstrated that RDH5 is an independent prognostic factor in HCC. In our study, we provided the first evidence that RDH5 plays a crucial role in suppressing proliferation and metastasis, and the RDH5 promoter is methylated in hepatocellular carcinoma. And as an important regulator, RDH5 can suppress the Hippo/YAP signaling pathway. Taken together, it revealed that RDH5 might be a potential therapeutic target in HCC patients. © The author(s).