AIDS, a serious fatal disease caused by the human immunodeficiency virus (HIV), is an epidemic disease for which no effective vaccine has been established. The current therapeutic interventions for AIDS have limited efficacy because they are unable to clear HIV infections and the continuous occurrence of resistant HIV strains. Therefore, the exploitation of new drugs to prevent the spread of AIDS remains a high priority. In this study, the effects of icariin and its metabolite anhydroicaritin on SIV/HIV replication were investigated. In CEM x 174 cells and PBMC cells, both icariin and anhydroicaritin can significantly inhibit HIV-1 or SIVmac251 replication. Furthermore, molecular docking studies revealed that icariin and anhydroicaritin can act on both HIV reverse transcriptase and protease but could not bind to integrase. Reverse transcriptase and protease inhibition biological assays showed that both icariin and anhydroicaritin could significantly inhibit only HIV reverse transcriptase. In summary, the two compounds can significantly inhibit HIV/SIV in vitro and their targets may be mainly involved with HIV reverse transcriptase.