Traditional templates matching methods have achieved good accuracy in markerless tumor tracking (MTT) on a Linac with the on-board imaging system. Nevertheless, its computational complexity hinders us from marching towards real-time tracking. Therefore, we aimed at evaluating both the accuracy and speed of our novel MTT method using Faster-RCNN, which is the most sophisticated RCNN based model trained via supervised learning. To this end, we use a dynamic thorax phantom that simulates the human chest anatomy and tumor motion in the lung. To feed the network model enough data, we generate digitally reconstructed radiography (DRR) from clinically used four-dimensional(4D) CT as the training dataset. Restricted by this, fluoroscopy images are preprocessed by histogram matching to ensure the maximum similarity between training dataset and testing dataset. Tracking errors were 0.92mm, 0.85mm, and 0.43mm correspond to target diameters of 12mm, 18mm, 27mm, respectively. The average end to end latency was around 220ms. In conclusion, the results verified that our proposed method was qualified for real-time MTT in the case of phantom study for its balance of accuracy and speed. © 2019 IEEE.