Accurate segmentation of rectal tumors is a basic and crucial task for diagnosis and treatment of rectal cancer. To avoid tedious manual delineation, an automatic rectal tumor segmentation model is proposed. A pretrained Resnet50 model was introduced for feature extraction. To reduce the complexity of the model, all layers after the 13th residual block of ResNet50 were removed, and three side-output modules were added to the hidden layer of ResNet50 to guide multiscale feature learning. The final boundaries of tumors were determined by fusion of the predictions from side-output modules. The proposed model was compared with two other models, and the effects of different region of interest (ROI) sizes, loss functions, and side-output fusion strategy were also evaluated. The models were trained and evaluated on data from four clinical centers; T2-weighted magnetic resonance images (T2W-MRIs) from 461 patients in the first clinical center were used for training, while T2W-MRIs from 51 patients in the same clinical center and 56 patients in three other clinical centers were used for performance evaluation. The proposed model was superior to the two other models and achieved an average Dice similarity coefficient of 82.39%, sensitivity of 86.32%, specificity of 92.25%, and Hausdorff distance of 12.10 px. In addition, when the ROI contained rectal tumors, the smaller the ROI size, the higher the segmentation accuracy. For a certain ROI size, there were no considerable differences in segmentation results among the loss functions. Compared to the models with single side-output module, the proposed model performed better. The results show that the proposed model has potential clinical applications in rectal cancer treatment, particularly with regard to therapeutic response evaluation and preoperative planning. © 2019 American Association of Physicists in Medicine.