Immunotherapy is a standard treatment for many tumor types. However, only a small proportion of patients derive clinical benefit and reliable predictive biomarkers of immunotherapy response are lacking. Although deep learning has made substantial progress in improving cancer detection and diagnosis, there is limited success on the prediction of treatment response. Here, we aim to predict immunotherapy response of gastric cancer patients using routinely available clinical and image data.We present a multi-modal deep learning radiomics approach to predict immunotherapy response using both clinical data and computed tomography images. The model was trained using 168 advanced gastric cancer patients treated with immunotherapy. To overcome limitations of small training data, we leverage an additional dataset of 2,029 patients who did not receive immunotherapy in a semi-supervised framework to learn intrinsic imaging phenotypes of the disease. We evaluated model performance in two independent cohorts of 81 patients treated with immunotherapy.The deep learning model achieved area under receiver operating characteristics curve (AUC) of 0.791 (95% CI 0.633-0.950) and 0.812 (95% CI 0.669-0.956) for predicting immunotherapy response in the internal and external validation cohorts. When combined with PD-L1 expression, the integrative model further improved the AUC by 4-7% in absolute terms.The deep learning model achieved promising performance for predicting immunotherapy response from routine clinical and image data. The proposed multi-modal approach is general and can incorporate other relevant information to further improve prediction of immunotherapy response.Copyright © 2023 Elsevier B.V. All rights reserved.