Multimodality medical imaging has played a significant role in lesion diagnosis and characterization. However, there are remaining challenges in the procedure of multimodality feature fusion based lesion characterization. First, large inter-modality variations make it difficult to harness the complementary information between modalities for better characterization. Subsequently, large intra-class and small inter-class variations due to the heterogeneity of neoplasm makes the classification more challenging. Finally, the relative importance of modalities for the characterization has not been thoroughly investigated, easily resulting in non-optimal fusion performance. In this study, we propose an attention guided discriminative and adaptive fusion (AGDAF) method based on deep learning architecture to address above three problems. Specifically, we first design a novel cross-modal intra- and inter-attention module to focus on learning both the intra-modality relations and inter-modality relations. Then, we introduce a discriminative feature learning loss to reduce the distance of features in the same class and increase the distance of features in different classes of neoplasm in single modalities. Finally, we design an adaptive weighting strategy to increase the contribution of modalities with relatively lower loss values and reduce the impact of modalities with large loss values for the final loss function. Experimental results of grading clinical hepatocellular carcinoma demonstrate that the proposed method significantly outperforms the previously reported multimodality feature fusion methods. In addition, ablation study also demonstrates the effectiveness of the proposed cross-modal intra- and inter-attention module, discriminative module, and adaptive weight adjustment module for multimodality feature fusion in lesion characterization.Copyright © 2022 Elsevier Ltd. All rights reserved.