Conventional feature selection methods based on clustering have been developed and optimized to focus on samples-based clustering, but less work has been done for features-based clustering. Especially, it is impossible to achieve prefect prediction results for the relatively small number of samples in high dimensional data, such as gene expression data. To overcome this problem, this paper proposes an efficient algorithm, K-Gravity, that groups interdependent features into clusters. Each feature cluster is treated as a single entity for classification evaluation. Unlike previous work that selects a subset of top feature groups from each cluster or top feature groups from all clusters to make up feature pools in final classification, a new classification evaluation method, Embedded Classification Learning (ECL) picks up some top feature groups and builds a classifier on each selected feature groups. The experiment results present that the proposed methods can achieve better feature clustering in final classification than conventional samples-based methods, such as K-Means clustering. Also the proposed evaluation methods, Embedded Classification Learning (ECL), can pay more attention to diversity between different feature groups and improve final classification accuracy further.