Development of high-throughput genotyping platforms provides an opportunity to identify new genetic elements related to complex cognitive functions. Taking advantage of multi-level genomic analysis, here we studied the genetic basis of human short-term (STM, n = 1623) and long-term (LTM, n = 1522) memory functions. Heritability estimation based on single nucleotide polymorphism showed moderate (61%, standard error 35%) heritability of short-term memory but almost zero heritability of long-term memory. We further performed a two-step genome-wide association study, but failed to find any SNPs that could pass genome-wide significance and survive replication at the same time. However, suggestive significance for rs7011450 was found in the shared component of the two STM tasks. Further inspections on its nearby gene zinc finger and at-hook domain containing and SNPs around this gene showed suggestive association with STM. In LTM, a polymorphism within branched chain amino acid transaminase 2 showed suggestive significance in the discovery cohort and has been replicated in another independent population of 1862. Furthermore, we performed a pathway analysis based on the current genomic data and found pathways including mTOR signaling and axon guidance significantly associated with STM capacity. These findings warrant further replication in other larger populations.