The thoracic aorta carries blood from the heart to the rest of the body. When this blood vessel becomes enlarged, it can rupture, leading to sudden cardiac death. Understanding the genetic basis for aortic size may permit novel therapies and in principle could identify individuals at high risk who should be screened. Here, we used deep learning to perform semantic segmentation in order to annotate magnetic resonance images of the thoracic aorta, and applied classical computer vision techniques for postprocessing and quality control. This approach permitted a genetic analysis that increased the number of significantly associated loci by nearly 10-fold compared to prior studies. We also provide proof of principle that a polygenic score identifies individuals at elevated risk compared to the general population, which may ultimately pave the way for identifying individuals who would benefit from screening.