There have been rapid advances at the intersection of AI and medicine over the last few years, especially for the interpretation of medical images. In this talk, I will describe three key directions that present challenges and opportunities for the development of deep learning technologies for medical image interpretation. First, I will discuss the development of transfer learning and self-supervised learning algorithms designed to work in low labeled medical data settings. Second, I will discuss the design and curation of large, high-quality datasets and their roles in advancing algorithmic developments. Third, I will discuss the real-world impact of AI technologies on clinicians’ decision making and subtleties for the promise of expert-AI collaboration. Altogether I will summarize key recent contributions and insights in each of these directions with key applications across medical specialties.