Complex animal behaviors are likely built from simpler modules, but their systematic identification in mammals remains a significant challenge. We use depth imaging to show that three-dimensional (3D) mouse pose dynamics are structured at the sub-second timescale. Computational modeling of these fast dynamics effectively describes mouse behavior as a series of reused and stereotyped modules with defined transition probabilities, which collectively encapsulate the underlying structure of mouse behavior within a given experiment. By deploying this 3D imaging and machine learning method in a variety of experimental contexts, we show that it unmasks potential strategies employed by the brain to generate specific adaptations to changes in the environment, and captures both predicted and previously-hidden phenotypes induced by genetic or neural manipulations. Further, we demonstrate its utility in automatically unblinding the behavioral effects of pharmacological manipulation. This work demonstrates that mouse body language is built from identifiable components and is organized in a predictable fashion; deciphering this language establishes an objective framework for characterizing the influence of environmental cues, genes and neural activity on behavior.