Here, we propose two different types of contrastive latent variable models to create a richer portrait of differential expression in sequencing data. These models disentangle the sources of transcriptional variation in different conditions, in the context of an explicit model of variation at baseline. Moreover, we describe a model-based hypothesis testing framework in the context of count data that can test for global and gene subset-specific changes in expression. We validate our model through extensive simulations and analyses with gene expression data from perturbation and observational sequencing experiments. We find that our methods can effectively summarize and quantify complex transcriptional changes in case-control experimental data. We then describe an extension to this model, called contrastive regression, that can be applied to data in which there is a continuous covariate associated with the case data, such as disease severity, treatment dose, or time, to characterize multivariate associations capturing variation exclusive to the case covariates.