Tuberculosis (TB) is the number one cause of death world-wide due to infection. 2 billion are infected and 10 million die each year. Understanding the immune response to TB is crucial to developing vaccines and improving treatment strategies. The immune response to infection with Mycobacterium tuberculosis (Mtb), the bacteria that causes TB, results in the formation of granulomas, spherical lesions, physically containing while immunologically restraining the bacteria. These lesions are a collection of immune cells, bacteria, dead tissue and other products including lipids, which serve as a food source and oxygen products. Two key adaptations of Mtb are a non-replicating phenotype and accumulation of lipid inclusions in response to hypoxic conditions. To explore how these adaptations influence granuloma-scale outcomes in vivo, we present a multiscale in silico model of granuloma formation in tuberculosis. We build on an Agent-based model, GranSim, that we have been curating continuously since 2004 and include a flux balance model (FBM) describing the metabolism of Mtb within GranSim creating a multi-scale model. The model comprises host immunity, Mtb metabolism, Mtb growth adaptation to hypoxia, and nutrient diffusion. We calibrated our model to in vivo data from nonhuman primates and rabbits and apply the model to predict Mtb population dynamics and heterogeneity within granulomas. We found that bacterial populations are highly dynamic throughout infection in response to changing oxygen levels and host immunity pressures. Our results indicate that a non-replicating phenotype, but not lipid inclusion formation, is important for long-term Mtb survival in granulomas.