The relative importance of visual-spatial and verbal working memory for mathematics performance and learning seems to vary with age, the novelty of the material, and the specific math domain that is investigated. In this study, the relations between verbal and visual-spatial working memory and performance in four math domains (i.e., addition, subtraction, multiplication, and division) at different ages during primary school are investigated. Children (N = 4337) from grades 2 through 6 participated. Visual-spatial and verbal working memory were assessed using online computerized tasks. Math performance was assessed at the start, middle, and end of the school year using a speeded arithmetic test. Multilevel Multigroup Latent Growth Modeling was used to model individual differences in level and growth in math performance, and examine the predictive value of working memory per grade, while controlling for effects of classroom membership. The results showed that as grade level progressed, the predictive value of visual-spatial working memory for individual differences in level of mathematics performance waned, while the predictive value of verbal working memory increased.