Born in Greenville, South Carolina, Anderson lost both parents when he was four years old, and was sent to live at the Jenkins Orphanage in Charleston, where he learned to play trumpet. Classmates gave him the nickname "Cat" (which he used all his life) based on his fighting style. He toured and made his first recording with the Carolina Cotton Pickers, a small group based at the orphanage. After leaving the Cotton Pickers, Anderson played with guitarist Hartley Toots, Claude Hopkins' big band, Doc Wheeler's Sunset Orchestra (1938–1942), with whom he also recorded, Lucky Millinder, the Erskine Hawkins Orchestra, Sabby Lewis's Orchestra, and Lionel Hampton, with whom he recorded the classic "Flying Home No. 2".