Originally from Tampa, Florida, Adderley moved to New York in the mid-1950s. His nickname derived from "cannibal", a title imposed on him by high school colleagues as a tribute to his voracious appetite.

His educational career was long established prior to teaching applied instrumental music classes at Dillard High School in Fort Lauderdale, Florida. Cannonball moved to Tallahassee, Florida, when his parents obtained teaching positions at Florida A&M University. Both Cannonball and brother Nat played with Ray Charles when Charles lived in Tallahassee during the early 1940s. Cannonball was a local legend in Florida until he moved to New York City in 1955, where he lived in Corona, Queens.

It was in New York during this time that Adderley's prolific career began. Adderley visited the Cafe Bohemia, where Oscar Pettiford's group was playing that night. Adderley had brought his saxophone into the club with him, primarily because he feared that it would be stolen, and he was asked to sit in as the saxophone player was late. That performance established his reputation.

Prior to joining Miles Davis' band, Adderley formed his own group with his brother Nat after signing onto the Savoy jazz label in 1957. He was noticed by Miles Davis, and it was because of his blues-rooted alto saxophone that Davis asked him to play with his group.

Adderley joined the Miles Davis Sextet in October 1957, three months prior to John Coltrane's return to the group. Adderley played on the seminal Davis records Milestones and Kind of Blue. This period also overlapped with pianist Bill Evans' time with the sextet, an association that led to recordingPortrait of Cannonball and Know What I Mean?.

His interest as an educator carried over to his recordings. In 1961, Cannonball narrated The Child's Introduction to Jazz, released on Riverside Records.