Rich was born in Manhattan to Jewish vaudevillians Bess (née Skolnik) and Robert Rich. His talent for rhythm was first noted by his father, who saw that Buddy could keep a steady beat with spoons at the age of one. He began playing drums in vaudeville when he was 18 months old, billed as "Traps the Drum Wonder." At the peak of Rich's childhood career, he was reportedly the second-highest paid child entertainer in the world (after Jackie Coogan).

At age 11 he was performing as a bandleader. He received no formal drum instruction and went so far as to claim that instruction would only degrade his musical talent. He also never admitted to practicing, claiming to play the drums only during performances and was not known to read music.