Eldridge played professionally from the age of 16, first with a touring carnival (where he imitated Coleman Hawkins' well-known tenor saxophone solo in Stampede) and later with obscure Midwestern bands. In 1930 he moved to New York and played in various dance bands in Harlem, including that of Teddy Hill; in 1932 he began a serious study of Louis Armstrong's style. From 1933 he worked in Pittsburgh and then in Baltimore before returning to New York, where his first recorded solos with Hill in 1935 immediately attracted attention; later that year he joined Fletcher Henderson's orchestra as head trumpeterand occasional singer.

In autumn 1936 he formed his own eight-piece band in Chicago with his older brother Joe Eldridge as saxophonist and arranger; the group broadcast nightly, and Eldridge took advantage of his position as leader to record several outstanding extended solos, including After You've Gone and Wabash Stomp. After a brief period studying radio engineering in 1938 Eldridge formed a ten-piece band, which the following year began a residency in New York at the Arcadia Ballroom and later at Kelly's Stable.

By this time Eldridge was widely regarded as the outstanding jazz trumpet soloist of his time, and he began to receive liberal offers from white swing bands. In 1941 he joined Gene Krupa, becoming one of the first black jazz musicians to be accepted as a permanent member of the brass section of a white big band. While with Krupa he recorded his celebrated ballad performance of Rockin' Chair and became a nationwide celebrity, particularly in a novelty hit, Let Me Off Up Town, with Anita O'Day. When Krupa's band broke up in 1943, Eldridge played as a freelancer and led his own band in New York for a while before taking a position in Artie Shaw's band in 1944. A year later, after many racial incidents had occurred while the band was on tour, he left Shaw to organize a big band of his own. Like most large jazz ensembles at this time, his group was financially unsuccessful, and Eldridge soon reverted to small group work. In 1948 he began a long association with Norman Granz's Jazz at the Philharmonic.

Although in the early 1940s Eldridge had taken a leading part in the jam sessions at Minton's Playhouse in New York, which later crystallized in bop, he was out of sympathy
with that style, and by the late 1940s his music was considered old-fashioned. In a crisis of confidence he moved to Paris in 1950 while on tour with Benny Goodman.

During his year in Paris he was lionized by the French jazz public, and made some of his finest recordings, including a version of Fireworks in a duo with Claude Bolling in which the two men reworked the ideas shared by Armstrong and Earl Hines in their recording of the same title (1928). After returning to the USA in April 1951 he joined the burgeoning mainstream jazz movement, performing in small groups with Benny Carter, Johnny Hodges, Ella Fitzgerald (1963-5), and, notably, Coleman Hawkins, with whom he made several outstanding albums for Verve. From 1970 until 1980, when he was incapacitated by a stroke, he led a traditional group at Ryan's in New York. Thereafter he performed occasionally as a singer, drummer, and surprisingly competent pianist.