Gordon was born in Fitchburg, Massachusetts. His first appearance on Broadway was in 1937 in the musical drama The Fireman’s Flame. From 1941-1945, he played the role of Officer Klein alongside Boris Karloff in the original cast of Arsenic and Old Lace on Broadway. He had an "Introducing" credit in the 1949 Marx Brothers film Love Happy.
In 1957, he guest starred on the ABC western series Tombstone Territory, starring Pat Conway and Richard Eastham, in the episode "Killer Without a Conscience". About this same time, he guest starred on Barry Sullivan's adventure/drama series, Harbormaster, set in maritime New England.
His film credits included roles in The Buccaneer (1958), Curse of the Undead (1959), Key Witness (1960), Roger Corman's Tower of London (1962), Hello Down There (1969) and Timerider: The Adventure of Lyle Swann (1982). He also worked with Corman again, playing the nasty Colonel Waxman in the cult classic Piranha (1978), starring alongside Bradford Dillman and Kevin McCarthy.
In 1959, Gordon starred, alongside Pernell Roberts, as Capt. Emil Tremaine in an episode of ABC's One Step Beyond entitled The Vision.
In 1960–1961, Gordon appeared as "Mercer" in two episodes of NBC's Outlaws Western series starring Barton MacLane. He was cast at that time in the NBC anthology series, The Barbara Stanwyck Show, and in the NBC sitcom, Car 54, Where Are You?
Gordon appeared twice in 1961 on ABC's Adventures in Paradise, starring Gardner McKay; he was cast as Stevens in "Mr. Flotsam" and as Red Munce in "Adam San".
In 1964, Gordon guest-starred in the episode "Between the Rats and the Finks" of CBS's drama series, Mr. Broadway, starring Craig Stevens, with fellow guest stars Larry Hagman and Dyan Cannon.
From 1965 to 1968, Gordon appeared in several episodes of the long-running prime-time soap opera Peyton Place alongside Actress Lee Grant as Gus Chernak, the alcoholic and vengeful father of Grant's character Stella Chernak. In 1966, Gordon costarred with trumpet player Jack Sheldon in the 16-segment CBS sitcom, Run, Buddy, Run, about the fictitious Buddy Overstreet who is on the run from the mob after "Buddy" overhears "Mr. D", played by Gordon, plotting the murder of a fellow gangster. In 1966, he and Robert Stack appeared together in an episode of The Lucy Show spoofing their roles from The Untouchables. In 1968, he played the security man in the "Sour Note" episode of It Takes A Thief, starring Robert Wagner.
Gordon retired from acting after playing himself in the 1989 film Ernest Goes to Splash Mountain, though he was the executive Producer of the Australian telefilm Feds: the Betrayal (1996) and Producer of the US/Chinese fantasy martial arts film Warriors of Virtue: the Return to Tao in 2002. For a time, he operated a dinner-and-show restaurant in Scottsdale, Arizona, called "Frank Nitti's Place", and in the early 1980s, a pizza restaurant of the same name in Kansas City, Missouri. He greeted patrons at the door in his typical pin stripe suit with a carnation in the lapel.
In 2003, he was reportedly unable to attend the funeral of Untouchables co-star Robert Stack because of poor health. Eight years later, Gordon died after a lengthy illness; he was two weeks shy of his 95th birthday. At the time of his death, Gordon lived in Santa Fe, New Mexico, with his wife Marla.