His most notable film roles are Carmine Rosato in The Godfather Part II (1974) and Albert Volpe in The Godfather Part III (1990). He is one of three actors to play two different roles in the Godfather films (the others being Frank Sivero, who played a young Genco Abbandando in Godfather Part II and a bystander to the fight between Sonny Corleone and Carlo Rizzi in The Godfather, and Sofia Coppola, who played Mary Corleone in Godfather Part III and the infant son of Carlo and Connie baptized in the final scenes of The Godfather as well as a child on the ship at the beginning of Godfather Part II).
On television he portrayed Dan Valenti on Phyllis from 1976-77. In 1978, he appeared as Di Carlo in the episode "The Intimate Friends of Janet Wilde" of the NBC crime drama series The Eddie Capra Mysteries. Between 1982 and 1983, Caridi played Angelo Martelli, Bruno Martelli's father, in the TV series Fame. Caridi went on to play Detective Vince Gotelli in the television series NYPD Blue from 1993-99.
According to studio executive Robert Evans in his 1994 memoir The Kid Stays in the Picture, Caridi was Director Francis Ford Coppola's first choice for the role of Sonny Corleone, but Evans insisted on James Caan because Caridi was too tall to play opposite Al Pacino. He appeared in the cult movie classic Kiss Meets the Phantom of the Park (1978), playing amusement park manager Calvin Richards, and appeared as Sam Giancana in the 1992 film Ruby.
The Academy of Motion Picture Arts and Sciences announced on February 3, 2004 that it was expelling Caridi for violating his agreement to safeguard their screeners, making him the first Academy member to be expelled.