Oliver started in amateur theatricals as an adolescent in Britain before pursuing a career as a jockey. However, he failed to gain an apprenticeship owing to his size. He joined the Merchant Navy at 16 and travelled the world, eventually settling in Sydney in 1956. In Australia he found work as a stockman, spending three years in this job, working his way around the country. He subsequently returned to the UK before beginning an overland trek across Asia to return to Australia. However, the journey was halted by illness and he returned to Britain where Oliver contacted Albert Finney who was then planning to produce the film Ned Kelly in Australia in 1963. Finney gave him a letter of introduction to an agency in Sydney and Oliver emigrated in 1963, later becoming an Australian citizen. Oliver became a busy theatre and television actor in Australia. He had many guest starring roles on Australian drama series, appearing frequently in the top-rated Crawford Productions police dramas Homicide, Division 4, Matlock Police and Cop Shop, and in Crawford's adventure series Hunter (1967). Oliver was then cast in serial Bellbird, playing the role of Tom Grey from 1969 to 1971, and appeared in the 1971 film Nickel Queen, directed by John McCallum. After this he returned to television guest roles for Crawfords, and other companies. He also played two roles in the Gerry Anderson series UFO – a Doctor in the episode entitled "Confetti Check A-OK" as well as a SHADO technician in the episode entitled "The Sound of Silence".
Oliver also served a stint as a presenter on Play School in 1967.
In 1972 he joined the cast of fledgling soap opera Number 96 playing the role of Janie Somers' new beau Jack Sellars. Back-slapping rough Diamond Jack, nicknamed "Jolly Jack Sellars" was intended as a guest character to appear for a run of just three weeks, but the makers of the show were impressed with his performance and the character was made into an ongoing lead regular in the serial.
Oliver became one of the Number 96's most popular cast members. During his run in the show he married fellow Number 96 actor Lynn Rainbow, who played Sonia Lansgard and opened a wine bar in Kensington, Sydney cunningly named Jack's Cellar. In late 1973, along with much of the show's regular cast including Lynn Rainbow, he reprised his television role in a feature film spinoff of the serial. Oliver stayed in the role in the series for more than two years, finally electing to leave the series in mid 1974. After leaving the serial he quickly returned to television guest starring roles and roles of varying size in feature films. He briefly returned to Number 96 in the role of Jack Sellars in September 1975. He has been married to his current wife Jan Oliver since 1985.
Oliver's film roles included ABBA: The Movie (1977). His primary role in the film is as ABBA's gruff bodyguard; however, he additionally appears in the film as a barman and as a chatty moustached taxi driver shown mainly from behind. Through the late 1970s and the 1980s, Oliver appeared in guest and regular roles in many Australian drama series and serials, including Prisoner, Holiday Island and Sons and Daughters.
Today Oliver is most known for his long-running role of Lou Carpenter in Neighbours. He first appeared for a handful of episodes as lovable rogue used-car salesman Lou, Madge Bishop's former flame, in 1988. In early 1992 the character was reintroduced to the series and has been a key character ever since. In 1996, Oliver was written out of the show but producers relented after numerous petitions from fans and he was quickly reintroduced. By 2009, he was the show's longest serving character, both in continuous and overall duration. From the 2009 season, Oliver reduced his role on Neighbours to a part-time regular member of the cast. Oliver appeared in a documentary special celebrating the show's 30th anniversary titled Neighbours 30th: The Stars Reunite, which aired in Australia and the UK in March 2015. In 2015, it was announced that Oliver had cut back further on his role as Lou and would only appear from time to time as a guest. In October 2016 it was announced that Oliver had quit the role after 28 years and his last appearance aired in December 2016.