James was born in Maidenhead, Berkshire, to a cardiologist father. She was educated at Downe House, a girls' independent school in Newbury, Berkshire. After graduating from the Drama Centre London in 1973, she began her career in repertory theatre.
James has been nominated four times for the BAFTA TV Award for Best Actress; for Dummy (1977), The Jewel in the Crown (1984), Band of Gold (1995) and The Sins (2000). She is also known as Mrs. Pincher, the breastfeeding mother in Little Britain (2004). Her film credits include Gandhi (1982), The Tall Guy (1989), Sherlock Holmes (2009), Alice in Wonderland (2010), The Girl with the Dragon Tattoo (2011) and 45 Years (2015).