James first went to the U.S. Air Force. In 1966 he visited New Orleans where musicians such as Nat Perillat, James Black, Roger Dickerson, Chuck Beatty, Germaine Basel, Richard Payne and Trevor Khoeler lasting impression left with him. After discharge from the Air Force studied at the University of Chicago with Rudolf Fahsbender. In 1969 he went to New York City where he worked at the post office and next to the Mannes College for Music with Julius Levine studied. Here he met Pharoah Sanders, with whom he played his first jazz concerts in New York. Soon he also worked with Alice Coltrane and Albert Ayler, with whom he undertook in 1970 a Canada tour. In 1971 he met the singer Melba Moore, with whom he appeared as part of the David Frost Revue.
In 1973, he was the first time with Gary Bartz to Europe. In 1974 he became a member of Art Blakey's Jazz Messengers. In 1976 he played with Dave Burrell at the wildflowers Loft Sessions. At the same time he met Betty Carter, with whom he formed a lifelong friendship and Woody Shaw, with whom he studied classical compositions like Bach's cello suites.
In 1976, James his own quartet, with whom he also toured Europe. There he met Dexter Gordon, with whom he recorded the album Homecoming in New York. Even with guitarist John Scofield, he was on tour in Europe and recorded the album Rough House. After the album Stafford James, he recorded his second album as a band leader with The Stafford James Ensemble 1978.
In the early 1980s he toured Australia, New Zealand, Japan, Mexico and South America. 1983 began working with Jimmy Heath. With the Woody Shaw Quintet, he has participated in a tour of the United States Information Service through North Africa, India and Syria.
In 1986, James' composition Ethopian suite for two double basses, strings ensemble, percussion and dancers, which was performed as part of the Celebrate Brooklyn Dance Festival. In addition, he arranged this year compositions by Duke Ellington for themselves, pianist Mulgrew Miller and cellist Akua Dixon. In 1987 he composed a Sonatina for Viola d' Amore and double bass, which was premiered in the U.S. in the Bates Recital Hall and at the European Music Festival 1988 in Stuttgart.
1989 James moved to Paris, where he performed with artists such as Pharoah Sanders, Barney Wilen, Monty Waters and Lavelle and founded the group Stafford James Project. In 1991 he took on the Ukrainian National Orchestra Igor Stravinsky's Firebird. In 1994 he was a juror at the Leverkusen Jazz Festival and played seventeen compositions for the WDR. In addition, he also worked as a trio, inter alia, with pianist Onaje Allen Gumbs and drummer Ronnie Burrage. In 1995 he composed Les Alpes aux Carpates for double bass, two panpipes, string orchestra, women's choir, synthesizers and drums.
In 1998 he undertook a tour through Belgium, Holland, Germany, Austria and Italy in the Quartet with Buster Williams, Miquel Diaz ( later Don Alias ) and Ronnie Burrage (later Jeff Boudreaux ).