A native of Pennsylvania, Anderson served as an officer in the Revolutionary War on the side of the Continental Army. For most of the war he held the office of paymaster in his regiment.
After the war he embarked on a legal career and was admitted to the Bar Association of Delaware. In 1791 he received the appointment as federal judge for the Southwest Territory, later from amongst others, the state of Tennessee. He was even five years on a member of the first Constitutional Convention of Tennessee.
1797 Anderson became politically active when the state Legislature of Tennessee elected him U.S. Senator. His predecessor, William Blount was expelled from the Congress. His original term ended on March 3, 1799; he remained for a further election by the Tennessee State Assembly in the Senate and took there now as the successor of Daniel Smith the second seat of the state a. After two more elections, he retired at his own request from Congress ultimately on 3 March 1815. During this time he was 1803-1804 president pro tempore of the Senate.
His political career was so but not yet completed. Joseph Anderson remained in Washington and was appointed Comptroller of the Treasury, a high-ranking position in the U.S. Treasury Department, which he held until 1836. The following year, he died in Washington.
He was buried in the Congressional Cemetery. In his honor, the Anderson County was named after him in Tennessee. His son, Alexander O. Anderson was also later Senator for Tennessee.