Robert Scott was born free, the son of Sarah Bell and Jesse Scott, a free man of color whose mother was a Pamunkey Indian. The Scott trio (Robert, his brother James, and his father) were well known musicians who traveled all over Virginia playing at dances at private homes, mountain resorts, and the University of Virginia
Scott married Nancy Colbert, probably the daughter of Burwell and Critta Colbert of Monticello. He was able to purchase her and some of their nine children out of slavery. In 1857 Robert Scott, who had more than three-quarters white ancestry, successfully petitioned the court to be declared "not negro"–an intermediate status between white and black or "mulatto."
Robert Scott lived in the Bell-Scott house on Charlottesville’s main street for almost ninety years. He was a rich source of recollections about Jefferson and, at his death, was described as "a man who in the course of a long life never failed to command the respect of his fellow citizens."