Nancy Scott was most likely the daughter of Monticello’s butler Burwell Colbert and his wife, Critta Hemings. In the late 1820s, Jefferson’s grandson Thomas Jefferson Randolph sold her to Albemarle County clerk Alexander Garrett, who in 1842 sold her to her husband, Robert Scott. The Scotts had at least nine children and lived on Charlottesville’s main street.
Nancy Scott’s oldest child, Elizabeth, was the subject of a remarkable story preserved orally by her descendants and validated by historical records. It states that Elizabeth, who took the surname of her stepfather, Robert Scott, was Thomas J. Randolph’s daughter and was sold to the Garrett family. She was recaptured, after trying to run away, and put on the auction block and sold to a Dr. Cox, who made her his mistress. They had five children, one of them Nannie Cox Jackson, a noted Charlottesville educator and community leader. When Emancipation came, in 1865, Elizabeth Scott determined to live independently and supported her family by working as a dressmaker.