Joseph Fossett was a son of Mary Hemings Bell (daughter of Elizabeth Hemings). Bell lived in Charlottesville as a free person after Jefferson sold her to her white common-law husband, though he  refused to allow Bell to purchase her oldest children, Joseph and Betsy; they remained enslaved at Monticello. Jefferson deemed Joseph Fossett a particularly efficient nail-maker and an effective house servant and had him trained as a blacksmith at age 16. As head blacksmith during Jefferson’s retirement, Joseph Fossett worked at an anvil in the blacksmith’s shop.

During Jefferson’s presidency, Fossett’s wife, Edith Hern Fossett,  was taken to Washington to be trained in the art of French cuisine. Three of the Fossetts’ ten children were born in the White House. In 1806, Jefferson sent an agent to recapture  Joseph Fossett who   left Monticello without Jefferson’s permission. Jefferson failed to realize Fossett was running to his wife in Washington. When Jefferson retired, the Fossetts were made head chef and head blacksmith at Monticello. Jefferson freed Joseph Fossett in his will, one of only five men to be freed in the will.  Jefferson did not free Edith Fossett or their children, and the family was separated and sold at auction by Jefferson’s heirs.