Wormley Hughes was the oldest son of Betty Brown; his father has not been identified. As a boy, he worked in the Monticello house and the Mulberry Row nailery. He became head gardener, preparing flower beds and planting seeds, bulbs, and trees. He also had charge of the valuable carriage and saddle horses in the Monticello stables. He dug the grave of his master, who had called him "one of the most trusty servants I have."
Wormley Hughes and his wife, Ursula Granger, a niece of Isaac Granger Jefferson, had twelve children. Hughes was informally freed by Jefferson’s daughter Martha Randolph, while the rest of his family was sold at the 1827 dispersal sale. Ursula and some of their children were acquired by the Randolphs, for whom Hughes continued to work. The Randolphs long remembered one of his expressions: "I am in no wise discouraged." Wormley and Ursula Hughes’s descendants include several ministers, as well as farmers, gardeners, blacksmiths, teachers, and archivists.