According to Virginia law, slaves freed after May 1806 were required to leave the state within one year or face reenslavement. From that time the number of manumissions (the legal freeing of slaves) dropped to a mere trickle. Although the 1806 law was not methodically enforced, Jefferson and other slaveholders considered it an impediment to freeing their human property. "The laws do not permit us to turn [our slaves] loose," wrote Jefferson in 1814. Thus, when he bequeathed freedom to five men in 1826, Jefferson petitioned the Virginia legislature for a special exemption from the law.
Efforts to rid the state of a population of free people of color numbering almost 50,000 by 1830 increased over the century. In 1833, in the wake of the Nat Turner rebellion, the governmental authorities sent a powerful message to Virginia's free blacks. They appropriated funds for colonizing freed slaves to West Africa and called for a special census in which all the free people of color in Virginia would be specifically asked about their willingness to emigrate. All 452 Albemarle County free blacks, including Joseph Fossett and Sally, Madison, and Eston Hemings, declined the offer.
As national tensions over slavery continued to intensify, so did Virginia's efforts to eliminate former slaves and other free blacks. 1850, the year of the Fugitive Slave Act, brought a wave of antipathy towards African Americans, which led to a tightening enforcement of the 1806 removal act and a larger appropriation of funds for transporting blacks to Africa. This tide of opinion broke over a former Monticello slave. Sally Cottrell Cole's right to remain in Virginia was brought before the court in November, when, as one of her advisers wrote, "there has been a general examination into the subject of Free Negroes remaining in this State." The heightened insecurity of free blacks in the south in 1850 becomes strikingly apparent in the elaborate legal arrangements undertaken to assist her effort to remain.
Most of the members of the Hemings family who were protected by special legislative permission to stay in Virginia had long since left for Ohio. In the last months of the year, those who remained (Tucker and Ann-Elizabeth Isaacs and Robert and James Scott) and dozens of other free blacks were brought before the court to answer the charge of "remaining in the Commonwealth without leave" and to make their cases for staying in Albemarle County. For the moment they escaped the harsh decree issued to some of those present in the court that day: leave the county within ten days or go to jail. In the case of Ann-Elizabeth Isaacs, the usually laconic clerk added this sentence to his entry: "No permission has been granted her to remain in the Commonwealth." These disturbing December incidents at the courthouse must have sealed her family's decision to sever its ties with Charlottesville forever.