Annefield was built about 1790 by Mathew Page, originally of Broadneck, Hanover County. The tract, originally 2258 acres, is located in Clarke County to which Mathew Page and his brother came to settle from Hanover. Mathew Page married Anne Randolph Meade in 1787 according to the Page family genealogy. She was the sister of Bishop William Meade and it was probably for her that Annefield was named. Mathew Page planned both the house itself and its lovely gardens. Some of the excellent interior carving is purported to have been done by Hessian soldiers who wandered now the Valley following the Revolution.

Mathew Pages' daughters sold Annefield following their father's death to Mr Thomas Carter. Mary Custis, wife of Robert E Lee, was born at Annefield in 1808 when her mother's coach stopped there during a journey. Mrs Thomas Carter supposedly freed thirty-four of her own slaves at Annefield and colonized them in Liberia.

The son of Thomas Carter, William Page Carter, grew up at Annefield and later, after serving in the Confederate Army, went on to become a noted poet, his poems mostly being about the Civil War. The most outstanding of his works was a collection of poems, Echoes of the Glen.

Annefield is a pleasing well-proportioned mansion which combines a basic eighteenth-century house design with some of the refinements of the Federal period. Although many features of the building would tend to place its stylistic date toward the middle of the eighteenth century, such things as the rather low silhouette of the roof, the Adamesque feature of the north east room, and the narrow muntins of the window sash reveal the structure's true 1790 date. Its position at the brow of a broad hill in the rolling slopes of the lower Shenandoah Valley gives it both a commanding view of the countryside and a justifiably prominent site. - NRHP, 12 November 1969