William Samuel Flemming Sr. purchased a large parcel of property from .Thomas Fitch in 1817. In 1835 he lost everything during the Second Seminole War. The Halifax area was abandoned until after the Civil War.
In 1876 William Samuel Flemming and Mary Flemming were residents of New Castle County, Delaware. The Flemmings still owned most of the land now comprising Holly Hill and William Flemming 's land holdings continued from there south through Port Orange, where he and his wife Mary lived. In 1877 he owned 4,000 acres on the Halifax River between Ormond and the recently settled site of Daytona.
In the summer of 1876 Flemming went to Philadelphia, which was preparing for the country's centennial, with the express purpose of influencing settlers to come to Florida. He got the promise of fifteen families.
The Flemmings and began to construct a simple frame dwelling on a portion of his river front property. The land was cleared just south of the Holly Hill Canal right on the river shore. Unfortunately William Flemming died on December 22, 1877 before construction was completed.