In 1803, Captain Walker gifted his son-in-law, Captain William Jefferds, Jr. “80 square rods of land, with love and affection.” In 1804, the two-story, Federal-style building was completed. Captain Jefferds and his wife, Sarah, had 11 children in the home, and lived there until the late 1800s. In 1944, the home was purchased by the Dougherty family, who converted the carriage house suites into rented apartments. The following owner, Constance Willy, was a rare book collector and reportedly was the wealthiest woman in Maine. She lived alone in the house from 1963 to 1980, after which the house became an inn.