The manor of Hilborough passed by descent through the Kailly or Caly, Clifton and Knyvett (of Buckenham) families during the medieval period. It was sold by Sir Philip Knyvett to Robert, Lord Rich, in 1611 and he as Earl of Warwick conveyed it in 1619 to Sir Ralph Hare.
It remained in the Hare family of Stow Bardolph until 1719 when Thomas Hare sold it to James Nelthorpe of Lynford Hall. About 1765 Ralph Caldwell or Cauldwell, steward of the Earl of Leicester and his tenant at Godwick Hall, purchased from the Nelthorpes. Caldwell reorganised the landscape at Hilborough and built the present Hall and the family remained in possession until Henry Berney Caldwell sold to the Duke of Wellington in 1858. Ralph Caldwell owned a small estate near Godwick, at Whissonsett, and its records are also present.
The neighbouring Clermont Estate in Threxton and Little Cressingham was created by the Irish peer William Henry Fortescue (1722-1806), Earl of Clermont, friend of the Prince of Wales and a noted sportsman. He built Clermont Lodge as a shooting box and it was extended by his nephew and heir William Charles Fortescue (1764-1829), Viscount Clermont. Following Viscount Clermont's death without issue the estate was sold to Sir Francis Lyttleton Holyoake Goodricke in 1844 and in 1858 it was purchased by the Duke of Wellington. In 1863 Arthur Richard, Duke of Wellington, conveyed the Hilborough and the Clermont Estates to John Remington Mills.
The Mills (originally Milne) family came from in Yorkshire but thrived as weavers in London in the 18th and early 19th centuries. The two sons of Samuel Mills (died 1847) of Russell Square both bought country estates: Thomas Mills, barrister and MP, at Tolmers in Hertfordshire (which his brother inherited) and John Remington Mills the Hilborough and Clermont Estates as stated above, retaining property and business interests in London and elsewhere. In 1867 he bought the Leheup estate at Ashwell in Hertfordshire, and in 1869 the Watton estate and an outlying farm at Tottenhill in Norfolk and property in Knaptoft, Leicestershire. His son Joseph Trueman Mills added the South Pickenham estate, Norfolk, and also acquired Stockgrove near Leighton Buzzard, Bedfordshire.
The Norfolk property came to Major John Charles Trueman Mills on the death of his uncle Henry Trueman Mills in 1933 and the records were deposited in the Record Office in 1987 after the death of his widow Ida Betty Mills and the sale of the estate.