The present house incorporates fragments of the Cistercian Nunnery called White Ladies, founded by Bishop Cantelupe, the friend of Simon de Mountfort, in 1250. Bishop Gifford, Cantelupe's successor, added to the endowment's and gave land bought from the de Flagge family. Some accounts say that Alice Flagge entered the Convent and brought to the Nunnery lands leading up to Perdiswell, part of which, after 700 years, is still called Flagge Meadow. The Nunnery was further endowed by 53 acres of land at Aston, which was afterwards called White Ladies Aston, but the foundation was never rich, and the nuns laboured hard at their farms beyond Pirie Wood. It was the only nunnery ever founded at Worcester.
At the dissolution of the religious houses by Henry VIII, the White Ladies fell to the Crown. In Elizabeth's 1's time it was why the Queen Elizabeth's School was able to move in 1868 from its old school house in St. Swithin's yard to the White Ladies of Whiston.
The most prominent of the medieval fragments is the west wall of the long range of the 18th century domestic buildings. It is of red sandstone and is part of the chapel dedicated to St Mary Magdalene in 1255, and had two lancet windows. The present White Ladies house is believed to have been built by Richard Blurton, whose wife was Mary Somers, aunt of the famous Lord Chancellor Somers. In the 18th century the Cookseys, Blurtons and Somers families which were tied by marriage , dwelt together in this long range of buildings. Lord Somer s as a boy was brought up in White Ladies. Holland Cooksey gives a charming account of the patriarchal life of the three families, each having a separate set of apartments, while a further set was reserved for their common use for meals and social mixing.
In 1842, when putting drains in the lawns in front of the house, brass and silver Roman and Greek coins were found, as well as human skeletons, thought to be from the Civil War period. During that troubled period the earlier house must have been considerable size for it was estimated as being capable of quartering 500 soldiers.