However, the borough of East Retford was established even earlier than that, having been founded in 1105 by Henry I to collect tolls form travellers crossing the River Idle.

Despite a fire which destroyed much of the town in the 16th century, Retford continued to grow and prosper, particularly with the coming of the Great North Road, which passed through the centre of the town, and the building of the Chesterfield Canal.

The spacious Market Square was laid out in the 18th century and is surrounded by elegant Georgian buildings. the original Georgian Town Hall, however, was replaced in 1868 by the present building, which has been described as looking like a French chateau. Inside are an impressive ballroom and council chamber. Also in the Market Square is the Broad Stone, the base of an old parish boundary cross.

During the plague years of the Middle Ages coins for trading were placed in vinegar in the hollow of the stone to prevent the spread of disease.

The East Retford parish church of St Swithun's was founded in the 12th century, but in 1651 the tower collapsed and wrecked much of the building.