Horatio grew up in the large vicarage with seven brothers and sisters, and they seem to have been a close family. However, tragedy struck when he was nine years old. His much-loved mother died. This must have been a devastating blow to the young boy. He also experienced other changes at the same time. He was sent to the Royal Grammar School in Norwich and then later his father transferred him to the nearby Paston Grammar School, in North Walsham.