Holy Trinity Church forms the centre of a neat, early Victorian Gothic Square. On the south side stands the Vicarage and four Almshouses. On the north side there are four more Almshouses and the Church Hall which was formally the Girls school. The whole ensemble was erected on the Grove Fields in 1839, and endowed by Miss Rachael Herbert in 1840. The architect was Mr Thomas H. Wyatt of London.
The buildings were erected and endowed at the expense of Miss Rachel Herbert of Little Hill, Pen-y-pound, Abergavenny. Miss Herbert was the daughter of Charles Herbert who made a fortune as a dealer in iron in Abergavenny and was a descendant of William Herbert, one of the illegitimate sons of Sir Richard Herbert of Coldbrook.
Rachel Herbert died in 1870 though her name still lives on through the charities she founded and the marble plaque to be found the north wall of Holy Trinity church.
The almshouses were provided by the Rachel Herbert Almshouses Charity for poor women, being members of the Church of England (as it was then), of good character, and of not less than 60 years of age who came from the parishes of Abergavenny, Llantilio Pertholey and Cwmyoy, all of which contained properties owned by Rachel Herbert.