Below the topsoil of its well-manicured world-famous gardens there's an industrial site that built a fortune and a city.