Some say it’s an eyesore. Others appeal to its history. Some want to see it repurposed, others call for its demolition. But before it became a site for public debates, the limestone structure at 4 E.