There are two candidates.
The Langren Hotel, which was replaced by a two story parking deck in the early 1980's. The parking deck serves the building that replaced the Nichols Shoe Co. building in the foreground -- the Akzona Building, perhaps the most hideous thing I.M Pei ever designed.
Or possibly, the Old Battery Park Hotel.
The New Battery Park Hotel replaced it in 1927.
Probably, for most grievous loss, the Langren wins out. Tearing down a grand dame hotel for an elegant high-rise might be defensible but tearing down an elegant hotel for a two-story piece of shit parking deck is not. From the 1960s on through the early 90's Americans were so relentlessly stupid when it came to urban issues it's a miracle anything worth loving survived to the present day at all.