If they bought the old Penn Club site and additional air rights, I think that they could build a decent tower. The footrprint isn't very small. It extends pretty deep to the west off of Fifth.
The total footprint here (in terms of north to south and east to west) seems to be about the same as that of 343 Madison.

KPF
The site also is much larger than the one for Aby's proposal next to the Chrysler Building.