That apartment building in the second last picture is such a mess. I don't like the thought of the hundred's of residence being forced out for a new development but purely esthetic wise that building needs to go. It has that extremely dated dead mall inside it and the property owners have not maintained the structure at all.
Just looking into it the name of the Property is The Charles Street Promenade and it's owned or by or managed for U of T to house students and their families. So technically if U of T builds new housing they could sell this prime bit of real-estate they own. The site is massive, 2x the size of The Ones site.
source:
https://universityfamilyhousing.utoronto.ca
Something massive could go here and not have to worry about any shadow impacts.