I enjoyed that series about the Government houses a lot, SHH !
One thing : Québec's government house is located inside the Citadel.
Gardes de cérémonie à la Citadelle de Québec by
abdallahh, sur Flickr
Before it was destroyed by fire in 1966, the House was located just outside of the old city, in the Bois-de-Coulonge Park, and looked like that :
Source : Wikipedia
In Toronto, according to Wikipedia, the situation is more complex. The first house exploded in 1813. The second one was sold and demolished in 1912. The third one was disused then demolished in 1961. I think the Lieutenant Governor now has some apartments inside of Queens Park... Maybe and Ontarian
forumer could give us some insights