Using a combination of old wards, federal election districts and 1941 census tract populations here's 1901 and 1941 populations of different areas of Toronto. I've divided into inner city/pre-1900 area which is Don River to Dovercourt south of Bloor, close to the original boundaries before the first annexations in the 1880s and the rest into outer city which was mostly built up between 1900 and 1930.
Inner districts:
Ward 2 (Cabbagetown/Don) 35,228 45,318 +29%
Ward 3 (Downtown) 32,815 23,252 -29%
Ward 4 (Kensington Market area) 40,465 46,055 +14%
Ward 5 (Trinity Bellwoods area) 31,589 59,844 +89%
Outer districts:
Ward 1 (Riverdale) 11,179 44,297 +296%
Ward 2 (Rosedale) 3,468 17,136 +394%
Ward 3 (Yorkville and Deer Park) 7,187 16,774 +133%
Ward 4 (Annex and Hill District) 5,773 22,677 +293%
Ward 5 (Christie and Hillcrest area) 4,338 32,972 +660%
Ward 6 (Parkdale and Brockton) 27,549 94,591 +243%
Ward 6 (Earlscourt area) N/A, 25,131
Ward 7 (West Toronto Junction) 6,091 48,742 +700%
Ward 8 (East Toronto/Beaches) 4,633 85,835 +1752%
Ward 9 (North Toronto) 2,061 72,953 +3439%
Municipal ward map here:
https://commons.wikimedia.org/wiki/Toronto/File:Toronto_ward_map_1964.PNG