I've thought about this as well. I think the reason we don't see anything proposed for Elgin has to do with it being an established commercial strip, one that is arguably the most consistent and successful in terms of street-walls and businesses lining it. While all the buildings are lowrises, save for one, there is only one building that has a parking lot (TD Bank). In order to build, developers would have to evict popular businesses and demolish small buildings on small lots in order to build something that would be no taller than six stories.
These new buildings wouldn't add much in terms of vibrancy to the street, as Elgin is already a very vibrant restaurant row. Not to mention there are some sizable (parking) lots within a few blocks that would be cheaper to developer (no demolition) and are perfect for taller buildings to absorb this growth. As well, these other properties would be less controversial than Elgin.
The other main streets on the other hand are not in such a favourable position. Rideau, Bank, Somerset-Wellington-Richmond, Main are not as developed as Elgin. There are a lot of large, vacant lots creating gaps along the street in neighbourhoods where developers probably feel they stand a better chance of selling units (well, not Somerset), and if storefronts are occupied they're not necessary the sort of businesses that make an area exciting. Kitchissippi does not have the density Centretown has, yet being so desirable developers can create more housing for people interested in living there. As well, they wouldn't have to demolish much other than a small automechanic building or gas station. Bank Street has larger lots on or near enough that a new building would fill gaps with stores and more people, making it much more vibrant (the condos building in the last 5 years have really helped to transform the dingy strip).
That doesn't mean we won't see something being built on Elgin, as we do have some spots that could be demolished to make way for something else. But considering there are better options elsewhere and Elgin is already great, no need to mess with what works. I'm fine with a lowrise street like Elgin, and you wouldn't want to anger Knox Church again.