I agree completely. The blocks surrounding my Prince of Wales Street are a perfect example - there's nothing wrong with the buildings, it's just that many of them have vinyl siding. That makes them look (to our eyes) cheaper and poorer than their clapboard-clad cousins elsewhere in the city.
The sad thing is, when the old clapboard was taken off and the vinyl put on, these neighbourhoods were the nicer ones.
Completely agree with banning vinyl.