I don't think it is a bad debate either way. It has to be discussed because if you just look wholistically at the area around them, the 'feel' isn't quite the same as the rest of downtown Vancouver. You can argue either way that they make it easier or more difficult to construct higher densities or bring a community feel, but it should still be looked at.
You also have to take into consideration the trends Vancouver council have been moving towards for the past 10-20 years. Every single council has been strongly against the notion of "moving people through Vancouver." That's why you've seen such large arguments against regional infrastructure improvements such as replacement of the Lion's Gate bridge, expansion of the road network into Vancouver from South Surrey/Delta, and even the Gateway HW1 expansion.
Same goes here at the macroscopic scale. I think Vancouver council current and past see the viaducts as surving only the purpose of moving people through Vancouver. As such they latch onto the notion that if the viaducts were gone and the infrastructure was made more community oriented (which does make it more difficult for cars) that less people will use downtown Vancouver as a means of getting through the area.
It would also push traffic down other roads.
The only issue I've had in thie entire debate is the notion of cutting off Georgia Street. I can understand that regionally the 'through' part of Georgia ends when you hit Howe/Seymour and that the viaduct itself isn't part of Highway 99. But it is still a major route, one of the very few major routes east-west through Downtown Vancouver (I'd argue it is the only major one) so making it terminate on 1 end would just be stupid.
But, any plan would be incomplete without looking at all options and all alternatives. So the questions have to be asked, what if. If they are to conclude that leaving Georgia as a through route on both ends is the best idea, they have to look at closing it off and finding out WHY that would be stupid. I can sit here and think it is stupid but that's just opinion and I'm not trying to argue the fact to voters in an official capacity that an official plan would require.
I do like DKaz' ideas at the top though. They are quite interesting.