From a planning point of view, the road network in the False Creek flats is a mess. Expo Blvd and Pacific Blvd around GM Place completely ignore the urban fabric - they're more like suburban roads and they really don't work.
My problem with the Viaducts is that they take up a lot of room and create a barrier in the flats. I agree they can be incorporated into the urban environment, but I can't see how it would create a welcoming urban pedestrian environment. To me, keeping the viaducts because they create a "big city feel" is just silly.
I'm not sure what do do with the Skytrain line because it does dip down to pass under the viaducts; the line obviously should stay and can be incorporated into the urban form; but the dip is a problem.
Not only do I think the viaducts should be removed, but I also think Pacific Blvd and Expo Blvd need rethinking.
The reconnected grid would be largely as seen on this 1929 map - new roads may be required south of Georgia on the old rail lands, but the streets would continue and extend the city grid creating a much more walkable city in that area.

source:
http://www.vanc.igs.net
Yes removing the viaducts would remove a fast route over the flats into the city, but we're not talking a big saving in travel time at all; in a city that halted the freeways and is proud not have a freeway in the city, surely removing the viaducts is in keeping with the city's vision to make the city green and less dependent on the automobile.
Somebody mentioned the need for routes for electric cars. Electric cars are not the answer; even if we have them, we've removed pollution but still have congestion. That won't go away if we cater to the car, even if the car is electric.