Metro Vancouver Transit Discussion

[Kisai]

Originally Posted by aberdeen5698

Let's not forget that each of the VCC's is triply redundant. They do in fact have backup systems, and I have no doubt that they have a UPS system for the control centre as well. I'm not privy to their system architecture, but unless these recent failures could have been avoided by additional redundancy then I think it's a bit premature to start proposing solutions that may be irrelevant.

Offhand, It's been stated already there are 4 VCC's, and if you say they are triple redundant, that suggests to me the "redundancy bottleneck" is the communication channel (Our version of Seltrac uses the inductive loops, while newer versions use 802.11 WiFi/Ethernet. See the PDF from two posts ago.) According to the Seltrac data I've read, each VCC can control 16 track loops. So this means "redundancy" isn't quite as simple as putting extra computer equipment at central control.

Even in the case of a fire, redundancy requires an off-site duplicate set of equipment. For cost reasons it seems like having an entire duplicate set of VCC's seems unnecessary.

Anyhow , between yesterday's and the previous outage, the weakest link in the system is not what motors are used but how they communicate. The Canada line also uses Seltrac.

This might be more suggestive of why we're getting breakdowns:
https://www.thalesgroup.com/en/content/t...g-contract-vancouvers-evergreen-lrt-line
THALES CANADA AWARDED SIGNALLING CONTRACT FOR VANCOUVER’S EVERGREEN LRT LINE
February 8, 2013

The guideways are elevated, at-grade and tunneled for a two kilometer stretch. The Evergreen Line will be in service in spring 2016.

Thales will upgrade the existing central control system and apply a two-stage implementation scheme to minimize any impact to existing SkyTrain service.

Maybe they're going to be throwing out the stuff that is failing and just need it to last another 2 years.

[GeeCee]

Originally Posted by queetz@home

Who says its going to be in the same room? And with regards to power failure in the control room, that can easily be mitigated by a backup power supply.

Supposedly the backup got cut as well.

[spm2013]

Originally Posted by queetz@home

There was no power outage in Metro Vancouver as far as I know.

And if the power issue is just on Skytrain, did Translink not pay the Hydro bill?

Its been over an hour...

The genius who thinks power outages only occur in the distribution network.


Credit

CBC: SkyTrain reopens after power outage

http://www.cbc.ca/news/canada/british-columbia/skytrain-reopens-after-power-outage-1.2713685

[queetz@home]

Originally Posted by Kisai

This might be more suggestive of why we're getting breakdowns:
https://www.thalesgroup.com/en/content/t...g-contract-vancouvers-evergreen-lrt-line
THALES CANADA AWARDED SIGNALLING CONTRACT FOR VANCOUVER’S EVERGREEN LRT LINE
February 8, 2013

Maybe they're going to be throwing out the stuff that is failing and just need it to last another 2 years.

If you are right, this only shows that Translink knew back then (when the signalling for the Evergreen Line was awarded) that the control system that we use today is bound to fail and decided to wait for the Evergreen Line to be completed to have a proper upgrade! Given how much delays that line had throughout its colourful history, if Christy didn't throw Lekstrom under the bus, no upgrades would have been done and this ticking time bomb would have continued indefinitely. Madness!

If this upgrade, in addition to the $20 million backup system, would have prevented these two issues, or even just the one last week, then why wasn't it done years ago?

This wasn't the first time Skytrain had massive shutdowns, and the only reason why the recent ones are so noticable was because they happened within days. Otherwise it would probably just be forgotten and the Skytrain cheerleaders (i.e. Darryl) would still be touting how perfect our system is, that it is the ultimate solution to bringing "no more cars" for our society. In the meantime, Skytrain users would be like playing Russian roulette whenever they use the system. So any trips that require on time arrival (i.e. doctor's appointments, job interviews, client meetings, schedule sensitive jobs, exams, etc), use a car, rent one if you don't have one.

[Xerx]

Originally Posted by queetz@home

If this upgrade, in addition to the $20 million backup system, would have prevented these two issues, or even just the one last week, then why wasn't it done years ago?

This wasn't the first time Skytrain had massive shutdowns, and the only reason why the recent ones are so noticable was because they happened within days.

This is gonna sound like a stupid answer but, probably because they are rare? Like once or twice a year rare?

[DKaz]

Drove to South Surrey Park & Ride this morning from Fleetwood, took the 351 then Canada Line. Takes an extra 15 minutes and $2 a day compared to driving to a 5 minute walk from Gateway Station then taking Expo Line but it's worth it.

[GlassCity]

News1130 ‏@News1130radio 1m
'@TransLink says human error led to yesterday's #SkyTrain shutdown. Says electrician accidentally tripped main breaker.

[Zassk]

Originally Posted by DKaz

Drove to South Surrey Park & Ride this morning from Fleetwood, took the 351 then Canada Line. Takes an extra 15 minutes and $2 a day compared to driving to a 5 minute walk from Gateway Station then taking Expo Line but it's worth it.

How was the train crowding at Bridgeport vs. at Gateway? Do you expect to be standing in either case?

[Kisai]

Originally Posted by queetz@home

If this upgrade, in addition to the $20 million backup system, would have prevented these two issues, or even just the one last week, then why wasn't it done years ago?

Speculatively speaking, The Olympics happened. The Evergreen line was supposed to happen before the Canada Line, and likewise both were originally supposed to be BOMBARDIER INNOVIA ART(This is what the complete Skytrain system is currently called) but the P3 meddling for the Olympics ruined that. The amount of political fighting over the Evergreen line, delaying it repeatedly didn't help either. This is why we have Translink and not letting the Mayors decide what gets built. See Toronto for how things don't get built.

Speaking of Evergreen line/Shutdown Yesterday:
http://globalnews.ca/news/1466677/human-...own-translink-to-offer-free-transit-day/
By Amy Judd Global News,July 22, 2014 10:51 am

VANCOUVER – Translink says it was human error that led to Monday’s SkyTrain shut down, causing about five hours of commuter chaos.

An experienced electrician was installing a new circuit breaker for the Evergreen Line at a power distribution panel when he accidentally tripped the main breaker feeding the critical systems at SkyTrain’s operations centre, which caused a system-side shut down of train controls.

So... this outage could have been prevented with a UPS right?

[queetz@home]

Translink running the show instead of the mayors was simply because Kevin Falcon was a petulant child and couldn't get what he wants.

Originally Posted by Xerx

This is gonna sound like a stupid answer but, probably because they are rare? Like once or twice a year rare?

Rarity doesn't mean you don't need a backup system, and especially so as the system ages. Its like insurance. Bad things may happen, but chances are it won't. But insurance companies sell it to you anyway for your home, car, life, etc., and since its rare, its easy money for them. And chances are you would be buying them despite the rarity of a bad event happening for peace of mind.

Of course, if something does happen, insurance companies will fight tooth and nail for you not to claim. At least in the case of Skytrain, if the primary system goes down because of faulty cards or electrical systems, the backup takes over and the trains won't halt to a grind.

I honestly don't get this resistance to a backup system anyway? Are you all so bloody defensive of Skytrain at its current form that you would try to justify this latest f*ckup by justifying not having a backup system???

These incidents are genuine mistakes caused by mismanagement that have to be corrected. There is absolutely no harm in taking corrective action, and even admitting that what the Translink management has done is wrong. Not everything is a pissing contest, LRT vs Skytrain, and so on!

[queetz@home]

Originally Posted by Kisai

Speaking of Evergreen line/Shutdown Yesterday:
http://globalnews.ca/news/1466677/human-...own-translink-to-offer-free-transit-day/
By Amy Judd Global News,July 22, 2014 10:51 am


So... this outage could have been prevented with a UPS right?

From one of the facebook comments...

Smith Ron · SFU/BCIT
shouldn't they have a back up system like a generator or second system to take over when this happens? you would think a multi million dollar system would have a secondary system to go to in event of an outage, or is that too simplistic? A plan B so to speak.
Reply ·
· 21 minutes ago

Bob Holowenko · BCIT
Absolutely! An enterprise grade UPS is not hard to install, not super expensive, and a generator backup is easy too...I think the major issue was the hard shutdown of these 1980s era computer systems...Probably running on 8088s...

The fact that it only took ONE circuit breaker to bring down the entire system with no backup in place to prevent it from shutting down...wow! Its like a dream or something for somebody wanting to cause some mischief in the Skytrain system. Maybe Malcolm Johnson is taking notes...

[Mac Write]

When someone is working on the power it's like a computer user with "root" access there are no safeties as your running as "God" and can do what ever you want with no one saying no.

[officedweller]

I like the G&M's infographic:
(apart from the fact the 62 km reference includes the unaffected Canada Line, and Surrey isn't labelled)

SkyTrain shutdown: How an electrical mishap crippled Vancouver's 62-kilometre transit system

The Globe and Mail
Published Tuesday, Jul. 22 2014, 7:00 PM EDT
Last updated Tuesday, Jul. 22 2014, 7:01 PM EDT

http://www.theglobeandmail.com/news/brit...ilometre-transit-system/article19717162/

[twoNeurons]

Originally Posted by quobobo

I have it on good authority (a coworker's dad is an engineer who works on Skytrain control systems) that this will be resolved in two hours or less.

He says there was a power outage in the computer control centre - this was completely unrelated to the issue last week. The control centre is back up now but they need to manually drive trains to each station before they can turn the entire system back on. Train attendants are in the process of doing that now.

How provincial to be affected by a power outage. It's pretty common to have a UPS system in cases like these (Uninterrupted Power Supply) and in some cases, a UPS connected to a generator!

[jlousa]

A backup system doesn't help if it's before the breaker. Most breakers would be after the power source to protect it. So whether they have a generator or not wouldn't have helped in this case.
I find it funny that it was caused due to construction of the evergreen line, What do you think Queetz should we cancel the line?

[twoNeurons]

Originally Posted by jlousa

A backup system doesn't help if it's before the breaker. Most breakers would be after the power source to protect it. So whether they have a generator or not wouldn't have helped in this case.
I find it funny that it was caused due to construction of the evergreen line, What do you think Queetz should we cancel the line?

Yes, I had replied to a previous post without reading the whole thread. Makes sense if it was a breaker that was tripped. I imagine looking at a breaker box with a bunch of unlabelled switches (like almost everyone's home circuit breaker box) and the electrician saying, "okay... does this breaker control the lights in the... << control center powers down like the sound of powering down the death star>>... uh oh. That's not good."

IT is comical that the evergreen was the cause of the problem!

[Alex Mackinnon]

Originally Posted by queetz@home

In the meantime, Skytrain users would be like playing Russian roulette whenever they use the system. So any trips that require on time arrival (i.e. doctor's appointments, job interviews, client meetings, schedule sensitive jobs, exams, etc), use a car, rent one if you don't have one.

Because cars never break down or get in to traffic jams or accidents. Yup, you're not being dramatic at all.

Also, you would still have a breaker after the UPS. If you end up with a short inside the computer, you would want a breaker to trip. The alternative is that the battery provides enough juice to light a bunch of things on fire. See how that affects your uptime.

[jsbertram]

Originally Posted by Kisai

Speculatively speaking, The Olympics happened. The Evergreen line was supposed to happen before the Canada Line, and likewise both were originally supposed to be BOMBARDIER INNOVIA ART(This is what the complete Skytrain system is currently called) but the P3 meddling for the Olympics ruined that. The amount of political fighting over the Evergreen line, delaying it repeatedly didn't help either. This is why we have Translink and not letting the Mayors decide what gets built. See Toronto for how things don't get built.

Speaking of Evergreen line/Shutdown Yesterday:
http://globalnews.ca/news/1466677/human-...own-translink-to-offer-free-transit-day/
By Amy Judd Global News,July 22, 2014 10:51 am


So... this outage could have been prevented with a UPS right?

Maybe, but you know there will be a Homer Simpson working there who would trip over the UPS power cord too.

[queetz@home]

Originally Posted by jlousa

What do you think Queetz should we cancel the line?

The Evergreen Line is already partly there, so no sense wasting all those columns, guideways and such (unless you want to go BERTS). But since we now know further Skytrain extensions can cause this incident, then lets cancel the Broadway Skytrain extension instead and make it an independent LRT or even a separate Rotem subway like the unaffected Canada Line...

Seriously though...

IT is comical that the evergreen was the cause of the problem!

The Evergreen Line as a whole did not cause it. It was an electrician who doesn't even work for EGRT who did this, he works for BCRTC. Granted he was only following orders, and while I don't usually side with the unions, I agree with them and think he is being scape goated.

Union says electrician unfairly blamed for SkyTrain meltdown

“Our initial review has discovered that management directed the member to do the work on an electrical panel despite the panel’s poor design and the high risk of it failing while working on it live,” O’Connell said, adding that under “common safety practice,” the panel should not have been worked on during service hours. “To our knowledge, the company (BCRTC) did not do a proper risk assessment of the work that was going to be undertaken on such a critical piece of electrical equipment.”

At the end of the day, this shows how vunerable this aging Skytrain system is to even the most simpliest of issues. All the more reason why we need backup systems, as well as more checks and balances. What could be so wrong with that?

And to add to that, why is Translink's Doug Kesley against an independent inquiry?

[spm2013]

Yeah and unions never protect their own even when they're at fault.