Metro Vancouver Transit Discussion

[queetz@home]

Originally Posted by spm2013

Yeah and unions never protect their own even when they're at fault.

Then answer this...why is he installing that circuit breaker during the day?

[aberdeen5698]

Originally Posted by queetz@home

At the end of the day, this shows how vunerable this aging Skytrain system is to even the most simpliest of issues.

"Aging" has nothing to do with the problem.

All the more reason why we need backup systems, as well as more checks and balances. What could be so wrong with that?

The question is this: what kind of backup system would have prevented this problem, and how much would it cost vs. how much would be saved by avoiding the infrequent occurrences of the problem.

[memememe76]

Originally Posted by aberdeen5698

The question is this: what kind of backup system would have prevented this problem, and how much would it cost vs. how much would be saved by avoiding the infrequent occurrences of the problem.

Not sure if this answers the question, but I am watching Global News while cooking dinner and the Translink Rep, Doug Kelsey, says a backup plan would be "at least $30 million dollars." TBH, the guy sounds rather blowhardish. Certainly, Translink could hire someone else who would be more media-friendly?

I don't know, 20-30 million dollars doesn't seem that much, no? Translink is kinda "lucky" that the weather was dry during these mishaps. What would've happened in a normal rainy Vancouver day? One of those passengers could've easily slipped on the platform and fell.

[cabotp]

Originally Posted by memememe76

Not sure if this answers the question, but I am watching Global News while cooking dinner and the Translink Rep, Doug Kelsey, says a backup plan would be "at least $30 million dollars." TBH, the guy sounds rather blowhardish. Certainly, Translink could hire someone else who would be more media-friendly?

I don't know, 20-30 million dollars doesn't seem that much, no? Translink is kinda "lucky" that the weather was dry during these mishaps. What would've happened in a normal rainy Vancouver day? One of those passengers could've easily slipped on the platform and fell.

And in that case I would blame the passengers for being in an area where they are not supposed to be.

In other words those who decide to walk along the platform should take full responsibility for anything that might happen to them.

[casper]

Originally Posted by twoNeurons

Yes, I had replied to a previous post without reading the whole thread. Makes sense if it was a breaker that was tripped. I imagine looking at a breaker box with a bunch of unlabelled switches (like almost everyone's home circuit breaker box) and the electrician saying, "okay... does this breaker control the lights in the... << control center powers down like the sound of powering down the death star>>... uh oh. That's not good."

IT is comical that the evergreen was the cause of the problem!

There have not been any reports yet of the specifics. I would hope they do a TapRooT investigation. I would hope that their preventative maintenance are a sufficiently structured that they don't have breaker panels with unlabeled switches.

[cabotp]

Originally Posted by casper

There have not been any reports yet of the specifics. I would hope they do a TapRooT investigation. I would hope that their preventative maintenance are a sufficiently structured that they don't have breaker panels with unlabeled switches.

I don't think it was a case of the electrician flipping the wrong breaker.

More likely during the installation of the breaker it caused a surge in the electrical network that tripped the other breaker.

[GMasterAres]

The issue is quite simple, management didn't perform a valid risk assessment and as a result, had a job done (electrical work) during a period of time that should not have had any work of this type being done. They should have did the electrical work after hours aka after 2am like they do most maintenance of this stature.

If the facts of the story are correct, then I agree with the Union that the worker should not yield blame because if the work was authorized, it would have been authorized by a manager whose head should be on the chopping block not the worker.

Regardless, it is a bit scary that a single electrical failure in a panel could bring the entire SkyTrain system to a grinding halt. That is kind of like saying 1 street lamp getting an electrical failure will cause all street lamps in a city to go out.

Then again, I know from personal experience work related that power is an often overlooked infrastructure system and many times taken for granted. I have worked on a project where someone plugged a device into a single circuit not verifying the load on that circuit. It subsequently blew out the entire circuits crashing systems on the circuit that did not have redundant power connections.

A few of those systems failed in coming back online and it took several days to recover all systems back to normal even though the power was restored within 5 minutes with this specific failure. With the work being done, the project actually had a fairly detailed risk assessment completed but in retrospect, power was overlooked.

Then again the work wasn't specifically power which is why in the case of SkyTrain I have less sympathy. The work was specific to power so missing this type of risk is either complete negligence or it was over-looked willingly by management.

[dpogue]

We still don't know at a detailed level exactly what the electrical failure affected. I believe with reasonable certainty that it definitely affected the control room workstations, but we don't know whether the actual VCCs shut down.

What we do know:

• There are 4 VCCs for the system, each one a cluster of 3 CPUs. These 3 CPUs all perform the same calculations, and 2 of them must agree before any command goes out to the system. This is sometimes called redundancy, but it's more accurately "error checking". Photos: 1 2 3
• Originally these were powered by OS/2. There's a mention of them being upgraded, but I can't find anything else that confirms that.
• The control operators use workstations running Windows (XP?) and Thales NetTrac SMC and SCADA software. Photos: 1 2
• There are redundant SMC servers, which are normally used for training new control operators, but can be used if the main SMC system goes down.
• If the SMC becomes unavailable, the VCC can still operate via a command-line "degraded" mode, but things like platform destination signs don't function. It's possible that degraded mode would actually involve commanding each train one by one.

We do not know:

• If there are backup VCCs. (Unlikely, since the VCC software is proprietary and it was "prohibitively expensive" to even get more mimic diagrams set up around the control centre).
• If there is a backup power supply for the VCCs.
• If there is a backup power supply for the control center workstations.

Thursday
On Thursday, VCC2 had a network hardware issue that prevented communication with the trains.

This caused all trains on that zone to time out and emergency brake to prevent collisions. Staff attended trains and manually drove them back to stations. Once all passengers were evacuated, staff had to wait for the VCC to be repaired before it could pick up and re-enter the trains.

The total delay was about 5 hours. The majority of that time was spent after evacuating passengers, waiting for the VCC to be repaired.


Monday
On Monday, an electrical failure caused (at the minimum) the control room workstations to lose power. This prevented the use of the on-train and in-station announcement system and all trains were held system-wide. Not sure of the exact time, but it was somewhere around 12:30.

First trains were held (system service hold), but with no ability to make announcements and no working intercom system on the trains, people broke out of trains and started walking, which forced control to de-energize the guideway. I suspect at this point they decided it was already enough of a mess that staff might as well just escort people off of trains via the side walkways.
Note that power had to have been restored to the control room in order to de-energize the guideway

By 13:15, the control room had power, workstations were operational and the SCADA system was being used to de-energize and re-energize the guideway. All field staff were manually driving trains back to stations or escorting passengers off of trains and walking the guideway between stations to ensure it was safe to re-energize. The entire system was re-energized by 14:30 at the latest.

Then all trains needed to be manually driven by staff to be picked up by the VCC for automation. It took several hours to coordinate spacing of the trains, ensuring that they had been re-entered into the system correctly, ensuring there were no on-board computer errors. A handful of trains had door faults due to passengers breaking out, which required vehicle techs to fix before the trains could be automated.

The system was fully operational again by 17:30.
Note, the times and details for this incident are reflective of the Expo line. The Millennium line was running earlier, but that might be just because there were fewer trains to re-enter.


Preventative measures
I'm still not sure we know enough of the details about the electrical failure to know exactly where backup power sources are needed, but for sure there should be a review of the existing systems to ensure that all critical components have more than one source of power with no single point of failure.

As for redundant VCCs, I suspect that all VCCs would need to be upgraded and replaced before that is really feasible. You don't want your backup system to be running on totally different hardware that's untested. I'm not sure that a backup would make problems clear up faster though, since you'd probably still time out all the trains when the main VCC went down.

[Axe]

They have a back up system, The breaker that was tripped was on the back up system. It was tripped by a fault ( or short circuit ). UPS systems like the one they have , still need breakers.

[WarrenC12]

Great detail dpogue. What I've learned is that a backup intercom system would have solved the vast majority of this issue.

[twoNeurons]

Originally Posted by casper

There have not been any reports yet of the specifics. I would hope they do a TapRooT investigation. I would hope that their preventative maintenance are a sufficiently structured that they don't have breaker panels with unlabeled switches.

Yes, I was just kidding around, relating it my experience at a residential home. I don't think I have EVER lived in a place where the breaker panel was labeled, resulting in a lot of guesswork when I had replace plugs or switches. At businesses where I have worked, everything is labelled specifically.

I would sincerely hope that the breaker panel for a city-wide transit system's control system would be labeled!

[MIPS]

Originally Posted by queetz@home

If this upgrade, in addition to the $20 million backup system, would have prevented these two issues, or even just the one last week, then why wasn't it done years ago?

It's a level of contingency we've never considered. You cannot lay blame on something that rears its ugly head after almost 30 years of operation.
Like I said before, unless it fails again in the near future it's safe to let this accident slide.

Originally Posted by queetz@home

From one of the facebook comments...
The fact that it only took ONE circuit breaker to bring down the entire system with no backup in place to prevent it from shutting down...wow! Its like a dream or something for somebody wanting to cause some mischief in the Skytrain system. Maybe Malcolm Johnson is taking notes...

It's not like the guy let a broom fall and bump a panel. This is about as close to the "the server loses power at 4AM every day because the janitor unplugs it for his floor buffer" myth as you can get, only it's an electrician who was not authorized to carry out work and as a result of his actions is now suspended and probably fired.
Edited: Actually, I'm reminded of that scene from Airplane.

Originally Posted by SFUVancouver

We need a plucky young adult who knows Unix to get in there with a 3D navigation interface and find the right box to fix everything; deus ex machina.

You rang?

Let me sum up the last five days with some quick photoshop jobs.

>>tagged for size<<
>>tagged for size<<
>>tagged for size<<

[Kisai]

Originally Posted by WarrenC12

Great detail dpogue. What I've learned is that a backup intercom system would have solved the vast majority of this issue.

It's probably not possible to do. It would be more cost effective to have a "in case of emergency stop, press this button. Do not force open doors. If there is no response in 3 minutes, call 310-xxxx or text YYYYY with this car number ZZZ"

Simply having a GPS-aware app on a smartphone would work too. (Isn't there one already? Maybe adding "DONT PANIC" push notifications to it would help .)

[WarrenC12]

Originally Posted by Kisai

It's probably not possible to do. It would be more cost effective to have a "in case of emergency stop, press this button. Do not force open doors. If there is no response in 3 minutes, call 310-xxxx or text YYYYY with this car number ZZZ"

Battery backup for the intercom? Can't be that bad. What are the train cars these days, $2M each?

[TransitJack]

All this talk of back up systems, remember:

Each VCC is made up of 3 CPUs: 2 must be in agreement and 1 is for redundancy

If 1 CPU fails, the VCC continues to operate.

Skytrain has 4 VCCs, each controlling a SEPARATE area of the ATC. 3 for the Mainline, 1 for the yard.

Canada Line has 2 VCCs, 1 for the Mainline, 1 for the yard.

Both OMCs have power supplied by Hydro and a Generator if Hydro is lost. To prevent any loss of power to the vital systems a UPS system exists which gives a few hours of juice.

When dealing with Power, there are many breakers to isolate systems, just like in your house. You can turn off the main feed in your home, or isolate just your dryer or furnace, see. Same for all electrical systems.

Last week, one VCC crapped out ( a card failed that took time to diagnose ). This week, the power (electrical) network was affected causing ALL VCCs to lose power (not crash). This causes all the trains to time out and need to be re-entered across the entire system (including the yard).

Big big headaches for passengers and staff, but unrelated issues between this week and last, not indicative of a failing infrastructure. All the trains Emergency Braked as designed and passengers were safe on board. Zero risk of trains colliding, for example.

There isn't really a "back up" available for Monday's problem as the failure was in the power supply between the main and back up systems and the actual computers. Best work around is NOT to perform this type of work during revenue service, I'd say.

[queetz@home]

Originally Posted by MIPS

Like I said before, unless it fails again in the near future it's safe to let this accident slide.

That's what you said regarding the July 17 incident, and then the Skytrain clunks up a few days later!

I know the two incidents are "unrelated" but ultimately, its how Translink manages the Skytrain is also in question. No backup system for the main control centre and the fact that it can only take one circuit breaker to bring down the entire system? What other things behind the scenes can cripple the aging Skytrain that we don't know off?

I watched the Global TV raw interview and do agree with memememe76 that Doug Kesley is very blowhardish. He is the Translink executive that would be accountable to this, but all suggestions of solutions, concerns or improvements he just brushes off. Then he easily passes the blame on passengers, who clearly had no choice but to find ways to get out of the sweltering rail cars!

Talk about such an arrogant know it all, and its attitudes like that which really annoys people. He is responsible for this mess, not the electrician or the immediate supervisors. He should own up to it since ultimately, the two issues and the subsequent effects are operational mismanagement issues more than anything.

[tybuilding]

Originally Posted by memememe76

Not sure if this answers the question, but I am watching Global News while cooking dinner and the Translink Rep, Doug Kelsey, says a backup plan would be "at least $30 million dollars." TBH, the guy sounds rather blowhardish. Certainly, Translink could hire someone else who would be more media-friendly?

I don't know, 20-30 million dollars doesn't seem that much, no? Translink is kinda "lucky" that the weather was dry during these mishaps. What would've happened in a normal rainy Vancouver day? One of those passengers could've easily slipped on the platform and fell.

The dangers of having a too efficient transit system: the SkyTrain meltdown

"Due to the provincially mandated requirement to become more efficient, TransLink has reduced the resiliency of the system.

For example, TransLink didn’t purchase a $20 million backup system that may have allowed SkyTrain service to be restored faster on Thursday. Also, TransLink doesn’t have the long-term funding to pay for the over $1 billion in upgrades that will be required to keep the SkyTrain from becoming overcrowded and running as reliably as possible over the coming decades."

"I don’t say this lightly; I’m very grateful that no one died due to the failures that shut down the SkyTrain system on Thursday and Monday. Other transit agencies, like Toronto’s TTC, have had people died when system failures occurred"

[MIPS]

Originally Posted by queetz@home

That's what you said regarding the July 17 incident, and then the Skytrain clunks up a few days later!

I never said they could brush off this latest incident. That one was completely avoidable.

No backup system for the main control centre and the fact that it can only take one circuit breaker to bring down the entire system?

It's already been noted that backup systems are in place at the OMC and it's more than likely they have been used however they're completely useless if a cord is kicked far enough up the chain.
Same applies for the three servers that sit in racks directly behind the chair I sit in. They're all surge protected and running through a 3kW UPS but if the leg of the chair bumps the rocker switch on the UPS the servers are going down no matter what I say or do.
Plus the VCC's have an EPO button (enough jargon yet? ). Pushing that would of done the exact same thing in relation to a breaker being accidentally flipped. You can't remove these and make the system any more reliable because then you cross the line of safety.

[queetz@home]

Apparently the Skytrain is having issues again this morning, but its *just* a switch problem this time...

‘Don’t worry, folks, we’re not going to leave you stranded’: Another day of SkyTrain glitches slows morning commute

SkyTrain commuters uttered groans Wednesday morning as SkyTrain control announcers came over the public address system to warn of morning system delays, just two days after human error caused a massive five-hour shutdown.

Just before 7 a.m., SkyTrain control announced that due to a switch problem with a train at Waterfront station, inbound trains throughout the system would be held for longer than usual outside or at stations.

The news was greeted by moans and frustrated outbursts from riders on at least one Millennium Line train.

From start to finish, the system delays lasted about three hours.

At about 7 a.m., TransLink issued a tweet advising the public to “expect minor delays and slower service from Commercial-Broadway to Waterfront as the problem train is removed.”

TransLink issued another tweet just after 8 a.m. advising riders that “delays/slower service [were] still ongoing in the downtown core. Please give yourself extra travel time this morning.”

Just before 9 a.m., the transit authority tweeted “delays will continue through the morning rush. sorry for any inconvenience and we thank you for your patience.”

A half-hour later TransLink explained to a follower that there were now “delays through the system due to the backup of cars in the downtown core.”

At about 10:20 a.m., TransLink tweeted that the switch at Waterfront was back in service and that “regular service [would be] returning to Waterfront shortly.”

Some riders tweeted it was taking 12 minutes to get between Main and Stadium stations.

Others on social media said it had taken 35 minutes to get from Broadway/Commercial to Waterfront station.

One rider claimed it took him 90 minutes to get from Nanaimo to downtown.

Still others reported that delays and station holds were happening at stations well outside the Commercial-Waterfront corridor.

In the past week, riders have endured two hours-long breakdowns in service.

[GlassCity]

I don't know anything about engineering or mechanics or anything like that, but something has to be done about this. Not only are you causing massive inconvenience to hundreds of thousands of people, but you're also contributing to the PR nightmare that is Translink. These past few weeks will definitely still be on the minds of voters as they head out for the transit referendum. These delays need to be stopped if we're going to save our region from LRT as well as having any hope of increased funding.