Metro Vancouver Transit Discussion - Page 604

MIPS · #**12061** Posted **Jul 18, 2014, 1:16 AM**

CBC is reporting that the entire SkyTrain network has shutdown. TransLink not saying much beyond computer/electrical problems. This is a biggie.
Service between Waterfront and Metrotown are in full-manual. All other stations are currently closed/not receiving trains. Shuttle buses are currently trying to fill the gaps all the way to King George.

Edited: CTV Says they've already had a few instances where passengers were pulling the door release handles and walking down the guideway, further preventing the system from restarting until they confirmed the track was clear.

spm2013 · #**12062** Posted **Jul 18, 2014, 1:37 AM**

Canada Line is unaffected as is Gregor's mythical Broadway Line.

quobobo · #**12063** Posted **Jul 18, 2014, 2:19 AM**

Translink says the issues will be fixed by 9PM at the earliest.

This is the worst outage I've seen - does anyone know how it compares to historical outages?

MIPS · #**12064** Posted **Jul 18, 2014, 2:33 AM**

Quote:

Originally Posted by spm2013

Canada Line is unaffected as is Gregor's mythical Broadway Line.

The Canada Line has independent VCC's to the SkyTrain.

My $20 is that either the network between the VCC's failed or multiple VCC's failed at the same time and once you pass the redundancy margin the system shuts down to save itself.

dpogue · #**12065** Posted **Jul 18, 2014, 3:37 AM**

Quote:

Originally Posted by MIPS

The Canada Line has independent VCC's to the SkyTrain.

My $20 is that either the network between the VCC's failed or multiple VCC's failed at the same time and once you pass the redundancy margin the system shuts down to save itself.

From the SkyTrain president (source):

Quote:

SkyTrain problem determined to be a failed card in the I/O rack of the main control computer. It has been replaced and the system is back.

Waders · #**12066** Posted **Jul 18, 2014, 4:03 AM**

i wonder how old that failed hardware component is. 28+ years? I am also surprised there is no built-in redundancy in the control system. May be the control system needs to be renewed.

quobobo · #**12067** Posted **Jul 18, 2014, 4:19 AM**

Quote:

Originally Posted by Waders

I am also surprised there is no built-in redundancy in the control system.

"No built-in redundancy" isn't quite accurate - many parts of the control system are redundant.

Sounds like either there was an issue with failover to a backup system, or there are some single points of failure in the system.

spm2013 · #**12068** Posted **Jul 18, 2014, 4:37 AM**

And the human factor:

Quote:

Some service was restored around 5:50 p.m., but soon after, a number of passengers on stopped trains left their cars and entered the guideway, which resulted in a system-wide hold.

http://www.vancouversun.com/Computer+gli...pdated/10039716/story.html#ixzz37n8zWy8b

MIPS · #**12069** Posted **Jul 18, 2014, 5:53 AM**

Quote:

Originally Posted by Waders

i wonder how old that failed hardware component is. 28+ years? I am also surprised there is no built-in redundancy in the control system. May be the control system needs to be renewed.

If I recall they did replace a bunch of hardware around late 1999 in part because of possible Y2K problems and also because the hardware was old (rumor was that the original VCC's were VAX systems). Back when it was just the Expo line there were just three VCC's. The redundancy was that the system would continue to operate if one computer failed. It was when two or more went down the system-wide shutdown engaged, or if the interconnect between all three kicked it.

dpogue · #**12070** Posted **Jul 18, 2014, 6:04 AM**

Quote:

Originally Posted by quobobo

"No built-in redundancy" isn't quite accurate - many parts of the control system are redundant.

Sounds like either there was an issue with failover to a backup system, or there are some single points of failure in the system.

I don't believe there is any sort of automatic failover system. There is a backup system that they use for training purposes, but it sounds like that's a last resort.

Meraki · #**12071** Posted **Jul 18, 2014, 6:17 AM**

From what I recall the VCC is segmented controlling different parts. There's one for the segment from Waterfront to somewhere midline (around Metrotown?) one for the rest of the Expo, and one for the Millennium line.

It sounds like the card failure was in the Millennium line rack, and problems with getting it back up caused them to delay/suspend eastern Expo line service.

dpogue · #**12072** Posted **Jul 18, 2014, 6:50 AM**

Quote:

Originally Posted by Meraki

From what I recall the VCC is segmented controlling different parts. There's one for the segment from Waterfront to somewhere midline (around Metrotown?) one for the rest of the Expo, and one for the Millennium line.

It sounds like the card failure was in the Millennium line rack, and problems with getting it back up caused them to delay/suspend eastern Expo line service.

VCC1 is from Waterfront to between Royal Oak and Edmonds.
VCC2 is from Edmonds to King George, but also includes the Millenium Line up to the pocket track between Braid and Lougheed.
VCC3 is the OMC and yard.
VCC4 is the Millennium Line (from Lougheed to VCC-Clark).

VCC2 is the one that went down today, so no service between Royal Oak and King George, but also no service on the Millennium Line for Braid and Sapperton.

ninjakafi_81 · #**12073** Posted **Jul 18, 2014, 7:18 AM**

Quote:

Originally Posted by MIPS

CBC is reporting that the entire SkyTrain network has shutdown. TransLink not saying much beyond computer/electrical problems. This is a biggie.
Service between Waterfront and Metrotown are in full-manual. All other stations are currently closed/not receiving trains. Shuttle buses are currently trying to fill the gaps all the way to King George.

Edited: CTV Says they've already had a few instances where passengers were pulling the door release handles and walking down the guideway, further preventing the system from restarting until they confirmed the track was clear.

I was caught up in this gong show this afternoon and it was just a cluster f*ck I tell you. People yelling at each other, transit police / security yelling at people, people cutting the line, and not enough buses. Sucked that they couldn't pull any bus at Metrotown as I would of loved riding the new XDE60 back to Surrey. Got to Metrotown Stn around 7 and left a bit after 9 and got home a bit past 10.

Some pics I took

Some of the articulated buses that we pulled from other runs, respectfully

A Surrey bus at Metrotown Stn

Meraki · #**12074** Posted **Jul 18, 2014, 7:33 AM**

Quote:

Originally Posted by dpogue

VCC1 is from Waterfront to between Royal Oak and Edmonds.
VCC2 is from Edmonds to King George, but also includes the Millenium Line up to the pocket track between Braid and Lougheed.
VCC3 is the OMC and yard.
VCC4 is the Millennium Line (from Lougheed to VCC-Clark).
VCC2 is the one that went down today, so no service between Royal Oak and King George, but also no service on the Millennium Line for Braid and Sapperton.

Awesome, thanks!

scryer · #**12075** Posted **Jul 18, 2014, 8:15 AM**

Yuck looks like hell. I'm glad that all I have to do is walk to work and school. But for those of you that had to deal with this, I feel sorry for you.

Vancity · #**12076** Posted **Jul 18, 2014, 8:46 AM**

yeah. heard it was a pretty bad situation. glad to hear that things are resolved, though.

maybe the computer system needs an upgrade. lol.

Kisai · #**12077** Posted **Jul 18, 2014, 10:41 AM**

I question why the Bus Driver Union leader would bring this up:

CKNW,Shane Woodford,July 18, 2014 12:59 am

Quote:

"Woods says across the country Skytrain technology is rarely used as other jurisdictions choose more reliable light rail"

The Skytrain is more reliable, because it has no drivers, grade separation so there are no accidents with vehicles and pedestrians, and this outage just proves how well it works. Computer failed, trains stopped, nobody died.

From the same article:

Quote:

Skytrain president Fred Cummings says the entire mess is due to one card.

“One of the computer cards in one of the main computer systems that communicate with the trains failed. Unfortunately it took a lot of time to trouble shoot and find out where the problem really was. We had to replace that card but as a result we had some major disruptions on the system.”

Cummings says this is not a sign of an aging Expo line.

Warmer days = higher chances of electronics failing.

Quote:

Originally Posted by quobobo

"No built-in redundancy" isn't quite accurate - many parts of the control system are redundant.

Sounds like either there was an issue with failover to a backup system, or there are some single points of failure in the system.

Well there was a mention of it being an i/o card, which suggests it started sending indecipherable data, making pin-pointing the problem more difficult, since even a defective part that still has power still has it's tx/rx lights blinking. If it's jamming communications for the rest of the network that's unavoidable.

Anyhow, I'd be more worried if this was something that happened on a monthly basis. Stuff breaks, the average computer part is only built to last 7 years, but there's no reason to replace it if continues to work.

whatnext · #**12078** Posted **Jul 18, 2014, 2:48 PM**

Quote:

Originally Posted by ninjakafi_81

I was caught up in this gong show this afternoon and it was just a cluster f*ck I tell you. People yelling at each other, transit police / security yelling at people, people cutting the line, and not enough buses. Sucked that they couldn't pull any bus at Metrotown as I would of loved riding the new XDE60 back to Surrey. Got to Metrotown Stn around 7 and left a bit after 9 and got home a bit past 10.

Some pics I took

This is why I love my car.

Alex Mackinnon · #**12079** Posted **Jul 18, 2014, 3:16 PM**

Quote:

Originally Posted by Kisai

Warmer days = higher chances of electronics failing.

Probably not. I'd bet it's in an air conditioned server room.

logan5 · #**12080** Posted **Jul 18, 2014, 3:25 PM**

How much would it cost to have a complete back up system? Just switch over to the back up system to get things running again, then try and isolate the problem. These breakdowns seem to be getting more frequent.

Also I'm wondering if there's any kind of emergency response plan for major breakdowns like this because it seems like complete chaos when a system failure happens.