Is the infrastructure in San Francisco as bad as it seems?

[vertex]

Thousands remain without power in S.F. (http://sfgate.com/cgi-bin/article.cgi?f=/c/a/2007/07/24/BAG9NR67253.DTL&tsp=1)

(07-24) 16:16 PDT SAN FRANCISCO -- At least 30,000 Pacific Gas and Electric Co. customers in San Francisco and the northern Peninsula lost power this afternoon after an explosion under a manhole cover on Mission Street, the utility said.

Brian Swanson, a spokesman for the utility, said power failures were reported throughout wide swaths of the east side of San Francisco, including downtown and at PG&E's own office on Beale Street near the Ferry Building.

The outage first occurred at about 1:50 p.m., and electricity flickered on and off at least five times before power was restored to some customers at about 3 p.m. Thousands of others were still without power at 4 p.m., Swanson said.

Swanson said the source of the power failure appears to be an explosion in a transformer vault under a manhole in a plaza at 560 Mission St. in San Francisco.

In August 2005, an explosion in an underground vault at Post and Kearny streets critically injured a woman who was walking by. At the time, PG&E blamed high levels of moisture in the attached high-voltage chambers and said it was checking the safety of about 1,000 other high-voltage chambers.

Today's problem was caused by a fault in the line, Swanson said, and there were no injuries reported.

"It is completely unrelated to what happened two years ago," he said.

Witnesses said they heard an explosion at about 1:50 p.m., then saw flames coming from the manhole.

Actor Torino Von Jones, 32, said he was filming a Fruit of the Loom commercial down the block at the time.

"We were standing over there waiting for the camera cue when we heard a big explosion," he said. "Flames came up taller than I am, and I'm 6-foot-2."

"Naturally, when you hear an explosion, you think the worst," said Von Jones, who nevertheless hurried back to work. "We're Fruit of the Loom -- we've got to make this commercial."

PG&E officials are investigating the cause of the explosion.

The outage briefly affected some Muni buses and trains, but all were back to normal by 3 p.m., a spokeswoman said.

E-mail Marisa Lagos at mlagos@sfchronicle.com.


A virtual who's-who of web 2.0 is offline right now; Craigslist, Technorati, Second Life, Netflix, LiveJournal all MIA.

This isn't the first time power outages have affected this area. I'm starting to get used to it, but you would think that these companies would have some kind of backup by this point.

Btw, if a mod thinks this should go into current discussions, feel free to move it.

[fflint]

One the one hand, San Francisco is an old city that could--like other old cities--use a total infrastructure transfusion. Alas--like other old cities--it won't get one.

On the other, we're talking about a spotty 2-hour power outage, for chrissakes. Big f'in whoop! You'll get your craigslist fix later tonight.

[vertex]

Yeah, only this isn't the first time this has happened. Second Life alone has had 3 outages in the 2 months since I joined, all of them stemming from their SOMA offices.

[Master Shake]

Nerds get political, very exciting.

I don't know about the electrical system, but the buses in SF are god awful, no AC, chuggin along on those weird wires, now that is infrasctructure that should be approved.

Websites should invest in generators and back-up locations.

[NewYorkYankee]

Like most old cities, SF has outdated infastructure. The money to put in new infastructure simply isn't in the budget of any major utlity. And the government is thoroughly uninterested in "maintanence" of any kind, becuase that isn't "sexy".


And take note, becuase when the "new cities" become "older cities", same shall happen.

[the94112]

I work in Starbucks in the Financial District, it was horrible, it was crowded, hot, and then BAM, power goes out, then we have to turn all the computers and alarms back on, just for it to go off again, a bunch more times after.

[krudmonk]

It's not that common. Stuff happens.

[LMich]

Like most old cities, SF has outdated infastructure. The money to put in new infastructure simply isn't in the budget of any major utlity. And the government is thoroughly uninterested in "maintanence" of any kind, becuase that isn't "sexy".

And take note, becuase when the "new cities" become "older cities", same shall happen.

I agree with most of that, but would argue that federal government is thoroughly uninterested in maintanence not so much because it's unsexy, but because they know they'd have to ask Americans for more of their money. That, alone, is one of the best motivators not to maintain infrastructure, as the word 'taxes' was made a slur decades ago. People bitch about our poor infrastructure when it goes horribly wrong, but they don't want to put in the money needed to keep it in even adequate condition. I guess as with any person we want to eat our cake, but then have it, too.

There are exceptions at the municipal level, though. I know my city is replacing the oldest part of our sewer system built in the 19th Century.

[BTinSF]

Nerds get political, very exciting.

I don't know about the electrical system, but the buses in SF are god awful, no AC, chuggin along on those weird wires, now that is infrasctructure that should be approved.

Websites should invest in generators and back-up locations.

Hmmm. It always helps to have a little information about what you saying.

- The busses: No, they aren't air conditioned. Niether are most homes in SF. It's an unusual summer day that breaks 70 degrees. Why should we waste energy when open windows do the job. Call 'em "green busses".

- The wierd wires: San Francisco, in the early 20th century, muscled through Congress the right to dam a valley in the Sierras called Hetch Hetchy and create both a pristine water source and a source of free electicity which it now uses to power a number of city utilities including the electric trolleys which, besides using free power, also are, of course, pollution free and, surprisingly to some, better able to climb the city's steep hills than diesel busses.

- Web sites and back up generators: The web sites in question are actually servers at a commercial "server farm" run by an independent company at 365 Main St. This building is a large one completely renovated in the late 90s (during the dot-com boom) with huge air conditioning, bank-like security and, supposedly, a secure power source specifically for this purpose. I am therefore unclear on how today's problem shut it down, but it's not from lack of maintenance or any other obvious cause. I'm sure the affected companies will find out.

- SF's power grid: We do seem to have a problem specifically with transformers that seem to blow up with some regularity. My condo lost power for 2 hours last Wednesday when a transformer went (we got a lot less publicity). The local utility, PG&E, had severe financial problems 5 years or so ago (after the state forced them to sell their power plants and buy power on the open market just before the infamous "CA energy crisis"). It's possible they have scrimped on maintenance, but they are a private publically traded company.

[Sonofsoma]

Wow... What a topic. (*The following is sure bring out the rabid defenders/apologists of SF's profoundly dysfunctional "progressive" govt in control of our City)

Despite an enormous budget of 6+ Billion Dollars, fattened with a once in a lifetime tax windfall generated by real estate transfer taxes, The City (and County) of San Francisco has virtually no capital expenditure program.
..with the exception of a City owned & operated Nursing Home now under construction that's years late and over budget by at least 100%,

The City's' infrastructure is a Disgrace. The roads are a mess. Major thoroughfares such as Mission, Divisadero, Battery, Van Ness and Valencia are in a state of disrepair that would be unimaginable elsewhere in America.

Consider a few of SF's $1Billion+ looming infrastructure projects not being seriously addressed:

- The total reconstruction of seismically unsafe SF General Hospital. (City politics, meanwhile holds up a new $2.1B CMPC Medical Center)

- Replacement of the grim Soviet-style Hall of Justice bldg. (home of Police HQ, most Courts and the main Jail)

- Cruise Ship Terminal. Believe it on not, San Francisco does not have one. (SF uses an ancient cargo wharf/shed that is easily the most decrepit cruise ship facility in the developed world. No Joke.)

- Billions of deferred maintenance work is needed to prevent what remains of The Port of SF's landmark piers from falling into the Bay.

- At SFO.. an inefficient and dangerous airfield configuration with runways that intersect like a tic-tac-toe board needs to be totally reconfigured. (Tourism is SF's biggest industry.. and 85% arrive by air)

And that's just for starters.

Getting back to the electrical power infrastructure in question... most outsiders would be SHOCKED to see the how the vast majority of San Francisco's neighborhoods are blighted with thousands of 100-year-old wooden phone poles and wires (electric, TV-cable, telephone and now, broadband) that connect to every home.

The average block in San Francisco's largest neighborhoods; the Sunset & Richmond, has about 20 utility lines strung between 30-40 poles. In addition, each home connects to the nearest pole with 4 or 5 wires suspended over lawns and across streets. The City does have a "utility under grounding plan" but.. it is estimated to take 122 years to complete

.. Sound backwards? It is. ..did I forget to mention most apartment buildings don't have individual water meters either?

[pip]

^not different from any other city?

[UglymanCometh]

Getting back to the electrical power infrastructure in question... most outsiders would be SHOCKED to see the how the vast majority of San Francisco's neighborhoods are blighted with thousands of 100-year-old wooden phone poles and wires (electric, TV-cable, telephone and now, broadband) that connect to every home.

The average block in San Francisco's largest neighborhoods; the Sunset & Richmond, has about 20 utility lines strung between 30-40 poles. In addition, each home connects to the nearest pole with 4 or 5 wires suspended over lawns and across streets. The City does have a "utility under grounding plan" but.. it is estimated to take 122 years to complete.


Does that have anything to do with the possibilty of underground wiring being very expensive (especially in a seismically active zone)? Here in Detroit, we have the same thing with the wooden utility poles on the vast majority of our streets (including major arteries).

[urbanflyer]

- At SFO.. an inefficient and dangerous airfield configuration with runways that intersect like a tic-tac-toe board needs to be totally reconfigured. (Tourism is SF's biggest industry.. and 85% arrive by air)


Sorry but it's nothing short of hyperbole to suggest SFO's configuration is 'dangerous'. Inefficient? Yes. Below the minimum standards required by the FAA for runway separation? Yes. Dangerous? No. SFO has far fewer total runway incursions and near-misses than LAX, owing largely to the fact that the frequent poor visibility calls for reduced operational capacity that slows everything down but also cuts ATC workload in half.

Not to mention that there haven't been any serious incidents at SFO since a JAL DC-8 landed in the bay and a Pan Am 747 struck runway end approach light structures, both in the 1960s.

[Abner]

^not different from any other city?

Yeah, Chicago has much worse things going on than that and nobody would argue that it's because city government has progressive ideals. In fact most of the things on that list are exactly the same here. Crumbling bridges and roads: check; deteriorating Soviet-quality criminal courts and jail: check; exploding transformers: check; airport with crossing runways: check; and in addition you can add catastrophically failing transit infrastructure and the worst freight rail congestion in the country. Everybody's got problems. And I fail to see how above-ground utility poles, which exist virtually everywhere, constitute decrepit infrastructure. In fact, how many cities in this country DON'T have above-ground utility wires?

[WonderlandPark]

True, SFO does have a great safety record, so does the Bay Area as a whole. Unlike Southern California.

[craeg]

How often has the power grid in the financial district of manhattan flickered on and off 5 times in the span of 90 minutes?
As someone who was directly affected by yesterdays outages, I have to say YES our infrastructure is as bad as it seems. PGE doesnt even know what happened yesterday. And those busses with no AC - argue all you want about summer temps but consider that in the winter those ac units can be removing the abundant amount of moisture that makes the busses moving saunas. Honesltly should be be paying how much hundreds of thousands for each bus and have them not throw in an AC compressor?
Yes power outages and disasters happen in every city - but they happen way more frequently here and with practically no consequence or change in behavior

[Gordo]

How often has the power grid in the financial district of manhattan flickered on and off 5 times in the span of 90 minutes?
As someone who was directly affected by yesterdays outages, I have to say YES our infrastructure is as bad as it seems. PGE doesnt even know what happened yesterday. And those busses with no AC - argue all you want about summer temps but consider that in the winter those ac units can be removing the abundant amount of moisture that makes the busses moving saunas. Honesltly should be be paying how much hundreds of thousands for each bus and have them not throw in an AC compressor?
Yes power outages and disasters happen in every city - but they happen way more frequently here and with practically no consequence or change in behavior

As BT mentioned, PG&E is a private company.

And as a user of buses in SF for many years, I can think of about once that I longed for AC on a bus - I would be screaming and yelling if Muni spent extra money to put AC on the buses.

[BTinSF]

There's so much cr*p being thrown around here indiscriminately, it's hard to clean up.

- Once again, PG&E is a private stockholder-owned company and the shortcomings of its maintenance program are only slightly to be blamed on the city (since the city and state do regulate it).

- SFO spends plenty of money (witness the new terminal complex) and also is a quasi-independent agency with its own bonding authority. Its problem is geography--no room to expand--not budgetary.

- The Hall of Justice: Well, in recent times we've built the new "glamour slammer" addition and a new jail in San Bruno. Again plenty being spent but maybe not being spent well since there was no budget to staff these facilities after they were built.

- The port: Another quasi-independent agency which is largely self-financed by renting the land it owns except the city politicians have so constrained what they they can rent it for that the income is not adequate. Still, there is money for the cruise terminal sitting in the bank if they could ever decide where to build it and what it should look like.

- SF General Hospital: Fewer and fewer cities are even bothering with a city-run major "hospital of last resort" these days and SF is planning a brand new one--what are you complaining about?

- The streets: Everybody's right. We could spend more on street repair. Who couldn't? But in SF some of the worst are state highways--like Van Ness and 19th Ave--so don't blame the city. Most forumers complain when money is budgeted for highways rather than public transit.

[craeg]

As BT mentioned, PG&E is a private company.

And as a user of buses in SF for many years, I can think of about once that I longed for AC on a bus - I would be screaming and yelling if Muni spent extra money to put AC on the buses.

I didnt say spend extra, I questioned why we have each bus hand-built at several hundred thousand dollars each - and they cannot just throw in an AC compressor.
Having a way to dehumidify the cabin in the winter would go a very long way to making using the bus in the rain much more comfortable.

[Top Of The Park]

....I had no idea

[mthd]

And as a user of buses in SF for many years, I can think of about once that I longed for AC on a bus - I would be screaming and yelling if Muni spent extra money to put AC on the buses.

i can think of a few times.... in almsot 10 years or so. the problem with SFs trollley bus network is not the buses, it's the streets and the congestion on the streets they use.

i can think of very few cities that have such a high percentage of zero emissions transit vehicles. light rail, bart, cable cars, single and artic trolley buses, etc. many lines improved their level of service tremendously when they switched to electric. before the 1 cali was electrified people had to get off the bus and run up the hill to get back on because the old diesel buses couldn't get up a hill carrying a crush capacity crowd. the newer electric buses are fast, quiet, nonpolluting, and generally clean.

i don't think san francisco's infrastructure is any worse in general than other cities of similar age - and keep in mind that there is only one city in the country with higher population density. things get a lot harder when you don't have lots of cheap land to build on.

as for SFO... i do not think there is an airport in the country i prefer. i can walk out the door at the office, get on a clean, fast, affordable train and be at the airport in 30 minutes. the airport itself is spacious, airy, and even beautiful in places. the main hall at the international terminal is among the greatest public spaces in the country. could the airport use another pair of runways? yes. but, again, we're not in denver or houston here. building another airport 30 miles away is not an option.

[Gordo]

I didnt say spend extra, I questioned why we have each bus hand-built at several hundred thousand dollars each - and they cannot just throw in an AC compressor.
Having a way to dehumidify the cabin in the winter would go a very long way to making using the bus in the rain much more comfortable.

Even if they can just "throw in" an AC compressor, AC compressors would cost money to maintain and use extra fuel/electricity to run. Incredible waste of resources in one of the mildest climates in the world.

[BTinSF]

The blackout blues
PG&E leaves its customers in the dark more often than do state's California's other big utilities

David R. Baker, Chronicle Staff Writer
Thursday, July 26, 2007

Pacific Gas and Electric Co. customers endure more frequent and longer-lasting blackouts than other Californians, state data show.

Tuesday's power outage in San Francisco and the Peninsula was no isolated incident. In 2006, the average PG&E customer lost power for more than 4 1/2 hours, according to statistics compiled by the utility and submitted to state energy regulators.

In contrast, Southern California Edison's average customer lost half as much time to blackouts -- not quite 2 1/2 hours. And residents of San Diego fared even better, spending less than an hour without power all year.

PG&E's performance exasperates many of its customers and has fueled unsuccessful attempts in several cities -- including the company's hometown of San Francisco -- to break away from the vast utility.

San Francisco Mayor Gavin Newsom expressed frustration Wednesday that the city had experienced another blackout.

"We've been through this so many times," he said after his staff met with PG&E representatives. "There's nothing more I can say. I've already said (everything) to them. Should I handcuff them? Arrest them? Should I bring them all to justice? Should I sue them? Obviously, we're not happy. Obviously, they're not happy."

PG&E still had not determined the cause of the blackout by Wednesday afternoon. But company representatives said they had made progress improving service in San Francisco in recent years, including installing another high-voltage line on the Peninsula to bring power into the city. Utility President Bill Morrow called Tuesday's outage unacceptable.

"The performance does not reflect the level and quality of service that we are committed to providing customers and that they expect from PG&E," Morrow said in a prepared statement.

Causes of Bay Area blackouts have varied from the startling to the mundane.

In 2003, for example, a fire at a San Francisco substation cut power to 120,000 customers the weekend before Christmas. In contrast, about 17,000 customers in the East Bay lost power last week in outages that the utility blamed on light rain, which mixed with dust to form mud, which caused equipment to short.

Many customers were stunned that such minor rainfall -- less than a tenth of an inch in most places -- could wreak such havoc.

"It was the most ridiculous thing I ever heard," said Rebecca Renfro, whose Berkeley home went without power for roughly 12 hours. "I thought, 'Wait a minute, this was not the first time in the history of the universe that there has been drizzle.' "

The company's blackouts occasionally have cost it money. PG&E agreed to pay $6.5 million in lieu of fines for the December 2003 outage. And the company's overall outage performance in 2005 triggered a $9.2 million penalty from the California Public Utilities Commission.

California's utilities keep tabs on their annual performance by compiling statistics that measure the frequency and length of blackouts year after year. One measurement -- known as the system average interruption duration index, or SAIDI -- adds up the duration of blackouts experienced during the year and divides that by the company's total number of customers.

PG&E's average customer went without power for 280.5 minutes last year, according to data compiled by the company and submitted to state energy regulators. Southern California Edison's average customer was blacked out for 142.3 minutes, and San Diego Gas and Electric's, 52.8 minutes.

The Sacramento Municipal Utility District recorded 99.3 minutes without power for the average customer last year. The Los Angeles Department of Water and Power did not have comparable data available for 2006.

Utilities use a similar statistic to measure how often blackouts occur. Called the system average interruption frequency index, or SAIFI, this measurement counts the utility's total number of outages during the year and divides that by the number of customers.

There again, PG&E fared worse than its fellow California utilities in 2006.

The company's average customer lost power 1.7 times, compared to 1.1 times for Southern California Edison, 0.5 times for San Diego Gas and Electric Co. and 1.4 times for the Sacramento Municipal Utility District.

PG&E representatives often point out that their electrical distribution system is larger than the other utilities', spanning the territory from the Humboldt County coast to Bakersfield. It serves more customers -- many of them living in remote and rugged terrain -- and endures more frequent storms than the southern part of the state.

But some utilities in states with far worse weather still manage to rack up better performance statistics than PG&E.

Wisconsin, which endures blizzards in the winter and severe thunderstorms in the summer, had only one of the state's five large utilities reporting more than 160 minutes of outages for the average customer in 2006. In neighboring Minnesota, the highest number was 118 minutes.

Ed Salas, PG&E's senior vice president for engineering and operations, said not all utilities compile their data in quite the same way, even if they use the same basic formula. That's true when comparing within California as well as state to state, he said.

"You don't really have a clean ability on an apples-to-apples basis to compare, from one utility to another," Salas said.

California energy regulators allow utilities to compile a second set of statistics excluding outages that strike during government-declared emergencies or freak weather events, like last July's record-setting heat wave. By that standard, PG&E's performance has been improving steadily for three years. The average customer last year experienced 150.8 minutes of blackouts, compared with 187.1 minutes the previous year and 205.1 minutes in 2003.

Without those exceptions, PG&E's performance has been worsening since 2004, when the utility's average customer experienced 205.3 minutes of blackouts.

Salas said excluding unusual weather from the statistics shows the system's underlying health. States along the Gulf Coast, for example, would have seriously skewed statistics if they counted outages during hurricanes.

"If you're in the South, I'm not sure that including the Katrina effects would really tell you how your system performs," Salas said.

Still, some of the company's critics claim PG&E is excluding too many storms from its data in an effort to make the utility look better.

The Utility Reform Network, a local watchdog group, recently filed a complaint with state regulators after PG&E requested a $151,899 bonus for meeting performance goals set by the state. The utility excluded blackouts caused by minor winter storms, said the group's staff attorney, Matt Freedman.

"It's easy to get a reward when you remove all the days when you have outages," said Freedman, whose group argues that PG&E should face a $5.1 million penalty instead.

http://sfgate.com/cgi-bin/article.cgi?f=/c/a/2007/07/26/BLACKOUTS.TMP

[fflint]

Is the corporation that holds a monopoly on providing electric power to San Franciscans as bad as it seems?

Yes.

[MayDay]

A tad off topic, but that's precisely why Cleveland went into default in 1978 (to avoid having such a corporate monopoly).

A local bank wanted Cleveland to sell their city-owned assets like Cleveland Public Power to repay some debts. Then-mayor Dennis Kucinich knew if that happened, electric power would have been under the sole control of a private corporation. Thus, the banks threw the city into default on those debts.

Just a little history lesson, kids :)

[harryc]

For a bit of perspective visit London - They make Chicago look all new and shiny.
Try taking the subway with no AC, they had signs and announcements reminding people to take water with them in the eventuality that the subway would be stuck.

[craeg]

PGE provides the power infrastructure in SF - we have no choice. Is the distinction that PGE sucks versus our infrastructure sucks important?

[BTinSF]

^^^Yes, because the argument was also made that the reason for the problems had something to do with liberal politics and/or governmental will when, in reality, it has more to do with capitalist parsimony and the desire to increase private profits.

Oh, and we do have a choice. As the Bay Guardian nevers stops saying, we COULD municipalize the power system in SF. I'm not saying we should, mind you.

[fflint]

Circuit breakers wipe out the Web
PG&E’s faulty equipment reveals the Internet’s vulnerability to a disruption of its power source

Verne Kopytoff
The San Francisco Chronicle
Thursday, July 26, 2007
http://sfgate.com/cgi-bin/article.cgi?f=/c/a/2007/07/26/BUQMR757R2.DTL

http://sfgate.com/c/pictures/2007/07/26/bu_poweroutages001.jpg
An employee enters the building housing mainframes at 365 Main Inc. that were silenced by the power outage. Chronicle photo by Penni Gladstone

All it took to wipe out some of the Internet's biggest sites Tuesday was some faulty PG&E electrical breakers that caused a blackout in downtown San Francisco.

Some of the Web's hottest destinations - Craigslist, Yelp, Second Life - were suddenly inaccessible from San Mateo to Singapore after back-up generators failed at the facility housing their computer equipment.

Although mostly fixed within 12 hours, the incident shows how easy it is to send major swaths of the online world to the dark ages. Sites that millions of people rely on can be knocked offline by freak accidents, not to mention major catastrophes, and this event served as a wake-up call to the executives that operate them.

"If the data center was that vulnerable to a power outage, what if something really catastrophic happened like an earthquake?" asked Derek Gordon, marketing vice president for Technorati, a search engine of blogs that was brought down for a couple of hours Tuesday after the blackout. "What does that say about the vulnerability of the Internet in the Bay Area?"

The troubles started when 365 Main, a key data center in downtown San Francisco that touts its "state-of-the-art electrical system," failed to get its backup generators started immediately after the power outage hit around 1:45 p.m. A number of companies that house their computer servers in the facility were suddenly offline, setting off a mad scramble to get the Web sites up and running.

Shoppers at RedEnvelope, an online retailer, couldn't buy monogrammed pillow soaps. Hipsters on Yelp, the review site, had to take a break from sharing their reports of fabulous and not-so fabulous restaurants. Users of online classified service Craigslist were out of luck in finding a second-hand futon.

The backup generators were turned on 45 minutes after the blackout started, a delay that 365 Main said it was still investigating yesterday. But it took some of the facility's customers anywhere from another hour to 11 hours to get their servers safely rebooted and their Web sites operational.

What the episode exposed is that some companies operate entirely from one data center, a decision described by some security experts as risky. In emergencies, such companies can't shift traffic to an alternative facility where they keep additional servers.

"There's all kinds of things that can happen from a power outage to a tornado to a backhoe," said Jason Needham, director of product management at F5 Networks, a Seattle company that sells software and equipment for data centers. "All these things seem far-fetched until they happen."

However, Needham said the trend is for companies to put all their eggs in one basket, so to speak, in an effort to save money. In fact, just hours before Tuesday's power outage, 365 Main put out a press release trumpeting the fact that RedEnvelope had moved all its operations to its facility and closed an unneeded center in the Midwest.

Data centers are usually designed with redundant equipment to ensure power during outages, earthquakes and floods. Backup electricity is supposed to kick in within seconds after an outage through a complex system that keeps servers humming without interruption.

Internet companies pay thousands of dollars, minimum, to house a few servers in such a secure environment. In most cases, the precautions work.

"The well-designed data centers, this should be handled pretty seamlessly," said Michael Foust, chief executive of Digital Realty Trust, a San Francisco company that owns 60 data centers globally, including one in San Francisco that was not affected by the outage.

However, the rare snafu or major disaster can create havoc, such as the Sept. 11, 2001, terrorist attacks or the flooding of New Orleans caused by Hurricane Katrina. In those cases, many companies were thrown offline.

Google, Yahoo and eBay all keep multiple data centers across the world, partly because proximity makes Web pages load faster and partly for security.

Jim Buckmaster, chief executive of Craigslist, which got back to normal operations after 11 hours of scrambling, said that his company is in the process of moving to a second center but that it will take a few more weeks.

Gordon, from Technorati, called opening several data centers ruinously expensive for thinly funded Internet startups, of which there are hundreds in the Bay Area. Only profitable companies can afford such an extravagance, he said, though he acknowledged that Technorati, which isn't profitable, is in the process of moving into a second facility.

Tuesday's outage "added to the sense of urgency," Gordon said.

"The lesson here is despite all of your planning and all of your promises, you are vulnerable."

E-mail Verne Kopytoff at vkopytoff@sfchronicle.com.

http://sfgate.com/cgi-bin/article.cgi?f=/c/a/2007/07/26/BUQMR757R2.DTL

[BTinSF]

^^^I was just about to post that. Now the truth comes out. The 365 Main data center scr*w*d up--and my guess is they will pay a price for that in lost business. As the article says, these places sell themselves on NOT doing axactly what happened: going down in a power failure. Within that company, somebody's head should roll because the backup power system should be something they test regularly.

[fflint]

Transit delayed in San Francisco

The San Francisco Chronicle
Friday, July 27, 2007

(07-27) 08:04 PDT SAN FRANCISCO -- Commuters should expect delays on both BART and Muni in San Francisco this morning.

Muni's K, L and M Metro lines are delayed because of overhead line damage at the West Portal Station. Trains are unable to enter the West Portal tunnel, a spokeswoman said, and the transit agency is providing bus shuttles between the West Portal and Van Ness stations. The shuttles are expected to remain in place throughout the morning commute.

The other Metro lines are running normal, the transit agency said.

BART is also experiencing delays. Trains on the Daly City line headed to the East Bay are now delayed 10 to 15 minutes.

Passengers at the Rockridge Station heading to Orinda on the Pittsburg/Bay Point line toward Daly City should also expect 25-minute delays, and passengers headed toward Pittsburg/Baypoint will have 10 minute delays.

The transit agency blamed equipment problems for the delays.

http://sfgate.com/cgi-bin/article.cgi?f=/c/a/2007/07/27/BAGLHR8AKT3.DTL

[sf_eddo]

Haha, the day of the power outage, I had lunch with good ol fflint, went back to my office, and promptly got stuck in my elevator when the lights went out. :)

[FourOneFive]

Oh, and we do have a choice. As the Bay Guardian nevers stops saying, we COULD municipalize the power system in SF. I'm not saying we should, mind you.

the real question is who would run the power system better in san francisco: PG&E or the city of san francisco. IMO PG&E is the lesser of two evils.

every utility in the country has its faults. look at con edison here in new york. ugh.