C&P still looking for cause of phone outage Officials suspect glitch in software programming

June 28, 1991|By Leslie Cauley

Chesapeake & Potomac Telephone Co. still doesn't know what caused the phone network in Maryland, Washington, Virginia and parts of West Virginia to crash Wednesday, and, unfortunately for phone users, there are no assurances that it won't happen again.

A master computer in Baltimore alerted C&P on Wednesday that there was trouble with the network, but by nightfall the cause and origin of the problem was still eluding engineers.

C&P engineers have determined that a main computer in Baltimore used to sort and route calls appeared to overload. The failed computer shifted traffic to its mate in Pikesville -- a backup feature that is supposed to prevent crashes -- but for some unknown reason the second computer also went into an overload mode.

According to Larry Plumb, a spokesman for Bell Atlantic Corp., C&P's parent company, the overflow from Baltimore wound up affecting two master computers in the Washington area. When those two computers shut down as well, the result was a telephonic burnout throughout the region.

The errant Baltimore computer, housed in a C&P facility at 323 St. Paul St., was under constant surveillance yesterday as engineers searched for an answer to the question that was on the lips of customers Wednesday: Why did it happen?

Kenneth Pitt, a spokesman for Bell Atlantic in Arlington, Va., said that it may be some time before the company has an answer to that question.

"Think of it an as airplane crash, and we're the National Transportation Safety Board," he said. "We're still looking for the black box."

At about noon Wednesday, the computer malfunction caused up to 5 million people to lose access to local phone service. The outage, the worst in memory at C&P, affected 2 million business and residential customers in Maryland alone.

By 9:30 p.m., the system had been fully restored. Since Wednesday evening, no new problems have been reported.

C&P engineers now are trying to determine why the system crashed. At this juncture, a software programming glitch is suspected.

"We're doing a lot of data analysis and a lot of self-diagnostic comparisons," said Jeanine Smetana, a C&P spokeswoman. "All that needs to be reviewed to see if we can determine the chronology of events."

Phil Freedenberg, president of Federal Engineering Inc., a Fairfax, Va.-based telecommunications consultancy, said that the precise reason for the system's crash may never be known.

The reason: Engineers may never be able to re-create the precise set of circumstances that led to the blowup, leaving the technical powers-that-be at C&P to engage in a protracted guessing game involving thousands of lines of computer code and an ever-changing kaleidoscope of calling patterns.

C&P's network is built around Signaling System 7, or SS7, a generic call-sorting system that is being installed by all the regional Bell phone companies.

SS7 systems use powerful computers, referred to as "call transfer points," to sort and route calls through a maze of electronic switches. Call transfer points are paired to back up each other in the event of a catastrophic failure. As a safety feature, all linkages within the system are duplicated and sometimes triplicated to provide alternative routing.

In short, SS7 systems are designed to make call-handling faster, more efficient and more reliable.

But Mr. Freedenberg said that it creates an interdependence among switching stations that has a definite drawback: When central transfer sites fail, the switches that depend on them fail as well.

"I'm not surprised C&P can't pinpoint the problem," Mr. Freedenberg said. "C&P may never be able to pinpoint it. That's the scary part: There's no guarantee that there's not another bug in it somewhere else that will send it off into Never-Never Land a week or month from now."

State and federal regulators, meanwhile, are waiting for the results of the Bell Atlantic investigation.

Frank Fulton, a spokesman for the Maryland Public Service Commission, said that the PSC already has asked Bell Atlantic to provide it with a full accounting of this week's events.

Likewise, the Federal Communication Commission has been in touch and has asked for a full accounting of the network failure.

Baltimore Sun Articles
Please note the green-lined linked article text has been applied commercially without any involvement from our newsroom editors, reporters or any other editorial staff.