An email nightmare that lasted nearly a week and affected up to 7,000 customers has finally come to an end, according to Whidbey Telecom officials.
On Friday, the South Whidbey-based telephone, internet and email provider announced that people may still be receiving a few old emails but that the new system is operating as designed and that customers should have full access to their accounts.
“I deeply, deeply regret the terrible effect this has had on our customers,” said co-CEO George Henny, who on Thursday returned from a trip to South Korea.
“We’re just sick about it,” he said.
Late last Friday, April 20, the company launched a planned update to its old email system. Officials had been working with the service provider’s Philadelphia-based vendor for months in preparation for the changeover and had been banking on a quick two-day transition.
However, minor glitches identified over the weekend snowballed into four days of major service interruptions and pulling-your-hair-out frustration for thousands of customers as they struggled with slow or no connection to their email accounts.
It was particularly hard on merchants and business owners. Dan Vorhis, owner of Muscle and Arm Farm in Freeland, relies on email to make a living. From receiving days-old orders to the uncertainty of what never came in, the situation was not only frustrating but downright bad for business.
“If people can’t communicate with me, that’s a problem,” said Vorhis, in an interview Wednesday. “One day is bad, two days is terrible, three days is unheard of, and here we are in day four and five.”
Drew Kampion, the distributor of an email list with about 3,350 South Whidbey subscribers, 1,200 of whom are Whidbey Telecom users, also saw severe service disruptions.
While he was able to publish several email batches over the weekend, he had to cancel all those scheduled for Tuesday and some for Wednesday morning. Email service was down for much of Thursday as well.
“It’s really slowed me down,” Kampion said.
However, for a man who’s personal motto is, “Life is a wave. Your attitude is your surfboard,” Kampion took the problems in stride.
“I got a day off so there was a silver lining to this,” Kampion said.
In a Friday morning interview, Whidbey Telecom COO Bruce Russell sat down with The Record to explain just what happened. He also apologized to those affected by the email snafu.
“We are so sorry that this happened with our customers,” Russell said. “This is certainly not to our standards, it’s not up to our expectations and definitely not to the expectations of our customers.”
As expected, there were some service interruptions over the weekend during the initial changeover. But by 9 a.m. Monday, the time the transition was supposed to be complete, it was clear there was a big problem as customers were experiencing a host of issues, from sluggish service to not being able to connect at all.
By Tuesday, things worsened as the new system had to be taken offline completely in order to troubleshoot and pinpoint the problem.
Company officials believed they had the situation largely under control by Wednesday when it was reported that the new system was “up and stable” though it was laboring under a massive amount of backlogged emails.
During a normal business day, Whidbey Telecom processes over 800,000 email transactions and over 35,000 emails per hour during peak traffic hours.
Yet, by Thursday, the company had to take the system offline again because it was still not performing as expected. The move resulted in customers having no email access for much of the day.
Technicians would eventually discover a faulty protocol in the new system and that may have been the source of the email fiasco. It was fixed and service was largely restored by early Thursday evening.
According to Russell, the company will be closely examining everything that happened over the past week to see if the faulty protocol was really the root problem or if there were other contributing factors.
“This (service loss) was the worst scenario; we never imagined we would be at this state,” Russell said. “This was totally unexpected.”
The company will also be looking at what could have been done better, particularly in the communications department. Despite posting numerous daily updates on its website and fielding thousands of calls, the service provider saw hefty criticism from customers who felt uninformed about what was happening.
“We’ll definitely be looking at that,” Russell said.
While no decisions have been made, the company is considering holding a public forum to gather ideas about what could be done better in the future, though Russell promised the events of this past week would never be repeated.
“Rest assured, this will never happen again,” he said.
Things got so bad, Russell acknowledged that Whidbey Telecom had begun helping some customers get set up with its subsidiary FiberCloud. Without a doubt, the company lost customers to other service providers as well, he said.
After 104 years of business, both Henny and Russell said Whidbey Telecom has earned a solid reputation and the trust of its customers with years of consistent and reliable service.
That trust was shaken over the last week and they said they are eager to roll up their sleeves and earn it back.
“We are deeply sorry,” Russell said. “This is not to our level of service, we have a high bar … we are going to gain that confidence back.”