PayPal's international Columbus Day weekend meltdown appears to be over. The site is up and responding to most visitors. Money is flowing and shipments are rolling again.
But like Mount St. Helens in 1980, the fallout from PayPal's cratering will continue for a long time. Sellers are still digging themselves out from the ashes, seeking buried treasures.
"My daily PayPal sweeps [electronic funds transfers from one's PayPal account to one's real bank] haven't been getting into my bank account for the last two days," wrote Rita A. Berkowitz, an eBay reseller of high-end electronics and computer components. "I checked my bank account and nothing, not even an immediate notice of pending transaction like I normally get."
Click here to read about how eBay is trying to woo VARs.
Some sellers have reported total failure of PayPal's e-mail notifications of payments received. Most cite less severe disruptions, with up to 30 percent of payments received failing to trigger e-mail notifications.
Many sellers rely upon e-mail notifications of received payments in their business processes, using the messages to trigger order pulling, packing, printing of shipping labels and other tasks. Today, they must resort to the payment detail pages on PayPal's site, further burdening the still fragile and sluggish system. This manual process also slows order processing and takes sellers away from other important tasks—such as pulling their money out of PayPal as fast as possible.
PayPal debit cards, issued by Bank One Corp. but processed through PayPal, are working again for virtually all cardholders. Some, leery of what might happen during the three-to-four-business-day limbo in which funds languish while being transferred from PayPal to real bank accounts, are raiding ATM machines to withdraw their maximum daily allowances of cash.
eBay maintains that PayPal began melting down during last Friday's "routine monthly update," implying that nothing unusual was done that might account for the concomitant mess.
But in fact, PayPal was heralding the imminent advent of its newly redesigned home page for at least 30 days prior to its official debut on Friday. It was much more than a face lift and monthly defoliation. Conversations on the PayPal Developers forum suggest that the code changes reached deeply into PayPal's API.
Indeed, at least one developer's complaint indicates that the new home page and underlying infrastructure were in place and causing problems a week before its ill-fated debut.
"It seems as though any number of us have experienced considerable problems with login over the past 5-7 days," a user known as ccrosby wrote on the PayPal Developers Web site on Monday. "Ever since PayPal enabled its new homepage last week these problems have manifested themselves."
PayPal representatives did not respond to requests for comment on issues related to the latest outage.
Sellers' attitudes toward the five-day interruption of their cash flows varies from rage to complacency.
"I'm just not going to sweat over the outage as my customers are not responding negatively to it. That's what counts, right?" said Kathleen McCarty, owner of Terra's Treasures. "I've had customers contacting me saying that if PayPal does not come up by tomorrow, they will follow my alternate payment instructions and send me a money order. No PayPal fees!"
eBay has posted the following explanation on its System Announcements page:
"These PayPal issues are the result of unforeseen problems that resulted when a new code base to upgrade the site architecture was introduced to the PayPal platform on Friday morning. The code worked well when tested and during the first hours of launch. Unfortunately, problems handling peak levels of traffic developed later in the day that created intermittent availability and errors for members. These problems have continued in varying degrees since Friday."
Sellers were angry and incredulous to learn that a critical "upgrade" had been implemented without peak-load stress testing and with no provisions for a quick rollback to the last working configuration in the event of trouble. PayPal representatives confirmed that PayPal's architecture does not permit rollbacks, unlike eBay's platform.
Click here to find out why Channel Insider columnist Wayne Rash thinks that testing is vital to any technology rollout.
Most sellers agreed with Andrew C. Auletta, managing partner of coin and currency dealer Brent-Krueger. "It was a day from hell."
"We told the rep that we're discontinuing PayPal as a payment option at our Web site until they call us to advise us of the exact nature of the problem and what they're doing to minimize such incidents in the future. They should consider compensating all of us for the damage they've caused."
Brent-Krueger now accepts three credit cards, money orders and checks. PayPal's woes may become wows for other payment systems. But the vast majority of sellers seem stuck in this ancient bit of folklore:
Place a frog in a pan of boiling water and he instantly leaps out. But place him in tepid water heated slowly to boiling and he will sit there until well done.
David Hakala is a veteran technology journalist and eBay seller based in Denver, Colo. He is currently writing his first book in over 10 years, under the entirely too long working title, "The Kingdom and the Power and the Glory of eBay: How and Why 'The World's Online Marketplace' Succumbed to Fundamentalist Right Wing Nut Jobs, And What You Can Do About It." He can be reached at dhakala@dahdistributing.com.
Check out eWEEK.com's Web Services Center for the latest news, reviews and analysis in Web services.
Commentary 
