77 Comments

I think what a lot of this comes down to is cost. We've spent a generation 'optimizing' every system to squeeze out cost and emergency planning is inherently a cost with a once in a blue moon payback. We've similarly optimized supply chains and healthcare so we can continue to enjoy lower prices and lower taxes.

I agree that we should put some of that cost back in the system, but history seems to prove that Canadians are cheap and don't like paying for things that don't provide immediate gratification. Moreover, most (but not all -- see the healthcare system) of the failures have been inconvenient to most people which doesn't provide much motivation to change behavior.

The problem is us and our sense of entitlement to new stuff for cheap!

Expand full comment

"Can someone please tell me why I apparently have put more thought into keeping my daily two-hour current events chat operating and online than Interac...?"

Because you have no Accounting department. Large companies have specialists that cut expenses, professionally. Their pressure is relentless. Where professional engineers are able to intercede, state that there are standards, unacceptable risks, Accounting can be fought off. In a large decision, involving many millions, the engineer MUST to have government laws/regulations to point to, or you will be overruled. Often they'll spend years overruling you, chopping away at the defenses bit by bit, but they get there.

"A failure of imagination, not of calculation", oddly, was my much-repeated line at the military conference I gave my "Titanic" talk at. They just didn't imagine the need for so many watertight compartments, for a double-hull - as had been on the earliest liners. Every generation of ships had fewer redundancies, less resiliency, more economic efficiency. From the "Great Eastern", with 50 watertight subdivisions, until you got the Titanic, with six. (Took sixty years. Accountants never sleep.) Then, the Titanic stimulated new REGULATIONS for watertight compartments, ships had to be forced to those designs, or economic pressures would relentlessly have made them more efficient, and less safe, again, in a few generations.

It's always that way: name a car safety feature that became universal because of free-market competition where customers always chose safety over savings. Never happened. They all had to be mandated. Computer network safety features are the same way: an efficient company will always beat a resilient company, except on that five-thousandth day; by then, the resilient companies are broke.

Matt's $150/month doesn't render him unable to out-compete other 'casters; but in a neck-and-neck competition for investors, a company with 7% profits, pays investors 14% more for their investment than one that makes 6%. If they can do that legally, then anything they can legally cut, will be, no matter how "prudent" your risk-managers say the extra network is.

http://www.cuug.ab.ca/branderr/titanic/

Expand full comment

I've been saying this since Friday. Rogers has SLAs that they violated and they'll pay the penalties and the market should take care of it. Bell and Telus reps will displace a lot of rogers volume over the next few months and then we'll all forget. We should be truly appalled that there are critical organizations who have carefully laid all of their eggs in one basket which is, frankly, full of egg sized holes.

Interac is a key operator in one of the most regulated industries in Canada and they don't have a redundant internet provider? That's Obscene. Health Services losing connectivity from a single outage? Criminal negligence. Even if it was difficult to imagine a systemwide failure from rogers prior to last year, this failure was a repeat performance. Anyone who was down as a result of the rogers failure is down because they evaluated their options and cut a corner.

NO ONE planning a critical service that requires connectivity just skips the discussion of redundant systems. These conversations weren't skipped, they were had, and they decided it wasn't important enough for redundant connections.

Rogers will probably lose my personal business, mostly because they seem to think it's "fixed" but i still lose connection for a minute at least every half hour or so. So their definition of fixed and mine are not the same. But they will get punished by the market. The market doesn't currently have a way to punish Interac, or the various critical government services that didn't appropriately plan. We need to deal with THOSE too or we're just rewarding the incompetence that got us here.

Expand full comment

Maybe if we had a proper Canadian Communications Commission reporting to Parliament through ISED that focused on maintaining a dependable, accessible, affordable $70 billion communications system . . . But no, we have the CRTC, reporting through Heritage that only cares about Cancon in a $19 billion broadcasting industry and only appoints leaders with broadcasting experience. Not that there’s anything wrong with broadcasters, of course.

Expand full comment

I particularly appreciated this commentary. Having worked in public service for many years, working on redundancy is a key part of risk management. Unfortunately redundancy costs something as your personal example shows. In business it is the cost of doing business.

In a hospital that is trying to deal with staff shortages and overcrowding, redundancy starts to slip down the cost priorities. Lord help anyone who suggest increasing revenues to address the issues. The cacophony of voices that arise to cut this service or that service at the mere suggestion of increased revenues makes it difficult since most suggestions are for services that the individual or business is asking to be cut is not a service that they use or access.

I also agree that we all need to build redundancy where we can, individually, locally, nationally in business or public sector. We need to recognize at the same time that there are many Canadians that can't afford to. And so the redundancy has to be seen as a public good and not necessarily an individual one. A public policy problem if I've ever seen one.

Expand full comment

I spent much of my career providing critical systems to major communications companies around the world including most of the Canadian ones. What happened at Rogers (a failed upgrade of their core network) shouldn't have happened but it wasn't a surprise. These companies have drifted away from the core mission, providing essential communication solutions to Canada, to value added content (i.e. the NHL) in order to drive more revenue. As a consequence, comms companies have been underspending on their networks since 2001 and have been driving more spending on revenue generation than network integrity. Like it or not they are more like Electric & water companies (which take delivering their services as an existential one) than entertainment companies and the CRTC should take it's regulatory role a lot more seriously. I'm not sure if this means they have to start divesting these assets/deals but the CRTC should demand a minimum up time rate and take customer sat a lot more seriously. This is going to take a lot more than a 5 day credit to fix this issue.

Expand full comment

I am a retired CA, part of the audit process every year is critical risk assessment. I think the risk assessments were obviously wrong when it came to Rogers, Interac and a few other places. Clearly this should have been addressed much better, BUT the auditors should have also flagged this...You can be sure NEXT year, after it has occurred, it will be better addressed! Something that caused this much hardship to in one day clearly needs to be addressed from the auditors standpoint too.

Expand full comment

We are a penny wise and pound foolish people. We just are, it's cultural. We hate opening our wallets to pay for nicer things, look at our health care system for instance, perhaps because this country is a series of protected guilds and interest groups squeezing people. We still have a colonial economic culture in many ways. I sometimes joke we have such high immigrant numbers because we need more suckers for our protected economy to rip off.

This has been going on too long, it's embarrassing. No, the Americans aren't going to take us over and Quebec won't seperate if we as Canadians push back against our mediocre elite and their featherbedding. But first we need to address our culture and stop behaving like colonials with such low expectations.

Expand full comment

I agree with the article but still am interested in what brought both the wireless and wired service down at Rogers and what Interac is doing to make sure they stay online in the future. With all the recent turmoil at Rogers and some rumours their technical people have been leaving I would like to know what exactly happened. As someone who worked in IT in the past, I know that often there will be a small number of people who REALLY know the system. Most technical people are competent but when something really weird knocks the whole system offline there will generally be that one person you need to have look at it. If you can’t get a hold of them or they have left the company you then have to muddle through until you find the problem.

Expand full comment
founding

Welcome back. "Always have a plan B" should be engraved in the eyeline of everyone involved in infrastructure training.

Expand full comment

Rogers needs to be held accountable. In the last 12 months there has been nothing but drama at Board level. Edward Rogers replaced an engineer with former Head of Accounting with goal to save money. There are a whole lot of other companies who need to do better - but 1st let’s acknowledge that Rogers did an unbelievably poor job. They should not be allowed to control more of the Canadian telecommunications market through purchase of Shaw.

Expand full comment

Good piece, Matt. Hope you had a good break.

At this point, I think you have enough material for a lengthy book (seriously) at what an utter basket case Canada has become, and the potential (probable) implications of the myriad of sheer ineptitude on display. I'll even provide you a title - Woe, Canada - We Lean, Off Guard, For Me.

One asks (as you repeatedly have) what productive, tangible things CAN Canada actually do at a world class level at this stage of our history?

The list of answers is embarrasingly brief - in fact, I can't think of a single thing.

In my half century living in Canada, I've watched public and private standards slowly but very steadily slipping away in every aspect of Canadian life. Mediocrity has become so acceptable, risk aversion so ingrained, entitlement so galling, that the majority of Canadians are now conditioned to accept that This Is Just The Way Things Are - meaning, they shrug their shoulders - and that's it! No outrage, no demands for better - just bland, indifferent, depressing acceptance for the declining state of affairs.

Evidence is everywhere - in tangible things like decaying roads and bridges, unreliable and overpriced utility and communication networks, and even overflowing garbages in public and private spaces. Canadian tolerance of galling, lazy ineptitude and outright, obvious corruption in public politicians and bureaucrats from the municipal through provincial to the federal levels is, actually, incredible.

We have so many in public and private life (elected, appointed, or 'employed') that have made careers out of accomplishing absolutely nothing. Instead of making things better, they have succeeded at failure so consistently that their legacy is making almost every aspect of Canadian life worse.

I continue to search in vain for a way out of this quicksand Canadians are all caught in.

There ARE good people in this country ready to change our national character from one I describe above to one Canadians can actually be proud of. The problem is, many of them are getting discouraged, or (most distressingly) are making active plans to leave Canada and never return.

If you do write a book, Matt - perhaps that can be the second part of it - after you detail the overwhelming rot, perhaps you can offer a plan, a prescription, a panacea for what ails Canada?

If so, I'll buy it. Book it.

Expand full comment

my company does work with many payment processors as well as loyalty card processors. Even the smaller ones try to have some sort of carrier independence. I really, really want to hear how Interac justified NOT bringing in multiple physical carriers. I have no inside knowledge on it so this is all pure speculation.

Worst case, total cheap incompetence on the banks who oversea it.

Best case, their data center (why just one?!?) was in a place where Rogers was the only company actually with fiber. Interac could have gone to Bell and actually purchased "Bell fiber" but it was actually delivered over Rogers infrastructure and not dark fiber but handed off at some CNI and Interact was none the wiser. We ( I work at an independent ISP) have ckts like this so its not unheard of, but generally someone that large would ask to see how much physical path redundancy there is as part of their due diligence.

As for hospitals and such, my guess is cost drivers and not having the technical staff. Its easier now, to provide "point and click" redundancy through technologies like SDWAN and more flexible VPN tecjnologies), but not that long ago having true carrier redundancy meant some somewhat specific technical knowledge (e.g. running your own ASN) which most orgs like hospitals would not have. Its one thing to have your home on multiple connections, its another where you are a large org with services (e.g. VPNs) tied to static IP addresses that people / other orgs connect TO. To have your own IP space (i.e. independent of the carriers you connect to) requires fairly esoteric skill sets. A hospital would be too small to have such technical skill among their staff so its gonna be harder. Like I said, this is getting a LOT easier but I suspect they are not there yet.

Expand full comment

The cost benefit analysis of redundancy is completely different for people and companies with competition and those without. If you are offline enough, your customers will go elsewhere. If a hospital is offline, do they suffer? No.

Expand full comment

If there wasn’t a fire ban on, I would start using smoke signals to contact my family. Trying to find out how to send a middle finger using smoke. Tricky stuff.

Expand full comment
founding

To paraphrase a joke from Bill Maher: I don't mind Canada, its just the Canadians I can't stand....

We, Canadians, collectively, are naive, complacent, lazy, and docile. Our national motto should be changed from 'A Mari usque ad Mare' to 'Oh Well, What Can You Do?'

I have some mixed feelings about the Truckers Convoy, but one thing that was extraordinary was the fact that Canadians actually got pissed enough to *disrupt* something, cause a scene, and be....not very nice. THE HORROR! But a nation-wide, commerce-disrupting, emergency services-threatening outage from a oligopoly telecom player, regulated by the government?

'Oh well, what can you do?'

Expand full comment