Garmin Connect ransomware attack – site still down after 24 hours

They probably do, and if it was something non-malicious (e.g. a power issue at a data center) then no problem, switch to the second site and continue, maybe a few minutes down time, hours tops.

In a malicious scenario this gets more complicated for two broad categories of reasons:

1) You've got to actually be confident you have located the source of the issue, second sites are usually connected to the primary (for data replication) and other offices (for maintence/control), as well as the internet (to serve customers).  If your second site isn't impacted already, simply failing over before being sure you've fixed the issue could be enough to allow the infection to spread.

2) Depending on the attack, the main payload (the encryption logic in the case of ransomware) could have been dormant for some time already whilst it spread, meaning that your backups may already contain corrupt data and/or dormant copies of the virus.  Restoring from backup may reintroduce the malware into the network environment.

Either way, in the case of a malicious scenario, the sensible thing to do is usually to fix the root cause of the problem before trying to stand the systems back up, even if they do have a "complete backup system".  They aren't critical national infrastructure, there is no overriding reason to wing it.

The secondary site could be critical for them. For Maersk they ended up sending an engineer to a site office in Ghana where a single solitary Domain Controller was still alive and unencrypted (a domain controller is the bible for the entire IT infrastructure - every user, every device, security and so on). Without that fortunate accident they would have been weeks longer in restoring service

Llewelyn wrote:

Please! Can anyone tell me why a world-class, worldwide company doesn't have a complete backup system?
My mind boggles at the lack of insight by the computer men of the company.

It won't be the computer people, it'll be the accountants.

It's cheaper to not have a decent backup system until the time when it suddenly becomes a lot more expensive.

Evidence for the prosecution m'lud

https://www.networkworld.com/article/3200105/british-airways-outage-like...

kil0ran wrote:

Evidence for the prosecution m'lud

https://www.networkworld.com/article/3200105/british-airways-outage-like...

Some basic rules that should be followed:

• You don't have a disaster recovery plan until it has been tested
• You don't have backups until you've tested them
• If you don't know what happens when equipment is powered off and on again, then it's not production-ready
• There's 2 types of people - those who have lost data and those that will lose data

One of our software vendors told us of one client who had a really good backup and recovery which they had tested. Then they had a flood in the basement and the only servers they had were in the basement...

So, they just bought new servers and restored from the off-site backup - job done!

Or do you mean that they only had on-site backups? (In the one location that is sensitive to floods)

I think the backup was offsite, but obviously they had nowhere to restore to.

I guess that they did buy new servers - but an expensive option and they would need a bit of configuration and additional vendor support for each system.

The moral is you don't know how good your back up is until crunchtime, no matter how good you think it is .

They do. It's also encrypted by the ransomware, or potentially they don't have any kit for the engineers to get access to it.

They've got two options:

1. Pay the ransom

2. Rebuild from scratch

It took Maersk (global logistics/shipping) two weeks to restore service when they were ransomed, Garmin don't have the same resources. Good long read here

https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-cra...

If it is a ransomware attack, then typically the data isn't actually stolen - it's just encrypted in place.

Which is as good as stolen (to Garmin - 'lost', anyway), unless they pay the ransomers a lot of money...

Yes - but it's not the same as 'stolen' as in users' credit card details and personal information being in the hands of the attackers, which is what Zebulebu seemed to be worried about.

True; fair enough.

Historically, yes, however over the last 12-18 months there's been an increase in hybrid attacks where data is both exfiltrated and then encrypted (e.g. Maze ransomware).

It's entirely possible they don't yet know what has been exfiltrated (if anything) so simply saying "they could have got to anything we hold, but we don't know yet" helps no one. 

As a couple of people have said, this is not just "Oh a system fell down, fail over to the DR site", you need to make sure you eliminate all instances of the malware and also make sure you aren't simply restoring corrupted backups that include the virus, otherwise you're straight back to square one.  That is substantially more complicated than your run of the mill fail over scenario.

What is the difference? They steal your credit card and ask you to pay a huge amount to get it back?

risoto wrote:

What is the difference? They steal your credit card and ask you to pay a huge amount to get it back?

The difference is they haven't stolen it*. It hasn't gone anywhere - it's still sitting on the victim company's servers, but just encrypted so it's unusable to them. You don't need to pay anything to get it back - you already know what your credit card details are. It's the company they demand payment from.

[* Unless, as FlyingPenguin said, it's not just launched a ransomware attack, but has also extracted the data. One would hope, though, that however flawed Garmin's systems were, they would at least have encrypted credit card / personal data, and that encryption should mean it would still be useless to the attackers**. Even if they hadn't, paying to get get your card details back would be pointless - you already have them, and there's nothing to stop the attackers keeping a copy, so you'd just cancel the cards.]

[** At least until quantum computing comes along, and then we're all screwed.]

Maybe so but if the ransomers have managed to gain access to Garmins internal IT setup so successfully to encrypt it on such a scale it basically borks all of their internal & external facing IT, and these things are usually firewalled and kept distinctly separate for a good reason,theres no telling what theyve peaked at,copied,or left behind in the time they had unrestricted access. This isnt someone accidentally loaded a virus from a floppy drive or clicked a bad link in an email.Disaster recovery shouldn't take more than 24hrs on a production environment,even a global one.

You obviously have had little dealings with such attacks.   Once they are in, the virus jumps from machine to machine encypting as it goes or in some cases only encypting a percentage of devices, leaving some users thinking they are ok, but in the background their machine keeps connecting to other machines and infecting them. 

It is not a question of just getting a tape out of the cupbaord and pressing restore.

You may have to go to each machine in turn, disconnected it from the network and then run fixes on each machine.  In a global compaine that can take days and days.  Only then can you trun on the whole network again.

No matter who you are, once you connect any part of the network to the outside world via the internet, USB, Disc, etc....your network can never be 100% secure, it is just impossible.

Judge them by how they deal with the situation they find themselves in.  If they did have rubbish security, their systems would be up and down all the time with issues, but in my experience that has not been the case.

and youve obviously watched far too many films and tv if you think thats how easy it is to spread malware across a whole office IT environment including voip call servers and a production front end level IT setup which would be specifically hardened for genuninely state sponsored hacking attacks, not spotty teens in their bedroom mining bitcoins

but who knows, I dont claim to know their network setup, I know Id never advise anyone who sought my opinion to allow that to be remotely possible precisely because its such an obvious flaw in their security and the damage it could cause, but maybe they are that naive to link them up so you could wipe it all out if they listen to obvious security expert con-sultants like yourself

Awavey wrote:

and youve obviously watched far too many films and tv if you think thats how easy it is to spread malware across a whole office IT environment including voip call servers and a production front end level IT setup which would be specifically hardened for genuninely state sponsored hacking attacks, not spotty teens in their bedroom mining bitcoins

but who knows, I dont claim to know their network setup, I know Id never advise anyone who sought my opinion to allow that to be remotely possible precisely because its such an obvious flaw in their security and the damage it could cause, but maybe they are that naive to link them up so you could wipe it all out if they listen to obvious security expert con-sultants like yourself

Get an over sized fizzy drink and some popcorn... have a read: https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-cra...

Awavey wrote:

if the ransomers have managed to gain access to Garmins internal IT setup

They may not have needed to - they may well have used someone who already had privileged access to the systems as a vector to introduce the malware, and it then spread itself from there.

There may also be a precautionary 'lockdown' aspect to everything being down as well - if you haven't yet identified how the infection spread, then it may be better to take everything down until you do, rather than bring affected systems back and risk re-infection from elsewhere (a 'second spike', if you will).

which is possible sure, but companies who take this stuff seriously, and Id have assumed Garmin would fall under that label but maybe they dont, would never have a setup that allowed malware that got into their office IT environment, which is identified in all security audits as the number 1 likely vector threat risk, to then propogate across into the production environment.

Because everyone knows thats the biggest risk of bringing down a live server, excepting asteroid strike,fire or flooding,

So you dont go nah it will be ok, you lock down production networks to the nth degree from office IT, never allow access from an untrusted domain, certainly never allow code deployments from them and then so what if your internal IT team is Fubar'd by malware, you wont have impacted any live customer data or access, you might not be able to deploy updates till you fix it, but youll still have a service.

Even if your live server did go down as a result, your disaster recovery process should have an alternate server,in an alternate secure location with a safe/secure back up spun up within 24hrs maximum downtime, and thats being generous most IT service level agreements insist on 3hrs max for live downtime recovery, beyond that it starts getting increasingly expensive in compensation.

Awavey wrote:

which is possible sure, but companies who take this stuff seriously, and Id have assumed Garmin would fall under that label but maybe they dont, would never have a setup that allowed malware that got into their office IT environment, which is identified in all security audits as the number 1 likely vector threat risk, to then propogate across into the production environment.

No one can 100% guarentee that.  Major government agencies and financial institutions with ooodles of resources (and regulators breathing down their necks) are still regularly breached, it's a question of when, not if, for a company like Garmin.

You can introduce controls with reduce the likelihood of compromise (e.g. firewalls, internal segmentation, IDS/IPS, SIEM/network activity monitoring & endpoint protection like AV and DLP), or reduce the impact (e.g. comprehensive backups, playbooks for major scenarios, incident response support retainers), but once the malware is past the external permieter you are reliant on the controls you do have in place preventing travesal of the internal network into production systems.  Unless your production system is air gapped from everything (including the external internet, which wouldn't be appropriate here), you have to be right for every possible threat event, the attacker only has to be right once.  And honestly, having led numerous security audits, I've never not raised a finding...

It's entirely possible they did something stupid (WannaCry brought down large parts of the NHS using a vulnerability that was patched by Microsoft a long time before the attack, heads should have rolled), but to view the recovery time in terms of normal service SLAs of three or four 9s availability ignores the complexities of dealing with (potentially competent, potentially specifically targeted) malware.

I imagine it's very difficult to communicate meaningfully to users while they are still investigating, working out what reports they need to make to regulators and law enforcement etc.