My self and the rest of LDC went to the MongoDb Coming To You day in London and amongst the many interesting sessions, there was one about MongoDb security, I’m not going to even post a link to the slides as the interesting part was not on them, it was an story about a company going under due to losing control of their AWS (Amazon Web service) access keys, their site was hacked, their servers terminated, their databases deleted and all their backups purged. Everything…. Dead
This gave me quite a shudder, I’m more than a little bit paranoid at the best of times, but it did remind me of off the old phrase to never “put all your eggs in one basket” and that applies as much to cloud platforms as to anything else, however its a tricky trap to escape from as the cloud service providers sell them selves as everything you could ever want and there is no need to have any other vendor, they make it sooo easy to just bolt on services using the same credentials, yes they do provide granularity and yes they provide 2 factor authentication, but they are always a single point of failure and there is no physical security measure if you suffer a major breach (you can’t pound down to the server room and yank the cables out the back of your router or servers) if the bad guys get your security keys then they ARE you.
So what to do?, cloud services are cheap and hideously powerful for their money, not using them robs you and your company of a major advantage in the Internet economy?
Well let’s relate this all back to personal experience and in particular LDCVia, after the conference we sat down and worked out some worst case scenario ( loss of the root account API keys, breach in a primary admins 2 factor authentication or a member of the LDC founders being possessed by a Brain Slug) and figured out how we would deal with them, not what we would do to prevent them, as we had already do everything we can to that end:
- 2 factor authentication on every account capable of doing anything at an infrastructure or admin level
- Not using primary accounts or their API keys for day to day activities.
- Multiple single function accounts and granulated security.
- Operating level accounts should not have access to database content
- Proper firewalls.
- etc. etc. etc.
But how much damage we would be looking at and what the hell we could do to mitigate and recover from it.
For how much damage?, cater to losing all of it, the whole lot lost or compromised 🙁 how are we going to deal with that >:( .
- Backups should not be on the same security domain as the servers and databases. hell they should not even be on the same cloud service (you create backup accounts with read only access to only databases, with no OS access) and the servers should have no idea the external backup provider exists, don’t forget to limit the source IP address for these accounts or else you are just giving people a nasty back door to your data.
- App servers and database servers should should be segregated in the same way if possible, now this will often depend on network connection speed as db access though a firewall is really going to slow down your apps, but often cloud providers provide good links between different accounts on their network.
- Inquire with your cloud provider about a separate authentication method in the case of a breach, a phone number to call and a list of real people and credentials such as you use for phone banking that can shut down all access so you can pick up the pieces,
- Talk and I do mean actually TALK with your provider to find out what provisions they have for this scenario type. and while you are there find out their notification schedule are you being told about their network changes? also what back door accounts to they insist on adding to your systems, how are they protected? … in short be knowledgeable about your cloud platform.
- Can you take full images of your machines and store them else where (not just backups), they are a lot faster to recover from
- Work out your full DR restore strategy, how fast can you get all the servers and database built and back on line, what gets built in what order, what do you tell your clients during that time, what’s the public message you display? get it all written down distributed to the right people and for heavens sake down store it in one place.
I think I need a brown paper bag and a quiet place to calm down (shudder)
