How do you protect a remote backup from a compromised account?
from eyesaremosaics@lemmy.zip to selfhosted@lemmy.world on 19 Jun 16:00
https://lemmy.zip/post/66460350
from eyesaremosaics@lemmy.zip to selfhosted@lemmy.world on 19 Jun 16:00
https://lemmy.zip/post/66460350
Let’s say you have access to a remote machine and use it to copy backups occasionally, eg with rsync. Your local machine has credentials stored that allow write access on the remote machine, however if the local account was compromised that could also allow access to the remote machine and the data stored there.
How can you grant access to an account to write remotely, but also protect the data from this account? One possibility could be to change the permissions on the data after it is copied to prevent deletion/interference, although I’m just making this up. Is there a standard practise for this?
threaded - newest
Encrypt before send, and if you want to have protection against deletions of the data have a cold backup offline other than during the copy.
What you are talking about is Immutability and an append only backup. The s3 file system and some others has Immutability built in. Not all backups can do append only.
I mean depends on the solution you are using, but you can have multiple accounts on the remote backup. IE so upon completion of the backup. The remote machine moves the backup to an offline or read only share (depending if you need those credentials to access the data again later),
Obviously most important thing is your credentials that make the backup… should be very limited in scope to just doing those backups.
I think you could do it somewhat like hetzner does for their storage boxes. You get an account that has read and write access to a directory and nothing outside. The accound can only run a limited set of commands, like ls, cat, nano, rsync etc. but has no access to commands that modify the filesystem.
Then you can use a copy on write fs like btrfs and make scheduled staggered snapshots.
I usually do 1x per year, 1x per month of current year, 4 per week of current montg, 7 per day in current week.
I have no clue what they use to limit the user accounts like that btw. but maybe that gives you a new jump off point for further research.
Nano and rsync are 100% designed to modify the filesystem. But yes the idea is correct.
Same with got over ssh, you restrict the connection to the got shell that can only do the things you want.
Are they? I thought they only write/modify/delete data to the fs, not change the fs itself.
Yeah precice phrasing ia hard sometime. I was refering to delete/modify of files as “changes to the fs”. Not sure how changing the actuall fs would be relevant to the backup question.
OP needs a restricted shell that can take backup data and write it to disk but not be able to modify anything that is already there. Nano and rsync can both do that.
OP asked:
So I was thinking that the account should not be able to delete the filesystem in an unrecoverable way. Like overriding the current fs with random data or an encrypted fs and filling it etc.
Like I said on a Hetzner storage box, multiple users get access to the same system, but each one only has file editing commands, not fs editing and they can only access their assigned directory. So if the system does scheduled snapshots (outside of that user’s scope of access) there is no way for a user to delete the files beyond recoverability. (no matter if their own files or other users files).
The user can still delete their own data. But because the fs is cow with snapshots (like btrfs) and they can not touch that, the data can be recovered easily.
Append only permissions.
Encrypted deltas.
Basically the time of the connection is the name of the only that folder that you have access to.
You can also setup a yubikey (or nitrokey) that requires you to physically process and would be immune to the host being compromised.
Restic has quite a good solution for this: https://restic.readthedocs.io/en/stable/060_forget.html#security-considerations-in-append-only-mode
Funny thing is that blockchains are actually good for this type of thing. Too bad crypto bros got a hold of it.
You’ve got the right idea with the permission change… the key is that you can have code executing on the remote side with different permissions. So the writer process has permission to write in one directory, and the turnsyle procees (often the root super-user) rotates the files or directories at a different time (or on a signal, sometimes).
WORM: write once, read many. Any good backup software supports this.
You could also keep offline backups. You can’t compromise what you can’t reach.
Are you connecting to the victim to push a backup into storage? If so, there’s SO much you should improve on that.
Go on.
What are the vulnerabilities that you’re afraid of? Can you answer this purely from the info OP gave, i.e. without making assumptions about what the server authorizes the phone to do? OP’s post does not indicate that they’re violating the principle of least privilege in any way.
If I wasn’t sure whether that was a risk, I would have asked whether that’s what was going on. Oh, wait: I did, right? I used a question mark and everything.
Append only mode.
Append only, like others are saying.
Do pull backups instead of push backups: Backup server connects to local machine.
What’s the rationale for this? Genuinely curious.
The reasoning is that your backup server should be more secure than production. Production has to have a bunch of stuff open in order to be useful and convenient. The backup server does not. It can be basically fully locked down.
To add - by doing pulls the backup server uses different credentials to run than the credentials used to perform pulls.
Backup server has it’s own credentials database, machines being backed up have their own database. Backup service in backup server uses appropriate credentials from machine being backed up to access the data there (shares, etc). So credentials from compromised machine are unrelated to credentials for backup server.
And if backups are done properly (full on a schedule, daily incrementals, or something similar) you should be able to revert to a known-good state with minimal data loss.
If the main site gets compromised the credentials there must be considered lost and known to che attackers.
with a pull backup that’s not an issue because the main site has no access to the remote system; it is a process on the remote site that has credentials to access the main site and not the other way around.
the remote system may
receiveretrieve a compromised copy of the data, but the attacker cannot tamper with previous backups so recovery is still possible.This is the main reason I had in my head about pull backups. Thanks for the explanation.
That makes sense. I use NFS, so there are other controls for security because “offsite” is another building on my property, but still in the same pool of subnets…
Why downvote this?
That’s an interesting idea
A system like proxmox backup server can do this scurely. There you can create a user that can only add new backups and read the existing ones, but cannot delete any or read anything else on the remote host.
Otherwise if you only care to protect the remote machine, then something like an ssh chroot jail would also work.
I have solved that by giving the distant machine the credentials to connect to the local machine. And the distant machine can’t be accessed from the outside.
Yes that sounds like pull backups, which is worth exploring
Scheduled snapshots (btrfs or zfs). If the compromised account deletes or modifies files, they’re still there in the past snapshots
Filesystem-level snapshots are quite space-efficient because they don’t make copies of all the files or even whole files; just the blocks that changed.
Just a small sidenote: If you do not trust your local machine you should think about why and how to change that.
Well I think she amount of caution is still appropriate no matter how much you manage to secure the local machine
Of course, of course :)
Personally as some extra spice as I worry about ransomeare, I have a few key files I check across my array that should never change. If any of their hashes are off, I abort immediately.
The suggestion I have heard is to have the remote machine connect to the machine on a schedule and pull the backups onto itself. Then your local machine doesn’t have direct access to the backups, making it harder to compromise the backups if hacked. But this also assumes the backup machine is locked down and isolated so it is lower risk than the local machine.
Yes that seems like one way to go. Although I am using rsync so maybe keeping the files in place and changing owner could be an option
For doing snapshots did that means the local system identifies the changes? Or it all gets copied each time?
Certainly sounds relevant, although overall it is quite a different approach than in currently using
Yep offline backups are useful, although it does require remembering & making the effort to do it each time