I have written before about syncing desktop stuff to a NAS (in my case a Drobo) using Rsnc for backup reasons before, but lets now take it a step further and sync our Drobo/NAS stuff with amazon s3, in this example I will be backing up my precious happy hardcore and audio book collection that all live in a Directory called “Audio” on one of my Drobo shares

1) First create a mount point for your Drobo/NAS connection, I created a folder called “/media/localAudio” (ensuring that the local user you will be backing up as has write rights to the folder)

2) Next ensure you have the samba file sharing utilities installed (smbfs), you can do this on a terminal prompt with

sudo apt-get install smbfs

3) See if you can now mount your share with “sudo mount -t smbfs //192.168.0.XXX/myshare/Audio/ -o username=stickfight,password=password”
This assumes that I am want to map the “Audio” directoy on the “myshare” share on the IP address 192.168.0.XXX, also that you have to log-on to your share to be able to read/write to it, if you don’t, just miss out the “-o username=stickfight,password=password” bit

4) Now letts connect to amazon S3, you will first need an amazon 3s bucket for this (or use an existing one), go here for instuctions on creating one, mine is called “stickfight-audio”

5) This bit is less easy, you need to install s3sf, you can get clear instructions from here

6) Right, s3sf uses FUSE to perform its connections, but we have to tell it where your security credeiatals are for your S3 bucket, so create a text file .passwd-s3fs in your home directory and put your security credentials in them in the following format bucketName:accessKeyId:secretAccessKey , e.g.

stickfight-audio:0VWEOIEWOIUREWOIUFDS2:t4SyQ6pGjldoi4898dsoierelke/auw2wB4Rs+

(no theses arn’t my bloody credentials)
and give it the following permissions

chmod 600 ~/.passwd-s3fs

7) Next we want a mount point for the s3 bucket on our system, I created a folder called “/media/s3Audio” (ensuring that the local user you will be backing up as has write rights to the folder)

8) Now we can mount our S3 bucket as a local drive with s3fs bucket_name /mount/point, e.g.

s3fs stickfight-audio /media/s3Audio

9) Next make sure you have Rsync installed with

apt-get install rsync

10) Finally you can run your Rsync command to do the backup e.g.

rsync -r -t -u --progress  /media/localAudio/ /media/s3Audio

NOTES:
“-r” = copies all the sub directories and file, normally you would use “-a” but that copies the file permissions as well which in this case I don’t want.
“-u” = Update, means it only copies only new or recently changed files.
“–progress” = makes the terminal output far more readable and tells you how far it gets now.
“/media/localAudio/ /media/s3Audio” = source and target directories.(the “/” at the end of localAudio stops it actually creating a localAudio folder in the root of your s3 bucket thus keeping your directory structures the same level)

Doing this will backup your data perfectly, but it will beat the hell out of your bandwidth, a program such as trickle can limit the damage, install it via

sudo apt-get install trickle

Then use it to alter your S3 mount point, so that it limits the upload speed (in this case to 512KB/s, but you can change it to what ever suits you)

trickle -u 512 s3fs stickfight-audio /media/s3Audio

If you get an error along the lines of “trickle: Could not reach trickled, working independently: No such file or directory” ignore it, its just a badly worded advisory

So thats it working, I’ve rolled all this up into a script file that I can run when it suits me (its too big for a schedule)

sudo mount -t smbfs //192.168.0.XXX/myshare/Audio/ -o username=stickfight,password=password
trickle -u 512 s3fs stickfight-audio /media/s3Audio
rsync -r -t -u --progress  /media/localAudio/ /media/s3Audio

There you go.