I have written before about syncing desktop stuff to a NAS (in my case a Drobo) using Rsnc for backup reasons before, but lets now take it a step further and sync our Drobo/NAS stuff with amazon s3, in this example I will be backing up my precious happy hardcore and audio book collection that all live in a Directory called “Audio” on one of my Drobo shares

1) First create a mount point for your Drobo/NAS connection, I created a folder called “/media/localAudio” (ensuring that the local user you will be backing up as has write rights to the folder)

2) Next ensure you have the samba file sharing utilities installed (smbfs), you can do this on a terminal prompt with

sudo apt-get install smbfs

3) See if you can now mount your share with “sudo mount -t smbfs //192.168.0.XXX/myshare/Audio/ -o username=stickfight,password=password”
This assumes that I am want to map the “Audio” directoy on the “myshare” share on the IP address 192.168.0.XXX, also that you have to log-on to your share to be able to read/write to it, if you don’t, just miss out the “-o username=stickfight,password=password” bit

4) Now letts connect to amazon S3, you will first need an amazon 3s bucket for this (or use an existing one), go here for instuctions on creating one, mine is called “stickfight-audio”

5) This bit is less easy, you need to install s3sf, you can get clear instructions from here

6) Right, s3sf uses FUSE to perform its connections, but we have to tell it where your security credeiatals are for your S3 bucket, so create a text file .passwd-s3fs in your home directory and put your security credentials in them in the following format bucketName:accessKeyId:secretAccessKey , e.g.

stickfight-audio:0VWEOIEWOIUREWOIUFDS2:t4SyQ6pGjldoi4898dsoierelke/auw2wB4Rs+

(no theses arn’t my bloody credentials)
and give it the following permissions

chmod 600 ~/.passwd-s3fs

7) Next we want a mount point for the s3 bucket on our system, I created a folder called “/media/s3Audio” (ensuring that the local user you will be backing up as has write rights to the folder)

8) Now we can mount our S3 bucket as a local drive with s3fs bucket_name /mount/point, e.g.

s3fs stickfight-audio /media/s3Audio

9) Next make sure you have Rsync installed with

apt-get install rsync

10) Finally you can run your Rsync command to do the backup e.g.

rsync -r -t -u --progress  /media/localAudio/ /media/s3Audio

NOTES:
“-r” = copies all the sub directories and file, normally you would use “-a” but that copies the file permissions as well which in this case I don’t want.
“-u” = Update, means it only copies only new or recently changed files.
“–progress” = makes the terminal output far more readable and tells you how far it gets now.
“/media/localAudio/ /media/s3Audio” = source and target directories.(the “/” at the end of localAudio stops it actually creating a localAudio folder in the root of your s3 bucket thus keeping your directory structures the same level)

Doing this will backup your data perfectly, but it will beat the hell out of your bandwidth, a program such as trickle can limit the damage, install it via

sudo apt-get install trickle

Then use it to alter your S3 mount point, so that it limits the upload speed (in this case to 512KB/s, but you can change it to what ever suits you)

trickle -u 512 s3fs stickfight-audio /media/s3Audio

If you get an error along the lines of “trickle: Could not reach trickled, working independently: No such file or directory” ignore it, its just a badly worded advisory

So thats it working, I’ve rolled all this up into a script file that I can run when it suits me (its too big for a schedule)

sudo mount -t smbfs //192.168.0.XXX/myshare/Audio/ -o username=stickfight,password=password
trickle -u 512 s3fs stickfight-audio /media/s3Audio
rsync -r -t -u --progress  /media/localAudio/ /media/s3Audio

There you go.

You will have to forgive this blog entry as most of you will know this tip, but it was years since I last used it and when I wanted it last night it completely went out of my head, so I’m putting it up here as an aide memoire

So here is the problem, you have a java collection (in this case a List) containing java custom classes.

List<Address>

and here is the Address class that this is a List of:

package com.ldc.classes;
public class Address {
    private String firstLine;
    private String secondLine;
    private String postCode;
    private int ownerId;
    public String getFirstLine() {
        return firstLine;
    }
    public void setFirstLine(String firstLine) {
        this.firstLine = firstLine;
    }
    public String getSecondLine() {
        return secondLine;
    }
    public void setSecondLine(String secondLine) {
        this.secondLine = secondLine;
    }
    public String getPostCode() {
        return postCode;
    }
    public void setPostCode(String postCode) {
        this.postCode = postCode;
    }
    public int getOwnerId() {
        return ownerId;
    }
    public void setOwnerId(int ownerId) {
        this.ownerId = ownerId;
    }
}

OK, You have got this List of Classes from something that the user has done, and you present them with this List, they then go “cant it be sorted by the Age of the person who lives there” , “errr well not really as the Address class does not contain the residents age, that is in a different class.

package com.ldc.classes;
public class Person {
    private int personId;
    private String firstName;
    private String lastName;
    private int age;  
}

“well just sort it out cant you”……..ffffssssss

Now I expect that most of you are thinking that it would be easier to do this with a SQL statement but believe me there are times when you cant or don’t want to do that, so we have to figure out a way or glue the 2 classes together and making them sortable, we are going to do that with a third class, and one that has a bit of magic in it called “implements Comparable”, our special class need to have the list of address in that we want to sort and something we want to sort them BY (in this case ownerID).

package com.ldc.classes;
public class SortableAddress implements Comparable<SortableAddress> {
    private Address address;
    private int ownerAge;
    public Address getAddress() {
        return address;
    }
    public void setAddress(Address address) {
        this.address = address;
    }
    public int getOwnerAge() {
        return ownerAge;
    }
    public void setOwnerAge(int ownerAge) {
        this.ownerAge = ownerAge;
    }
    public int compareTo(SortableAddress compareForm) {
        int compareAge = ((SortableAddress) compareForm).getOwnerAge(); 
        return this.ownerAge - compareAge;
    }
}

You can see that this class has 2 special bits in it, the first is the “implements Comparable” at the top and the second is the “CompareTo” at the bottom which I have tried to make as simple as possible so you can just swap out your values and get it working on your classes. what this means is you can now use, the standard Collection.sort that you would normally use to sort a list, but use it on your class, here is an example implementing that:

package com.ldc;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import com.ldc.classes.Address;
import com.ldc.classes.SortableAddress;
public class SortMyClass {
    // OK we have a unsorted list of addresses and we are going to pass them to the function "getasortedListofAddress"
    public List<Address> getasortedListofAddress(
            List<Address> unsortedAddressList) {
        //first lets make a empty sortable address list (our special class)        
        List<SortableAddress> sortableAddressList = new ArrayList<SortableAddress>();
        //and loop though the unsorted list of addresses adding each on to the special class
        for (Address tempAddress : unsortedAddressList) {
            SortableAddress sortableAddress = new SortableAddress();
            sortableAddress.setAddress(tempAddress);
            //here we are getting the owners age which is the value we will be sorting by, to the special class
            sortableAddress.setOwnerAge(getTheOwnersAge(tempAddress
                    .getOwnerId()));
            sortableAddressList.add(sortableAddress);
        }
        //we now have a fully populated class that can be sorted using the standard Collections.sort
        Collections.sort(sortableAddressList);
        //hey presto we have sorted the list
        //now just run though the special class, to move the Address back into their old format, but now they are ordered!!!
        List<Address> listOfSortedAddressToReturn = new ArrayList<Address>();
        for (SortableAddress tempAddress : sortableAddressList) {
            listOfSortedAddressToReturn.add(tempAddress.getAddress());
        }
        return listOfSortedAddressToReturn;
    }
    private int getTheOwnersAge(int ownersID) {
        // some code that gets the owners age from its ID
        int ownersAge = 24;
        return ownersAge;
    }
}

give me a kick if you want me to explain it in more detail, but that that should be enough to provide a practical solution to most of you (and me when I next need it)

Old Comments
————
##### Mark(23/05/2012 14:43:57 GDT)
Oh, I agree generic would be best in real life, I was trying to do the easiest example to follow, but Dear readers Kerr is right when you actually put this bugger into practice
##### Kerr Rainey(23/05/2012 14:24:05 GDT)
I’d try to make this a little more generic by replacing int ownerAge (and associated getters/setters) with Comparable comparable.

Then you push down what it is that the SortableAddress is comparing. Since Integer is Comparable and we have auto boxing, this would be a drop in replacement for what you already have, bar the refactoring of the getter/setter names.