Private GIT server on AWS

Source code control is essential for LDC and something we have to keep up to date on, GIT is currently the ‘IN’ flavor of source control , and while places like GITHUB make it very easy, quite a few of our clients simply wont allow their source code on a none dedicated/personally secured OS instance (also once you start to get in to LDC’s number of current and previous projects/repositories, a dedicated server is cheaper) . so it was time to build a new dedicated GIT server on AWS (using their “Amazon Linux” base build, which is built off CentOS)

NOTE: this is a document in progress
TODO: create a persistent data area and mount point for storing the repositories (in case the instance gets terminated)
TODO: store this back as a AMI in case of loss of instance

1) we build a new instance via the “Launch Instance” wizard at https://console.aws.amazon.com/ec2 using the following options ( I’m only stipulating options that might need changing)

AIM: Quick Start – “Basic 32-bit Amazon Linux AMI”
Instance Type: “micro” (no need for power for this and we want to keep it cheap)
Termination Protection: YES (goodness knows when this is not set as default)
TAG: NAME – “LDC GIT Server”
Key Pair: “LDCdev” ( we already have a key pair for dev work, but you might need to create on, if you do, remember where you put your private key file (*.pem) as you’ll be screwed if you loose it)
Firewall: create a new “security group” containing: SSH, HTTP, HTTPS and TCPIP: 9148

2) Now we have a launched EC2 instance, lets log on via SSH (linux and Mac boys will find this bit easy, windows users, may I recommend PuTTY)

ssh -i LDCdev.pem ec2-user@xxx.xxx.xxx.xxx

you note, you have to provide a user of “ec2-user” rather than “root” as root wont work, and you have to provide the private key file you stipulated when you launched the instance. with the xxx.xxx.xxx.xxx, we use an ip address as we use the elastic IP address function, if you are not using that then you will need to use the public DNS that amazon provides ( select your running instance and on the “description” tab below, you will find it near the bottom, it will look something like “ec2-50-55-94-157.compute-1.amazonaws.com”)

if you get an error on log-on regarding your private key being too open, you have to secure it better at file level on your client machine, in Linux you do it like this

chmod 400 LDCdev.pem

3) now we start to install git, thankfully amazon Linux has a version of git built into our repository, so all you need to do it enter

sudo yum install git

next create the user that will store the repositories

sudo useradd -m -d /home/git -u 1005 git

and set its password

sudo passwd git

4) and a second user so we can create RSA key pair (GIT uses security keys rather than passwords)

sudo useradd -d /home/gitolite-admin gitolite-admin

sudo passwd gitolite-admin

su – gitolite-admin

ssh-keygen

follow the instructions and remember to pick a good password

Copy this key somewhere public on the server so we can get to it later

cp ~/.ssh/id_rsa.pub /tmp/gitlite-admin.pub

exit

5) next install gitolite(which I prefer to gitosis) to handle the repository management, we cant use yum for this as its not in the amazon repositories

git clone git://github.com/sitaramc/gitolite
cd gitolite
src/gl-system-install

now as the git user

su – git
gl-setup /tmp/gitlite-admin.pub

6) Now you will be taken take to the config document in vi, just exit

exit

7) At this point the oddities of git on AWs should be done and we can move over to the formal gitolite documentation at http://sitaramc.github.com/gitolite/doc/2-admin.html

Old Comments
————

##### Mark Myers(30/05/2011 23:27:55 GDT)
i expect we will be using the SSL, as we often provide the client direct access to their source code it will be better, i will keep updating this document to make it a good guide so if you have any updates you want let me know (ill try integrating egit with domino eclipse and see what happens

##### Nathan T. Freeman(30/05/2011 23:09:10 GDT)
Mark, thanks for this. GIT is a killer SCM, and we’ve been working to get an internal implementation running with gitolite as well.

Are you using the simple strategy of creating SSH accounts for each user, or have you tried using gitolite’s more elaborate identity control techniques with SSL?

Also, if you want to integrate git control with Domino projects, you might find this handy: { Link }

Drop Box on Servers

I like Dropbox, just about everyone I know uses it all the time, but one place I think it is under utilised is on the server, by this I don’t mean big corporate infrastructures, I mean cloud based servers or multiple hosted servers, a perfect example was last weeks migration, LDC were moving a box from one hosting provider to another, to transfer the data I merely split rar’ed it up on the current box (giving the rar creation directory as a folder on drop box), and grinned when I saw it appearing on the new server (the speed was comparable to FTP and with far less manual work)

but before that, we have found drop box invaluable on a daily basis for admin work, lets look at a couple of the directories on our Admin Drop Box account

“Source” – This Contains all the current software and patches that we use on our servers, not only does this mean that its very easy to get patches to all servers (download it once on my pc and drop it in), and that you never have to think “where can i get the patch from” when on a server, but if you install dropbox first when you are building a new box, all the files you need are there straight away.

“Restore” – Explains its self, how often have you restored a file and its on your local machine rather than a server or vs versa.

“Docs” – contains all the aide memoire’s that you always seem to need at the worst times when fixing a server, contains text files of ip address, network settings and any other relevant documentation that you MIGHT need, again, easy to keep up to date, just change it once on your local machine.

All data is transferred over SSL so ports are not a problem and the data is as secure as anything can be in our connected world.

why make your life difficult.

Jungle Disk Server Version

During a recent review of LDC’s hardware and systems, its was decided to change backup strategies, as most of us had been using jungleDisk /Amazon S3 for our laptop backups, we decided to give the server version a go.

Well i am very impressed with it, it behaves very much like any enterprise backup solution i have used, you install a bit of agent software on each server then monitor and control the jobs from a separate bit of control software (which is also installed by default on each server). it then backups your choices to your Amazon S3 account (jungle disk them selves do NOT backup your data, although they make the integration to your Amazon S3 account very very easy {as well as giving you about 10 gig of free data per server})

swapping from server to server is easy

and the configuration and job set-up are nearly the same as the standard jungle disk

pricing for the full blow version is $5 per month + what ever S3 storage costs your encure, so very “cloud” (makes a sick noise).

Oh, was immediately asked by Matt white, “does it handle open files?” , the answer would appear to be yes

what is not made so clear is that, if you are backing up a windows machine you need to have the “Windows Volume Shadow Copy” Service running (not an issue with Linux and Mac as they already cope with such things) , the way you can tell if you have missed this step is seeing the following error when tying to back up a locked file

“VSS_E_OBJECT_NOT_FOUND”

but if that is the limit to problems, then I’m more than happy.

Pros
-Cheap
-Easy to use
-Uses Amazon S3 so no more “We cant find the tape”
-You can get to the files directly without the backup software

Cons
-No application specific bolt ons (which might be viewed as a pro)
-Limited granularity of controls
-Obviously heavy on the bandwidth

Domino Appendix
As a largo portion of those who read this blog are IBM Domino users a note on using this on Domino:
Now Domino does not support VSS as far as i know (God knows why not, but i can make a guess) if this IBM note from 2007 is still correct (and judging by this ideaJam.net post it is)

So I’m afraid you are going to have to use the old stand-by’s of ensuring that databases are replicated, and use scripts to flush the cache before backup and/or stop services http,agent manager etc etc, to ensure the databases are handle free. our test restores all worked perfectly (including names.nsf), but as a lot of our stuff in on AWS now we also keep drive snapshots (cant be to careful with client data)

(sigh)

My First Full Lotusphere session

Woop Woop, I have my first full Lotusphere session! I will be presenting a Show ‘n’ Tell entitled “Write Better Java Code: Debugging, Logging, and Unit Tests” with the great Julian Robichaux, the master of openlog (a tool I use on every Domino app I write)

Anyway, quickly onto the point of this blog entry: After every conference’s sessions are announced there is the usual “it’s only the same set of people that get them each year” grumbles. However, 4 years ago I was not part of the Lotus community at all and this year I get to present properly at IBM’s leading collaboration event and scare the living shit out of myself in front of hopefully not too many people, so how?

  • I did my local LUG circuit (UKLUG & ILUG) which incuded rejections and one total cock up 🙁 .
  • I did LotusIdol last year (a year in which I purchased my own ticket).
  • I followed Gab Turtle’s guidelines on writing abstracts, then passed my attempts round to a couple of peers.This year I submitted 3 abstracts (2 joint) and got 1 session. Of the other 2, 1 had to be retracted due to time conflicts with my co-submitter, the other was a tad fringe and a bit of a gamble, but I may try it again on a LUG this year.

    Soooo, Lotusphere is not a closed shop, new people present each year, and I’m one of that lucky number.

Old Comments

Mark Myers(05/12/2011 23:43:48 GMT)

@joe and to a drink together after

Joe Litton(05/12/2011 23:38:36 GMT)

Your session last year (LotusIdol) was great. Looking forward to your session at LS12!!

Mark Myers(05/12/2011 23:36:29 GMT)

@David @Bruce thank you VERY much

Bruce(05/12/2011 23:34:42 GMT)

Mark – congratulations!!!!!!!! ROCK STAR

David Leedy(05/12/2011 23:22:15 GMT)

Congrats Mark! Looking forward to your session!

Mike McGarel(06/12/2011 00:43:22 GMT)

Congratulations!

Paul Withers(06/12/2011 00:11:20 GMT)

Congratulations, well deserved and a great topic. Looking forward to it

Mary Beth Raven(06/12/2011 03:44:29 GMT)

Congratulations!I volunteer to check all of your “bad grammer” 🙂 and I am sure you’ll do a great job because you’ll be speaking about a topic you know and that you feel passionate about. See you there. I volunteer to sit in first row and heckle you:)

Carl Tyler(06/12/2011 06:20:20 GMT)

Hard to believe its your first time. You’ll be great.

Mark Barton(06/12/2011 09:30:29 GMT)

Well done Mark – I know Java is a subject close to your heart.

Ben Poole(06/12/2011 08:12:29 GMT)

Congratulations old bean; LS won’t know what hit it! I wish I could be there to heckle.

Mark Myers(06/12/2011 07:50:36 GMT)

thank you every one, im sure Julian will be gentle with me

Mark Myers(06/12/2011 10:19:05 GMT)

@ben i wish you were there as well

@Mark same goes for you, it would be awesome

ChrisC(06/12/2011 13:17:27 GMT)

Congrats Mark – look forward to attending!

Mark Myers(06/12/2011 15:43:19 GMT)

thanks Chris, i better make it a good one