About Me

My photo
I know the last digit of PI

Monday, July 04, 2011

Creating GIT projects based on SVN dumps

Following script will create new GIT projects based on SVN dump files.
As prerequisite you must have setup Gerrit codereview system. You need to allow the Gerrit user to directly push to master branch, otherwise all SVN commits need to be reviewed/verified, and for existing project this could means hundreds of commits.

Script:

#!/bin/bash
#!/bin/bash
GERRIT_IP=10.1.1.1
GERRIT_USER=xxx
GERRIT_PORT=29418

if [ -z "$1" ]; then
echo "Provide SVN dump file as first argument"
exit -1
fi

if [ ! -f $1 ]; then
echo "SVN dump $1 doesn't exists"
exit -1
fi

if [ -z "$2" ]; then
echo "Provide project name (that will be created in GIT) as second argument"
exit -2
fi

if [ -e $2 ]; then
echo "File or directory with name $2 already exists"
exit -2
fi


CURR_DIR=`pwd`

rm -Rf svnrepo
svnadmin create svnrepo/

svnadmin load svnrepo/ < $1
git svn clone file://$CURR_DIR/svnrepo/ $2
ssh -p $GERRIT_PORT -l $GERRIT_USER $GERRIT_IP gerrit create-project --name $2

# add origin
echo -e "[remote \"origin\"]" >> $2/.git/config
echo -e "\tfetch = +refs/heads/*:refs/remotes/origin/*" >> $2/.git/config
echo -e "\turl = ssh://$GERRIT_USER@$GERRIT_IP:$GERRIT_PORT/$2" >> $2/.git/config
echo -e "\tpush = HEAD:refs/for/master" >> $2/.git/config

# add originmaster
echo -e "[remote \"originmaster\"]" >> $2/.git/config
echo -e "\tfetch = +refs/heads/*:refs/remotes/origin/*" >> $2/.git/config
echo -e "\turl = ssh://$GERRIT_USER@$GERRIT_IP:$GERRIT_PORT/$2" >> $2/.git/config
echo -e "\tpush = HEAD:refs/heads/master" >> $2/.git/config

# add master branch config
echo -e "[branch \"master\"]" >> $2/.git/config
echo -e "\tremote = originmaster" >> $2/.git/config
echo -e "\tmerge = refs/heads/master" >> $2/.git/config


cd $2
git push originmaster
cd ..


In nutshell The script creates a dummy SVN repository, then use git-svn command to create local git project, creates a new project in Gerrit, adds Gerrit as remote repository to the local GIT configuration and finally pushes the changes to Gerrit

In order to use the script you need to provide a SVN dump file as first argument, and GIT project name as second argument

Saturday, July 02, 2011

Install Gerrit

1) Install tomcat.

2) Download/build gerrit.war to tomcat websapps directory

3) Create gerrit directory
mkdir /usr/share/gerrit


4) Initialize gerrit configurations
java -jar gerrit.war init -d /usr/share/gerrit


- Choose default values for all configuration except
- Choose authentication method HTTP
- Enter tomcat as "run-as user"
- Choose to Update/copy gerrit.war
- Choose to use Bouncy Castle

5) Edit /usr/share/gerrit/etc/gerrit.conf
and it modify it to looks like this:

[gerrit]
basePath = git
[database]
type = H2
database = db/ReviewDB
[auth]
type = LDAP
[sendemail]
smtpServer = localhost
[container]
user = tomcat
javaHome = /usr/lib/jvm/jdk1.6.0_26/jre
[sshd]
listenAddress = MYIP:8418
[httpd]
listenUrl = http://*:8282/
[cache]
directory = cache
[ldap]
server = ldap://MYIP:10389
username = uid=gerrit,ou=users,ou=system
password = gerrit
accountBase = ou=Users,dc=MYHOST
accountPattern = (&(objectClass=person)(uid=${username}))
accountFullName = displayName
accountEmailAddress = mail

groupBase = ou=Groups,dc=MYHOST
groupMemberPattern = (&(objectClass=groupOfUniqueNames)(uniquemember=${dn}))


Where you need to replace MYIP, MYHOST with the IP and the host name of the machine.
Note the LDAP configuration. We need to add gerrit user to ou=system and create the MYHOST domain structure.

6) Install ApacheDS LDAP server for user management. (There is a RPM package for Fedora, so just download and follow installation instructions).

7) We need to configure ApacheDS.
Open /var/lib/apacheds/default/conf/server.xml and add new partition
<jdbmPartition id="MYHOST" suffix="dc=MYHOST" optimizerEnabled="true" syncOnWrite="true" cacheSize="100"/gt;

Replace MYHOST with machine hostname.

Remove anonymous access
<defaultDirectoryService ... allowAnonymousAccess="false" ... >



7) Start the service
service apacheds start default



8) Install and Apache Directory Studio and connect to LDAP server
ldap://MYHOST:10389

where MYHOST is the hostname/IP address of the machine
The default username is "uid=admin,ou=system" and password is "secred"

Go to ou=system, and select uid=admin. Change the userPassword attribute with new password


9) Add new user to ApacheDS. Open Apache Directory Studio and import following LDIF

dn: uid=gerrit,ou=users,ou=system
objectClass: organizationalPerson
objectClass: person
objectClass: inetOrgPerson
objectClass: top
cn: gerrit administrator
sn: gerrit
displayName: Gerrit administrator
uid: gerrit
userPassword:: e1NIQX1PNWNIRFViTTFtUWlxT2U0UG1sbjdZUjRCVGc9

It contains a user gerrit and password gerrit

9) Create init.ldif file containing

#########################################################
# Root node for domain
#########################################################
dn: dc=MYHOST
objectClass: domain
objectClass: extensibleObject
objectClass: top
dc: MYHOST

#########################################################
# Root node for Users
#########################################################
# The node contains all users
dn: ou=Users,dc=MYHOST
objectClass: organizationalUnit
objectClass: top
ou: Users

#########################################################
# Root node for Groups
#########################################################
# Each group contains the user Ids assigned to the group
dn: ou=Groups,dc=MYHOST
objectClass: organizationalUnit
objectClass: top
ou: Groups

#########################################################
# Groups
#########################################################
dn: cn=admins,ou=Groups,dc=MYHOST
objectClass: groupOfUniqueNames
objectClass: top
cn: admins
description: Administrators group
uniquemember: uid=user1,ou=Users,dc=MYHOST

dn: cn=developers,ou=Groups,dc=MYHOST
objectClass: groupOfUniqueNames
objectClass: top
cn: developers
description: Developers group
uniquemember: uid=admin,ou=system
uniquemember: uid=user1,ou=Users,dc=MYHOST
uniquemember: uid=user1,ou=Users,dc=MYHOST

dn: cn=guests,ou=Groups,dc=MYHOST
objectClass: groupOfUniqueNames
objectClass: top
cn: guests
description: Guests group
uniquemember: uid=admin, ou=system

#########################################################
# Users
#########################################################
dn: uid=user1,ou=Users,dc=MYHOST
objectClass: organizationalPerson
objectClass: person
objectClass: extensibleObject
objectClass: uidObject
objectClass: inetOrgPerson
objectClass: top
cn: John Smith
givenname: John
sn: Smith
displayName: John Smith Jr.
mail: johnsmith@MYHOST
ou: Users
uid: user1
userpassword:: e1NIQX1zOXFuZTB3RXFWVWJoNEhRTVpIK0NZOHlYbWM9

dn: uid=user2,ou=Users,dc=MYHOST
objectClass: organizationalPerson
objectClass: person
objectClass: extensibleObject
objectClass: uidObject
objectClass: inetOrgPerson
objectClass: top
cn: Joe Doe
givenname: Joe
sn: Doe
displayName: terminator
mail: JoeDoe@MYHOST
ou: Users
uid: user2
userpassword:: e1NIQX1vWWdjQnU3SmJibVFISHUvNUJ4Q28vQ09uTFE9


Replace MYHOST with the name of the host. If your host have full domain name, then replace dc=MYHOST, with dc=mysubdomain,dc=mydomain,dc=com

The file describes a simple Groups/User hierarchy with 3 groups: admins,developers,users and 2 users: user1 (password:user1) and user2 (password:user2)

10) Using Apache Directory Studio import init.LDIF into LDAP server

11) Start tomcat service
service tomcat7 start


12) Stop tomcat service
service tomcat7 stop


13) Copy Bouncy castle jars to /usr/share/tomcat7/webapps/gerrit/WEB-INF/libs
cp /usr/share/gerrit/lib/bcprov-jdk16-144.jar /usr/share/tomcat7/webapps/gerrit/WEB-INF/lib


14) Start tomcat service and now you must be able to login to gerrit system with user1/user1 or user2/user2

15) Generating public/private keys.
Windows:
Download puttygen.exe and use it to generate a new private/public key. Use the menu Conversion / Export OpenSSH key to export the private key. Copy the OpenSHH public key (from the textbox)

Linux:
ssh-keygen -t rsa

Will generate /home/user/.ssh/id_rsa and /home/user/.ssh/id_rsa.pub
Copy the content of id_rsa.pub

16) Login into gerrit go to settigs, SSH key and paste the OpenSSH key (generated from the puttygen or ssh-keygen). Don't forget to click "Add"

17) Testing ssh connection. From Linux shell (or cygwin on windows boxes)
ssh -p 8418 -i <path to the private key> <gerrit IP/host>


18) Creating new project.
ssh -p 8418 -i <path to the private key> <gerrit IP/host> gerrit create-project -n <project name>


19) Go to Gerrit, choose "Admin" / "Projects" and select the newly created project.
Go to "Access" and add Submit,Push,Read permissions to "Registered Users" group

20) Cloning the newly created project for the first time.

git config --global user.name "Your Name"
git config --global user.email you@example.com
git config --global core.autocrlf false

git clone ssh://GERRIT_HOST:8418/PROJECT_NAME.git
cd PROJECT_NAME
git config remote.origin.push HEAD:refs/for/master
git config branch.master.remote origin
git config branch.master.merge refs/heads/master
echo Hello > readme.txt
git add readme.txt
git commit -m "Initial commit"
git push

After that the warning messages "You appear to have cloned an empty repository." or "remote HEAD refers to nonexistent ref, unable to checkout." will disappear.