Updates from February, 2016 Toggle Comment Threads | Keyboard Shortcuts

  • gio 12:08 am on February 25, 2016 Permalink  

    GitLab 8.5 and the new CI 

    With GitLab 8.5 we are introducing some great new features, between them also a brand new CI service.

    To have a working CI follow this quick start.

    If you don’t know how to setup your runner or you need one: ask me. We have a VM with docker available to register new runners.

     

  • gio 11:53 pm on February 23, 2016 Permalink  

    GitLab: installing version 8.5.0 

    • One: 
      sudo apt-get install curl openssh-server ca-certificates postfix
    • Two: 
      ~$ curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
~$ sudo apt-get install gitlab-ce=8.5.0-ce.1
    • Three: 
      sudo gitlab-ctl reconfigure
     

  • gio 8:51 pm on February 23, 2016 Permalink  

    GitLab: bakup and restore 

    WARNING: You can only restore a backup to exactly the same version of GitLab that you created it on.

      :: Create a backup of the GitLab system:

    • sudo gitlab-rake gitlab:backup:create

      the backup file will be created at /var/opt/gitlab/backups/
      or where specified in the gitlab config file.

      You can skip the backup for come components using:

      sudo gitlab-rake gitlab:backup:create SKIP=db,uploads
    • Please be informed that a backup does not store your configuration files. One reason for this is that your database contains encrypted information for two-factor authentication. Storing encrypted information along with its key in the same place defeats the purpose of using encryption in the first place!
      At the very minimum you should backup /etc/gitlab/gitlab-secrets.json (Omnibus) or /home/git/gitlab/.secret (source) to preserve your database encryption key.
      :: Restore a previously created backup:

    • We will assume that you have installed GitLab from an omnibus package and run 
      sudo gitlab-ctl reconfigure

      at least once.

      Note: that you need to run gitlab-ctl reconfigure after changing gitlab-secrets.json.

      First make sure your backup tar file is in /var/opt/gitlab/backups (or wherever gitlab_rails[‘backup_path’] points to).

      sudo cp 1393513186_gitlab_backup.tar /var/opt/gitlab/backups/
    • Next, restore the backup by running the restore command. You need to specify the timestamp of the backup you are restoring. 
      # Stop processes that are connected to the database
sudo gitlab-ctl stop unicorn
sudo gitlab-ctl stop sidekiq
 
# This command will overwrite the contents of your GitLab database!
sudo gitlab-rake gitlab:backup:restore BACKUP=1393513186
 
# Start GitLab
sudo gitlab-ctl start
 
# Check GitLab
sudo gitlab-rake gitlab:check SANITIZE=true

    source: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/backup_restore.md

     

  • gio 9:59 pm on August 17, 2015 Permalink
    Tags: , , tips   

    Git and coding: tip and tricks 

    • Socks Proxy: To access our GitLab website from IPs outside the Internet Archive’s network you need a socks proxy. To do so you can:
      • You can open a sock proxy with: 
        ssh -N -D <port> username@archive.org

        and configure manually your browser or your network to use the socks proxy.

      • if you are using OSX you can use this script to make the socks proxy setup easier.

    • Sublime Text: if you are using sublime text to develop your code, you will appreciate this how-to use sublime text over ssh.

    • Memory problem: if running git push you have this error: fatal: Out of memory, calloc failed, you can configure git to use only one thread for “packing”: 
      git config --global pack.threads 1

      another “solution” is to remove the limit:

      ulimit -v unlimited

    • Git prompt:
      • If you’re a Bash user, you can tap into some of your shell’s features to make your experience with Git a lot friendlier. Git actually ships with plugins for several shells, but it’s not turned on by default. Take a look to: Git prompt [2].
      • If you’re using zsh, and also make use of oh-my-zsh, many themes include git in the prompt. It is recommended that you set git config –local oh-my-zsh.hide-dirty 1 within the petabox repo to prevent a slow prompt.

    • How to rename the author info for all the commits in a repo:
      • for a single commit: 
        git commit --amend --author "New Author Name <email@address.com>"
      • for all the commits in a repo: 
        git filter-branch --commit-filter 'if [ "$GIT_AUTHOR_NAME" = "Josh Lee" ];
  then export GIT_AUTHOR_NAME="Hobo Bob"; export GIT_AUTHOR_EMAIL=hobo@example.com;
fi; git commit-tree "$@"'
     

    • kelsey 2:58 am on August 18, 2015 Permalink

      Here is some info I found the hard way on Mac & case-sensitive git repos http://kelsey.blog.archive.org/2015/08/18/mac-case-sensitive-git-repos/

    • pooh 5:34 am on August 18, 2015 Permalink

      i have a script in ~tracey/scripts/post-commit that one can copy to petabox/.git/hooks/ to make the “update the $Id….$ thing in the file” (mostly useful for deriver hackers now, etc.)

      traceypooh [3:25 PM]
      (it re-checks out file after commit, so your version now has the $Id….$ string updated in the file you just committed)

    • pooh 5:34 am on August 18, 2015 Permalink

      [petabox tree file last mod times]
      ~tracey/scripts/post-checkout is nice to add in petabox/.git/hooks/ when one first clones a tree (technically, clone the tree, then add this hook, then checkout a file — it will take a *long* time to run, then remove the hook). this will make all files have modtime of their last commit (which can be v. helpful to determine quick age/modtimes of files at a “ls” glance) for those who like that kind of thing

    • kelsey 10:31 pm on September 25, 2015 Permalink

      That pesky error that we ran into before

      % git clone git@git.archive.org:ia/petabox.git -v
Cloning into 'petabox'...
fatal: protocol error: bad line length character: No s

      Most likely means that the user is trying to clone a repo that they don’t have access to. Add the user to the project members, and you’re good to go

  • gio 12:17 am on May 7, 2015 Permalink  

    GitLab: Continuous Integration at ci.archive.org 

    GitLab is shipped with a Continuous Integration Server, our instance is reachable at https://ci.archive.org.
    Adding a gitlab project it is possible to run jobs on specialized nodes called runners.
    At the moment we have only one runner, configured to execute the jobs in a docker container, with Ubuntu.14.04, python, php5. The docker image is build using this Dockerfile.

    for sys-admins:

     

  • gio 8:50 pm on May 5, 2015 Permalink  

    Git: how to split a repo 

    You can use this command to split out a dir from a repo

    git filter-branch --subdirectory-filter folder_to_split_out/ -- master

    If you want tor remove a file or a folder from a repo:

    git filter-branch --index-filter "git rm -r --cached --ignore-unmatch file_to_remove" --prune-empty
     

  • gio 11:30 pm on April 21, 2015 Permalink  

    CReP: IA dev workflow using git-svn 

    • WARNING:: if you are working on vm-home to avoid out-of-memory problems do:

      ulimit -v unlimited
    • Pull the last commits to the local master branch with:

      git checkout master
git svn rebase

      • create a working branch

      git checkout -b devBranch
    • use this branch to develop your feature and fix the bugs
      and commit your work to the local git repository with

      git commit -m 'bla bla bla bla'
    • create a test branch like the master

      git checkout master
git svn rebase
git checkout -b sandMaster

      and merge all the commits from the devBranch branch in an atomic commit (using –squash)

      git merge --squash devBranch

      check what is going on

      git status
git diff --staged

      if everything looks ok, commit it

      git commit
    • Now you can check if this code is working correctly
      to be sure that the commit to the SVN will work correctly…
      If everything is fine…
    • …let’s do the same commit to the master branch

      git checkout master
git merge --squash devBranch
git status
git diff --staged
git commit
git svn rebase
    • Let’s do a dry-run and check the patches

      git svn dcommit --dry-run
git diff-tree 25e084b618580d69f5b57d8f9c1ca37045cf65dd~1 25e084b618580d69f5b57d8f9c1ca37045cf65dd -p
    • If everything is correct it’s time to commit to the SVN remote repo
      • git svn dcommit
    • You can push the code using the pi interface
     

  • gio 10:21 pm on March 30, 2015 Permalink  

    GitLab for the Internet Archive 

    Tips for admins:

    :: Install and configure GitLab.

    :: How to convert a SVN repository to GIT.

    :: How to add users in bulk on GitLab.

    Cookbooks for users:

    :: Pro Git. a good book with everything you need to know to use Git.

    :: Git – SVN. a crash course to switch from svn to git.

    :: Become a Git guru. a good tutorial.

     

  • gio 9:24 pm on March 27, 2015 Permalink  

    GitLab: how add users in bulk 

    For this task we will use the GitLab API.

    1. Obtain the PRIVATE-TOKEN

    Do you need a token to prove to be authenticated

    giovanni@vm-gittest:~/git$ curl "http://vm-gittest.us.archive.org/api/v3/session" --data 'login=root&password=ThePassword'  | python -mjson.tool
 
{
    "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd197acbd4d13d7dd61?s=40&d=identicon",
    "bio": null,
    "can_create_group": true,
    "can_create_project": true,
    "color_scheme_id": 1,
    "created_at": "2015-03-23T17:41:18.649Z",
    "email": "admin@example.com",
    "id": 1,
    "identities": [],
    "is_admin": true,
    "linkedin": "",
    "name": "Administrator",
    "private_token": "2xyxyxysyxsyxsUe8",
    "projects_limit": 10000,
    "skype": "",
    "state": "active",
    "theme_id": 2,
    "twitter": "",
    "username": "root",
    "website_url": ""
}

    2. Create the users

    You can add a new user on GitLab using the command:

    curl --header "PRIVATE-TOKEN: YourPrivateToken" -d "email=user@archive.org&password=defaultPassword&username=username$name=name" "http://vm-gittest.us.archive.org/api/v3/users"

    To add users in bulk we need a file containing all the users to be created like:

    mark = mark <mark@archive.org>
zella = zella <zella@archive.org>
uie = uie <uie@archive.org>
kers = kers <kers@archive.org>
.....

    and run the command:

    for i in `cat authors-transform.txt | awk {'print "email="$1"@archive.org&password=defaultPassword&username="$1"&name="$1""'}`; 
   do  curl --header "PRIVATE-TOKEN: YourPrivateToken" -d $i "http://vm-gittest.us.archive.org/api/v3/users"; 
done

    WARNING: at this point all the users will receive a confirmation email to activate their account.

     

  • gio 11:32 pm on March 26, 2015 Permalink  

    Git: how to convert a SVN repository to GIT 

    For the conversion we are following the John Albin’s document:
    http://john.albin.net/git/convert-subversion-to-git

    1. Retrieve a list of all Subversion committers

    From the root of your local Subversion checkout, run this command:

    svn log -q | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors-transform_alpha.txt

    Git require the emails as username, so you have to edit each line:

    from:

    fred = fred <fred>

    to:

    fred = fred <fred@archive.org>

    for our purpose we can do that quickly running this command:

     sed "s/>/@archive.org>/g" authors-transform_alpha.txt > authors-transform.txt

    2. Clone the Subversion repository using git-svn

    This will do the standard git-svn transformation (using the authors-transform.txt file you created in step 1) and place the git repository in the “~/temp” folder inside your home directory.

    git svn clone svn://home.us.archive.org/petabox -A authors-transform.txt --stdlayout --prefix=origin/ ~/temp

    3. Convert svn:ignore properties to .gitignore

    cd ~/temp
git svn show-ignore > .gitignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'

    4. Push repository to a new project on GitLab

    git remote add gitlab git@git.domain.com.au:dev-team/favourite-project.git
git push --set-upstream gitlab master

    Congratulations you have done!

    —The following instructions are useful only if you are not using GitLab and you want a bare repository—


    4b. Push repository to a bare git repository
    Then push the temp repository to the new bare repository.

    cd ~/temp
git remote add bare ~/petabox.git
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare

    5. Rename “trunk” branch to “master”
    Your main development branch will be named “trunk” which matches the name it was in Subversion. You’ll want to rename it to Git’s standard “master” branch using:

    cd ~/new-bare.git
git branch -m trunk master

    6. Clean up branches and tags
    git-svn makes all of Subversions tags into very-short branches in Git of the form “tags/name”. You’ll want to convert all those branches into actual Git tags using:

    cd ~/new-bare.git
git for-each-ref --format='%(refname)' refs/heads/tags |
cut -d / -f 4 |
while read ref
do
  git tag "$ref" "refs/heads/tags/$ref";
  git branch -D "tags/$ref";
done

     

← Older posts

c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel