Updates from April, 2016 Toggle Comment Threads | Keyboard Shortcuts

  • gio 10:47 pm on April 22, 2016 Permalink
    Tags: coverage, python, tests   

    Python Coverage 

    :: Install

        pip install coverage

    Command line usage:

    • run – Run a Python program and collect execution data.
    • report – Report coverage results.
    • html – Produce annotated HTML listings with coverage results.
    • xml – Produce an XML report with coverage results.
    • annotate – Annotate source files with coverage results.
    • erase – Erase previously collected coverage data.
    • combine – Combine together a number of data files.
    • debug – Get diagnostic information.
        $ coverage help
    $ coverage help run

    Execution
    :: run

        $ coverage run my_program.py

    :: report

        $ coverage report -m
    Name                      Stmts   Miss  Cover   Missing
    -------------------------------------------------------
    my_program.py                20      4    80%   33-35, 39
    my_other_module.py           56      6    89%   17-23
    -------------------------------------------------------
    TOTAL                        76     10    87%

    : a nicer report

        $ coverage html

    ## Branch coverage
    In addition to the usual statement coverage, coverage.py also supports branch coverage measurement. Where a line in your program could jump to more than one next line, coverage.py tracks which of those destinations are actually visited, and flags lines that haven’t visited all of their possible destinations.

        $ coverage run --branch myprog.py

    (source: https://coverage.readthedocs.org)

     

  • gio 9:59 pm on August 17, 2015 Permalink
    Tags: , , tips   

    Git and coding: tip and tricks 

    • Socks Proxy: To access our GitLab website from IPs outside the Internet Archive’s network you need a socks proxy. To do so you can:
      • You can open a sock proxy with: 
        ssh -N -D <port> username@archive.org

        and configure manually your browser or your network to use the socks proxy.

      • if you are using OSX you can use this script to make the socks proxy setup easier.

    • Sublime Text: if you are using sublime text to develop your code, you will appreciate this how-to use sublime text over ssh.

    • Memory problem: if running git push you have this error: fatal: Out of memory, calloc failed, you can configure git to use only one thread for “packing”: 
      git config --global pack.threads 1

      another “solution” is to remove the limit:

      ulimit -v unlimited

    • Git prompt:
      • If you’re a Bash user, you can tap into some of your shell’s features to make your experience with Git a lot friendlier. Git actually ships with plugins for several shells, but it’s not turned on by default. Take a look to: Git prompt [2].
      • If you’re using zsh, and also make use of oh-my-zsh, many themes include git in the prompt. It is recommended that you set git config –local oh-my-zsh.hide-dirty 1 within the petabox repo to prevent a slow prompt.

    • How to rename the author info for all the commits in a repo:
      • for a single commit: 
        git commit --amend --author "New Author Name <email@address.com>"
      • for all the commits in a repo: 
        git filter-branch --commit-filter 'if [ "$GIT_AUTHOR_NAME" = "Josh Lee" ];
  then export GIT_AUTHOR_NAME="Hobo Bob"; export GIT_AUTHOR_EMAIL=hobo@example.com;
fi; git commit-tree "$@"'
     

    • kelsey 2:58 am on August 18, 2015 Permalink

      Here is some info I found the hard way on Mac & case-sensitive git repos http://kelsey.blog.archive.org/2015/08/18/mac-case-sensitive-git-repos/

    • pooh 5:34 am on August 18, 2015 Permalink

      i have a script in ~tracey/scripts/post-commit that one can copy to petabox/.git/hooks/ to make the “update the $Id….$ thing in the file” (mostly useful for deriver hackers now, etc.)

      traceypooh [3:25 PM]
      (it re-checks out file after commit, so your version now has the $Id….$ string updated in the file you just committed)

    • pooh 5:34 am on August 18, 2015 Permalink

      [petabox tree file last mod times]
      ~tracey/scripts/post-checkout is nice to add in petabox/.git/hooks/ when one first clones a tree (technically, clone the tree, then add this hook, then checkout a file — it will take a *long* time to run, then remove the hook). this will make all files have modtime of their last commit (which can be v. helpful to determine quick age/modtimes of files at a “ls” glance) for those who like that kind of thing

    • kelsey 10:31 pm on September 25, 2015 Permalink

      That pesky error that we ran into before

      % git clone git@git.archive.org:ia/petabox.git -v
Cloning into 'petabox'...
fatal: protocol error: bad line length character: No s

      Most likely means that the user is trying to clone a repo that they don’t have access to. Add the user to the project members, and you’re good to go

  • gio 11:30 pm on April 21, 2015 Permalink  

    CReP: IA dev workflow using git-svn 

    • WARNING:: if you are working on vm-home to avoid out-of-memory problems do:

      ulimit -v unlimited
    • Pull the last commits to the local master branch with:

      git checkout master
git svn rebase

      • create a working branch

      git checkout -b devBranch
    • use this branch to develop your feature and fix the bugs
      and commit your work to the local git repository with

      git commit -m 'bla bla bla bla'
    • create a test branch like the master

      git checkout master
git svn rebase
git checkout -b sandMaster

      and merge all the commits from the devBranch branch in an atomic commit (using –squash)

      git merge --squash devBranch

      check what is going on

      git status
git diff --staged

      if everything looks ok, commit it

      git commit
    • Now you can check if this code is working correctly
      to be sure that the commit to the SVN will work correctly…
      If everything is fine…
    • …let’s do the same commit to the master branch

      git checkout master
git merge --squash devBranch
git status
git diff --staged
git commit
git svn rebase
    • Let’s do a dry-run and check the patches

      git svn dcommit --dry-run
git diff-tree 25e084b618580d69f5b57d8f9c1ca37045cf65dd~1 25e084b618580d69f5b57d8f9c1ca37045cf65dd -p
    • If everything is correct it’s time to commit to the SVN remote repo
      • git svn dcommit
    • You can push the code using the pi interface
     

  • gio 11:33 pm on March 23, 2015 Permalink  

    GitLab: Install and configure 

    Installing and configure a gitlab server on vm-gittest.us.archive.org

    For the basic requirements please check the document
    https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/requirements.md

    Install and configure the necessary dependencies:

    giovanni@vm-gittest:~$ sudo apt-get install openssh-server
giovanni@vm-gittest:~$ sudo apt-get install postfix

    Download and install debian package for Ubuntu 14.04

    giovanni@vm-gittest:~$ wget https://downloads-packages.s3.amazonaws.com/ubuntu-14.04/gitlab_7.9.0-omnibus.2-1_amd64.deb
giovanni@vm-gittest:~$ sudo dpkg -i gitlab_7.9.0-omnibus.2-1_amd64.deb

    We want to run CPU_NUM+1 uinicorn workers editing the file /etc/gitlab/gitlab.rb adding the lines:

    unicorn['worker_timeout'] = 60
unicorn['worker_processes'] = 3

    Re-configure and start GitLab

    giovanni@vm-gittest:~$ sudo gitlab-ctl reconfigure

    Start GitLab

    giovanni@vm-gittest:~$ sudo gitlab-ctl start

    The default root credentials are:

    Username: root 
Password: 5iveL!fe

    The servers require some minutes to be usable, before you will receive a 502 status code.

    WARNING
    To make sure the UI is reachable only within the IA network we have to edit the file
    /var/opt/gitlab/nginx/conf/gitlab-http.conf
    adding the lines

    server {
...
...
 
  allow  123.12.34.0/21;
  deny   all;
...
...
}

    The git test vm is reachable at: http://vm-gittest.us.archive.org/
    (the admin/root password is the usual one)

     

  • gio 11:32 pm on February 9, 2015 Permalink
    Tags: debug   

    Print a Python stacktrace of a running process 

    To print a Python stacktrace of a running process you need two script

    :: The ignore-error.py

    class IgnoreErrorsCommand (gdb.Command):
    """Execute a single command, ignoring all errors.
Only one-line commands are supported.
This is primarily useful in scripts."""
 
    def __init__ (self):
        super (IgnoreErrorsCommand, self).__init__ ("ignore-errors",
                                                    gdb.COMMAND_OBSCURE,
                                                    # FIXME...
                                                    gdb.COMPLETE_COMMAND)
 
    def invoke (self, arg, from_tty):
        try:
            gdb.execute (arg, from_tty)
        except:
            pass
 
IgnoreErrorsCommand ()

    :: and the gdb script pygdb

    • be sure gdb and python2.7-dbg are installed: apt-get install gdb python2.7/dbg
    # sudo apt-get install gdb python2.7-dbg
# to run:
# sudo gdb python
# (gdb) bt_pid <pid>
source /home/samuel/tmp/ignore-errors.py
 
define bt_pid
    attach $arg0
    t a a ignore-errors py-bt
    t a a bt
    detach
end

    To run this script and have the stacktrace:

    giovanni@ol-home:~$ sudo gdb python
(gdb) source pygdb
(gdb) bt_pid <PID>
     

c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel