Updates from August, 2015 Toggle Comment Threads | Keyboard Shortcuts

  • gio 9:59 pm on August 17, 2015 Permalink | Reply
    Tags: , , tips   

    Git and coding: tip and tricks 

    • Socks Proxy: To access our GitLab website from IPs outside the Internet Archive’s network you need a socks proxy. To do so you can:
      • You can open a sock proxy with:
        ssh -N -D <port> username@archive.org

        and configure manually your browser or your network to use the socks proxy.

      • if you are using OSX you can use this script to make the socks proxy setup easier.
    • Sublime Text: if you are using sublime text to develop your code, you will appreciate this how-to use sublime text over ssh.
    • Memory problem: if running git push you have this error: fatal: Out of memory, calloc failed, you can configure git to use only one thread for “packing”:
      git config --global pack.threads 1

      another “solution” is to remove the limit:

      ulimit -v unlimited
    • Git prompt:
      • If you’re a Bash user, you can tap into some of your shell’s features to make your experience with Git a lot friendlier. Git actually ships with plugins for several shells, but it’s not turned on by default. Take a look to: Git prompt [2].
      • If you’re using zsh, and also make use of oh-my-zsh, many themes include git in the prompt. It is recommended that you set git config –local oh-my-zsh.hide-dirty 1 within the petabox repo to prevent a slow prompt.
    • How to rename the author info for all the commits in a repo:
      • for a single commit:
        git commit --amend --author "New Author Name <email@address.com>"
      • for all the commits in a repo:
        git filter-branch --commit-filter 'if [ "$GIT_AUTHOR_NAME" = "Josh Lee" ];
          then export GIT_AUTHOR_NAME="Hobo Bob"; export GIT_AUTHOR_EMAIL=hobo@example.com;
        fi; git commit-tree "$@"'
    • kelsey 2:58 am on August 18, 2015 Permalink | Reply

      Here is some info I found the hard way on Mac & case-sensitive git repos http://kelsey.blog.archive.org/2015/08/18/mac-case-sensitive-git-repos/

    • pooh 5:34 am on August 18, 2015 Permalink | Reply

      i have a script in ~tracey/scripts/post-commit that one can copy to petabox/.git/hooks/ to make the “update the $Id….$ thing in the file” (mostly useful for deriver hackers now, etc.)

      traceypooh [3:25 PM]
      (it re-checks out file after commit, so your version now has the $Id….$ string updated in the file you just committed)

    • pooh 5:34 am on August 18, 2015 Permalink | Reply

      [petabox tree file last mod times]
      ~tracey/scripts/post-checkout is nice to add in petabox/.git/hooks/ when one first clones a tree (technically, clone the tree, then add this hook, then checkout a file — it will take a *long* time to run, then remove the hook). this will make all files have modtime of their last commit (which can be v. helpful to determine quick age/modtimes of files at a “ls” glance) for those who like that kind of thing

    • kelsey 10:31 pm on September 25, 2015 Permalink | Reply

      That pesky error that we ran into before

      % git clone git@git.archive.org:ia/petabox.git -v
      Cloning into 'petabox'...
      fatal: protocol error: bad line length character: No s

      Most likely means that the user is trying to clone a repo that they don’t have access to. Add the user to the project members, and you’re good to go

  • gio 10:08 pm on August 3, 2015 Permalink | Reply
    Tags: index, , reindexing, ,   

    OL: reindexing the Solr search 

    To reindex the OpenLibrary’s Solr search index making it consistent with the db data we can use the script:


    here you can find the source.

    Here same basic usage:

    /olsystem/bin/olenv python /opt/openlibrary/openlibrary/scripts/ol-solr-indexer.py --config /olsystem/etc/openlibrary.yml --bookmark ol-solr-indexer.bookmark --backward --days 2

    /olsystem/bin/olenv is the script to load the right virtualenv.
    --config /olsystem/etc/openlibrary.yml is the OpenLibrary yml configuration file.
    --bookmark ol-solr-indexer.bookmark is the location of the last scan timestamp (YYYY-MM-DD hh:mm:ss) bookmarked.
    --backward / --forward the direction to do the reindexing
    --days the number of days to reindex.

    the script can run in daemon mode. At the moment we still using the new-solr-updater for the partial updates…

Compose new post
Next post/Next comment
Previous post/Previous comment
Show/Hide comments
Go to top
Go to login
Show/Hide help
shift + esc