Recent Updates Page 3 Toggle Comment Threads | Keyboard Shortcuts

  • gio 10:02 pm on April 21, 2015 Permalink
    Tags: ergonomy   

    Ergonomics for Healthy Working 

    IMG_1268 (1)

     

  • gio 5:44 pm on April 21, 2015 Permalink
    Tags: cluster republisher,   

    CReP: how to RePublisher-checkout a "lost" item 

    When a book is moved to a new datanode you have to RePublisher-checkout the book again.

    First you have to know the directory of the item,
    to do so at manage page:

    https://archive.org/manage/testmytestgiogio0000gio

    click on Locate directory of item in cluster.

    for our example: /25/items/testmytestgiogio0000gio/

    then run the RePublisher-checkout.php:

    https://iaxxxxxx.us.archive.org/RePublisher/RePublisher-checkout.php?id=testmytestgiogio0000gio&itemPath=/25/items/testmytestgiogio0000gio

    NOTE: be sure the RePub State for the item is 12.

     

  • gio 9:29 pm on April 14, 2015 Permalink
    Tags: ,   

    CReP: run common code from your petabox tree 

    to be sure that the Cluster RePublisher is running the code on your petabox tree,
    be sure these two php variables are set:

    $_SERVER['PETABOX_HOME'] = '/home/giovanni/petabox';
$_SERVER['DOCUMENT_ROOT'] = $_SERVER['PETABOX_HOME'] . '/www/datanode/';
     

    • gio 11:55 pm on May 17, 2016 Permalink

      In /etc/hosts on local machine, map `xxx-gio.archive.org` to datanode’s IP (and that’s a literal `xxx`). Then, thanks to config in `petabox/etc/nginx/archive.conf` which lives on every data node, it should use the code in your vm-home1 home directory.

  • gio 6:37 pm on April 1, 2015 Permalink
    Tags: lending, , waitinglists   

    OL: how to fix a common problem with the waitinglists 

    For the OL admins: the following solution is implemented with the Upload loan info button, on the Borrow – Administration page.

    The lending system is managed through Internet Archive.
    When there is a connection problem between IA and OL during a loan update request (returning or borrowing) it is possible that the waitinglist stalls in an undetermined status.

    To fix this situation we are using the script:
    /opt/openlibrary/openlibrary/scripts/update-loans.py

    The script accept three commands:

        if cmd == "update-loans":
        borrow.update_all_loan_status()
    elif cmd == "update-waitinglists":
        waitinglist.prune_expired_waitingloans()
        waitinglist.update_all_waitinglists()
    elif cmd == "update-waitinglist":
        waitinglist.update_waitinglist(sys.argv[2])
    else:
        usage()

    To update and fix a waitinglist we use the update-waitinglist command:

    python scripts/openlibrary-server openlibrary.yml runscript scripts/update-loans.py update-waitinglist <InternteArchiveItemId>

    Notes:
    :: If the script does not run correctly and you receive this error message:
    “Required security token not privided or didn’t match.”
    it means you don’t have the ia_ol_shared_key in yours openlibrary.yml

    :: Do not run the script as root

     

  • gio 10:00 pm on March 31, 2015 Permalink  

    OL: Interacting with memcached 

    On any host:

    $ cd /olsystem/etc
$ . /opt/openlibrary/venv/bin/activate
$ python

    in python:

    >>> import yaml
>>> import memcache
>>> y = yaml.safe_load(open('openlibrary.yml'))
>>> mc = memcache.Client(y['memcache_servers'])

    where:

    >>> y['memcache_servers']
['ol-mem0:11211', 'ol-mem1:11211', 'ol-mem2:11211']

    -: to GET the memcache entry:

    >>> mc.get('ia.get_metadata-"houseofscorpion00farmrich"')

    -: to DELETE a memcached entry:

    >>> mc.delete('ia.get_metadata-"houseofscorpion00farmrich"')

    [ref: http://dev.blog.archive.org/2014/02/14/manually-deleting-stale-cache-entries-from-ol-memcache]

     

  • gio 10:21 pm on March 30, 2015 Permalink  

    GitLab for the Internet Archive 

    Tips for admins:

    :: Install and configure GitLab.

    :: How to convert a SVN repository to GIT.

    :: How to add users in bulk on GitLab.

    Cookbooks for users:

    :: Pro Git. a good book with everything you need to know to use Git.

    :: Git – SVN. a crash course to switch from svn to git.

    :: Become a Git guru. a good tutorial.

     

  • gio 9:24 pm on March 27, 2015 Permalink  

    GitLab: how add users in bulk 

    For this task we will use the GitLab API.

    1. Obtain the PRIVATE-TOKEN

    Do you need a token to prove to be authenticated

    giovanni@vm-gittest:~/git$ curl "http://vm-gittest.us.archive.org/api/v3/session" --data 'login=root&password=ThePassword'  | python -mjson.tool
 
{
    "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd197acbd4d13d7dd61?s=40&d=identicon",
    "bio": null,
    "can_create_group": true,
    "can_create_project": true,
    "color_scheme_id": 1,
    "created_at": "2015-03-23T17:41:18.649Z",
    "email": "admin@example.com",
    "id": 1,
    "identities": [],
    "is_admin": true,
    "linkedin": "",
    "name": "Administrator",
    "private_token": "2xyxyxysyxsyxsUe8",
    "projects_limit": 10000,
    "skype": "",
    "state": "active",
    "theme_id": 2,
    "twitter": "",
    "username": "root",
    "website_url": ""
}

    2. Create the users

    You can add a new user on GitLab using the command:

    curl --header "PRIVATE-TOKEN: YourPrivateToken" -d "email=user@archive.org&password=defaultPassword&username=username$name=name" "http://vm-gittest.us.archive.org/api/v3/users"

    To add users in bulk we need a file containing all the users to be created like:

    mark = mark <mark@archive.org>
zella = zella <zella@archive.org>
uie = uie <uie@archive.org>
kers = kers <kers@archive.org>
.....

    and run the command:

    for i in `cat authors-transform.txt | awk {'print "email="$1"@archive.org&password=defaultPassword&username="$1"&name="$1""'}`; 
   do  curl --header "PRIVATE-TOKEN: YourPrivateToken" -d $i "http://vm-gittest.us.archive.org/api/v3/users"; 
done

    WARNING: at this point all the users will receive a confirmation email to activate their account.

     

  • gio 11:32 pm on March 26, 2015 Permalink  

    Git: how to convert a SVN repository to GIT 

    For the conversion we are following the John Albin’s document:
    http://john.albin.net/git/convert-subversion-to-git

    1. Retrieve a list of all Subversion committers

    From the root of your local Subversion checkout, run this command:

    svn log -q | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors-transform_alpha.txt

    Git require the emails as username, so you have to edit each line:

    from:

    fred = fred <fred>

    to:

    fred = fred <fred@archive.org>

    for our purpose we can do that quickly running this command:

     sed "s/>/@archive.org>/g" authors-transform_alpha.txt > authors-transform.txt

    2. Clone the Subversion repository using git-svn

    This will do the standard git-svn transformation (using the authors-transform.txt file you created in step 1) and place the git repository in the “~/temp” folder inside your home directory.

    git svn clone svn://home.us.archive.org/petabox -A authors-transform.txt --stdlayout --prefix=origin/ ~/temp

    3. Convert svn:ignore properties to .gitignore

    cd ~/temp
git svn show-ignore > .gitignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'

    4. Push repository to a new project on GitLab

    git remote add gitlab git@git.domain.com.au:dev-team/favourite-project.git
git push --set-upstream gitlab master

    Congratulations you have done!

    —The following instructions are useful only if you are not using GitLab and you want a bare repository—


    4b. Push repository to a bare git repository
    Then push the temp repository to the new bare repository.

    cd ~/temp
git remote add bare ~/petabox.git
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare

    5. Rename “trunk” branch to “master”
    Your main development branch will be named “trunk” which matches the name it was in Subversion. You’ll want to rename it to Git’s standard “master” branch using:

    cd ~/new-bare.git
git branch -m trunk master

    6. Clean up branches and tags
    git-svn makes all of Subversions tags into very-short branches in Git of the form “tags/name”. You’ll want to convert all those branches into actual Git tags using:

    cd ~/new-bare.git
git for-each-ref --format='%(refname)' refs/heads/tags |
cut -d / -f 4 |
while read ref
do
  git tag "$ref" "refs/heads/tags/$ref";
  git branch -D "tags/$ref";
done

     

  • gio 6:29 pm on March 26, 2015 Permalink
    Tags: , howto, petabox   

    Petabox: git-svn quick how-to 

    How to commit:

    • rebase: git svn rebase
    • edit files vim foo.ff
    • git add foo.ff
    • git commit -m 'note about the commit'
    • git svn rebase
    • if the rebase fails because of memory problems: ulimit -v unlimited and rebase again
    • dry commit git svn dcommit --dry-run
    • check the patch git diff-tree ea56092a94b7b0266cdfbb08f69245d7761bba09~1 ea56092a94b7b0266cdfbb08f69245d7761bba09 -p
    • git svn dcommit
     

  • gio 5:25 pm on March 26, 2015 Permalink
    Tags: stats   

    OL: graphs statistics 

    OL statistics are present in two different graphics:

    Screen Shot 2015-03-26 at 10.17.14 AM

    Screen Shot 2015-03-26 at 10.18.25 AM

    They are generated through two scripts:

    :: /opt/openlibrary/openlibrary/scripts/ipstats.py runs on ol-www1
    that create the graph:Screen Shot 2015-03-31 at 3.50.06 PM

    :: /opt/openlibrary/openlibrary/scripts/store_counts.py runs on ol-home
    To generate the count stats of the past n days, execute the command:

    giovanni@ol-home:~$ sudo -s
root@ol-home:/home/giovanni# su openlibrary
openlibrary@ol-home:/home/giovanni$ source /opt/openlibrary/venv/bin/activate
(venv)openlibrary@ol-home:/home/giovanni$ cd /opt/openlibrary/openlibrary/scripts/
(venv)openlibrary@ol-home:/opt/openlibrary/openlibrary/scripts$

    and run the command:

    $ python store_counts.py /opt/openlibrary/olsystem/etc/infobase.yml /opt/openlibrary/olsystem/etc/openlibrary.yml /opt/openlibrary/olsystem/etc/coverstore.yml  n

    this creates the graphs: Screen Shot 2015-03-31 at 3.49.59 PM

    The scripts run as scheduled in the /etc/cron.d/openlibrary

    0 * * * * openlibrary /olsystem/bin/verify-node.sh ol-home && /olsystem/bin/olenv $SCRIPTS/store_counts.py /opt/openlibrary/olsystem/etc/infobase.yml /opt/openlibrary/olsystem/etc/openlibrary.yml /opt/openlibrary/olsystem/etc/coverstore.yml 1
0 * * * * www-data /olsystem/bin/verify-node.sh ol-www1 && /olsystem/bin/olenv $SCRIPTS/ipstats.py  /opt/openlibrary/olsystem/etc/openlibrary.yml
59 23 * * * openlibrary /olsystem/bin/verify-node.sh ol-home && /olsystem/bin/olenv $SCRIPTS/store_counts.py /opt/openlibrary/olsystem/etc/infobase.yml /opt/openlibrary/olsystem/etc/openlibrary.yml /opt/openlibrary/olsystem/etc/coverstore.yml 1
59 23 * * * www-data /olsystem/bin/verify-node.sh ol-www1 && /olsystem/bin/olenv $SCRIPTS/ipstats.py  /opt/openlibrary/olsystem/etc/openlibrary.yml

    When, for some reason, the graphs are broken you have to run the scripts manually.
    Be careful running them within the right days window.

    See the code for the details:
    https://github.com/internetarchive/openlibrary/blob/master/scripts/ipstats.py
    https://github.com/internetarchive/openlibrary/blob/master/scripts/store_counts.py

     

← Older posts

Newer posts →

c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel