OL: How to generate the sitemaps
First you need the last ol_dump file.
This file is generated on ol-home using as source the dump file ol_dump.txt.gz. To know how to generate it please check the post: OL: how to generate the dump files
:: Generate sitemaps on ol-home
anand@ol-home:/1/var/tmp/sitemaps$ python /opt/openlibrary/openlibrary/scripts/2009/01/sitemaps/sitemap.py ../dumps/ol_dump_2015-01-31/ol_dump_2015-01-31.txt.gz .... Wed Feb 4 12:48:03 2015 writing sitemaps/sitemap_works_1700.xml.gz 39979 Wed Feb 4 12:48:03 2015 writing sitemaps/sitemap_works_1701.xml.gz 39963 Wed Feb 4 12:48:04 2015 writing sitemaps/sitemap_works_1702.xml.gz 39975 Wed Feb 4 12:48:04 2015 writing sitemaps/sitemap_works_1703.xml.gz 39947 Wed Feb 4 12:48:05 2015 writing sitemaps/sitemap_works_1704.xml.gz 39347 Wed Feb 4 12:48:05 2015 writing sitemaps/sitemap_works_1705.xml.gz 39691 Wed Feb 4 12:48:05 2015 writing sitemaps/sitemap_works_1706.xml.gz 38423 Wed Feb 4 12:48:06 2015 writing sitemaps/sitemap_works_1707.xml.gz 30851 Wed Feb 4 12:48:06 2015 writing sitemaps/sitemap_works_1708.xml.gz 3447 Wed Feb 4 12:48:06 2015 writing sitemaps/siteindex.xml.gz 19891 Wed Feb 4 12:48:06 2015 done anand@ol-home:~$ |
:: Copy sitemaps to ol-www1
anand@ol-www1:~$ sudo mkdir -p /1/var/lib/openlibrary/sitemaps anand@ol-www1:~$ sudo rsync -av rsync://ol-home/var_1/tmp/sitemaps/sitemaps /1/var/lib/openlibrary/sitemaps/ ... sitemaps/sitemap_works_1707.xml.gz sitemaps/sitemap_works_1708.xml.gz sent 94,519 bytes received 329,106,541 bytes 73,155,791.11 bytes/sec total size is 328,678,782 speedup is 1.00 anand@ol-www1:~$ |
:: Verify sitemaps are available
anand@ol-www1:~$ curl -I https://openlibrary.org/static/sitemaps/siteindex.xml.gz HTTP/1.1 200 OK Server: nginx/1.1.19 Date: Wed, 04 Feb 2015 16:48:50 GMT Content-Type: text/plain Content-Length: 14689 Last-Modified: Wed, 04 Feb 2015 12:48:06 GMT Connection: keep-alive Accept-Ranges: bytes anand@ol-www1:~$ |