Theiling Online    Sitemap    Conlang Mailing List HQ    Attic   

Re: ANNOUNCE: Constuff Mirroring on

From:Henrik Theiling <theiling@...>
Date:Thursday, July 3, 2008, 2:07

Iain E. Davis writes:
>... >> For mirroring, I use 'wget' and for archiving 'rsync'. Both are >> standard tools under Linux. > > Ah! It hadn't occurred to me to use wget that way. I tend to use it to get a > single file or page. :) > > I'll set something up a little later. :)
For a secondary archiver, you can use a very short script. Assuming your archive is going to be in /var/www/archives/attic, then the following will get the url list from my server, and the files from the original location: ---------------------------------------------------------------------- #! /bin/sh set -euC BOT_ID='Conlang Archive Bot' BOT_ME='Iain' cd /var/www/archives/attic wget -q -N -U "$BOT_ID ($BOT_ME)" cat urls.txt | while read URL; do wget -q -m -U "$BOT_ID ($BOT_ME)" "$URL" done ---------------------------------------------------------------------- A crontab entry running once a day or once a week (that should really be enough) would do the rest. **Henrik