Autarchy of the Private Cave » mirror https://bogdan.org.ua Tiny bits of bioinformatics, [web-]programming etc Wed, 28 Dec 2022 16:09:04 +0000 en-US hourly 1 https://wordpress.org/?v=3.8.27 My favourite command line for mirroring with wgethttps://bogdan.org.ua/2011/01/09/my-favourite-command-line-for-mirroring-with-wget.html https://bogdan.org.ua/2011/01/09/my-favourite-command-line-for-mirroring-with-wget.html#comments Sun, 09 Jan 2011 12:30:27 +0000 http://bogdan.org.ua/?p=1237 wget ––continue ––mirror ––page-requisites ––adjust-extension ––convert-links ––backup-converted ––limit-rate=500k ––wait=2 URL

(short form: wget -cmpEkK ––limit-rate=500k -w 2 URL)

  • ––continue does no harm on initial run, but when run again this instructs wget to pick up where it was interrupted the previous time
  • ––mirror turns on several options useful for mirroring (read wget manual for details)
  • ––page-requisites instructs wget to also fetch linked resources, which would otherwise be skipped
  • ––adjust-extension (or ––html-extension in older wget versions) will append the .html extension to files without it, but with document type text/html or equivalent
  • ––convert-links will convert the links for convenient local viewing
  • ––backup-converted helps to avoid re-downloading some files, especially when ––convert-links is specified
  • ––limit-rate=500k and ––wait=2 prevent overloading the target website (being nice to someone offering information you need is a must)

Share

]]> https://bogdan.org.ua/2011/01/09/my-favourite-command-line-for-mirroring-with-wget.html/feed 0