Can you make sure you're running 1.9.0? I added more logging that prints the version number, and I don't see it in the above log. 1.9.0 should fix the 520 issue (or at least if they don't show up more than 5 times in a row).

Is there a parameter for updating archiver? Like a --update or something.

Originally posted by @Melonadev in https://github.com/agude/wayback-machine-archiver/issue_comments/711156395

I'm using the latest version of pip (20.2.4), and I'm still having troubles with reinstalling archiver:

C:\Users\yewhe>pip install wayback-machine-archiver
Collecting wayback-machine-archiver
  Using cached wayback_machine_archiver-1.9.0-py3-none-any.whl (7.1 kB)
Collecting requests
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\python38\lib\site-packages (from requests->wayback-machine-archiver) (1.25.10)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Requirement already satisfied: chardet<4,>=3.0.2 in c:\python38\lib\site-packages (from requests->wayback-machine-archiver) (3.0.4)
Installing collected packages: idna, certifi, requests, wayback-machine-archiver
  WARNING: Failed to write executable - trying to use .deleteme logic
ERROR: Could not install packages due to an EnvironmentError: [WinError 2] The system cannot find the file specified: 'c:\\python38\\Scripts\\archiver.exe' -> 'c:\\python38\\Scripts\\archiver.exe.deleteme'

Awesome!

Hi @Melonadev!

This isn't a bug with Archiver, but I'm happy to try to help.

Permissions Issue

The problem you're having is a filesystem permissions issue. pip is being run by your user account but is trying to write to a location that only an administrator can write to.

You have two options:

1. Run pip install --user wayback-machine-archiver. This will install the packages to your user directory instead of the system directory, so only you will be able to use them, but you don't need administrator privileges.
2. Run pip install wayback-machine-archiver but from an administrator shell/terminal. This will install it for everyone, and since you're the admin you can install to the system directory instead of your user directory. The DOWNSIDE of this method is that you might have some issues running the script as your normal user. I suggest going with the \1. option above.

Other Notes

Since you have already installed Archiver, you should be able to run pip with --upgrade, which upgrades an already installed package. Something like pip install --upgrade --user wayback-machine-archiver.

Let me know how it goes!

@Melonadev I've released 1.9.0, which should fix the 520 errors you are seeing. It got rid of them in a test run I did. Let me know if you see 520s again!

I haven't been able to reproduce this. Can you use pip to upgrade to the newest version (1.9.0) and run:

archiver --file ./fest.txt --rate-limit-wait 30 --log DEBUG > out.log 2>&1

That's what it would be on Linux, not sure on Windows. The > out.log 2>&1 is just saving the debug and error log to a file, but you can leave those out and copy/paste the output here as well.

I tried 60 for this file: jass.txt

...then 520 Server Error: UNKNOWN for url

Microsoft Windows [Version 10.0.18363.1082]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\yewhe\Downloads>archiver --file jass.txt --rate-limit-wait 60
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2017-ppl-lifetime-achievement-award/
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2017-ppl-lifetime-achievement-award/
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2013-uk-vocalist-of-the-year/
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2013-uk-vocalist-of-the-year/
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/and-the-judges-are/
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/and-the-judges-are/
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2020-prs-for-music-uk-jazz-act-of-the-year/
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2020-prs-for-music-uk-jazz-act-of-the-year/
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2017-ppl-lifetime-achievement-award/
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\yewhe\AppData\Local\Programs\Python\Python38-32\Scripts\archiver.exe\__main__.py", line 7, in <module>
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 243, in main
    pool.map(partial_call, archive_urls)
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2017-ppl-lifetime-achievement-award/

Originally posted by @Melonadev in https://github.com/agude/wayback-machine-archiver/issue_comments/710761066

Looks like retry is not triggering for these! I'm actually surprised any passed. Adding 520 to the list of codes to retry for.

Logging in text:

DEBUG:root:Arguments: Namespace(archive_sitemap=False, file='./fest.txt', jobs=1, log_file=None, log_level='DEBUG', rate_limit_in_sec=30, sitemaps=[], urls=[])
INFO:root:Parsing sitemaps
INFO:root:Reading urls from file: ./fest.txt
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/talks-2020
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/wellness
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/hub
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/talks-2019-2
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/2020-lineup
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/personnel
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/videos-2020
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/photos-2020
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/past-shows
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/2020-friday-jan-10th
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/2020-saturday-jan-11th
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/2020-friday-jan-17th
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/marathon-map
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2020/1/23/thank-you-for-joining-us-at-2020-winter-jazzfest
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2020/1/16/wjf-closing-night-show-with-mark-guiliana-beat-music-improvisations-at-nublu-is-sold-out
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475-bj3fd
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/archive
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/about
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/sponsorship
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/contact
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/talks-2018
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/piedmont-blues-a-search-for-salvation
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/amendola-vs-blades-w/-skerik-mark-guiliana-space-heroes
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/jazz-for-kids
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/12/13/opening-night-dj-set-just-added-on-jan-8th-gilles-peterson-lefto-and-kassa-overall-at-nublu
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/12/13/artemis-just-added-to-eubanks-evans-experience-allison-miller-boom-tic-boom-at-lpr-on-jan-13th
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/11/29/revive-yo-feelings-a-musicians-wellness-benefit-with-robert-glasper-terrace-martin-more-just-added-to-jan-11th-manhattan-marathon
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/11/29/marathon-artist-lineups-by-day-just-announced
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/11/7/just-announced-seu-jorge-at-the-town-hall
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/10/31/new-shows-just-announced-tickets-on-sale-friday-nov-1-at-12-pm-et
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/10/30/tickets-on-sale-and-new-shows-announced
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/10/24/tickets-on-sale-friday-1025-at-12-noon-et
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/10/16/2020-nyc-winter-jazzfest-initial-lineup-announced
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/10/6/were-proud-to-be-part-of-prs-foundations-international-keychange-program
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/10/6/2020-wjf-dates-announced-january-9-18-more-details-to-come
DEBUG:root:Creating archive URL for https://www.winterjazzfest.com/news/2019/10/6/thx-for-attending-2019-wjf-see-you-next-year
DEBUG:root:Archive URLs: {'https://web.archive.org/save/https://www.winterjazzfest.com/past-shows', 'https://web.archive.org/save/https://www.winterjazzfest.com/2020-friday-jan-10th', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/11/7/just-announced-seu-jorge-at-the-town-hall', 'https://web.archive.org/save/https://www.winterjazzfest.com/talks-2020', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/12/13/artemis-just-added-to-eubanks-evans-experience-allison-miller-boom-tic-boom-at-lpr-on-jan-13th', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/11/29/revive-yo-feelings-a-musicians-wellness-benefit-with-robert-glasper-terrace-martin-more-just-added-to-jan-11th-manhattan-marathon', 'https://web.archive.org/save/https://www.winterjazzfest.com/piedmont-blues-a-search-for-salvation', 'https://web.archive.org/save/https://www.winterjazzfest.com/', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/11/29/marathon-artist-lineups-by-day-just-announced', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/6/were-proud-to-be-part-of-prs-foundations-international-keychange-program', 'https://web.archive.org/save/https://www.winterjazzfest.com/videos-2020', 'https://web.archive.org/save/https://www.winterjazzfest.com/hub', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/30/tickets-on-sale-and-new-shows-announced', 'https://web.archive.org/save/https://www.winterjazzfest.com/2020-friday-jan-17th', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/16/wjf-closing-night-show-with-mark-guiliana-beat-music-improvisations-at-nublu-is-sold-out', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/6/2020-wjf-dates-announced-january-9-18-more-details-to-come', 'https://web.archive.org/save/https://www.winterjazzfest.com/photos-2020', 'https://web.archive.org/save/https://www.winterjazzfest.com/contact', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/24/tickets-on-sale-friday-1025-at-12-noon-et', 'https://web.archive.org/save/https://www.winterjazzfest.com/talks-2018', 'https://web.archive.org/save/https://www.winterjazzfest.com/amendola-vs-blades-w/-skerik-mark-guiliana-space-heroes', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added', 'https://web.archive.org/save/https://www.winterjazzfest.com/archive', 'https://web.archive.org/save/https://www.winterjazzfest.com/2020-lineup', 'https://web.archive.org/save/https://www.winterjazzfest.com/about', 'https://web.archive.org/save/https://www.winterjazzfest.com/jazz-for-kids', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/31/new-shows-just-announced-tickets-on-sale-friday-nov-1-at-12-pm-et', 'https://web.archive.org/save/https://www.winterjazzfest.com/wellness', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/12/13/opening-night-dj-set-just-added-on-jan-8th-gilles-peterson-lefto-and-kassa-overall-at-nublu', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/6/thx-for-attending-2019-wjf-see-you-next-year', 'https://web.archive.org/save/https://www.winterjazzfest.com/sponsorship', 'https://web.archive.org/save/https://www.winterjazzfest.com/2020-saturday-jan-11th', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/23/thank-you-for-joining-us-at-2020-winter-jazzfest', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/16/2020-nyc-winter-jazzfest-initial-lineup-announced', 'https://web.archive.org/save/https://www.winterjazzfest.com/talks-2019-2', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475-bj3fd', 'https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax', 'https://web.archive.org/save/https://www.winterjazzfest.com/personnel', 'https://web.archive.org/save/https://www.winterjazzfest.com/marathon-map'}
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/past-shows
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): web.archive.org:443
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/past-shows HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175158/https://www.winterjazzfest.com/past-shows HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/2020-friday-jan-10th
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/2020-friday-jan-10th HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175246/https://www.winterjazzfest.com/2020-friday-jan-10th HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015062242/https://www.winterjazzfest.com/2020-friday-jan-10th HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/11/7/just-announced-seu-jorge-at-the-town-hall
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2019/11/7/just-announced-seu-jorge-at-the-town-hall HTTP/1.1" 520 0
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/11/7/just-announced-seu-jorge-at-the-town-hall
Traceback (most recent call last):
  File "/home/user/bin/anaconda3/lib/python3.8/site-packages/wayback_machine_archiver/archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "/home/user/bin/anaconda3/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/11/7/just-announced-seu-jorge-at-the-town-hall
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/hub
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): web.archive.org:443
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/hub HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175409/https://www.winterjazzfest.com/hub HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015062614/https://www.winterjazzfest.com/hub HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/30/tickets-on-sale-and-new-shows-announced
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2019/10/30/tickets-on-sale-and-new-shows-announced HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175457/https://www.winterjazzfest.com/news/2019/10/30/tickets-on-sale-and-new-shows-announced HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015063403/https://www.winterjazzfest.com/news/2019/10/30/tickets-on-sale-and-new-shows-announced HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/2020-friday-jan-17th
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/2020-friday-jan-17th HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175542/https://www.winterjazzfest.com/2020-friday-jan-17th HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015063821/https://www.winterjazzfest.com/2020-friday-jan-17th HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/16/wjf-closing-night-show-with-mark-guiliana-beat-music-improvisations-at-nublu-is-sold-out
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2020/1/16/wjf-closing-night-show-with-mark-guiliana-beat-music-improvisations-at-nublu-is-sold-out HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175628/https://www.winterjazzfest.com/news/2020/1/16/wjf-closing-night-show-with-mark-guiliana-beat-music-improvisations-at-nublu-is-sold-out HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015064736/https://www.winterjazzfest.com/news/2020/1/16/wjf-closing-night-show-with-mark-guiliana-beat-music-improvisations-at-nublu-is-sold-out HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/6/2020-wjf-dates-announced-january-9-18-more-details-to-come
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2019/10/6/2020-wjf-dates-announced-january-9-18-more-details-to-come HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175714/https://www.winterjazzfest.com/news/2019/10/6/2020-wjf-dates-announced-january-9-18-more-details-to-come HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015064342/https://www.winterjazzfest.com/news/2019/10/6/2020-wjf-dates-announced-january-9-18-more-details-to-come HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/photos-2020
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/photos-2020 HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175758/https://www.winterjazzfest.com/photos-2020 HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/contact
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/contact HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175848/https://www.winterjazzfest.com/contact HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015062327/https://www.winterjazzfest.com/contact HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/24/tickets-on-sale-friday-1025-at-12-noon-et
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2019/10/24/tickets-on-sale-friday-1025-at-12-noon-et HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017175931/https://www.winterjazzfest.com/news/2019/10/24/tickets-on-sale-friday-1025-at-12-noon-et HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015062507/https://www.winterjazzfest.com/news/2019/10/24/tickets-on-sale-friday-1025-at-12-noon-et HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/talks-2018
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/talks-2018 HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180014/https://www.winterjazzfest.com/talks-2018 HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/amendola-vs-blades-w/-skerik-mark-guiliana-space-heroes
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/amendola-vs-blades-w/-skerik-mark-guiliana-space-heroes HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180057/https://www.winterjazzfest.com/amendola-vs-blades-w/-skerik-mark-guiliana-space-heroes HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015063904/https://www.winterjazzfest.com/amendola-vs-blades-w/-skerik-mark-guiliana-space-heroes HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180141/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015064037/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): web.archive.org:443
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180230/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/archive
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/archive HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180313/https://www.winterjazzfest.com/archive HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015064904/https://www.winterjazzfest.com/archive HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/2020-lineup
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/2020-lineup HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180415/https://www.winterjazzfest.com/2020-lineup HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/about
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/about HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180518/https://www.winterjazzfest.com/about HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015063007/https://www.winterjazzfest.com/about HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/jazz-for-kids
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/jazz-for-kids HTTP/1.1" 520 0
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.winterjazzfest.com/jazz-for-kids
Traceback (most recent call last):
  File "/home/user/bin/anaconda3/lib/python3.8/site-packages/wayback_machine_archiver/archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "/home/user/bin/anaconda3/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.winterjazzfest.com/jazz-for-kids
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): web.archive.org:443
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475 HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180657/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475 HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015063738/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475 HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/23/thank-you-for-joining-us-at-2020-winter-jazzfest
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2020/1/23/thank-you-for-joining-us-at-2020-winter-jazzfest HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180746/https://www.winterjazzfest.com/news/2020/1/23/thank-you-for-joining-us-at-2020-winter-jazzfest HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015064122/https://www.winterjazzfest.com/news/2020/1/23/thank-you-for-joining-us-at-2020-winter-jazzfest HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/10/16/2020-nyc-winter-jazzfest-initial-lineup-announced
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2019/10/16/2020-nyc-winter-jazzfest-initial-lineup-announced HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180831/https://www.winterjazzfest.com/news/2019/10/16/2020-nyc-winter-jazzfest-initial-lineup-announced HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015064948/https://www.winterjazzfest.com/news/2019/10/16/2020-nyc-winter-jazzfest-initial-lineup-announced HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/talks-2019-2
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/talks-2019-2 HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017180915/https://www.winterjazzfest.com/talks-2019-2 HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015062105/https://www.winterjazzfest.com/talks-2019-2 HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475-bj3fd
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475-bj3fd HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017181011/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax-mygwj-jm475-bj3fd HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017181055/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015064515/https://www.winterjazzfest.com/news/2020/1/2/new-wjf-talks-just-added-gcpax HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/personnel
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/personnel HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017181139/https://www.winterjazzfest.com/personnel HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015064820/https://www.winterjazzfest.com/personnel HTTP/1.1" 200 0
DEBUG:root:Sleeping for 30
INFO:root:Calling archive url https://web.archive.org/save/https://www.winterjazzfest.com/marathon-map
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.winterjazzfest.com/marathon-map HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201017181229/https://www.winterjazzfest.com/marathon-map HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /web/20201015062702/https://www.winterjazzfest.com/marathon-map HTTP/1.1" 200 0
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/user/bin/anaconda3/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/user/bin/anaconda3/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/user/bin/anaconda3/lib/python3.8/site-packages/wayback_machine_archiver/archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "/home/user/bin/anaconda3/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/11/7/just-announced-seu-jorge-at-the-town-hall
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/bin/anaconda3/bin/archiver", line 8, in <module>
    sys.exit(main())
  File "/home/user/bin/anaconda3/lib/python3.8/site-packages/wayback_machine_archiver/archiver.py", line 243, in main
    pool.map(partial_call, archive_urls)
  File "/home/user/bin/anaconda3/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/user/bin/anaconda3/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.winterjazzfest.com/news/2019/11/7/just-announced-seu-jorge-at-the-town-hall

In testing for #18, I have gotten a few 520s as well, the first few resolved successfully, but the last one blew up the program. Logging here.

I think I'm not handling error statuses in the best way possible, probably here: https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py#L38

One thing to do is handle statuses differently. 520 generally means a transient error, but 500 generally means rate limiting (which is unrecoverable).

Around 10 minutes into running archiver for a txt file:

Here's the text file: bluemaxima.txt

This happens: 500 Server Error: Internal Server Error for url

C:\Users\yewhe\Downloads>archiver --file bluemaxima.txt
ERROR:root:500 Server Error: Internal Server Error for url: https://web.archive.org/save/https://bluemaxima.org/flashpoint/datahub/Game_Master_List
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://web.archive.org/save/https://bluemaxima.org/flashpoint/datahub/Game_Master_List
ERROR:root:500 Server Error: Internal Server Error for url: https://web.archive.org/save/https://bluemaxima.org/flashpoint/datahub/Special:RecentChangesLinked/Unity_Curation
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://web.archive.org/save/https://bluemaxima.org/flashpoint/datahub/Special:RecentChangesLinked/Unity_Curation
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://web.archive.org/save/https://bluemaxima.org/flashpoint/datahub/Game_Master_List
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\yewhe\AppData\Local\Programs\Python\Python38-32\Scripts\archiver.exe\__main__.py", line 7, in <module>
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 243, in main
    pool.map(partial_call, archive_urls)
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://web.archive.org/save/https://bluemaxima.org/flashpoint/datahub/Game_Master_List

Does this mean that after the error link, the archiver doesn't run anymore? In other words, does archiver interrupt the process and ignore the remaining urls in the text file? Does archiver retry the link after a period of time?

That looks like a separate issue, I've opened a new bug: #19

I tried 60 for this file: jass.txt

...then 520 Server Error: UNKNOWN for url

Microsoft Windows [Version 10.0.18363.1082]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\yewhe\Downloads>archiver --file jass.txt --rate-limit-wait 60
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2017-ppl-lifetime-achievement-award/
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2017-ppl-lifetime-achievement-award/
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2013-uk-vocalist-of-the-year/
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2013-uk-vocalist-of-the-year/
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/and-the-judges-are/
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/and-the-judges-are/
ERROR:root:520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2020-prs-for-music-uk-jazz-act-of-the-year/
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2020-prs-for-music-uk-jazz-act-of-the-year/
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 38, in call_archiver
    r.raise_for_status()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\requests\models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2017-ppl-lifetime-achievement-award/
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\yewhe\AppData\Local\Programs\Python\Python38-32\Scripts\archiver.exe\__main__.py", line 7, in <module>
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\site-packages\wayback_machine_archiver\archiver.py", line 243, in main
    pool.map(partial_call, archive_urls)
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "c:\users\yewhe\appdata\local\programs\python\python38-32\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
requests.exceptions.HTTPError: 520 Server Error: UNKNOWN for url: https://web.archive.org/save/https://www.jazzfmawards.com/awards/2017-ppl-lifetime-achievement-award/

Originally posted by @Melonadev in https://github.com/agude/wayback-machine-archiver/issue_comments/710761066

Interesting error! I've never seen this one before.

I would guess it's a single URL from your list that has some weird script redirection, or something else like that. I'll rerun with logging and see what happens.

The problem with that is that it's not the specific URL, it's that the Internet Archive is rate limiting you.

Archiver used to just chug through the full list even when one failed, but as soon as one URL failed all subsequent URLs would also fail. The only solution was to stop, wait for the rate limit to end, and try again.

Hi @Melonadev!

When Archiver errors (as above) that means it does halts and does not back up the remaining items on the list.

It should retry multiple times, but I wonder if I'm short-circuiting that here: https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py#L38

Anyway, the most common reason for a 500 error is the Internet Archive rate-limiting you. My suggestion is to turn the --rate-limit-wait parameter higher! It defaults to 5 seconds; I'd try 30 or even 60.

I'm trying to fix the problem I reported in #3876, although we haven't agreed that it is a problem. :-) This modifies code originally committed as #2127.

This PR removes the first replaceAll() in insertAsQuote() which looks for two or more "line feed" (\n) characters optionally separated by spaces with a single line feed. If this PR is merged, text will be pasted with the same white space it has before quoting.

We might want to think about how this function should work (that is, for a set of input texts, what should the output be) and then rework it to do that. For example, I'm not sure the final .replaceAll("\n$", "") is doing what it should be. I think it's supposed to strip remaining empty lines at the end of the text but for my tests strings it doesn't seem to. (It does do that if you move it before the "insert >" step.)

So that is to say, I'm happy to keep tweaking this PR until it meets your approval!

On 20.04 Mate I'm seeing this issue with dual horizontal monitors.

My monitors are next to each other, so I have a left and a right monitor. Using the Mutiny panel layout, if the left monitor is primary (so the dock on the left side of the left monitor) the windows maximize and snap without overlapping the dock.

If I make the right monitor primary (so the dock is on the left side of the right monitor, hence at the "seem" between the two monitors) the windows maximize under the dock.

I have tried switching between nvidia-driver-450 and Nouveau and also between Marco Adaptive and Marco (No compositing) but the bug persists.

Glad to hear it! :-)

I get the following error message: "NameError: global name 'Retry' is not defined". Any idea how to fix it?

😬 Sorry, that code was a quick sketch, and it looks like I missed some imports.

You'll need:

from urllib3.util.retry import Retry

P.S. Plex will only run python 2.7

I feel your pain! My Raspberry Pis only support 2.7 right now, so I'm trying to keep Archiver working on it as well!

Why write a data post when I can ramble for 2 pages?!

Why make a data post when I can ramble for 2 pages?!

There are several posts that should link to my negotiating post (PR #32).

Here is an incomplete list:

• The 2020 interviewing post
• The salary posts

Looks like this is the line the does it: https://github.com/iNPUTmice/Conversations/blob/c9e6653e33676c3df8502eb7a18d5dc24b7e7750/src/main/java/eu/siacs/conversations/ui/widget/EditMessage.java#L145

General information

• Version: 2.8.10+pcr
• Device: Pixel 3
• Android Version: Android 11 (stock)
• Server name: self hosted
• Server software: prosody 0.11.5
• Installed server modules: I'd have to look...
• Conversations source: PlayStore

Steps to reproduce

1. Copy text with empty lines
2. Select 'Paste as Quote'
3. All empty lines are stripped

Expected result

If I copy and paste the following text:

This is the first line.

This is the second line.

Gets pasted as: I would expect:

> This is the first line.
>
> This is the second line.

Actual result

> This is the first line.
> This is the second line.

Preserving the empty lines would be nice during paste, especially for multi-paragraph quotes where the empty lines separate the paragraphs.

It breaks for some objects (like lines).

Most recent post at the last minute...

    CardRank(rank),

Most recent post at the last minute...

Yup @tataganesh! Looks like TF Serving will do it for you naturally!

@lauhaide: The program will throw an error and terminate when it fails. Right here:

https://github.com/agude/wayback-machine-archiver/blob/master/wayback_machine_archiver/archiver.py#L36-L41

I run this script to back up my personal site every evening. It's about 100 pages, and I run with --rate-limit-wait=60. It completes most of the time, but every few weeks it'll error out due to rate limiting from the Internet Archive.

So I don't have an exact number for you, but I would say closer to 30-60 seconds than 1-2. :-)

Hi @lauhaide!

This script doesn't overwrite, as it were, because it asks The Wayback Machine to save a snapshot of the current page. As you can see here with the Yahoo.com archive there are multiple snapshots stored each day.

As for recovering the timestamp, you could get that from the Wayback Machine itself (you'll see each snapshot is timestamped on the Yahoo page for example), but that's not something this tool supports.

If you are on Linux, you could do something like this:

archiver https://yahoo.com --log DEBUG 2>&1 | ts '[%Y-%m-%d %H:%M:%S]'

That would timestamp every line of the debug program like this:

[2020-08-11 13:17:12] DEBUG:root:Arguments: Namespace(archive_sitemap=False, file=None, jobs=1, log_file=None, log_level='DEBUG', rate_limit_in_sec=5, sitemaps=[], urls=['https://yahoo.com'])
[2020-08-11 13:17:12] INFO:root:Adding page URLs to archive
[2020-08-11 13:17:12] DEBUG:root:Page URLs to archive: ['https://yahoo.com']
[2020-08-11 13:17:12] DEBUG:root:Creating archive URL for https://yahoo.com
[2020-08-11 13:17:12] INFO:root:Parsing sitemaps
[2020-08-11 13:17:12] DEBUG:root:Archive URLs: {'https://web.archive.org/save/https://yahoo.com'}
[2020-08-11 13:17:13] DEBUG:root:Sleeping for 5
[2020-08-11 13:17:18] INFO:root:Calling archive url https://web.archive.org/save/https://yahoo.com
[2020-08-11 13:17:18] DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): web.archive.org:443
[2020-08-11 13:17:18] DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://yahoo.com HTTP/1.1" 301 0
[2020-08-11 13:17:30] DEBUG:urllib3.connectionpool:https://web.archive.org:443 "HEAD /save/https://www.yahoo.com/ HTTP/1.1" 200 0

Which you could use to get a rough timestamp from.

Local sitemaps (prefixed with file://) are not handled correctly when also using --archive-sitemap-also.

Remote sitemaps should be separated out and only they should be backed up.

This bug is in 1.7.0 and 1.7.1.