Slow on RPi: manage retrievecontentpack download

[holta]

Summary

Running "kalite manage retrievecontentpack download en" takes many hours/days to complete on KA Lite 0.17.4 (repeatedly over past weeks, on 2 different ISP's, Time Warner/Spectrum business line and AT&T DSL home line).

It sometimes begins quickly, but then progress takes about an hour to increase by each additional 1% step, such that the 0.9GB (929,916,955 bytes) en.zip file never downloads realistically.

The command-line workaround is to run the following 2 steps, which completes about 100X faster — in a few short minutes — instead of hours/days:

wget http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip
kalite manage retrievecontentpack local en en.zip

System information

• Operating system: Raspbian Lite 2018-04-18 or Raspbian Lite 2018-06-27
• Hardware: Raspberry Pi 3 B+
• KA Lite Version: 0.17.4
• ISP's confirmed to exhibit this problem:
  • Time Warner Spectrum cable business line: 87 mbit/sec down, 10mbit/sec up
  • AT&T DSL home line: 16 mbit/sec down, 1.5mbit/sec up

How to reproduce

1. Install KA Lite 0.17.4 on Raspbian
2. Run "kalite manage retrievecontentpack download en" over either ISP (AT&T or Time Warner/Spectrum)
3. Observe that the download takes many hours (or several days) to complete.

Screenshots

Progress: [==========================                        ] (53%)root@raspber
root@raspberrypi:/home/pi#

Real-life consequences (anything community should be aware, for instance how it affects your deployment)

Internet-in-a-Box install scripts (http://download.iiab.io/6.6) have become unusable across major US ISP's (such as AT&T and Time Warner/Spectrum) that have this problem.

Time Warner/Spectrum has about 22 million broadband customers and AT&T has about 16 million broadband customers. It is possible that other US and int'l ISP's are affected as well. We have confirmed that Comcast customers (another major US ISP) are not affected to date.

Internet-in-a-Box (https://github.com/iiab) has documented the workaround for command-line-savvy users (at http://FAQ.IIAB.IO #27) under "Mandatory English Pack taking too long to download?"

However, sadly, many/most all others are giving up in frustration :(

Thanks @benjaoming for contacting me if there are more specific diagnostics we can ask implementers to provide, towards resolving this tedious-but-very-real-issue preventing people from using KA Lite!

[benjaoming]

Thanks a lot for the report @holta - it sounds really weird.

What download speeds are you producing if you just fetch the package directly from here?

http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/

[holta]

What download speeds are you producing if you just fetch the package directly from here?

http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/

wget and browser[*] all download the 0.9GB en.zip completely within less than 5min using the above-mentioned Time Warner/Spectrum cable/business line — and in less than 10min with the above-mentioned AT&T DSL/home line.

Regardless whether the user(s) in question used Chrome, Edge or Firefox on Windows 10 for comparison — this very rapid download of en.zip was also confirmed with wget from Raspbian Lite.

Whereas the equivalent download when running "kalite manage retrievecontentpack download en" takes overnight (or days, yep!)

[holta]

Typo fixed in above bug report...what I meant was...and the problematic command in question is:

kalite manage retrievecontentpack download en

(I had accidentally pasted in "kalite manage help retrievecontentpack", apologies!)

[benjaoming]

Whereas the equivalent download when running "kalite manage retrievecontentpack download en" takes overnight (or days, yep!)

Yaiks, but did this use to work? Or was it always slow?

[benjaoming]

Thanks a lot for the thorough description @holta

[holta]

Whereas the equivalent download when running "kalite manage retrievecontentpack download en" takes overnight (or days, yep!)

Yaiks, but did this use to work? Or was it always slow?

Good Question! I wish I knew.

FWIW: AT&T and Time Warner/Spectrum are both very large, but also regional ISP's, used in different parts of the USA. It's (certainly possible) that prior implementers/users in those areas just gave up when facing this issue.

Conversely it's also possible that major ISP's are implementing severe new bandwidth shaping / bandwidth throttling techniques that are hitting us very hard now, but that might not have existed in prior years. Certainly the USA's net neutrality regulation ended on 2018-06-11 — whether or not this is related I have no idea :/

[benjaoming]

I'm unable to reproduce this on a 100 mbit connection. It took only 3 min 21 seconds.

I do smell some kind of bandwidth throttling.

Would you be able to perhaps do some tests like trying with VPN? See: https://www.highspeedinternet.com/resources/how-can-i-tell-if-my-internet-is-being-throttled-by-my-isp/

[benjaoming]

Maybe they red-flagged you somehow?

2. Some ISPs have been accused of throttling an internet user’s speeds based on the user’s internet activities. Although that wasn’t allowed under net neutrality laws, it still happened on occasion. In such cases, the user was often participating in high-bandwidth activities, like downloading videos via torrent files. The ability of an ISP to change your internet speed on the fly based on your activities is part of what makes throttling difficult to detect.

[holta]

Maybe they red-flagged you somehow?

Highly doubtful when the same exact thing happens across different ISP's and different network environments in different places.

Note we've reconfirmed since Tuesday that the implementation of "kalite manage retrievecontentpack download en" is indeed the problem:

Running "wget http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip" and then "kalite manage retrievecontentpack local en en.zip" works flawlessly within ~10 min across these ISP's.

To Be Clear: "kalite manage retrievecontentpack download en" fails to complete in these exact same situations.

[holta]

Would you be able to perhaps do some tests like trying with VPN? See: https://www.highspeedinternet.com/resources/how-can-i-tell-if-my-internet-is-being-throttled-by-my-isp/

I would be. But realistically, it's not at all easy to ask distant users to modify their networks.

Particularly when they already have a 2-line workaround (yes they were completely stuck — and of course are very happy now with the ~100X speed improvement ;-)

[benjaoming]

@holta I wasn't proposing VPN as some work around, just as a means for identifying the issue :)

If you are victim of throttling, then I think a lot of activists downloading data from various places will find them selves at risk. Not just from using a Learning Equality server for fetching a KA Lite content pack, but I'm sure there's many other resources that would end up in such filters :/

Anyways, if you could catch the error where at the same instance you have limited bandwidth without VPN and then a VPN gives you bandwidth, then I think we have at least found the cause, although a VPN might of course also be throttled :)

[holta]

Not just from using a Learning Equality server for fetching a KA Lite content pack

It's not the Learning Equality server.

It's the Learning Equality client (downloader) that's the problem here. Again (just FYI, among other choices you face!) if you chose to reimplement "kalite manage retrievecontentpack download en" as follows, all the problems would go away:

wget http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip
kalite manage retrievecontentpack local en en.zip

[holta]

RECAP: the proof that https://github.com/learningequality/ka-lite/blob/master/kalite/distributed/management/commands/retrievecontentpack.py is failing is that Internet-in-a-Box has replaced it (if this is what "kalite manage retrievecontentpack download en" calls) with wget http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip and all the problems have gone away.

For the approx 40 Million families/businesses who use AT&T and Time Warner/Spectrum as their ISP, they are now back in business according to all tests so far.

Whereas every known download to date (over those ISP's) came to an absolute crawl right around 20% of the way into the download of http://pantry.learningequality.org/downloads/ka-lite/0.17/content/contentpacks/en.zip when using "kalite manage retrievecontentpack download en"

Again, I apologize I don't know the reasons. But if other KA Lite users can benefit from this workaround IIAB has developed, the more the merrier.