Why is wget so slow

For some reason, this is taking around 2. I know this is a year old but this exact problem plagued us for days. There are a wide variety of issues that could be causing this.

What version of openssl is being used by wget - there could be an issue there. What OS is this running on full information would be useful there. There could be some form of download slowdown being enforced due to the agent ID being passed by wget implemented on the site to reduce the effects of spiders.

Is the certificate on the client site valid? You may want to specify --no-certificate-check if it is a self-signed certificate. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. I'm trying to wget an entire site to transfer from Server A to Server B. It works, but incredibly slowly. I tested this by uploading a mb psd file to my server, and grabbing it with wget ftp.

Transfer speeds were But when I try and grab the whole site minus the psd file it takes 5 minutes to transfer 92mb of html files. It's a wordpress install with a theme and images basically. Is this because it opens a new connection for each file? I tried doing this with php's ftp library, but that took just as long if not longer.

I'm using:. It is always slower to transfer many files than it is to transfer one big file. Zip the contents of your website and the transfer will go much faster. You can use tar to do this. It's not as simple as -r I admit but you could try this: Make a list of the files, separate this list in e. You make a request with wget for www. Your server picks a nameserver round robin from your resolv.

It picks DB Your IPv6 configuration is not working so we wait and wait and time out. Then we pick another address DB We wait again. Now we get Now we contact the server. It has an IPv6 address and and IPv4 address. Our name server hands us the IPv6 address and we wait again.

Then we try the IPv4 address. We start to figure out how we are going to encrypt the traffic. I checked my internet speed using Speedtest and I got.

I configured my network using NetworkManager and it doesn't give me any issues when it comes to downloading from the net or using anyother application like qbittorrent. My kernel version is. So what am I doing wrong in here or am I missing something that is causing this issue? Any sort of information in regards to this would be really appreciated.

Use a tool like reflector to pick better mirrors? I"m confused. So you're getting speed tests that report 30mb download speeds but when you try to download speed test files using wget, it's really slow.

What does this have to do with pacman? Now to answer eschwartz, it has to do with pacman since this same situation repeats itself with pacman as well. Within pacman I'm getting kbps at all time which downloading packages through it. This same situation repeats itself with yay , wget and even git clone though I'm not concerning about that in this post atleast.

Hope I was able to clear the doubts in regards to my issue. If you have any more doubts then please revert back to me. And that will do exactly nothing for you. You need to replace your mirrorlist with those that reflector lists. As for comparing speedtest, speedtest-cli, and that wget command output - those are apples and oranges. That data is meaningless. Speedtest will use a different server each time and gives wildly different results on each use.

The hetzner test is using the same remote server notwithstanding any load balancing done at their end but even there I get wildly ranging results that do not reflect local connection speed, but rather traffic on the wider internet between the local network and remote server.

It also won't fix pacman, because if git clone and wget have issues too, the issue is obviously not the mirrorlist. It's on the user's end. I don't even take those "speedtest" results with a grain of salt I take them with a beach of the Great Salt Lake: they're meaningless. But pacman running slower does mean something and can be troubleshot or troubleshooted?

Thanks for your clarification Trilby. Yes I replaced my mirrorlist with the ones listed through reflector. I followed the Wiki honestly on this part. But as you mentioned the mirrorlist doesn't affect the wget or git clone so what's the thing that I specifically need to look for to solve the issue.

Though I was already aware of the fact that git clone or wget doesn't concerns itself with mirrorlist so I didn't mention it previously.