Cfrtjryk

I've got a very weird SSH behavior and it's making me a bit nervous. Basically the progress meter is totally bogus: so bogus that it's nearly useless. And I fear that there may be something else going on (more on this below).

Basically I only have a 60 KB/s upload link or so, yet every single scp I initiate starts saying it's doing 2 MB/s.

Then it invariably tries to "fix" that number, slowly converging toward the real value.

Then, for "big" files (a few MB or more), it invariably stalls at 100% for many seconds (and eventually it succeeds and I get back to the prompt).

The output looks like this:

...

test.tgz 16% 2112KB   2.1MB/s   00:04 ETA

test.tgz 17% 2208KB   1.7MB/s   00:06 ETA

test.tgz 18% 2320KB   1.2MB/s   00:08 ETA

test.tgz 19% 2448KB   1.1MB/s   00:08 ETA

test.tgz 20% 2576KB 942.2KB/s   00:10 ETA

test.tgz 21% 2704KB 697.3KB/s   00:14 ETA

test.tgz 22% 2832KB 576.3KB/s   00:16 ETA

test.tgz 23% 2960KB 478.3KB/s   00:20 ETA

test.tgz 24% 3088KB 399.0KB/s   00:23 ETA

test.tgz 25% 3216KB 334.7KB/s   00:27 ETA

test.tgz 26% 3344KB 282.6KB/s   00:32 ETA

test.tgz 27% 3472KB 240.4KB/s   00:37 ETA

test.tgz 28% 3600KB 185.6KB/s   00:48 ETA

test.tgz 29% 3728KB 161.9KB/s   00:54 ETA

test.tgz 30% 3856KB 142.7KB/s   01:01 ETA

...

(it's on a single line: I did paste multiple lines here, which I got using scp -vvv to better explain my issue).

At the end, when the output is invariably stuck at 100%, there's actually quite a lot of data missing: I've checked this on the other end and the file is definitely not there in its entirety yet.

All the measurements in percentage are basically pointless: when it says that 80% of a 12 MB file are there, there are only about 65% on the server.

What can explain this?

I'm posting here because I was wondering how these numbers would look like if a man-in-the-middle attack was happening close to my system (maybe on a compromised router or something).

I'm using Linux and SSH / SCP since many years and I don't remember that these numbers were that off.

EDIT

Download works as expected: if I scp from the remote host to my computer, then the %, ETA and KB/s are all correct.

This behaviour is easily explained through output buffer size and TCP window settings.

First, when receiving data, you either have the bits or you don't. Your local scp knows how much it is expecting and how much it has received so far, so it can give you an accurate assessment of the progress and estimated time remaining.

When you are sending data, you don't have any information about how much of that data has actually reached the receiver yet. Your local machine will have an output buffer that holds data after it has been "sent" by the application (scp) and before it is actually transmitted on the network. Additionally, TCP allows a certain amount of data to be "in flight" between the sender and the receiver.

When sending data, scp only sees how much it has handed off to the OS for eventual transmission. Filling up the output buffers happens really quickly, so that's why scp measures a high transmission rate initially. As the transfer progresses, this value converges toward the real transmission rate. After you have handed all the data off to your OS, it still has to reach the other side and that's why it appears to be "stuck" at 100% for several seconds at the end.

Modern OSes and TCP networks have increased the TCP window size (see TCP window scale option) to account for "long fat networks" with both high bandwidth and high latency. That is why you may be seeing this behaviour more often than you have in the past.

I recently came across a very similar issue where WireShark was showing TCP ZeroWindow errors any time I would try to push data over ssh beyond a certain rate. I finally traced this back to something in my route screwing up with IP quality of service handling. Adding this config to my sshd_config and ssh_config fixed it:

IPQoS=af21 cs1

These settings will be the default in a newer SSH version than I currently have: https://www.openssh.com/txt/release-7.8