Michael Fleming

Forewarned is Forearmed

On a call recently with a customer troubleshooting a slow SCP file transfer with Openssh in preparation for a large data migration into the cloud, it became clear that the packets were being marked with an unknown dscp value, variously marked as 0x01 and 0x08, which translate to dscp values 0x04 and 0x02, see figure 1 below.

Openssh 7.4 was installed on the cloud server and a capture taken server side indeed proved that it was the Openssh application that was marking the packets, should we be worried and was it somehow implicated in the slow transfer seen by the customer?

The first thing was to find out if this was normal behavior or a bug and after doing the required internet research we find out it is indeed a function of the application. The full story can be found here

By default Openssh 7.4 and earlier uses two types of packet marking, 0x01 for normal secure shell and 0x08 for secure copy, which translate to 'lowdelay' for interactive sessions and 'throughput' for non-interactive sessions. As dscp values 0x04 and 0x02 don’t map to anything, hence Wireshark flagging it as unknown, then any intermediate routers with QoS configured should just ignore the marking and treat the packets as best effort CS0. To err on the side of caution we decided to set the IPQoS field to CS0, subsequent packet captures proved that Openssh was behaving as expected and marking the the traffic CS0 which was what the client required.

The article goes on to explain that in future Openssh would use recognized dscp values for both lowdelay and throughput and that these would map to AF21 and CS1 respectively out of the box.

The astute reader will now be joining the dots and seeing the oncoming issue for bulk scp file tranfers in a QoS enabled network! Yep all that scp file transfer has just ended up in your scavenger queue, and that’s likely not what you were intending.

Unless you are aware of this out of the box behavior then you may be in for a shock and many man hours spent troubleshooting the issue, just as before the default can be changed to suit your network and application requirements in a QoS enabled network.

Having visibility into the network traffic helped identify an issue the client was unaware of and whilst not the culprit of the slow transfer in this case, it also identified future issues with upgrading the openssh client software.

The data in the capture was anonymized using Tracewrangler by @PacketJay

Link to ssh_config manual page