Nmap does its best to find the speed at which it can accurately find the state (open, closed, or filtered) for every port. The timing system is complex, and it has some worst-case scenarios that can lead to very slow scans. One of these cases is when the target is rate-limiting TCP connection resets (RST), which are the responses Nmap receives when a port is closed. The majority of ports on a system are usually closed, and in order to save resources or perhaps to stymie scanning attempts, the target's OS may choose only to issue a RST once per second.
When Nmap detects RST rate-limiting, it must slow its probes down to match that rate, otherwise a port that is closed may be marked "filtered" because no response was received, consistent with a firewall dropping traffic to that port. This slowdown behavior happens gradually, so the first few ports are not as much affected as later ones. Also, it only affects closed ports, so the commonly-used ports between 1 and 1000 are less likely to contribute to slowness.
There's a workaround for this problem, but it comes with a loss in accuracy. If you don't care about knowing the difference between filtered and closed ports, you can use the
--defeat-rst-ratelimit option to not let rate-limited RSTs affect Nmap's timing. Nmap will continue sending at an appropriate rate for the network, detecting dropped packets and slowing down when necessary, but being perfectly happy marking closed ports as filtered. The set of open ports should be exactly the same, which is all that most people want. In fact, you can even add
--open to avoid printing info about closed and filtered ports altogether.