Save moving the "new" row

Previously, we actually moved one more line back than we had to.
This only moves lines back that need to remain where they were.

This shaves another couple seconds off the benchmark.