Fix scrolling when there's no scrollbar.

The code was moving unscrolled lines back after scrolling the
majority of them... if all lines were scrolled, moving some back
actually broke scrolling.