Since that's the most used/assumed byte order for IP addresses, let's make
that the default and eliminate a lot of unnecessary byte-swapping.

This fixes the issue reported by Keyop in IRC, introduced in commit 29b93c32