Fix crash when cleanup() was called twice in a row, e.g. load_cfg failure

Need to nullify the xpms after destroying/freeing (as is done in other
servers already).