Solved
OK, so I found the problem, I'll post it here for anyone else who needs it
It's not immediately obvious, but the meta data settings boxes in "Settings" only applies to new pages you create AFTER you change these settings
You need to open the "Pages" tab and click "Edit description...