I'm having a strange issue where google can see robot.txt exists but cannot always read it.
I've tried all of the obvious like encoding, permissions etc etc,
but no joy.
I look in /var/log/nginx/access.log and I find
66.249.69.26 - - [23/Jul/2018:05:36:36 +0000] "GET /robots.txt HTTP/1.1"...