@orenk did you find what was causing it? I'm experiencing the exact same problem
WP sites will be randomly unavailable for just a few minutes near 4 am every other day. It's a bare metal server on OVH, just mention it in case it's a vendor specific configuration.
I am using pingdom for external...