I hope that no one will mark me as spam...
I would suggest you to use a cache plugin for WordPress or go with CDN.
May be you are facing such problem because of server location.
Let's say, Hostgator's server location is in the USA and your website is Australian then you will face loading time...