As some may have noticed particularly on Sunday evenings (our peak load) some members see
an annoying message "The server is a bit busy at the moment. Please try again in 1 to 3 minutes".
Most of the time the server copes, but peak load is a problem.
We self cap the server usage and the reason we do this is to prevent more drastic uncontrolled crashes.
It is also caused by the fact that AP is growing.
We are currently hosted on a Virtual Private Server (VPS) and if the load average goes over 5 the site software temporarily blocks access.
This check is done every 60 seconds or so and takes another 60 or so seconds to clear (as the load drops off).
Note: Since we moved from shared hosting to the current VPS server AP has been much more stable.
Rick and I have been discussing the issue and are planning to increase server CPU capacity.
We have some options.
- we increase the plan we are currently on which would double our hosting costs
- we find an alternate provider with greater performance (at a similar current cost) and move hosts
- we move to a dedicated server at a much higher cost, maybe on-sell the extra capacity to other parties to defray costs
- we move to an Aussie host with better performance at higher cost but with better network response
AP is provided for free, and paid for via our great sponsors, the shop, and commissions via click through adverts.
Extra costs in hosting means less prizes etc. so we are likely to go option 2.
Please bear with us while we work through this issue over the next few weeks.
Technical background. (For those interested or hosting your own site)
Our current hosting plan uses a technology called Virtuozzo and if we move hosting it will be to a Xen (or Xen like) based platform.
- Virtuozzo - virtualization at the kernel level, allows more on the same hardware but greater impact from other users, better value for hosting providers
- Xen / VMware / HyperV - virtualisation at the hardware level, allows complete separation of the virtual machines
(this is the technology (VMware) we use at work for nearly 100 virtualized servers for nearly 2,000 desktops)
Providers we are considering include the following, the last two being Australian have some speed advantages, but are more expensive, esp. the bandwidth.
I'm open to advice and suggestions from anyone with experience in this area.
FYI Relocation plan summary if we change providers...
- Buy new service
- Tell all the members what is happening and when
- Ensure domain names set at 60 minute TTL (so new IP address gets propagated quickly)
- Setup new technical environment
- rsync copy existing site to new server and database (~10GB attachments, comp entries etc)
- Run in test mode on new server
- Make sure backups are correct on new server
- Offline old AP site
- Change DNS to new server change on old server as well
- rsync any last changes
- Copy database latest from old to new server
- Make AP live on new server
- Mailout to all member 12-24 hours after go live
- Cancel the old service