User Tag List

Thanks useful information Thanks useful information:  18
Results 1 to 12 of 12

Thread: AP server performance - hosting - "server busy" messages

  1. #1
    It's all about the Light!
    Tech Admin
    Kym's Avatar
    Join Date
    15 Jun 2008
    Location
    Modbury, Adelaide
    Posts
    9,641
    Mentioned
    18 Post(s)
    Tagged
    0 Thread(s)

    AP server performance - hosting - "server busy" messages

    As some may have noticed particularly on Sunday evenings (our peak load) some members see
    an annoying message "The server is a bit busy at the moment. Please try again in 1 to 3 minutes".
    Most of the time the server copes, but peak load is a problem.

    We self cap the server usage and the reason we do this is to prevent more drastic uncontrolled crashes.
    It is also caused by the fact that AP is growing.

    We are currently hosted on a Virtual Private Server (VPS) and if the load average goes over 5 the site software temporarily blocks access.
    This check is done every 60 seconds or so and takes another 60 or so seconds to clear (as the load drops off).
    Note: Since we moved from shared hosting to the current VPS server AP has been much more stable.


    Rick and I have been discussing the issue and are planning to increase server CPU capacity.
    We have some options.
    1. we increase the plan we are currently on which would double our hosting costs
    2. we find an alternate provider with greater performance (at a similar current cost) and move hosts
    3. we move to a dedicated server at a much higher cost, maybe on-sell the extra capacity to other parties to defray costs
    4. we move to an Aussie host with better performance at higher cost but with better network response


    AP is provided for free, and paid for via our great sponsors, the shop, and commissions via click through adverts.
    Extra costs in hosting means less prizes etc. so we are likely to go option 2.


    Please bear with us while we work through this issue over the next few weeks.

    _____________________

    Technical background. (For those interested or hosting your own site)

    Our current hosting plan uses a technology called Virtuozzo and if we move hosting it will be to a Xen (or Xen like) based platform.
    Why? http://hostcube.com/blog/48/virtuozzo-vs-xen/
    In summary:
    • Virtuozzo - virtualization at the kernel level, allows more on the same hardware but greater impact from other users, better value for hosting providers
    • Xen / VMware / HyperV - virtualisation at the hardware level, allows complete separation of the virtual machines
      (this is the technology (VMware) we use at work for nearly 100 virtualized servers for nearly 2,000 desktops)


    Providers we are considering include the following, the last two being Australian have some speed advantages, but are more expensive, esp. the bandwidth.



    I'm open to advice and suggestions from anyone with experience in this area.

    FYI Relocation plan summary if we change providers...

    1. Buy new service
    2. Tell all the members what is happening and when
    3. Ensure domain names set at 60 minute TTL (so new IP address gets propagated quickly)
    4. Setup new technical environment
    5. rsync copy existing site to new server and database (~10GB attachments, comp entries etc)
    6. Run in test mode on new server
    7. Make sure backups are correct on new server
    8. Offline old AP site
    9. Change DNS to new server change on old server as well
    10. rsync any last changes
    11. Copy database latest from old to new server
    12. Make AP live on new server
    13. Mailout to all member 12-24 hours after go live
    14. Cancel the old service
    regards, Kym Gallery Honest & Direct Constructive Critique Appreciated! ©
    Digital & film, Bits of glass covering 10mm to 500mm, and other stuff



  2. #2
    Member
    Join Date
    25 Mar 2010
    Location
    Buderim
    Posts
    1,260
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    All fun and games to be had here!

    You obviously are aware of the myriad of advantages hardware virtualisation has over software virtualisation.

    My opinion (without being privvy to traffic profiles etc) is that moving to Australian servers is probably a needless expense. I anticipate that network bandwidth/activity and response time is currently acceptable (certainly feels fine to me) and network response wouldn't have a big impact on server loads. A saturated network link could be a symptom of high load if, for example, the network was saturated due to massive disk loads, but the networking shouldn't be the cause of high loads. I would have thought here that the loads would be php compilations and sql transactions. While it's nice to have 20-30ms response times to the server and be able to eliminate international link congestion and faults, at the end of the day is the extra network response worth the extra cost? In my experience, it has always come back as no (Australian bandwidth is too expensive!) and a US server has been the weapon of choice.

    Kym have you done optimisations on the site, like using some php caching, disabling compression (sacrifices bandwidth for the sake of freeing up cpu cycles) etc? Has the server had unnecessary processes and web server/php modules disabled? While individually they don't seem to do much, as a whole they could make a huge difference.

  3. #3
    Member
    Join Date
    25 Mar 2010
    Location
    Buderim
    Posts
    1,260
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Another thought - I don't know if any hosting companies offer it, but VMWare were working on technology to automatically provision increased resources for period of high load (eg. bring on extra memory, allow additional CPU etc). I've only just started working with VMWare again after an absence of 12 months so it could well be available now - it may be worth looking into if this is offered anywhere, especially if it only needs to ramp up for say a few hours a week - it could be more cost effective than having to be well oversubscribed resource-wise 90% of the time...

  4. #4
    It's all about the Light!
    Tech Admin
    Threadstarter
    Kym's Avatar
    Join Date
    15 Jun 2008
    Location
    Modbury, Adelaide
    Posts
    9,641
    Mentioned
    18 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by dieselpower View Post
    Kym have you done optimisations on the site, like using some php caching, disabling compression (sacrifices bandwidth for the sake of freeing up cpu cycles) etc? Has the server had unnecessary processes and web server/php modules disabled? While individually they don't seem to do much, as a whole they could make a huge difference.
    A few optimisations

    - Minimal processes, i.e. turned off all but one stats package etc
    - MySQL tuning scripts used to check config as needed based on live running
    - Nightly MySQL DB optimize (after the backup)
    - Fast CGI PHP, with acceleration (byte code cache)
    - memcached server (option in vBulletin - see stats below)
    - mod_deflate (text, not images) - and turned off PHP compression
    - mod_expires - cache static content

    cache.png

    Edit: The cache contains a stuff that is needed on every page hit,
    I implemented memcached after two weeks on the VPS, it was the most noticable performance improvement.
    FastCGI was the next most noticable.
    Last edited by Kym; 12-02-2012 at 9:47pm.

  5. #5
    Member
    Join Date
    25 Mar 2010
    Location
    Buderim
    Posts
    1,260
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    She's pretty trim then - does compression on/off make a noticeable difference in the high load periods? I have seen it make the difference before - but it didnt totally resolve the problem but delayed the need to upgrade substantially.

    As for hosts - rack space are great (use them at work) but they are not cheap...


    Sent from my iPhone using Tapatalk

  6. #6
    It's all about the Light!
    Tech Admin
    Threadstarter
    Kym's Avatar
    Join Date
    15 Jun 2008
    Location
    Modbury, Adelaide
    Posts
    9,641
    Mentioned
    18 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by dieselpower View Post
    She's pretty trim then - does compression on/off make a noticeable difference in the high load periods?
    As FastCGI has several PHP processes waiting for requests I can see the Apache httpd load separately from the PHP via top.
    httpd is only 1-3% of all load, mysql is 2-4%, PHP is by far the biggest factor, so turning of compression at peak times is near pointless.
    In fact turning off GZIP compression in PHP and leaving it to Apache helped a lot.

    I appreciate the comments as it helps me review what we have done.

    I have a feeling 99% of the members think we talking another language

    Rackspace are good but would at least double the hosting cost, as they charge bandwidth extra.

  7. #7
    Member rene52's Avatar
    Join Date
    18 Nov 2009
    Location
    Benalla
    Posts
    1,008
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Kym,

    Correct that 99% will not be understanding what you guys have been saying lol. I am definately no experet in MySQL or PHP and am not sure what affects the changes you have done on the accesses. Also, whether the virtual Server is here in Aus or overseas is not necessarily the issue here as I agree would have minimal affect if any on how we access the systems.

    I do agree with the process that you have outlined above as the way to go (once you work out what and wher this is going for an upgrade).

    I haven't been on here too long and enjoy the photos and the people on here and would hate to be hit with messages that I need to try again later (I might be one of the few that only do any updates or uploads on weekdays rather than Sunday. ALso I am not so sure I would like to see this being a pay-per-use site either as it may discourage some people from using it.

    I wish you well in your fact finding and your porcesses.

  8. #8
    Member
    Join Date
    25 Mar 2010
    Location
    Buderim
    Posts
    1,260
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by Kym View Post
    I have a feeling 99% of the members think we talking another language
    On the other hand if 99% of members started talking about their work specifics we'd likely be sitting back scratching our heads too!

    Have you thought of having multiple front-end servers talking back to the one database (a quick google indicates this is possible on vBulletin) and using a load balancer of sorts to spread the load, even if it's just DNS round-robin? The trade-off though is that you need to ensure all data not held in the database is replicated, again a potential management overhead but it could be cost effective.

    I found this 'cloud' load balancing host, I'm sure there's more out there. Price seems very good but it's UK based:
    http://www.tsohost.co.uk/clustered-hosting.php

    It looks like more capacity of some description is the answer - it's a tough one on a shoestring budget, and a heck of a lot of work to move it all over.

    I can't think of anything else that you haven't already tried to increase performance locally, aside from code optimisation but being the code guru you are, you would have done that long ago (if you even needed to!).

    Wouldn't it be nice if there was a data center manager/operator/owner who is in to photography that would be willing to sponsor the site by hosting it at a significantly discounted rate (or for free!).

    Shout if I can be of any assistance with research/relocation/testing/etc - I'd be happy to help.

  9. #9
    Fishy bricat's Avatar
    Join Date
    06 Apr 2010
    Location
    Melbourne
    Posts
    777
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Clear as mud to me!!!!!
    Cheers Brian.

    Canon 7D Kit lenses EFS 18-55 IS EFS 55-250 IS EF28-90 Canon EF 2xll Extender Sigma DG150-500 OS Speedlight 420EX. 580EX

  10. #10
    It's all about the Light!
    Tech Admin
    Threadstarter
    Kym's Avatar
    Join Date
    15 Jun 2008
    Location
    Modbury, Adelaide
    Posts
    9,641
    Mentioned
    18 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by dieselpower View Post
    On the other hand if 99% of members started talking about their work specifics we'd likely be sitting back scratching our heads too!


    Have you thought of having multiple front-end servers talking back to the one database (a quick google indicates this is possible on vBulletin) and using a load balancer of sorts to spread the load, even if it's just DNS round-robin? The trade-off though is that you need to ensure all data not held in the database is replicated, again a potential management overhead but it could be cost effective.
    Yes, but cost is the issue.

    I found this 'cloud' load balancing host, I'm sure there's more out there. Price seems very good but it's UK based:
    http://www.tsohost.co.uk/clustered-hosting.php
    Thanks, I'll have a look. But for now 4xCPU from Arvixe for similar $$ looks good.

    It looks like more capacity of some description is the answer - it's a tough one on a shoestring budget, and a heck of a lot of work to move it all over.
    That's what RIck and I think. The move is quite easy having done it 3 times now, it just requires a co-ordinated plan.

    I can't think of anything else that you haven't already tried to increase performance locally, aside from code optimisation but being the code guru you are, you would have done that long ago (if you even needed to!).
    I've sent some code improvements back to vBulletin and they have implemented my suggestions into the core product.
    (in one case 200 queries for a specific page was dropped to only 28 by use of cache)
    I've also done some optimisations to AP specific bits, eg. the Competition (X/Y) open/voting count in the menu is only updated after one hour per user,
    but forced if someone votes or enters a comp.

    Wouldn't it be nice if there was a data center manager/operator/owner who is in to photography that would be willing to sponsor the site by hosting it at a significantly discounted rate (or for free!).


    Shout if I can be of any assistance with research/relocation/testing/etc - I'd be happy to help.
    Your feedback has been quite helpful.
    Last edited by Kym; 14-02-2012 at 9:29am.

  11. #11
    Member
    Join Date
    04 Jan 2009
    Location
    Brisbane
    Posts
    40
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi,
    Just posting so that I canaccess the site.
    Regards.

  12. #12
    Administrator ricktas's Avatar
    Join Date
    24 Jun 2007
    Location
    Hobart
    Posts
    15,641
    Mentioned
    10 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by gordon_l34 View Post
    Hi,
    Just posting so that I canaccess the site.
    Regards.
    Read the email you got when you went inactive. Site Matters posts will not re-activate your account,

    Have a look at this: http://www.ausphotography.net.au/for...nactive_status
    "It is one thing to make a picture of what a person looks like, it is another thing to make a portrait of who they are" - Paul Caponigro

    Constructive Critique of my photographs is always appreciated
    Nikon, etc!

    RICK
    My Photography

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •