Web Development

Hi all,

My company run a CMS (typo3) on LAMP on RH linux.
Basically all users have same typo3 page as home page and thousands of users starts their browser every morning causing server very slow or even to hang (swap full because of too many apache processes)..

99% of the traffic is this home page which take 2s in a non busy time to be delivered.
I improve this time to 1.5s using php cache. Others mysql/apache tunning have so far not given me any significant improvement (I have been trhu web pages on apache/mysql tunning and test differents things).
As far I have seen mysql is ok and httpd processes are taking all cpu/mem.

My question is :
1 - As this home page (php page, 30K) in 99% of the traffic how can i improve the process not getting 1000 times the same request to my apache server.
I have no way to modify this home page policy, I can just work on the infrastructure side. I was thinking on some proxy/cache server but have no experience on that. Any efficient and open source solution ?

2 - Apart from php cache any hint to help me improve apache performance ?
I did not find anything else number of thread and related stuff in apache.

It is bad that you can't modify optimize the home page, 30k php page sounds like a lot, combined with your users' concurrency (how much is it ? average, peaks ? ). Apache can be tuned in many ways, depending on the usage. Ironically, this is what I'm working on currently, so I will have to ask about the version of apache you're using. Depending on the answer I should be able to provide more info. Also, let us know if you are using SSL somewhere to serve content, and, if it's not secret, what are the hardware parameters of the server. For better PHP handling, I can suggest FastCGI - I have seen very good results with it in the past.

1a. APC cache was the quickest way to get the most improvement in speed on my server. Memcached is also an option, but only has benefits above APC when you can distribute the caching load to backend servers or if you have multiple front-end servers.

1b. Squid. See Squid (software) - Wikipedia, the free encyclopedia. You put your LAMP server on another port, and proxy it using SQUID. Squid will cache as much as can be cached and fetch from your LAMP server. If most of your content is dynamic, this may not help much.

2a. You absolutely must minimize swap in such an environment. If you start going to swap, your system loses.

In sysctl.conf, change:
vm.swappiness = 1
and "tune" your swap to use a swap file that's about 80M. If you need more swap then that, your system will crash. Better to let Apache think it cannot get more memory and return 503 (server busy) errors than to wait forever.

2b. You can play around with the following:
2c. More tuning at the TCP level. See sysctl parameters such as:Also, LOWER the amount of shared memory, to ensure you have more memory for HTTP. (Note: This may not work if you are running MySql or postgres on the same server!)
2d. If you have the memory, put all your HTTP/PHP files onto a ramdisk. Otherwise, install everything on a FAST internal (or SAN) disk. When I say FAST, I mean hardware-RAID1 15k RPM SCSI Fiberchannel. Store all log files on a SEPARATE disk, which can be any junk drive.

Thanks for you answers.

My apache version is : Apache/2.0.52 with 4x2.8 mhz xeon with 3gb RAM and 2gb swap.
On dev I got VM machine with 2x1.86Mhz xeon and 2.5gb RAM and 2Gb swap (I do all tests there).

99% of the traffic is when users starts their browser around 9h.
A lot of connections (may be 1000 users).

I do my tests on dev with curl and ab but did not experience any swap pb (i did not start more than 100 thread each time).

I use eaccelerator for cache because of my boss choice, I tried mmcache and got slightly better results.

1a : I will try APC.

1b : as typo3 is php script and home page is index.php I wonder how squid will help. I thought that squid is not easy to setup, is it worth to try ??


2a : I agree with you better having 503 than swapping.
I understand vm.swappiness, but it's not clear on what you say about tunning swap, is it trying different values on swappiness ? It looks tricky :-)

2b : Some parameters are not clear for me especially ServerLimit/maxclient. I am a little bit reluctant to change that as I am not sure of what I am doing. I'l take deeper look in documentation.
keepalive is on and I will try to play with the value.
But most of my connections are 1 time connection, user start browser ask home page and that's all.
For 1 connection like that I was gessing that I do not need a high keepalive as I got only 1 http request, I am right ?
Is there any relation between tcp keepalive and apache keepalive ?

Finally as I tried to emphasize, I got 99% request on the same page which is dynamic page BUT do not change 99% of the time.
Can I cache that kind of things in any proxy or cache to avoid going thru all the layers (browser httpd mysql httpd ...).

And as otheus said I rather like to have 503 but quick response than having server crashing.
Maybe the swap tuning is my best help in this case ???

My disk are internal HD that I can't change.

See below for how to reduce the amount of swap, which I recommend you do. I'll explain more details below.

Set those pre-fork settings VERY high. I'll explain more details below.

Nevermind. Benchmarks show eAccelerator is faster. Keep it.

I'm not really sure. If the ENTIRE home page is from PHP, then it won't. However, if there's a lot of media, static JSS, static CSS on the page, it might help. Then again, if it's just one page, and there's a fixed number of media files, it won't -- HTTP and the OS should cache most of that.

After you move /var/log to its own disk, find the disk least used in your system. Say it's already mounted as /vol3.

Code:

$ dd if=/dev/zero of=/vol3/swapfile bs=4k count=25000   ## TWEAK count
$ swapon /vol3/swapfile
$ swapoff /swap

That's it. Now go into /etc/fstab and change the /swap mountpoint. Finally, make sure rc.sysinit does the right thing. Do:

Code:

grep -n -E "swapon|mount .*-a"

If mount comes before swapon, you're fine.

Nothing to fear here.

This is straight from Apache documentation. In your case, you want to set the maximum servers to something like 1024 (!) and the MinSpareServers to something like 128 or maybe even 512.

You got it! You might not need at all, but I am guessing your home page also consists of graphics, JS source files, CSS files, etc., all coming from your server. So its better to keep it on, but to timeout after only a few seconds.

Only if you have lots of connections all over the internet. If everyone is coming from a LAN or something, you don't need to touch it.

If you know the parameters that change it, then sure! Just pre-generate the page and serve that page. If the condition is determined on IP or COOKIE or some header info, you can use Mod_rewrite to give redirect the user to the dynamic one.

What's MySQL doing in the picture? Is the DB read-only? If so, can you fit it on a RAM-disk?

Run this command before and after each set of changes you make, assuming you make one change per day. Post the early and improved results here.

Code:

 
sar -b -r -s 08:00:00 -e 09:30:00

What is the your current max clients settings in both Apache and MySQL?

What are your keep alive and connection timeout settings in MySQL?

Many folks set these incorrectly and it causes problems.

Also, did you run FireBug (FireFox plugin) and look at the performance numbers?

Did you run mysqltuner.pl and tuning-primer.sh ? If so, please post the results.....

These numbers are too high and will cause performance problems.

FYI, our site is quite busy (do you have more traffic than this site?) and our numbers are much lower!

Code:

Timeout 20

ServerLimit 250
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 1000

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 2

...

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers         30
    MinSpareServers      25
    MaxSpareServers      100
    MaxClients           250
    MaxRequestsPerChild  4000
</IfModule>