Apache / Tomcat / APR Issues [RHEL]


 
Thread Tools Search this Thread
Operating Systems Linux Red Hat Apache / Tomcat / APR Issues [RHEL]
# 1  
Old 11-29-2010
Apache / Tomcat / APR Issues [RHEL]

Evening,

I'm posting for help here, because I'll be honest I've reached the end of my tether, hopefully someone can give me some assistance and help me maintain a level of sanity...

I maintain a number of webservers on RHEL 5 64Bit (Red Hat Enterprise Linux Server release 5.3 (Tikanga)), the smallest with 8GB of RAM, the largest with 12GB. As I serve these behind a load balancer I thought it was best to not pack them with RAM.

In short my problem is that I consume MaxClients within 2 minutes. At present my MaxClients setting within prefork.c is at 600. HTTPD has taken this setting and will run to the level.

My solution is as follows:

LoadBalancer: The LB will take connections on port 80 and 443 for our websites using VIP's (Virtual IP) and then pass to an internal web server based on connection count in the load balancer. SSL termination is completed on the load balancer.

Web Servers:
Our front facing application is HTTPD (httpd-2.2.16 - RPM) which takes all connections and then proxies traffic onto Tomcat 6.0.18 (tar.gz from Website) using AJP13 mounts (same server). Our website is built using Java hence the need for Tomcat. I'm using JRE 1.6.17 with a symlink to the java binary in /usr/bin/java so any process can call Java.
Catalina has the JRE_HOME option defined.
I have attempted to install APR using the following commands, as my predecessors' tell me that unless we're running APR what I'm seeing with happen.

Our Database is MSSQL, using the JTDS.JDBC connection pool.

I have installed APR using the following (instructions passed down from previous, and working on other servers)
Code:
wget http://archive.apache.org/dist/apr/apr-1.3.3.tar.gz
wget http://archive.apache.org/dist/apr/apr-util-1.3.4.tar.gz

tar -zxvf apr-1.3.3.tar.gz
cd apr-1.3.3
./configure --prefix=/opt
make
make install

tar -zxvf apr-util-1.3.4.tar.gz
cd apr-util-1.3.4
./configure --prefix=/opt --with-apr=/opt/bin
make
make install

However, if I run rpm -qa | grep apr I get the following results.

Code:
rpm -qa | grep apr
apr-1.2.7-11.el5_3.1
apr-devel-1.2.7-11.el5_3.1
apr-util-ldap-1.3.9-1.jason.2
apr-util-devel-1.3.9-1.jason.2
apr-util-1.3.9-1.jason.2

Note the -Jason, signifies the files came from the Jason Litka Repo found at http://www.jasonlitka.com/yum-repository this is because one of our sites required mod_subsititute which is not installed as part of 2.2.3 (stock on RHEL5)

The command apr-1-config --version, tells me im running 1.2.7.

I've also ensured that libtcnative is installed using

Code:
cd /opt/apache-tomcat-6.0.18/bin/
tar -zxvf tomcat-native.tar.gz
cd /opt/apache-tomcat-6.0.18/bin/tomcat-native-1.1.14-src/jni/native
./configure --with-apr=/opt/bin --with-java-home=/opt/jdk1.6.0_17
make
make install
cd /usr/lib
ln -s /usr/local/apr/lib/libtcnative-1.so libtcnative-1.so
ln -s /usr/local/apr/lib/libtcnative-1.so.0 libtcnative-1.so.0

Inside $CATALINA_HOME/logs/catalina.out I can see that APR is loaded

Code:
INFO: Loaded APR based Apache Tomcat Native library 1.1.14.
Nov 29, 2010 9:46:14 PM org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].

However, I also see the following, above this statement.
Code:
Nov 29, 2010 9:46:14 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
Nov 29, 2010 9:46:14 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'minSpareThreads' to '25' did not find a matching property.
Nov 29, 2010 9:46:14 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'maxSpareThreads' to '75' did not find a matching property.
Nov 29, 2010 9:46:14 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
Nov 29, 2010 9:46:14 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'minSpareThreads' to '25' did not find a matching property.
Nov 29, 2010 9:46:14 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'maxSpareThreads' to '75' did not find a matching property.

All files within $CATALINA_HOME are owned by the user tomcat6. FYI: $CATALINA_HOME is /opt/apache-tomcat-6.0.18

My tomcat setenv.sh file is as follows (changes depending on amount of ram):

Code:
export CATALINA_OPTS="-Xmx8500M -XX:PermSize=512m -Dcom.sun.management.jmxremote.port=8999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
export JAVA_OPTS="-Dhut.keyfiles.location=/lib/app/keyfiles"
export APP_CONF="/lib/elysium/conf"
export JRE_HOME=/opt/jre1.6.0_17

I cannot see errors as to why we're hitting MaxClients within any *-error_log file within httpd log directory. Catalina.out does show exception errors, such as NullPointerException, which my development team tell me is normal (our code has bugs).

I have a script which calls pstree -G | grep httpd and returns the value and inserts into a log file, every 2 minutes. When this script goes above a threshold, which I believe will not recover from, it issues an httpd stop/start and tomcat6 stop/start

Excerpt from file
Code:
2010-11-29 21:12:01 GMT - 0
2010-11-29 21:14:04 GMT - 263
2010-11-29 21:16:04 GMT - 458
2010-11-29 21:18:01 GMT - 199
2010-11-29 21:20:01 GMT - 79
2010-11-29 21:22:01 GMT - 588
2010-11-29 21:24:01 GMT - 331
2010-11-29 21:26:01 GMT - 528
2010-11-29 21:28:01 GMT - 206
2010-11-29 21:30:01 GMT - 523
2010-11-29 21:32:01 GMT - 391
2010-11-29 21:34:01 GMT - 551
2010-11-29 21:36:01 GMT - 468
2010-11-29 21:38:01 GMT - 551
2010-11-29 21:40:01 GMT - 570
2010-11-29 21:42:01 GMT - 262
2010-11-29 21:44:01 GMT - 377
2010-11-29 21:46:01 GMT - 600
2010-11-29 21:48:02 GMT - 467
2010-11-29 21:50:01 GMT - 451
2010-11-29 21:52:01 GMT - 331
2010-11-29 21:54:01 GMT - 218
2010-11-29 21:56:02 GMT - 144
2010-11-29 21:58:03 GMT - 209
2010-11-29 22:00:01 GMT - 146
2010-11-29 22:02:01 GMT - 285
2010-11-29 22:04:01 GMT - 197
2010-11-29 22:06:01 GMT - 118
2010-11-29 22:08:01 GMT - 120
2010-11-29 22:10:01 GMT - 182
2010-11-29 22:12:01 GMT - 159

My restart threshold is 450. If httpd is running 450 processes, the services will restart

Additional:

Code:
ps -ylC httpd --sort:rss
S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
Z    48 15165 12513  0  75   0     0     0 exit   ?        00:00:00 httpd <defunct>
S    48 15169 12513  0  75   0 16684 48743 429493 ?        00:00:00 httpd
S    48 15166 12513  1  75   0 16688 48826 semtim ?        00:00:00 httpd

Code:
free -m
             total       used       free     shared    buffers     cached
Mem:         12004      11919         85          0        239       4968
-/+ buffers/cache:       6710       5294
Swap:         4094         14       4080

At time of writing this message, the stats are as follows

Web Server: 129 running processes
Load Balancer: 290 active connections (260 HTTP / 30 SSL)

Can anyone assist as to why we're reaching MaxClients. Is APR not setup properly? I'm literally at my wits end. Other servers within the solution are working fine, including connectivity to the database. Which I have confirmed is working from the servers having this issue. Switches, cables and network cards all changed. The server recently having this issue, is brand new.

Thank you for your time, and I welcome any questions to help solve this.

Kai
# 2  
Old 01-15-2011
cpickering,

as far as i know you cant query in RPM a SRC build. Pleas see your HTTP configs and analyze from there the minserver / maxserver.
# 3  
Old 01-20-2011
Agree with csorhand about the RPM query.
You can try to remove the apr 1.2.7 and reinstall apr 1.3.3.
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Building an apache-tomcat server

Hi, I have been assigned a task to build a server with these requirements : > has multiple tomcats running under the same apache. > tomcat & jdk has to be binded with apache. > latest openssl with all necesarry "mod" to bind with apache & tomcat. I am fairly new to... (1 Reply)
Discussion started by: anaigini45
1 Replies

2. Linux

Apache httpd configuration - Issues with APR

Hi I have tried setting up of Apache http server - httpd-2.4.25. During configuration, I understand it needs APR to be setup. Hence I have downloaded APR & APR-Util. Performed, tar xvfC apr-1.5.2.tar /root/httpd-2.4.25/srclib/apr What is happening is there is another directory... (1 Reply)
Discussion started by: videsh77
1 Replies

3. UNIX for Dummies Questions & Answers

Help needed to install Apache Tomcat

I want to setup apache tomcat on my linux distribution . I Have downloaded apache tomcat from below mentioned link. Administration Web Application: Apache Tomcat - Apache Tomcat 5 Downloads The setup was in following format. apache-tomcat-5.5.29-admin.tar.gz Below are the command i... (7 Replies)
Discussion started by: pinga123
7 Replies

4. Web Development

Apache to Tomcat forwarding not working properly

Hi, I am facing some problem with Apache to Tomcat forwarding. I have Tomcat (6.0.18) behind Apache (2.2). I want to use ‘https' for receiving request from the client. This request, when forwarded by Apache to Tomcat, will use ‘http'. My application's name is ‘abc', which is deployed in... (1 Reply)
Discussion started by: Technext
1 Replies

5. UNIX for Advanced & Expert Users

Error Tomcat + Apache

Hi, I have a machine with Unixware 7.1.3 and Apache 2 and Tomcat 4.0.3 it's works, but in the archive log from mod_jk.log ajp13_process_callback - write failed : Error ajp13_process_callback - write failed : Error ajp13_process_callback - write failed : Error... (1 Reply)
Discussion started by: By_Jam
1 Replies

6. UNIX for Dummies Questions & Answers

difference between Tomcat apache and weblogic

Hi, can anyone explain the difference in terms of application server, web server concepts? Thanks james (0 Replies)
Discussion started by: james94539
0 Replies

7. Programming

starting with APR (Apache Portable Runtime)

Hi I am new to APR and I started programming using APR 1.2.2. So please guide me how to start learning it? (0 Replies)
Discussion started by: sumsin
0 Replies

8. SCO

HELP SCO 5.07 and Installing Apache Tomcat 4.01

I need to install Apache TOmcat 4.01 on SCO Openserver 5.07 rather than using the default 4.1.29 version which comes with the original release of 5.07 anyone got any ideas hw to go about this, really struggling having downloaded the source and binariys and getting nowhere fast (0 Replies)
Discussion started by: ccarcher
0 Replies
Login or Register to Ask a Question