Sponsored Content
Operating Systems Linux Red Hat How to troubleshoot a 1000 nodes Apache cluster? Post 302750895 by thmnetwork on Wednesday 2nd of January 2013 06:36:54 PM
Old 01-02-2013
The load balancer is going to be a choking point (bandwidth and CPU utilization-wise). If it's bandwidth you may consider increasing the number of links. You can try to increase links through interface bonding. If it's CPU utilization, you might consider multiple load balances (kind of like google.com does) so you can spread load amongst several physical machines. I'd recommend doing that latter anyways, for availability reasons.

Before anyone can give you specific advice though, you need to localize the performance issue. For example, if you run a jMeter test against the serving nodes directly is that faster? If it is, then the load balancer is the choke point. If not then you may check to see if the network links are being saturated with some sort of command-line bandwidth checking tool (I use bwm-ng). If that's not the issue then move onto CPU, memory, etc. Once you've done that you should be in a better position to do something about the performance problems.

It's also possible that it's an application-level latency. For example if a particular website is served by a particular DB cluster and that cluster is going slow it may slow down the end user's experience.
 

10 More Discussions You Might Find Interesting

1. High Performance Computing

Bonding, IP alias, Virtual IP, 2 nodes cluster

Hi ! I have a simple setup of 2 PC (with linux Red-Hat) where the first PC is the primary machine and the second the backup. I use DRBD for data replication and Red-Hat cluster suite for HA (High Availability). I have tested both. Now I NEED a COMMON IP ADDRESS (or Master/unique IP address) for... (0 Replies)
Discussion started by: Danny Gilbert
0 Replies

2. UNIX for Dummies Questions & Answers

IP Alias, Bonding or Virtual IP, 2 nodes Cluster, which one to use ?

Hi ! I have a simple setup of 2 PC (with linux Red-Hat) where the first PC is the primary machine and the second the backup. I use DRBD for data replication and Red-Hat cluster suite for HA (High Availability). I have tested both. Now I NEED a COMMON IP ADDRESS (or Master/unique IP address) for... (3 Replies)
Discussion started by: Danny Gilbert
3 Replies

3. Emergency UNIX and Linux Support

Rebooting 3 to 1 Cluster nodes.

hello Gurus, My current set up is 3 to 1 Cluster (SUN Cluster 3.2) running oracle database. Task is to reboot the servers. My query is about the procedure to do the same. My understanding is suspend the databases to avoid switchover. Then execute the command scshutdown to down the cluster... (4 Replies)
Discussion started by: EmbedUX
4 Replies

4. Red Hat

Centos/rhel 5 cluster 3 nodes with out Quorum

Hi all, i have 3 nodes cluster (Centos 5 cluster suit) with out quorum disk, node vote = 1, the value of a quorum = 2, when 2 nodes going offline, cluster services are destoys. How i can save the cluster and all services(move all services to one alive node) with out quorum disk when other... (3 Replies)
Discussion started by: Flomaster
3 Replies

5. Solaris

Need advise on setting up solaris 10 2 nodes cluster

I am new to setting up sun solaris 10 cluster, I have 2 sun sparc t3-1 servers (identical), going to use them as web servers (sun one java web server 7), looking for data replication and real time fail over. My question is do I need external storage to configure the cluster? or I can just use... (3 Replies)
Discussion started by: spitfire2011
3 Replies

6. Solaris

What is the procedure to reboot cluster nodes

Hi we have 2 solaris 10 servers in veritas cluster. also we have oracle cluster on the database end. now we have a requirement to reboot both the servers as it has been running for more than a year. can any one tell what is the procedure to bring down the cluster services in both the nodes... (7 Replies)
Discussion started by: newtoaixos
7 Replies

7. AIX

Re-cluster 2 HACMP 5.2 nodes

Hi, A customer I'm supporting once upon a time broke their 2 cluster node database servers so they could use the 2nd standby node for something else. Now sometime later they want to bring the 2nd node back into the cluster for resilance. Problem is there are now 3 VG's that have been set-up... (1 Reply)
Discussion started by: elcounto
1 Replies

8. UNIX for Advanced & Expert Users

Arbitrator for 2 nodes ocfs cluster

Is there any way to create a arbitrary node for ocfs2 on a virtual machine (others are physical servers) so it won't go panic when one of physical server goes down? This is for load balanced application servers. Any setting example or tips? Thanks. (0 Replies)
Discussion started by: malayo
0 Replies

9. Red Hat

RedHat Cluster: Nodes won't see each other

Hi All; I try to build a Redhat Cluster (CentOS 6) on vmware. But each node sees the other down like: # clustat Cluster Status for mycluster @ Wed Apr 8 11:01:38 2015 Member Status: Quorate Member Name ID Status ------ ---- ... (0 Replies)
Discussion started by: Meacham12
0 Replies

10. Red Hat

RedHat Cluster: Nodes won't see each other

Hi All; I try to build a Redhat Cluster (CentOS 6) on vmware. But each node sees the other down like: # clustat Cluster Status for mycluster @ Wed Apr 8 11:01:38 2015 Member Status: Quorate Member Name ID Status ------ ---- ... (1 Reply)
Discussion started by: Meacham12
1 Replies
cpu_sup(3erl)						     Erlang Module Definition						     cpu_sup(3erl)

NAME
cpu_sup - A CPU Load and CPU Utilization Supervisor Process DESCRIPTION
cpu_sup is a process which supervises the CPU load and CPU utilization. It is part of the OS_Mon application, see os_mon(7) . Available for Unix, although CPU utilization values ( util/0,1 ) are only available for Solaris and Linux. The load values are proportional to how long time a runnable Unix process has to spend in the run queue before it is scheduled. Accord- ingly, higher values mean more system load. The returned value divided by 256 produces the figure displayed by rup and top . What is dis- played as 2.00 in rup , is displayed as load up to the second mark in xload . For example, rup displays a load of 128 as 0.50, and 512 as 2.00. If the user wants to view load values as percentage of machine capacity, then this way of measuring presents a problem, because the load values are not restricted to a fixed interval. In this case, the following simple mathematical transformation can produce the load value as a percentage: PercentLoad = 100 * (1 - D/(D + Load)) D determines which load value should be associated with which percentage. Choosing D = 50 means that 128 is 60% load, 256 is 80%, 512 is 90%, and so on. Another way of measuring system load is to divide the number of busy CPU cycles by the total number of CPU cycles. This produces values in the 0-100 range immediately. However, this method hides the fact that a machine can be more or less saturated. CPU utilization is therefore a better name than system load for this measure. A server which receives just enough requests to never become idle will score a CPU utilization of 100%. If the server receives 50% more requests, it will still scores 100%. When the system load is calculated with the percentage formula shown previously, the load will increase from 80% to 87%. The avg1/0 , avg5/0 , and avg15/0 functions can be used for retrieving system load values, and the util/0 and util/1 functions can be used for retrieving CPU utilization values. When run on Linux, cpu_sup assumes that the /proc file system is present and accessible by cpu_sup . If it is not, cpu_sup will terminate. EXPORTS
nprocs() -> UnixProcesses | {error, Reason} Types UnixProcesses = int() Reason = term() Returns the number of UNIX processes running on this machine. This is a crude way of measuring the system load, but it may be of interest in some cases. Returns 0 if cpu_sup is not available. avg1() -> SystemLoad | {error, Reason} Types SystemLoad = int() Reason = term() Returns the average system load in the last minute, as described above. 0 represents no load, 256 represents the load reported as 1.00 by rup . Returns 0 if cpu_sup is not available. avg5() -> SystemLoad | {error, Reason} Types SystemLoad = int() Reason = term() Returns the average system load in the last five minutes, as described above. 0 represents no load, 256 represents the load reported as 1.00 by rup . Returns 0 if cpu_sup is not available. avg15() -> SystemLoad | {error, Reason} Types SystemLoad = int() Reason = term() Returns the average system load in the last 15 minutes, as described above. 0 represents no load, 256 represents the load reported as 1.00 by rup . Returns 0 if cpu_sup is not available. util() -> CpuUtil | {error, Reason} Types CpuUtil = float() Reason = term() Returns CPU utilization since the last call to util/0 or util/1 by the calling process. Note: The returned value of the first call to util/0 or util/1 by a process will on most systems be the CPU utilization since system boot, but this is not guaranteed and the value should therefore be regarded as garbage. This also applies to the first call after a restart of cpu_sup . The CPU utilization is defined as the sum of the percentage shares of the CPU cycles spent in all busy processor states (see util/1 below) in average on all CPUs. Returns 0 if cpu_sup is not available. util(Opts) -> UtilSpec | {error, Reason} Types Opts = [detailed | per_cpu] UtilSpec = UtilDesc | [UtilDesc] UtilDesc = {Cpus, Busy, NonBusy, Misc} Cpus = all | int() | [int()]() Busy = NonBusy = {State, Share} | Share State = user | nice_user | kernel | wait | idle | atom() Share = float() Misc = [] Reason = term() Returns CPU utilization since the last call to util/0 or util/1 by the calling process, in more detail than util/0 . Note: The returned value of the first call to util/0 or util/1 by a process will on most systems be the CPU utilization since system boot, but this is not guaranteed and the value should therefore be regarded as garbage. This also applies to the first call after a restart of cpu_sup . Currently recognized options: detailed : The returned UtilDesc (s) will be even more detailed. per_cpu : Each CPU will be specified separately (assuming this information can be retrieved from the operating system), that is, a list with one UtilDesc per CPU will be returned. Description of UtilDesc = {Cpus, Busy, NonBusy, Misc} : Cpus : If the detailed and/or per_cpu option is given, this is the CPU number, or a list of the CPU numbers. If not, this is the atom all which implies that the UtilDesc contains information about all CPUs. Busy : If the detailed option is given, this is a list of {State, Share} tuples, where each tuple contains information about a processor state that has been identified as a busy processor state (see below). The atom State is the name of the state, and the float Share represents the percentage share of the CPU cycles spent in this state since the last call to util/0 or util/1 . If not, this is the sum of the percentage shares of the CPU cycles spent in all states identified as busy. If the per_cpu is not given, the value(s) presented are the average of all CPUs. NonBusy : Similar to Busy , but for processor states that have been identified as non-busy (see below). Misc : Currently unused; reserved for future use. Currently these processor states are identified as busy: user : Executing code in user mode. nice_user : Executing code in low priority (nice) user mode. This state is currently only identified on Linux. kernel : Executing code in kernel mode. Currently these processor states are identified as non-busy: wait : Waiting. This state is currently only identified on Solaris. idle : Idle. Note: Identified processor states may be different on different operating systems and may change between different versions of cpu_sup on the same operating system. The sum of the percentage shares of the CPU cycles spent in all busy and all non-busy processor states will always add up to 100%, though. Returns {all,0,0,[]} if cpu_sup is not available. SEE ALSO
os_mon(3erl) Ericsson AB os_mon 2.2.5 cpu_sup(3erl)
All times are GMT -4. The time now is 01:53 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy