Sponsored Content
Operating Systems Linux Red Hat How to troubleshoot a 1000 nodes Apache cluster? Post 302750895 by thmnetwork on Wednesday 2nd of January 2013 06:36:54 PM
Old 01-02-2013
The load balancer is going to be a choking point (bandwidth and CPU utilization-wise). If it's bandwidth you may consider increasing the number of links. You can try to increase links through interface bonding. If it's CPU utilization, you might consider multiple load balances (kind of like google.com does) so you can spread load amongst several physical machines. I'd recommend doing that latter anyways, for availability reasons.

Before anyone can give you specific advice though, you need to localize the performance issue. For example, if you run a jMeter test against the serving nodes directly is that faster? If it is, then the load balancer is the choke point. If not then you may check to see if the network links are being saturated with some sort of command-line bandwidth checking tool (I use bwm-ng). If that's not the issue then move onto CPU, memory, etc. Once you've done that you should be in a better position to do something about the performance problems.

It's also possible that it's an application-level latency. For example if a particular website is served by a particular DB cluster and that cluster is going slow it may slow down the end user's experience.
 

10 More Discussions You Might Find Interesting

1. High Performance Computing

Bonding, IP alias, Virtual IP, 2 nodes cluster

Hi ! I have a simple setup of 2 PC (with linux Red-Hat) where the first PC is the primary machine and the second the backup. I use DRBD for data replication and Red-Hat cluster suite for HA (High Availability). I have tested both. Now I NEED a COMMON IP ADDRESS (or Master/unique IP address) for... (0 Replies)
Discussion started by: Danny Gilbert
0 Replies

2. UNIX for Dummies Questions & Answers

IP Alias, Bonding or Virtual IP, 2 nodes Cluster, which one to use ?

Hi ! I have a simple setup of 2 PC (with linux Red-Hat) where the first PC is the primary machine and the second the backup. I use DRBD for data replication and Red-Hat cluster suite for HA (High Availability). I have tested both. Now I NEED a COMMON IP ADDRESS (or Master/unique IP address) for... (3 Replies)
Discussion started by: Danny Gilbert
3 Replies

3. Emergency UNIX and Linux Support

Rebooting 3 to 1 Cluster nodes.

hello Gurus, My current set up is 3 to 1 Cluster (SUN Cluster 3.2) running oracle database. Task is to reboot the servers. My query is about the procedure to do the same. My understanding is suspend the databases to avoid switchover. Then execute the command scshutdown to down the cluster... (4 Replies)
Discussion started by: EmbedUX
4 Replies

4. Red Hat

Centos/rhel 5 cluster 3 nodes with out Quorum

Hi all, i have 3 nodes cluster (Centos 5 cluster suit) with out quorum disk, node vote = 1, the value of a quorum = 2, when 2 nodes going offline, cluster services are destoys. How i can save the cluster and all services(move all services to one alive node) with out quorum disk when other... (3 Replies)
Discussion started by: Flomaster
3 Replies

5. Solaris

Need advise on setting up solaris 10 2 nodes cluster

I am new to setting up sun solaris 10 cluster, I have 2 sun sparc t3-1 servers (identical), going to use them as web servers (sun one java web server 7), looking for data replication and real time fail over. My question is do I need external storage to configure the cluster? or I can just use... (3 Replies)
Discussion started by: spitfire2011
3 Replies

6. Solaris

What is the procedure to reboot cluster nodes

Hi we have 2 solaris 10 servers in veritas cluster. also we have oracle cluster on the database end. now we have a requirement to reboot both the servers as it has been running for more than a year. can any one tell what is the procedure to bring down the cluster services in both the nodes... (7 Replies)
Discussion started by: newtoaixos
7 Replies

7. AIX

Re-cluster 2 HACMP 5.2 nodes

Hi, A customer I'm supporting once upon a time broke their 2 cluster node database servers so they could use the 2nd standby node for something else. Now sometime later they want to bring the 2nd node back into the cluster for resilance. Problem is there are now 3 VG's that have been set-up... (1 Reply)
Discussion started by: elcounto
1 Replies

8. UNIX for Advanced & Expert Users

Arbitrator for 2 nodes ocfs cluster

Is there any way to create a arbitrary node for ocfs2 on a virtual machine (others are physical servers) so it won't go panic when one of physical server goes down? This is for load balanced application servers. Any setting example or tips? Thanks. (0 Replies)
Discussion started by: malayo
0 Replies

9. Red Hat

RedHat Cluster: Nodes won't see each other

Hi All; I try to build a Redhat Cluster (CentOS 6) on vmware. But each node sees the other down like: # clustat Cluster Status for mycluster @ Wed Apr 8 11:01:38 2015 Member Status: Quorate Member Name ID Status ------ ---- ... (0 Replies)
Discussion started by: Meacham12
0 Replies

10. Red Hat

RedHat Cluster: Nodes won't see each other

Hi All; I try to build a Redhat Cluster (CentOS 6) on vmware. But each node sees the other down like: # clustat Cluster Status for mycluster @ Wed Apr 8 11:01:38 2015 Member Status: Quorate Member Name ID Status ------ ---- ... (1 Reply)
Discussion started by: Meacham12
1 Replies
HTOOLS(1)							   Version 2.5.2							 HTOOLS(1)

NAME
htools - Cluster allocation and placement tools for Ganeti SYNOPSIS
hbal cluster balancer hspace cluster capacity computation hail IAllocator plugin hscan saves cluster state for later reuse DESCRIPTION
htools is a suite of tools designed to help with allocation/movement of instances and balancing of Ganeti clusters. htools is also the generic binary that must be symlinked or hardlinked under each tool's name in order to perform the different functions. Alternatively, the environment variable HTOOLS can be used to set the desired role. Installed as hbal, it computes and optionally executes a suite of instance moves in order to balance the cluster. Installed as hspace, it computes how many additional instances can be fit on a cluster, while maintaining N+1 status. It can run on models of existing clusters or of simulated clusters. Installed as hail, it acts as an IAllocator plugin, i.e. it is used by Ganeti to compute new instance allocations and instance moves. Installed as hscan, it scans the local or remote cluster state and saves it to files which can later be reused by the other roles. COMMON OPTIONS
Options behave the same in all program modes, but not all program modes support all options. Some common options are: -p, --print-nodes Prints the node status, in a format designed to allow the user to understand the node's most important parameters. If the command in question makes a cluster transition (e.g. balancing or allocation), then usually both the initial and final node status is printed. It is possible to customise the listed information by passing a comma-separated list of field names to this option (the field list is currently undocumented), or to extend the default field list by prefixing the additional field list with a plus sign. By default, the node list will contain the following information: F a character denoting the status of the node, with '-' meaning an offline node, '*' meaning N+1 failure and blank meaning a good node Name the node name t_mem the total node memory n_mem the memory used by the node itself i_mem the memory used by instances x_mem amount memory which seems to be in use but cannot be determined why or by which instance; usually this means that the hyper- visor has some overhead or that there are other reporting errors f_mem the free node memory r_mem the reserved node memory, which is the amount of free memory needed for N+1 compliance t_dsk total disk f_dsk free disk pcpu the number of physical cpus on the node vcpu the number of virtual cpus allocated to primary instances pcnt number of primary instances scnt number of secondary instances p_fmem percent of free memory p_fdsk percent of free disk r_cpu ratio of virtual to physical cpus lCpu the dynamic CPU load (if the information is available) lMem the dynamic memory load (if the information is available) lDsk the dynamic disk load (if the information is available) lNet the dynamic net load (if the information is available) -v, --verbose Increase the output verbosity. Each usage of this option will increase the verbosity (currently more than 2 doesn't make sense) from the default of one. -q, --quiet Decrease the output verbosity. Each usage of this option will decrease the verbosity (less than zero doesn't make sense) from the default of one. -V, --version Just show the program version and exit. UNITS Some options accept not simply numerical values, but numerical values together with a unit. By default, such unit-accepting options use mebibytes. Using the lower-case letters of m, g and t (or their longer equivalents of mib, gib, tib, for which case doesn't matter) explicit binary units can be selected. Units in the SI system can be selected using the upper-case letters of M, G and T (or their longer equivalents of MB, GB, TB, for which case doesn't matter). More details about the difference between the SI and binary systems can be read in the units(7) man page. ENVIRONMENT
The environment variable HTOOLS can be used instead of renaming/symlinking the programs; simply set it to the desired role and then the name of the program is no longer used. REPORTING BUGS
Report bugs to project website (http://code.google.com/p/ganeti/) or contact the developers using the Ganeti mailing list (ganeti@google- groups.com). SEE ALSO
Ganeti overview and specifications: ganeti(7) (general overview), ganeti-os-interface(7) (guest OS definitions). Ganeti commands: gnt-cluster(8) (cluster-wide commands), gnt-job(8) (job-related commands), gnt-node(8) (node-related commands), gnt- instance(8) (instance commands), gnt-os(8) (guest OS commands), gnt-group(8) (node group commands), gnt-backup(8) (instance import/export commands), gnt-debug(8) (debug commands). Ganeti daemons: ganeti-watcher(8) (automatic instance restarter), ganeti-cleaner(8) (job queue cleaner), ganeti-noded(8) (node daemon), ganeti-masterd(8) (master daemon), ganeti-rapi(8) (remote API daemon). Ganeti htools: htools(1) (generic binary), hbal(1) (cluster balancer), hspace(1) (capacity calculation), hail(1) (IAllocator plugin), hscan(1) (data gatherer from remote clusters). COPYRIGHT
Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 Google Inc. Permission is granted to copy, distribute and/or modify under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. On Debian systems, the complete text of the GNU General Public License can be found in /usr/share/common-licenses/GPL. Ganeti HTOOLS(1)
All times are GMT -4. The time now is 11:10 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy