Sponsored Content
Top Forums UNIX for Advanced & Expert Users Gurus needed to diagnose severe performance degradation Post 302349670 by DBA_guy on Tuesday 1st of September 2009 01:23:09 PM
Old 09-01-2009
Error Gurus needed to diagnose severe performance degradation

Hi everyone, newbie forum poster here. I'm an Oracle DBA and I require some guidance from the Unix gurus here about how to pinpoint where a problem is within a Solaris 9 system running on an 8 CPU Fujitsu server that acts as our Oracle database server. Our sysadmins are trying their best to resolve the issue but none of us are 100% sure where the issue resides - I'm hoping people here can help shed some light on things or help point us in a new/better direction.

Environment:
Server: Fujitsu P650, (7 cpu in use, 48GB RAM) Solaris 9 Generic_122300-22 sun4us sparc FJSV,GPUZC-M
Old Storage: EMC Clarion fibre attached storage
New Storage: NetApp storage, 3040 Controller, NFS mounted volumes via multi trunked1GB ethernet connection (not round robin)
Database: Oracle 9i

Problem: We are migrating our storage from fibre EMC to NFS NetApp and are encountering huge performance degradation... pin pointing where the problem is has been problematic. (As a DBA I seriously questioned this move, but this point is now moot as the money has been spent and we have to deal with it.)

Detail: We've been slowly migrating our databases off of the fibre EMC to NFS NetApp Some of our high performance databases struggled mightily on the NetApp storage and there has been lots of finger pointing as to why.

Symptoms: Over time (hours to days) database jobs and response times nosedive - lots of hooting and hollering from the business Smilie System response time can be extremely slow.. simple commands “df -h” “ls” slow in responding. However system load is typically minimal, almost non-existant. (low 1's and 2's for load) however at times we can see high kernel processing times.

Advice from our sysadmins: Current advice from one admin is that the Fujitsu server is older hardware that is not built for this kind of transaction processing. They have been monitoring “counters on the PCI bus (66MHz) and are seeing overflow issues” (forgive me if this isn't well articulated) and noticing that it “has problems keeping up”. Another sys admin feels that the PCI bus has nothing to do with it and that it is networking related: specifically that while we have trunking in place to the NetApp filer, it is not round robin and as a result the pipe from the server to the storage is too small for any given transaction (which from Oracle will necessarily be single threaded) Having conflicting reports from the sys admins is not great.

Are there any recommendations on where the problem possibly lies? (obviously this is very difficult to do from a few paragraphs). Or perhaps more realistically, aside from looking at top/prstat to see low load, iostat to see ok I/O processing times, sysadmins checking counters on a PCI bus, is there any other tools, either available in Solaris or 3rd party that can be used to definitively say “AHA! That is definitely where the bottleneck is!”

Many thanks in advance..
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

SED GURUS - Help!

I wish to substituite a string on each line but ONLY if it appears within double-quotes: this_string="abc#def#geh" # Comment here I wish to change the "#" characters within the double quoted string to "_": this_string="abc_def_geh" # Comment here ... but as you see, the "comment" hash... (2 Replies)
Discussion started by: Simerian
2 Replies

2. Solaris

error notification and diagnose

Hi All, How does Solaris 9/10 alert the server? Where do you get the error on the server? Is there some kind of verifying of errors (like in AIX, CERTIFY resources or diagnose)? Please let me know. Thanks, itik (4 Replies)
Discussion started by: itik
4 Replies

3. Red Hat

Severe Error while starting the System

Dear All, I am facing a unknown error, I start the Linux (RHEL 4 update 6) as usual. After starting the various services(like network,sendmail,portmap etc) a error appears suddenly. The error looks like : Post_create: setxattr failed, rc=28 (dev=hda2 ino=772685) Post_create: setxattr... (2 Replies)
Discussion started by: akhtar.bhat
2 Replies

4. Shell Programming and Scripting

Performance degradation with KSH93

Hi, I have a script that calls an external program to perform some calculations and then I read with "grep" and "sed" values from the output files. I've noticed that performance of KSH93 degrades with every iteration. The output files are all the same size, so I don't understand why after the... (2 Replies)
Discussion started by: i.f.schulz
2 Replies

5. Shell Programming and Scripting

Performance monitoring help needed.

How would i check for following? 1)open ports in my linux machine. 2)Hard disk read speed. 3)Hard disk write speed. (2 Replies)
Discussion started by: pinga123
2 Replies

6. AIX

Diagnose high disk write IO

Hi, say for example if there is high disk write IO in one disk (detected from NMON), how to we identify what processes is writing on that particular disk? (3 Replies)
Discussion started by: ngaisteve1
3 Replies

7. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

8. Shell Programming and Scripting

Shell script to diagnose the network

i have learnt a little bit of shell scripting but not alot. i want to write a script to diagnose the network using ping and another script to traceroute. how would i do this? (6 Replies)
Discussion started by: stefanere2k9
6 Replies

9. AIX

Ld: 0711-851 SEVERE ERROR:

I need to install python 3.3.0 to AIX 6.1 I created folder where I want to install I downloaded files archive from python official website I extracted it into new folder and ran; 1)./configure --with-gcc="xlc_r" --with-cxx="xlC_r" --disable-ipv6 --prefix=my_folder CXX=xlC_r... (2 Replies)
Discussion started by: AIX_30
2 Replies
NetApp::Aggregate(3pm)					User Contributed Perl Documentation				    NetApp::Aggregate(3pm)

NAME
NetApp::Aggregate -- OO class for creating and managing NetApp filer aggregates SYNOPSIS
use NetApp::Filer; use NetApp::Aggregate; my $filer = NetApp::Filer->new({ .... }); my @aggregate_names = $filer->get_aggregate_names; my @aggregates = $filer->get_aggregates; my $aggregate = $filer->get_aggregate( 'aggr01' ); DESCRIPTION
This class encapsulates a single NetApp filer aggregate, and provides methods for querying information about the aggregate and it's sub- objects (eg. volumes), as well as methods for managing the aggregate itself. INSTANCE METHODS
General Instance Methods get_filer Returns the NetApp::Filer object representing the filer on which the aggregate exists. get_name Returns the name of the aggregate as a string. get_states, get_statuses, get_options Each of these methods returns a list of strings, each of which represents a single state, status, or option for the aggregate. NOTE: All you English grammar pluralization rules fanatics can give up trying to convince the author to call that one method get_stati. get_state( $state ), get_status( $status ), get_option( $option ) Each of these methods returns the value for the specified state, status or option. If that particular key wasn't present, then this method will return undef. This makes it easy to tell the difference between a key that doesn't exist, and one that has a false value. get_plex Returns the NetApp::Aggregate::Plex object representing the plex on which the aggregate lives. Volume Specific Methods get_volume_names Returns a list of the volume names which are contained within this aggregate. get_volumes Returns a list of NetApp::Volume objects, each of which represents one of the volumes in the aggregate. get_volume( $name ) Returns a single NetApp::Volume object representing the specified volume. If that volume doesn't exist on the aggregate, then a fatal exception is raised. create_volume( %args ) This method creates a flexible volume in the aggregate, and returns the NetApp::Volume object representing the new volume. The arguments are as follows. All values are simple strings, unless otherwise noted. $aggregate->create_volume( # Required arguments name => $name, size => $size, # Optional arguments space => 'none' | 'file' | 'volume', language => $language, source_filer => $source_filer, source_volume => $source_volume, ); Both the source_filer and source_volume arguments must be given when creating a flexcache volume. The space and language arguments may not be specified with the source_filer/source_volume arguments. destroy_volume( %args ) Destroys the specified volume. Note that since this API is not designed to be used interactively, the -f (force) argument is always used. Be sure you really want to destroy the volume, programatically. $aggregate->destroy_volume( # Required argument name => $name, ); The $name must be a string, and it must be one of the volumes in the $aggregate. Qtree Specific Methods get_qtree_names Returns a list of strings, each of which is the name of a qtree on the aggregate. get_qtrees Returns a list of NetApp::Qtree objects, each of which represents a single qtree on the aggregate. get_qtree( $name ) Returns a single NetApp::Qtree object for the specified qtree name. The name must in the form of a pathname, for example: /vol/volume_name/qtree_name The qtree_name is optional if querying the object for a volume's qtree. This method simply returns nothing if the specified qtree doesn't exist on the aggregate. Snapshot Specific Methods get_snapshots Returns a list of NetApp::Snapshot objects for each of the snapshots of the aggregate. get_snapshot( $name ) Returns a single NetApp::Snapshot object matching the specified name, if it exists for the aggregate. create_snapshot( $name ) Creates a snapshot of the aggregate with the specified name. delete_snapshot( $name ) Deletes a snapshot of the aggregate with the specified name. get_snapshot_deltas Returns a list of NetApp::Snapshot::Delta objects for each snapshot delta for the aggregate. get_snapshot_reserved Returns a string representing the amount of reserved space, as a percentage. This string does NOT include the % sign. set_snapshot_reserved( $percentage ) Sets the snapshot reserved space to the specified percentage, which should also NOT include the % sign. get_snapshot_schedule Returns a NetApp::Snapshot::Schedule object representing the snapshot schedule for the aggregate. set_snapshot_schedule( %args ) Sets the snapshot schedule for the aggregate based on the arguments passed. The argument syntax is: $aggregate->set_snapshot_schedule( weekly => $weekly, daily => $daily, hourly => $hourly, hourlist => [ $hour1, $hour2, $hour3, .... ], ); TO BE IMPLEMENTED
NOTE: Currently, all of the following methods have yet to be implemented, but will be soon. This documentation serves as a guideline for how to implement the perl API for each associates CLI function. NetApp::Aggregate->create( ... ) my $aggregate = NetApp::Aggregate->create( # Required arguments filer => $filer, # NetApp::Filer object name => $aggregate_name, # Required but mutually exclusive arguments # Either 'disks' OR 'diskcount and/or disksize' disks => [ [ $disk1, $disk2, .... ], [ $diskn, $diskn+1, .... ], ], diskcount => $diskcount, disksize => $disksize, # Optional arguments raidtype => 'raid0' | 'raid4' | 'raid-dp', raidsize => $raidsize, disktype => 'ATA' | 'FCAL' | 'LUN' | 'SAS' | 'SATA' | 'SCSI', rpm => $rpm, mirrored => $boolean, ); $aggregate->add( ... ) $aggregate->add( # Required arguments name => $aggregate_name, # Required but mutually exclusive arguments # Either 'disks' OR 'diskcount and/or disksize' disks => [ [ $disk1, $disk2, .... ], [ $diskn, $diskn+1, .... ], ], diskcount => $diskcount, disksize => $disksize, # Optional arguments raidgroup => $raidgroup, force => 1, ); $aggregate->destroy() NOTE: This always uses the -force option, since this API is not interactive. $aggregate->offline() $aggregate->offline( # Optional arguments cifsdelaytime => $cifsdelaytime, ); $aggregate->online() NOTE: It is unclear whether or not we should always imply -f (force => 1), or whether we should treat the prompted scenario as an error, and raise an exception. Since forcing an aggregate online can result in data loss when -f is used, perhaps we should force that state to be cleaned up first. $aggregate->rename( $newname ) $aggregate->restrict( ... ) $aggregate->restrict( # Optional arguments cifsdelaytime => $cifsdelaytime, ); perl v5.14.2 2008-11-26 NetApp::Aggregate(3pm)
All times are GMT -4. The time now is 10:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy