Today (Saturday) We will make some minor tuning adjustments to MySQL.

You may experience 2 up to 10 seconds "glitch time" when we restart MySQL. We expect to make these adjustments around 1AM Eastern Daylight Saving Time (EDT) US.


Network related issues

Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Network related issues

Oflate we are finding a few servers experiencing severe slowness. What would be the commands that I need to try to postmortem the situation?
# 2  
What OS? Your other post mentions AIX. Getting data from the past is really difficult unless you had already set up monitoring or auditing.

If you have detailed logs from applications, sometimes you can infer that application A has been taking longer and longer times to complete.

Many kinds of problems are sporadic or are hard to reproduce. These can only be found by creating monitors before the fact.

Please give us more system details: specific OS, main application(s) for the system.
Example: AIX 7.3, sybase server on SAN.
# 3  
Some wild guesses:-
  • Loss of access to DNS server (slow reverse IP lookup for auditing, so slow login or application)
  • Database locks - hugely dependant on your application
  • Missing database index causing full table scans
  • Poor data queries, e.g. get all records from the database then check each in turn on criteria rather than building the condition into the query
  • Database logs files filling and flushing too slowly
  • Exhausting real memory causing paging (potentially DB consuming too much real memory)
  • Network speed conflict, e.g. if NIC is 10M-half and switch is 100M-full, it will work, but any file transfer will cripple it with lots of dropped packets.
  • IO issues, especially with NFS or an HA cluster if you fail over
  • Scheduled work, e.g. current stock summary
  • Ad-hoc jobs, e.g. current stock summary
  • Resources stealing by another LPAR if the definitions allow it
  • Large write volume to direct disk (e.g. local) rather than cached disk (RAID or SAN etc.)
  • High NFS contention especially with other seemingly unrelated servers

You can see it is a very very VERY wide spread of options so far - and the list is a long way from being exhaustive. You need to be a fair bit more explicit about what you have (including OS) what goes slow, what's happening at the time, what dependencies you have with other servers.



Robin
# 4  
Most *NIX systems (AIX, Linux, Solaris, BSD) have some kind of system and accounting records. You can run
Code:
sar

to see if it is properly deployed on your system. If you run it and get loads of output, you may be in luck. To use it, refer to the man pages. Typically you want to check options for memory and swap usage, CPU usage, and I/O activity.

If it's not installed, consider deploying this first before installing some complex monitoring software; it's a very standard unix utility that has been around for ages, but the implementation and features vary from platform to platform. For Linux install the sysstat package.

On most systems, sar's data is collected through another program which is run as a cronjob. On a typical RedHat/CentOS Linux system, you will find /etc/cron.d/sysstat to contain:

Code:
* * * * * root /usr/lib64/sa/sa1 -S XALL 1 1

which I immediately change to

Code:
*/5 * * * * root /usr/lib64/sa/sa1 -L -S XALL 10 30

The original form collects data once per minute, which is often simply not enough granularity to get a feel for rapid changes to the system, the kind that cause instability and crashes. Also, if memory becomes extremely sparse, cron might not be able to spawn the job every minute.

My form, however, spawns a new job every 5 minutes. It writes 30 records, one every 10 seconds. The corresponding reports contain enough detail to know very precisely when the problem started. You will need an additional 1.5 GB of disk space on /var/log if you do this.

If you want graphs and pretty output, you may be able to export the data into graphing engines or spreadsheets. Linux's sar has such a program (sadf), and other related projects can slurp of the data and present graphs.
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Network related script

First of all,I would like to introduce about me, This is my own try and this is not my homework,and I study myself reading shell script pdf guide from net and learn shell scripts bit by bit.I am self study learner.I try to work out shell scripts.please help to resolve this type of problem.when I... (1 Reply)
Discussion started by: kannansoft1985
1 Replies

2. HP-UX

Network Connectivity Issues

Newbie with UNIX here. Currently troubleshooting a UNIX terminal we have. I determined it to be bad and swapped it out with a known good terminal. I went in and changed the IP address and host name to reflect the old terminal. Although now there is no connectivity. I swapped out the NIC... (1 Reply)
Discussion started by: kevinlord190
1 Replies

3. Solaris

Swap space related issues, how to recognise the newly attached disk

hi!:) i got a problem....:wall::wall: i got several disks in my diskarray. I attached new disk to allocate it to the swap space. The problem is : how to recognise the newly attached disk? I've one more requirement -:wall:- i want to run dns service on another port number.how can i do... (2 Replies)
Discussion started by: vamshigvk475
2 Replies

4. AIX

Network related errors

Hi How to check if my AIX server has any network related errors ? (2 Replies)
Discussion started by: samsungsamsung
2 Replies

5. UNIX for Dummies Questions & Answers

Check for network issues

Hi all, How do i check for network issues on a server (solaris & linux) - interms of connectivity, collision, congestion, whatever that's impacting the connection to/from the server? Please advise. Thanks. :confused: (4 Replies)
Discussion started by: greencored
4 Replies

6. Programming

Fork syscall and related issues

Hi all, i just started started learning system programming and want to pursue a career in the sys prog area. below is the program that use a fork() call. i read in one of the tutorials that parent process and child process uses different address spaces and runs concurrently. that meas each... (2 Replies)
Discussion started by: MrUser
2 Replies

7. Solaris

Solaris 10 network, process, database related comands

Hi everbody, Can anyone let me know the resources for list of network, process, database related commands of solaris10 possibly with little bit of explanation. Thanks in advance, Chandra Sekhar. (1 Reply)
Discussion started by: chandoo.java
1 Replies

8. UNIX for Dummies Questions & Answers

Network printer issues

Hello, I have been having a problem with printing to a network printer on my LAN, I am able to ping all ports from the server and the printer. Print request just stay in the print que the only way to print is from the parallel line to the server. Any ideas on what can be going wrong?? Lp sched is... (1 Reply)
Discussion started by: ostac
1 Replies

Featured Tech Videos