Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

What process is writing to disk?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users What process is writing to disk?
# 22  
Old 10-29-2008
Quote:
Originally Posted by otheus
It's the download, not the process Smilie
Of course,
I thought the provider (of the C version) would eventually have a PHP based web site Smilie
# 23  
Old 10-29-2008
re 10: I'm the author of collectl. If there's something inconsistent with the documentation could you let me know what is?

While I don't have any quick answers to the problem of finding the guilty process, one thought that comes to mind is to try a smaller monitoring interval, something you can to do with most of the 'standard' utilities. For example you could do

collectl -i.1 --vmstat -oT

to see the same output vmstat would show, only prefaced with a timestamp and you'd see one sample every second. If that's not enough you could try 0.01 seconds or anything in between but that's probably overkill. 9-)

you could also try

collectl -i.1 --top

to see what top might show or even

collectl -i.1 -sD

to see what iostat would show. far too many other commands to show here, but you could always look at collectl.sourceforge.net

-mark
# 24  
Old 10-30-2008
Quote:
Originally Posted by radoulov
Ah ...
Did you try to stop mysql (if possible given your SLA, of course Smilie)?
Here's the weird thing. I've stopped just about everything. The one thing I cannot stop is FTP and NFS. Samba, syslog, mysql, postfix, cron have all been shut down temporarily with no effect.

NFS is not exporting the disk in question. FTP has several log files open. The culprit might be /var/log/wtmp as the other logs are relatively idle (only when users are actually transferring, which is relatively rate).

Code:
proftpd 12155  ftp    3w   REG        8,4   5086871 12767115 /var/log/proftpd/replicate.log
proftpd 12155  ftp    6w   REG        8,4   1924515 12767452 /var/log/proftpd/auth.log
proftpd 12155  ftp    7w   REG        8,4   4340157 12767454 /var/log/proftpd/access.log
proftpd 12155  ftp    8u   REG        8,4      4936  9755364 /var/run/proftpd/proftpd.scoreboard
proftpd 12155  ftp   11w   REG        8,4  68647296 12767550 /var/log/wtmp
proftpd 12155  ftp   12w   REG        8,4   4668852 12767913 /var/log/proftpd/transfer.log

I rotated the wtmp file. The new one is less than 5k in size, so I think I can eliminate it. Watching "du -sh /var/log/proftpd" leads me to conclude that it is also not changing in size.

Any known subversive software that writes to disk a lot?
Last edited by otheus; 10-30-2008 at 06:23 AM..
# 25  
Old 10-30-2008
Here's what I plan to do ... (does anyone have a better idea?) On a periodic basis, run the following command and compare the subsequent runs' outputs to the first run.
Code:
find / -xdev -type d -exec du -s "{}" ";" >other-file-system/du-output

# 26  
Old 10-30-2008
I would definitely try to trace the rest of the processes one by one (not exactly, here I'm tracing the mysqld process and follow the forks).
Something like this:

Code:
strace -Tfe pwrite64 -p <pid>

Code:
# strace -Tfe pwrite64 -p 2158
Process 32381 attached with 19 threads - interrupt to quit
[pid 32381] pwrite64(10, "\200>u\327\0\364\0\21\0\0\336\0178^\202\\\37&\0\201\307\0\2\0\2\200\4\200\0048\333\2"..., 512, 4721664) = 512 <0.000042>
[pid 32381] pwrite64(10, "\200>u\327\1\324\0\21\0\0\336\0178^\202\\\37&\0\201\307\0\2\0\2\200\4\200\0048\333\2"..., 512, 4721664) = 512 <0.000021>
[pid 32381] pwrite64(10, "\200>u\327\2\0\0\21\0\0\336\0178^\202\\\37&\0\201\307\0\2\0\2\200\4\200\0048\333\2"..., 1024, 4721664) = 1024 <0.000022>
[pid  2170] pwrite64(4, "\22\177\266\324\0\0\0007\377\377\377\377\377\377\377\377\0\0\0\0|\353\254\364\0\2\0\0\0\0\0\0"..., 81920, 1048576) = 81920 <0.000117>
[pid  2164] pwrite64(4, "2`\32e\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0|\353\256\303\0\0\0\0\0\0\0\0"..., 32768, 81920) = 32768 <0.000095>
[pid  2164] pwrite64(4, "\22\177\266\324\0\0\0007\377\377\377\377\377\377\377\377\0\0\0\0|\353\254\364\0\2\0\0\0\0\0\0"..., 16384, 901120) = 16384 <0.000073>
[pid  2164] pwrite64(4, "Fd7\370\0\0\r{\377\377\377\377\377\377\377\377\0\0\0\0|\353\256YE\277\0\0\0\0\0\0"..., 16384, 56541184) = 16384 <0.000067>
[pid  2164] pwrite64(4, "\247\30\243\257\0\0\25\340\0\0\24\34\0\0\24:\0\0\0\0|\353\256\303\0\2\0\0\0\0\0\0"..., 16384, 91750400) = 16384 <0.000067>
[pid  2162] pwrite64(9, "\0\0\0\0\0\0\336\17\0\0\0\0|\353\256\303\0\230\16\303\0\20\0\0\377\377\377\377\377\377\377\377"..., 512, 1536) = 512 <0.000070>

Or use -c to count write system calls.

This should show the write intensive processes.
# 27  
Old 10-30-2008
This is helping narrow it down a bit:
Code:
{ date; sudo find /var -xdev -mindepth 1 -type d -exec du -s "{}" ";" ; } |sudo sdiff -s -w 95  logfiles/sa/du-var -

From there, fuser, then strace. Thanks guys!

(The culprit was "multilog" from the daemontools suite. After ionice'ing it, the w/s count still stays high, but I think things are smoother. Anyway, we'll offload this process onto another disk or system.)
Last edited by otheus; 10-30-2008 at 07:25 AM.. Reason: update
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Open Source

Help with writing Shell Script to automate process using multiple commands

Hello! Need help to write a Linux script that can be run from windows using command/Cygwin/any other way. I am new to scripting, actually i am trying to automate server health check like free disk space, memory along with few services status, if any services is not running then start services ,... (7 Replies)
Discussion started by: Sayed Ibrahim
7 Replies

2. Emergency UNIX and Linux Support

How to find which process using, while getting High Disk I/O?

In Our Production server I/O was very high, I Recived mail that Disk I/O was high, is it possible how to find which process Used this much I/O ? Iam Using Ubuntu server 12.04. Linux 3.9.3-x86_64-server33 (Li473-1200) 07/23/2014 _x86_64_ (8 CPU) 12:05:01 AM DEV ... (5 Replies)
Discussion started by: babinlonston
5 Replies

3. Ubuntu

Reading disk usage per process

Hi, I am using iostat and /proc/<pid>/status to determine the disk usage per process. I have a question regarding buffer cache. When I am reading a 10MB file, the counters I get show that <5MB of the file is read. Do the counters in iostat and /proc/<pid> reflect the amount of data that is... (2 Replies)
Discussion started by: fidelity
2 Replies

4. UNIX for Dummies Questions & Answers

Writing a loop to process multiple input files by a shell script

I have multiple input files that I want to manipulate using a shell script. The files are called 250.1 through 250.1000 but I only want the script to manipulate 250.300 through 250.1000. Before I was using the following script to manipulate the text files: for i in 250.*; do || awk... (4 Replies)
Discussion started by: evelibertine
4 Replies

5. Linux

How to follow up disk mirroring process?

Hi, The OS is SuSE Enterprise 11 and the system is HP WS460c G6 Blade with hardware disk array RAID 1 mirror. One disk was just replaced and the disk mirroring process is on its way. My question is how to follow up / monitor the disk mirroring process? I know hpacucli can do the job, but there... (0 Replies)
Discussion started by: aixlover
0 Replies

6. UNIX for Dummies Questions & Answers

Which process is doing all the writing

Hi We are running an IBM P570 with AIX and Unidata. The disk monitor in nmon is showing that one of our logical volumes is hitting 100% most of the time, and that 98% of it is write. I am trying to identify the top processes in terms of disk IO, obviously particularly write so that we... (4 Replies)
Discussion started by: idwalton72
4 Replies

7. Shell Programming and Scripting

Which Process is writing this Log file!!

Hello , Well I have some /tmp files which are growing very quickly..Can anyone suggest me a way to find which process is logging into this file :confused: ? Thanks very much in Advance!! Mohammed (2 Replies)
Discussion started by: Mohammed
2 Replies

8. AIX

disk i/o per process on aix 5.2

hi, i am working on aix 5.2, i want to check disk i/o for any running process. please if anybody can help me. thanks Aqeel Anwar (1 Reply)
Discussion started by: system-admin
1 Replies

9. UNIX for Advanced & Expert Users

disk i/o per process

Dear All, We are using AIX 5.2 with IBM pseries servers. I want to check disk io for a running process. Please if anybody can help me. Thanks. Aqeel Anwar (1 Reply)
Discussion started by: aqeelcu@hotmail
1 Replies

10. UNIX for Dummies Questions & Answers

Writing file to disk takes time

Hi All, I am working on a Solaris machine. When i use a particular software to generate some files, the log shows around 0 to 3 secs for generating each file. But the same when i see on the disk it shows double the time difference between two file generation. For example if file A takes 0 secs... (7 Replies)
Discussion started by: nileshkarania
7 Replies
Login or Register to Ask a Question