Shell Programming and Scripting

Mark:

I am looking for the cause or clue of what my clients reports as an issue for him.
He has an AWS Instance that was Ubu 9.04, so we upgraded his Instance to 12.04 LTS.
This cloning process included an upgrade of MySQL and MongoDB,

Client's description in red...

Since this upgrade, he reports that he sees in htop a CPU (there's 8 CPUs) spike of anywhere from 1 or 2 CPUs that lasts for "several seconds" where the IO field entries are working pretty hard.
"The io-wait happens very frequently. I suppose I can stare at HTOP for a while to see how often, but I'm guessing about every 5 minutes (big guess). "
and
"ok. Let me watch.... there it goes. I've attached a screenshot that shows 2 CPU's going into io-wait (there were actually 3 at one point). This went on for about 7 seconds. My knee-jerk reaction is ‘it's mongo', because the IO in htop (sorted by IO) is the highest. However, that value is really no higher (and even lower) than other values when there is no io-wait happening. So I think it's something that's writing to disk, but not much data (so low IO because of all of the waiting" says the client.

So since I also would suspect mongod, I fire up "htop -u mongodb" and seems accurate.
I've also ran "htop -u mongodb -s IO_WRITE_RATE" and witnessed similar activity.

so, the trail I am following thus far, is...

Code:

dbpath=/mnt/data-indexes/mongodb
/dev/md1       xfs     50G   29G   22G  57% /mnt/data-indexes
ARRAY /dev/md1 level=raid0 devices=/dev/sdj,/dev/sdk

The usual sysstat tools installed and I usually check sar data but that only collects every 10m, so I installed collectl

And this is my DaemonCommands entry since I modified it to slice up my task by the Hour.

Code:

DaemonCommands = -f /var/log/collectl -r00:00,7,60 -m -F60 -s+YZ

The last thing I ran is

Code:

collectl --procfilt $(pidof mongod) --dskopts z -sD --dskfilt sd[jk] -oT --headerrepeat 100 -R 1d >> mongod_playback.out

and that shows Pct as hitting "100" 3681 times.

Why I use --hr 100? It looked "better" on screen to be honest.

Thank you for your time.
I appreciate it.

actually one other thing I just spotted, it you want to filter on a pid, you need to preface the number with a 'p'.
-mark

both

Code:

--procfilt -p $(pidof mongod)
--procfilt $(pidof mongod)

both seem to put out identical data...?

and most importantly, (I just noticed also)

Code:

collectl V3.6.0-1 (zlib:2.033,HiRes:1.972101)

Thank you,

JJ of c9