03-06-2009
I too have done similar testing but slightly different. Rather than simply time the creation of 100 files, which can be very misleading OR timing the creation of 1M files which is no better, I prefer to look at what is happening across the system resources during the entire event.
I run collectl with a monitoring interval of 1 second, logging to a file or simply watching the system in real time. When I create a million files I can watch the cpu periodically increase. In fact, when getting in the higher ends of files I can actually see spike in cpu load. This is something you can't see when just doing end-to-end numbers.
Another interesting test is to set up an alarm in your script to write out the number of files created every 10th (or even hundredth) of a second. You'll be amazed to see how linearly the number of files created/second drops over time as well as how things periodically slow down but are not visible when only looking at second-level samples.
You can also run collectl at a monitoring interval of 0.1 seconds and see micro-spikes in CPU load as well. This is something most people miss because none of the existing tools can deal with sub-second reporting.
-mark
9 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I reinstalled my Linux box with RedHat 7.2 and used the ext3 journaling file system. This thing is a pig now. There isn't much running on the box, and performance is sad. (1 Reply)
Discussion started by: 98_1LE
1 Replies
2. Shell Programming and Scripting
Hi all, I have a php file that grabs xml, parses it and updates my db accordingly. I want to automate the execution of this process, rather than having to hit the url manually.
I have been looking into using cron to execute a script to do this, however i'm not exactly sure what command i would... (1 Reply)
Discussion started by: restivz77
1 Replies
3. Programming
hi,
in my application, i have set up to capture SIGINT and execute a handler.the problem is whenever i hit C-c, multiple SIGINT are sent to the application.I have blocked the SIGINT right after catching the first one but it is unsuccessful.Here is what i do :
jmp_buf main_loop;
int... (1 Reply)
Discussion started by: Sedighzadeh
1 Replies
4. Shell Programming and Scripting
I have a unix shell script (ex.sh) written.
How to find out how many users (incl. myself) have run this .sh ?
I can insert code snipet at top of script if need be.
- Ravi (2 Replies)
Discussion started by: ravi368
2 Replies
5. Shell Programming and Scripting
Hi.. I need to get the following output from the input file like this
INPUT
GRM1 GRM1 0
GRM1 ABC1 1
GRM1 FEQ1 2
GRM1 SED1 3
ABC2 GRM1 0
ABC2 ABC2 1
ABC2 FEQ1 2
ABC2 BED1 3
SED1 SED1 0
SED1 SED1 1
SED1 SED1 2
SED1 ABC1 3
OUTPUT: (7 Replies)
Discussion started by: empyrean
7 Replies
6. SuSE
Hello,
I'm having trouble looking for info for SUSIE on this CVE-2012-4681.
This is basically the newest Java hit. It is mostly a web browser issue but I would like to see if the versions on our servers are vulnerable. I already found the pages/info for Solaris and RHEL.
Any help would be... (4 Replies)
Discussion started by: bitlord
4 Replies
7. Cybersecurity
Hello,
I'm having an issue with VNC. Security at work says that they scanned my servers (Solaris, RHEL, SLES) and found that you don't need a password to access a VNC session. I have tested this and you can't login to the VNC session without a password. Can someone tell what the Retina scanner... (1 Reply)
Discussion started by: bitlord
1 Replies
8. Shell Programming and Scripting
Hello,
I am looking to hit a URL using curl and click on submit button so that I can get the results. The below is the code
<input name="tos_accepted" id="tos_accepted" class="button" type="submit" value="Yes, I Agree"/>
<input name="tos_discarded" id="tos_discarded"... (1 Reply)
Discussion started by: Kochappa
1 Replies
9. Shell Programming and Scripting
Hi Gurus,
I need to merge two files.
file1 (small file, only one line)
this is first linefile2 (large file)
abc
def
ghi
... I use below command to merge the file, since the file2 is really large file, the command read whole file2, the performance is not good.
cat file1 > file3... (7 Replies)
Discussion started by: green_k
7 Replies
LEARN ABOUT DEBIAN
genplotfiles
GENPLOTFILES(1) colplot GENPLOTFILES(1)
NAME
genplotfiles - utility to help automate creation of collectl plot file from raw ones
SYNOPSIS
/opt/hp/colplot/genplotfiles.pl -outdir [-switches]
DESCRIPTIONS
This utility is not intended to meet all plot file creation needs, but quite frankly should meet most. Features not provided with this
utility are best realized by running collectl directly in playback mode (see collectl -p). It is therefore important to understand what
this utility will and will not do for you before building an automation strategy around it.
The algorithm used is fairly simple - genplotfiles examines each collectl raw file to see it it falls within the data ranges (if specified)
and contains a matching hostname (if specified). If a plot file already exists, the timestamps are compared and the raw file skipped if
its newer than the plotfile.
Any raw files that pass all these tests are then passed to collectl with their timestamp wildcarded. The result is all raw files for that
host/date will be processed. This means if you copy over collectl raw files multiple times over the course of the day using rsync, you can
immediately genererate up-to-date plots for them using genplotfiles. Furthermore, you won`t even need any date parameters if the older
plot files are alreay up to date.
For an example of what a cron job that aggregates multiple collectl logs onto a single system for plotting purposes, see crontab.sh in the
same directory as this script.
SWITCHES
All switches are optional with the exception of -outdir:
-colbin path-to-collectl
This points to collectl and is only necessary if /usr/sbin is not in your path or you`ve chosen to install collectl in a non-stan-
dard directory.
-debug number
Primarily for development, will provide intermediate processing information based on its value. See souce for details.
-force
Always generate plot files even in there is an existing one that is newer than the latest raw file. This will be necessary if you
want to rerun this script and just change the value of -subsys.
-help
Print a summary of available switches.
-host string
Only process files that contain this string in their hostname portion of the filename.
-indir directory
Directory where to look for collectl logs. The default is /var/log/collectl and is therefore only necessary if your logs are in a
non-standard location.
-outdir directory
Directory into which to write the plottable files. This field is required.
-fromdate date
Ignore files with a datestamp in their name less that this value. The default is 20050101 and so defaults to all files. The value
of `yesterday` is also recognized to simplify scripts that call this.
-thrudate date
Ignore files with a datestamp in their name with a datestamp greater than this value. The default is the current day and one can
also specify a value of `yesterday`.
-subsys string
Instead of using the default subsystems for which data was collected, generate output files based on this set. There are basically
two reasons for doing this. The first is that you may want to reduce the size of the plot files, though with compression there is
rarely much point in this. The second is if you`ve asked collectl to collect summary data but want to plot detail data.
See the colplot FAQ for more information.
AUTHOR
This program was written by Mark Seger (mjseger@gmail.com).
Copyright 2005 Hewlett-Packard Development Company, L.P.
SEE ALSO
LOCAL
OCTOBER 2005 GENPLOTFILES(1)