IF you start testing methods on a file be aware of the effect file caching by the OS and disk controllers. You will get completely bogus results if you are not aware of this. I/O wait time is the biggest time consumer. Disks are at the very best 10 times slower than memory unless you have SSD.
Pretend you try sed and get this answer:
Code:
time sed -n '$=' input_file
real 0m2.098s
user 0m0.516s
sys 0m0.338s
Great - that took 2.098 seconds of wall time.
Let's try wc -l
Code:
time wc -l input_file
real 0m0.778s
user 0m0.416s
sys 0m0.338s
Wow. wc -l was faster.
No. A lot of the file data was still in cache. So there was no I/O wait. Why, because you ran against the same file. As you read thru a file the system will attempt to cache all or parts of it, depending on available resources.
The file data in the cache slowly goes away as other users read/write the same disk. After a while the file is no longer cached. How long that is, I cannot say. Solaris will use part of free memory as file cache, so will Linux. Add this to what the disk controller caches and some large chunks of really huge files can be in memory.
SAN storage behaves in a similar way, but is a lot more complex. SAN is generally slower than direct disk, then some systems have the faster directio options, the fastest storage is raw disk (bypassing the filesystem and kernel code for filesystem support). Oracle will do this for its database files if configured.
Also you can tune a filesystem.
If you have to speed up file I/O look into SSD for desktops.
These 2 Users Gave Thanks to jim mcnamara For This Post:
IF you start testing methods on a file be aware of the effect file caching by the OS and disk controllers. You will get completely bogus results if you are not aware of this. I/O wait time is the biggest time consumer. Disks are at the very best 10 times slower than memory unless you have SSD.
Pretend you try sed and get this answer:
Code:
time sed -n '$=' input_file
real 0m2.098s
user 0m0.516s
sys 0m0.338s
Great - that took 2.098 seconds of wall time.
Let's try wc -l
Code:
time wc -l input_file
real 0m0.778s
user 0m0.416s
sys 0m0.338s
Wow. wc -l was faster.
No. A lot of the file data was still in cache. So there was no I/O wait. Why, because you ran against the same file. As you read thru a file the system will attempt to cache all or parts of it, depending on available resources.
The file data in the cache slowly goes away as other users read/write the same disk. After a while the file is no longer cached. How long that is, I cannot say. Solaris will use part of free memory as file cache, so will Linux. Add this to what the disk controller caches and some large chunks of really huge files can be in memory.
SAN storage behaves in a similar way, but is a lot more complex. SAN is generally slower than direct disk, then some systems have the faster directio options, the fastest storage is raw disk (bypassing the filesystem and kernel code for filesystem support). Oracle will do this for its database files if configured.
Also you can tune a filesystem.
If you have to speed up file I/O look into SSD for desktops.
thank you so much for the detailed explanation. i've always wondered why sometimes i get faster response and other times i get a much slower response when running the same command on a file. now i know. thanks a million.
IF you start testing methods on a file be aware of the effect file caching by the OS and disk controllers. You will get completely bogus results if you are not aware of this. I/O wait time is the biggest time consumer. Disks are at the very best 10 times slower than memory unless you have SSD.
Pretend you try sed and get this answer:
Code:
time sed -n '$=' input_file
real 0m2.098s
user 0m0.516s
sys 0m0.338s
Great - that took 2.098 seconds of wall time.
Let's try wc -l
Code:
time wc -l input_file
real 0m0.778s
user 0m0.416s
sys 0m0.338s
Wow. wc -l was faster.
No. A lot of the file data was still in cache. So there was no I/O wait. Why, because you ran against the same file. As you read thru a file the system will attempt to cache all or parts of it, depending on available resources.
The file data in the cache slowly goes away as other users read/write the same disk. After a while the file is no longer cached. How long that is, I cannot say. Solaris will use part of free memory as file cache, so will Linux. Add this to what the disk controller caches and some large chunks of really huge files can be in memory.
SAN storage behaves in a similar way, but is a lot more complex. SAN is generally slower than direct disk, then some systems have the faster directio options, the fastest storage is raw disk (bypassing the filesystem and kernel code for filesystem support). Oracle will do this for its database files if configured.
Also you can tune a filesystem.
If you have to speed up file I/O look into SSD for desktops.
Yeah i thought so when i tested...varying times..Very good food for our thoughts Jim...thanks..
How to find count total number of pattern in a file …
File contains : a.txt
-------------
aaa bbb nnn
ccc aaa bbb
aaa ddd aaa
aaa aaa aaa
grep -c aaa a.txt
Op: 4 ( But my requirement is should count the total no of patterns as 7 ) (4 Replies)
Hello Friends,
I know you all are busy and inteligent too...
I am stuck with one small issue if you can help me then it will be really great.
My problem is I am having some files i.e.
Input.txt1
Input.txt2
Input.txt3
Now my task is I need to check the total number of rows in... (4 Replies)
i have a file in following format
1 32 3
4 6 4
4 45 1
45 4 61
54 66 4
5 65 51
56 65 1
12 32 85
now here the total number of lines are 8(they vary each time)
Now i want to select only those lines in which the values... (6 Replies)
how to count the total number of lines of all the files under a directory using perl script..
I mean if I have 10 files under a directory then I want to count the total number of lines of all the 10 files contain. Please help me in writing a perl script on this. (5 Replies)
Hi
I have some files that contain be anything up to 100k lines - eg. file100k
I have another file called file5k and I need to produce filec which will contain everything in file100k minus what matches in file 5k..
ie.
File100k contains
1FP
2FP
3FP
File5k contains
2FP
I would... (2 Replies)
Hello, I need help in appending the line number of each line to the file and also to get the total number of lines. Can somebody please help me.
I have a file say:
abc
def
ccc
ddd
ffff
The output should be:
Instance1=abc
Instance2=def
Instance3=ccc
Instance4=ddd
Instance5=ffff
... (2 Replies)
Hi have following file
|abcd
2|abcd
|sdfh
|sdfj
I want to find total number of files haivng nothing in feild 1 using awk
will command
awk -F "|" '( $1=="") {print NR}' test_awk
will work??? (4 Replies)
I am trying to get a total number of tapes w/out headers or footers in a ERV file and append it to the file. For some reason I cannot get it to work. Any ideas?
#!/bin/sh
dat=`date +"%b%d_%Y"`
+ date +%b%d_%Y
dat=Nov16_2006
tapemgr="/export/home/legato/tapemgr/rpts"... (1 Reply)
10-01-2012
