05-19-2015
That its 75 times slower via NFS than disk is a pretty big hint.
Yes, NFS versus disk matters quite a lot. Its asynchronous nature means that an operation you'd consider atomic on a local disk mean a lot of "who has this file? is this file busy? Has anyone updated it since I saw it last? I'm taking this file, LAST WARNING, FILE TAKEN. Everyone who might be reading this file, I have updated it, flush your buffers! I have now CLOSED THE FILE, free for all takers" etc etc messaging back and forth. Imagine the overhead this creates for programs using huge clutters of tiny files! All that asking and waiting. If any of it can be done in parallel that can help, having 3-4 requests flying at once instead of 1, but a shared filesystem is by definition a lot more back-and-forth than a dedicated one.
The traditional answer to "why is my file tree walker slow" is "Is it NFS?" and the traditional reply to that is something along the lines of "Why -- does that matter?" followed by "I configured it to skip the NFS, much better now".
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi experts.
In one thread i have asked you how to grep the string from the below sample file-
Unfortunately the script did not gave proper output (it missed many strings). It happened may be i did gave you the proper contents of the file
That was the script-
"$ perl -00nle'print join... (13 Replies)
Discussion started by: thepurple
13 Replies
2. Shell Programming and Scripting
hi,
i am facing a problem in merging two files using awk,
the problem is as stated below,
file1:
A|B|C|D|E|F|G|H|I|1
M|N|O|P|Q|R|S|T|U|2
AA|BB|CC|DD|EE|FF|GG|HH|II|1
....
....
....
file2 :
1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies
3. Shell Programming and Scripting
Hi,
I am new to perl:
I need to write perl script to list all the files present in directory and mail should be come to my inbox with all the files present in that directory.
advanced thanks for valuable inputs.
Thanks
Prakash GR (1 Reply)
Discussion started by: prakash.gr
1 Replies
4. Shell Programming and Scripting
Hi All,
I have a file (Names.txt) and the contents of the file is give below.
$ cat Names.txt
FF313207008.txt
FF223207007.txt
FF143207006.txt
FF372150600.txt
FF063407005.txt
FF063307005.txt
$
From these given file names I want to find the files which has the 6th index value as 2. So... (5 Replies)
Discussion started by: krish_indus
5 Replies
5. Shell Programming and Scripting
I have two files which I would like to compare and then manipulate in a way.
File1:
pictures.txt 1.1 1.3
dance.txt 1.2 1.4
treehouse.txt 1.3 1.5
File2:
pictures.txt 1.5 ref2313 1.4 ref2345 1.3 ref5432 1.2 ref4244
dance.txt 1.6 ref2342 1.5 ref2352 1.4 ref0695 1.3 ref5738 1.2... (1 Reply)
Discussion started by: linuxkid
1 Replies
6. Shell Programming and Scripting
Folks I need your help cuz I've a file with 100,000 records that need to be compared against a passwd file (300) and then create a third one with the data in the first one and the passwd from the second one set in it.
The format of the first file is:
host xxxxxx "" 0,0 Closed control00/... (4 Replies)
Discussion started by: ranrodrig
4 Replies
7. Shell Programming and Scripting
Hi All,
I am using the awk command to replace ',' by '\t' (tabs) in a csv file. I would like to apply this to all .csv files in a directory and create .txt files with the tabs.
How would I do this in a script?
I have the following script called "csvtabs":
awk 'BEGIN {
FS... (4 Replies)
Discussion started by: ScKaSx
4 Replies
8. Shell Programming and Scripting
Hi dear users,
I need to compare numeric columns in two files. These files have the following structure.
K.txt (4 columns)
A001 chr21 9805831 9846011
A002 chr21 9806202 9846263
A003 chr21 9887188 9988593
A003 chr21 9887188 ... (2 Replies)
Discussion started by: jcvivar
2 Replies
9. Shell Programming and Scripting
Hi,
I have a situation to compare one file, say file1.txt with a set of files in directory.The directory contains more than 100 files.
To be more precise, the requirement is to compare the first field of file1.txt with the first field in all the files in the directory.The files in the... (10 Replies)
Discussion started by: anandek
10 Replies
10. Shell Programming and Scripting
I have two files, these have to be selectively merged into two other files. In addition there will require to be a edit to the last field, where the date format is changed.
The first file is a csv file with around 300k lines the data is now nearly 20 years old and I have been asked to move this... (7 Replies)
Discussion started by: gull04
7 Replies
nfsiod(8) System Manager's Manual nfsiod(8)
NAME
nfsiod, biod - The local NFS compatible asynchronous I/O daemon
SYNOPSIS
nfsiod [ numthreads ]
DESCRIPTION
The nfsiod daemon runs on an NFS compatible client machine and spawns several IO threads to service asynchronous I/O requests to its
server. The I/O threads improve performance of both NFS reads and writes. Both try to enlist the aid of an idle I/O thread. If none is
available, the process itself issues the request to the server and waits for the reply.
The optimum number of I/O threads to run depends on many variables, such as how quickly the client will be writing, how many files will be
accessed simultaneously, and the behaviour of the NFS server. For use with a Tru64 UNIX server, 7 is a good number of I/O threads for most
systems.
When reading, if the client believes the process is reading a file sequentially, it requests an I/O thread to read a block ahead of what
the process is currently requesting. If the readahead completes before the process asks for that block, then the subsequent read system
call for that data completes immediately and does not have to wait for the NFS request to complete. Read ahead will be triggered again so
the read may find that next block available as well.
When writing a file, the client takes the process's data, passes the request to an I/O thread and immediately returns to the process. If
the process is writing data faster than the network or server can process, then eventually all the I/O threads become busy and the process
has to handle a NFS write itself. This means the process has to wait until the server finishes the write. For Tru64 UNIX servers, the NFS
block size is 8Kb and UFS tries to cluster I/O 64Kbs at a time. If the client is running with 7 I/O threads, 8 write requests can be in
progress at once. This allows the client and server to write data 64Kbs at a time and is the reason for recommending 7 I/O threads.
Unlike nfsd, each client thread can use either UDP or TCP. However, if TCP mounts are active, the nfsiod process will time out, close idle
TCP connections, and acknowledge any connections closed by the server.
The nfsiod process is also responsible for syncing the access time and modify times for special files and named pipes (fifos). Because I/O
to these files does not go through the NFS server, NFS clients have to directly update the access time and modify time attributes.
The client threads are implemented as kernel threads; they are part of Process ID 0, not the nfsiod process. The ps axml command displays
idle I/O threads under PID 0. Idle threads will be waiting on nfsiod_wait. Therefore, if 7 I/O threads are configured, only 1 nfsiod
process is displayed in the output from the ps command, although 7 client threads are available to handle NFS requests.
FILES
Specifies the command path Specifies the file for logging NFS activity.
RELATED INFORMATION
Commands: nfsd(8), nfsstat(8)
Daemons: async_daemon(2) delim off
nfsiod(8)