Sponsored Content
Top Forums Shell Programming and Scripting Finiding Files with Perl or awk? Post 302944476 by Corona688 on Tuesday 19th of May 2015 03:04:53 PM
Old 05-19-2015
Quote:
Originally Posted by Michael Stora
Even a 2-3x improvement would help tramendously.
Computers do not work that way.

A slow system call is slow in any language, and the slower it is, the less there is to be gained by 'optimizing' it.

Suppose your program is spending 98% of its time waiting for NFS and 2% of its time actually running. If you find a 200% faster proram, it will be spending 1% of its time actually running and 99% of its time waiting on NFS with theoretical a speed gain of 1% and a realistic speed gain of absolutely zip.

This is also why you can't turbo charge a slow disk with a fast program. No matter how fast your program is, the underlying I/O can't actually move faster.

You might be able to parallelize it, but only to a point.

Perhaps your network connection or NFS can be fine-tuned? That's beyond my expertise, though.

P.S. The find command is not 'known to be slow', certainly not slower than any other file tree walker I know. If you don't understand why it's 'slow' when used on huge file trees, you don't actually know what it's doing.

Last edited by Corona688; 05-19-2015 at 04:12 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl or awk/egrep from big files??

Hi experts. In one thread i have asked you how to grep the string from the below sample file- Unfortunately the script did not gave proper output (it missed many strings). It happened may be i did gave you the proper contents of the file That was the script- "$ perl -00nle'print join... (13 Replies)
Discussion started by: thepurple
13 Replies

2. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

3. Shell Programming and Scripting

perl script for listing files and mailing the all files

Hi, I am new to perl: I need to write perl script to list all the files present in directory and mail should be come to my inbox with all the files present in that directory. advanced thanks for valuable inputs. Thanks Prakash GR (1 Reply)
Discussion started by: prakash.gr
1 Replies

4. Shell Programming and Scripting

Finiding filenames with specific index string

Hi All, I have a file (Names.txt) and the contents of the file is give below. $ cat Names.txt FF313207008.txt FF223207007.txt FF143207006.txt FF372150600.txt FF063407005.txt FF063307005.txt $ From these given file names I want to find the files which has the 6th index value as 2. So... (5 Replies)
Discussion started by: krish_indus
5 Replies

5. Shell Programming and Scripting

Comparison and editing of files using awk.(And also a possible bug in awk for loop?)

I have two files which I would like to compare and then manipulate in a way. File1: pictures.txt 1.1 1.3 dance.txt 1.2 1.4 treehouse.txt 1.3 1.5 File2: pictures.txt 1.5 ref2313 1.4 ref2345 1.3 ref5432 1.2 ref4244 dance.txt 1.6 ref2342 1.5 ref2352 1.4 ref0695 1.3 ref5738 1.2... (1 Reply)
Discussion started by: linuxkid
1 Replies

6. Shell Programming and Scripting

Compare two files and set a third one using awk or perl

Folks I need your help cuz I've a file with 100,000 records that need to be compared against a passwd file (300) and then create a third one with the data in the first one and the passwd from the second one set in it. The format of the first file is: host xxxxxx "" 0,0 Closed control00/... (4 Replies)
Discussion started by: ranrodrig
4 Replies

7. Shell Programming and Scripting

Apply 'awk' to all files in a directory or individual files from a command line

Hi All, I am using the awk command to replace ',' by '\t' (tabs) in a csv file. I would like to apply this to all .csv files in a directory and create .txt files with the tabs. How would I do this in a script? I have the following script called "csvtabs": awk 'BEGIN { FS... (4 Replies)
Discussion started by: ScKaSx
4 Replies

8. Shell Programming and Scripting

Compare intervals (columns) from two files (awk, grep, Perl?)

Hi dear users, I need to compare numeric columns in two files. These files have the following structure. K.txt (4 columns) A001 chr21 9805831 9846011 A002 chr21 9806202 9846263 A003 chr21 9887188 9988593 A003 chr21 9887188 ... (2 Replies)
Discussion started by: jcvivar
2 Replies

9. Shell Programming and Scripting

awk command to compare a file with set of files in a directory using 'awk'

Hi, I have a situation to compare one file, say file1.txt with a set of files in directory.The directory contains more than 100 files. To be more precise, the requirement is to compare the first field of file1.txt with the first field in all the files in the directory.The files in the... (10 Replies)
Discussion started by: anandek
10 Replies

10. Shell Programming and Scripting

Awk or Perl - to selectively merge two files.

I have two files, these have to be selectively merged into two other files. In addition there will require to be a edit to the last field, where the date format is changed. The first file is a csv file with around 300k lines the data is now nearly 20 years old and I have been asked to move this... (7 Replies)
Discussion started by: gull04
7 Replies
RPIECE(1)						      General Commands Manual							 RPIECE(1)

NAME
rpiece - render pieces of a RADIANCE picture SYNOPSIS
rpiece [ -v ][ -x xres ][ -y yres ][ -X xdiv ][ -Y ydiv ][ -F|R syncfile ][ -T timelim ] [ $EVAR ] [ @file ] [ rpict options ] -o picture octree DESCRIPTION
Rpiece renders a RADIANCE picture a piece at a time, calling rpict(1) to do the actual work. This is useful for running multiple rpict processes on cooperating machines to render a single picture, which is a shared file specified with the -o option. The overall picture dimensions will be xres by yres (or smaller, depending on the -pa option and other view options), and the picture will be rendered in xdiv by ydiv pieces. There are two basic methods for telling rpiece which piece(s) of a picture to render. The explicit method is to write on the standard input the X and Y position of the desired piece(s), where X runs from zero to xdiv-1 and Y runs from zero to ydiv-1. (The lower left piece of a picture corresponds to (0,0) in this system.) Alternatively, the implicit specification method uses a synchronization file to deter- mine which piece is to be rendered next. Specified with the -F option, syncfile initially contains the values for xdiv and ydiv, so the -X and -Y options are unnecessary. (However, they are used if syncfile does not exist.) The first rpiece process puts a lock on syncfile and modifies its contents before starting work on the first piece of the image. It writes the X and Y position of the piece it will work on, so the next rpiece process to modify syncfile will start on the next piece. (When it finishes with its piece, it appends the index to the end of syncfile.) This procedure continues until all the pieces are done, at which point all of the rpiece processes will terminate. The -R option may be used instead of -F if some of the pieces were not properly finished by previous (killed) runs of rpiece. This option should be used by at most one rpiece process, which must be started first and with no other rpiece processes running or else it will reren- der the same pieces other processes have begun. Once the recover process is started, you may start other rpiece processes using the -F option to run simultaneously. If some processes die during execution, leaving one or more half-finished pieces in the picture even though the other processes think the work is done, you may run a single rpiece with the -R option by itself to repair the holes. The -v flag switches on verbose mode, where rpiece reports to the standard output after each piece begins and after each piece is finished. Options may be given on the command line and/or read from the environment and/or read from a file. A command argument beginning with a dollar sign ('$') is immediately replaced by the contents of the given environment variable. A command argument beginning with an at sign ('@') is immediately replaced by the contents of the given file. EXAMPLE
First rpiece process is started on the machine "goober": goober% echo 1 8 > syncfile goober% echo -F syncfile -x 1024 -y 1024 -vf view -o picture octree > args goober% rpiece @args & Second rpiece processes is started on the machine "sucker": sucker% rpiece @args & NOTES
Due to NFS file buffering, the network lock manager is employed to guarantee consistency in the output file even though non-overlapping writes are used. This would tend to slow the process down if rpiece were to wait for this I/O to complete before starting on the next piece, so rpiece forks separate processes to hang around waiting for I/O completion. The number of processes thus designated is set by the MAXFORK macro in the program (compiled in the src/util directory). If the fork call is slow on a system, it may actually be better to set MAXFORK to zero. In other cases, the network lock manager may be so slow that this value should be increased to get the best utilization. The output picture is not run-length encoded, and can be quite large. The approximate size (in kilobytes) can be computed by the simple formula: filesize = xres*yres/256 Make sure that there is enough space on the filesystem to hold the entire picture before beginning. Once the picture is finished, the ra_rgbe(1) program with the -r option may be used to convert to a run-length encoded picture for more efficient storage, although pfilt(1) or any of the other Radiance picture filters will do the same thing. The ALRM signal may be used to gracefully terminate an rpiece process after it finishes the current piece. This permits other currently running or subsequently started rpiece process(es) to continue rendering the picture without loss. The -T option will send the ALRM signal to rpiece after the specified number of (decimal) hours. This is the best way to force a time limit on the computation, since information will not be lost, though the process may continue for some time afterwards to finish its current piece. BUGS
This program may not work on some systems whose NFS lock manager is unreliable. In particular, some System V derivative UNIX systems often have problems with the network lock manager. If the output is scrambled or rpict aborts with some ambient file related problem, you should just remove the ambient file and go back to normal rendering. AUTHOR
Greg Ward SEE ALSO
getinfo(1), pfilt(1), ra_rgbe(1), rpict(1), ximage(1) RADIANCE
10/1/98 RPIECE(1)
All times are GMT -4. The time now is 04:13 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy