Sponsored Content
Top Forums Shell Programming and Scripting Increase the performance of find command. Post 303041939 by jim mcnamara on Saturday 7th of December 2019 09:28:02 AM
Old 12-07-2019
This standard library call: nftw (or ftw)
IBM Knowledge Center

supports the find command traversing directory file trees - i.e., searching and locating files.

Assuming you want to keep the command you already have (and I am not sue that Rudi's suggested test is valid because of file and directory caching ):

A limiting factor is known to be the number of sub-directories in the file tree, and possibly the number of available open file descriptors - a per process limit.
If you can parallelize your code using several processes it may improve performance. I'm not sure this will help much because it depends on the number of sub-directories being large to gain any benefit. The developers who write system code try to maximize throughput.

What I'm saying is: performance enhancement work is subjective and often a misplaced resource and a waste of programmer time.
Suppose your command runs in one minute in production. Then you work hard and get it down to 35 seconds. The user perception of "slow" will still be there, so you have to get it down to maybe 6 seconds to make users happy and see it as "faster". In this case getting an order of magnitude improvement may not be possible.

And in this case you would have to do something about directory caching messing up testing because (you check this yourself) once you open a directory the system caches it for speedier access. Use the time command and rerun the command to see what I mean:
Code:
time [my long command goes here]
#write down the result
time [my long command goes here]
# write down the result and compare the two resulting times

This User Gave Thanks to jim mcnamara For This Post:
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

improve performance by using ls better than find

Hi , i'm searching for files over many Aix servers with rsh command using this request : find /dir1 -name '*.' -exec ls {} \; and then count them with "wc" but i would improve this search because it's too long and replace directly find with ls command but "ls *. " doesn't work. and... (3 Replies)
Discussion started by: Nicol
3 Replies

2. Solaris

What is the command to increase filesystem on solaris

I wanted to know what is the process or command to increase a filesystem on solaris. For example the /tmp directory. (3 Replies)
Discussion started by: strikelit
3 Replies

3. Shell Programming and Scripting

Increase Performance

I have written a code using AWK & sed to compare two files. The structure of the files is like this" Format is this: <bit code> <file code> <string> Follwoed by any numbers of properties lines whic start with a "space" 10101010101111101 XX abcd a AS sasa BS kkk 1110000101010110 XX... (1 Reply)
Discussion started by: sandeep_hi
1 Replies

4. Shell Programming and Scripting

Increase sed performance

I'm using sed to do find and replace. But since the file is huge and i have more than 1000 files to be searched, the script is taking a lot of time. Can somebody help me with a better sed command. Below is the details. Input: 1 1 2 3 3 4 5 5 Here I know the file is sorted. ... (4 Replies)
Discussion started by: gpaulose
4 Replies

5. Shell Programming and Scripting

SLEEP command performance

Hi, I wanted to run a particlar script for every 20 minutes. I dont have crontab in my server. Hence i ran this script in a loop by providing the command sleep 1200 Now i wanted to know is there any performance issue if this job keeps on execute in the server. Thanks, Puni (1 Reply)
Discussion started by: puni
1 Replies

6. Shell Programming and Scripting

Awk : find progressive increase in numbers

NR_037575 -0.155613339079513 -0.952655362767482 -1.42096466949375 -0.797042023687969 -1.26535133041424 -0.468309306726272 NR_037576 0.59124585320226 0.408702582537126 0.888885242203586 -0.182543270665134 0.297639389001326 0.480182659666459... (4 Replies)
Discussion started by: quincyjones
4 Replies

7. Shell Programming and Scripting

Performance issue while using find command

Hi, I have created a shell script for Server Log Automation Process. I have used find xargs grep command to search the string. for Example, find -name | xargs grep "816995225" > test.txt . Here my problem is, We have lot of records and we want to grep the string... (4 Replies)
Discussion started by: nanthagopal
4 Replies

8. Solaris

8 character limit for ipcs command , any way to increase # of chars ?

Hello All, We have a working script which identifies and kills ipcs resources which havent been correctly killed during normal shutdowns. It is working fine and dandy however there are some issues now. Environment: SunOS 5.10 Generic_148888-03 sun4u sparc SUNW,SPARC-Enterprise ... (4 Replies)
Discussion started by: icalderus
4 Replies

9. Shell Programming and Scripting

Increase command length for ksh shell on Redhat Linux

I have a ksh shell script and i need to pass arguments which are generated by data pulled from a database. When the argument to the shell script is too long (about 4000 charecters) the below is the issue observed. I copy the command which is 4000 charecters long from the logs and paste it... (7 Replies)
Discussion started by: mohtashims
7 Replies
cksum(1)						      General Commands Manual							  cksum(1)

NAME
cksum - Displays the checksum and byte count of a file SYNOPSIS
cksum [file...] STANDARDS
Interfaces documented on this reference page conform to industry standards as follows: cksum: XCU5.0 Refer to the standards(5) reference page for more information about industry standards and associated tags. OPTIONS
None OPERANDS
The path name of a file to be checked. If no file is specified, standard input is used. DESCRIPTION
The cksum command reads the files specified by the file argument and calculates a 32-bit checksum Cyclic Redundancy Check (CRC) and the byte count for each file. If no files are specified, standard input is read. The checksum, number of bytes, and file name are written to standard output. If standard input is used, no path name is printed. The cksum command can be used to compare a suspect file copied or communicated over noisy transmission lines against an exact copy of a trusted file. The comparison made by the cksum command may not be cryptographically secure; however, it is unlikely that an accidentally damaged file will produce the same checksum as the original file. [Tru64 UNIX] The cksum command uses a different algorithm than the sum command to calculate the 32-bit checksum CRC. The cksum command uses a CRC algorithm based on the Ethernet standard frame check. In addition, the sum block count is an octet count in cksum. [Tru64 UNIX] The IEEE Std 1003.2-1992 and X/Open CAE specifications require a slightly different method of calculating checksum values. To obtain these new standards, set the environment variable CMD_ENV to xpg4 as follows: export CMD_ENV=xpg4 [Tru64 UNIX] To maintain compatibility if the environment variable CMD_ENV is not set, the default action of the cksum command reports the traditional checksum values. The CRC checksum is obtained in the following way: The encoding is defined by the generating polynomial: G(x) = x^32 + x^26 + x^23 + x^22 + x^16 + x^12 + x^11 + x^10 + x^8 + x^7 + x^5 + x^4 + x^2 + x + 1 Mathematically, the CRC value corresponding to a given file is defined by the following procedure: The n bits to be evaluated are consid- ered to be the coefficients of a mod 2 polynomial M(x) of degree n-1. These n bits are the bits from the file, with the most significant bit being the most significant bit of the first octet of the file and the last bit being the least significant bit of the last octet, padded with zero bits (if necessary) to achieve an integral number of octets, followed by one or more octets representing the length of the file as a binary value, least significant octet first. The smallest number of octets capable of representing this integer is used. M(x) is multiplied by x^32 (that is, shifted left 32 bits) and divided by G(x) using mod 2 division, producing a remainder R(x) of degree less than or equal to 31. The coefficients of R(x) are considered to be a 32-bit sequence. The bit sequence is complemented and the result is the CRC. EXIT STATUS
The following exit values are returned: All files were processed successfully. An error occurred. EXAMPLES
To display the checksum and the size, in bytes, of file1 and file2, enter: cksum file1 file2 3995432187 1390 file1 3266927833 20912 file2 This output shows that the checksum of the file1 file is 3995432187 and contains 1390 bytes, and that the checksum of the file2 file is 3266927833 and contains 20912 bytes. ENVIRONMENT VARIABLES
The following environment variables affect the execution of cksum: [Tru64 UNIX] If this variable is set to xpg4, the method of calculating checksum values specified by IEEE Std 1003.2-1992 (POSIX.2) and X/Open CAE specifications is used. Provides a default value for the inter- nationalization variables that are unset or null. If LANG is unset or null, the corresponding value from the default locale is used. If any of the internationalization variables contain an invalid setting, the utility behaves as if none of the variables had been defined. If set to a non-empty string value, overrides the values of all the other internationalization variables. Determines the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multibyte characters in arguments). Determines the locale for the format and contents of diagnostic messages written to standard error. Determines the location of message catalogues for the processing of LC_MESSAGES. SEE ALSO
Commands: sum(1), wc(1) Standards: standards(5) cksum(1)
All times are GMT -4. The time now is 07:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy