speed test +20,000 file existance checks too slow Post: 302267641

Sponsored Content

Top Forums UNIX for Advanced & Expert Users speed test +20,000 file existance checks too slow Post 302267641 by nullwhat on Saturday 13th of December 2008 03:01:53 AM

12-13-2008

Registered User

speed test +20,000 file existance checks too slow

Need to make a very fast file existence checker. Passing in 20-50K num of files

In the code below ${file} is a file with a listing of +20,000 files. test_speed is the script. I am commenting out the results of <time test_speed try>.

The normal "test -f" is much much too slow when a system call inside awk or perl. basic grep on +20,000 files is super fast, why does doing a file existence test slow it down so much.

Yes i am on try 55, and still i can not get this thing to go faster. I think try 55 would be very fast but i can not actauyly pass a file listing of +20,000 into a for loop becuase i run out of memory. anyone have any ideas on how to speed up a file check inside awk or perl or chell?

This would be fast if it actually worked

how can i pipe into pram $1 ?

awk '{print $10}' ${file} | if [ -f $1 ];then echo 1; else echo 0; fi

how can you pipe into an if statement?

Quote:

#!/bin/ksh

file=spySD.Dec10_aha~
u=aha2231

user=$USER

## No file existance test.

## time test_speed 1
## real 0m3.32s
## user 0m0.68s
## sys 0m0.19s

if [[ $1 = 1 ]];then
awk -v u=${u} '$5~u {print}' ${file} > /tmp/junk_${user}_f1
fi

## With existence test: Try 22

## time test_speed 22
##
## real 3h13m25.76s
## user 1h14m20.86s
## sys 52m23.13s

if [[ $1 = 22 ]];then

awk -v u=${u} '
$5~u {
sysA="if [[ -f " $10 " ]] ;then echo 1;else echo 0;fi"
sysA | getline chk
close(sysA)
if(chk=="1") {print}
}
' ${file} > /tmp/junk_${user}_f2

fi

## With existance test: Try 3
## This is slow too....

if [[ $1 = 3 ]];then

awk -v u=${u} '
$5~u {
sysA="ls " $10 " | grep -c " $10 " 2>/dev/null"
sysA | getline chk
close(sysA)
if(chk=="1") {print}
}
' ${file} > /tmp/junk_${user}_f3

fi

## With existence test: Try 55
if [[ $1 = 55 ]];then
for i in `awk '{print $10}' ${file}`
do

[ -f $i ] && echo 1 || echo 0

done > /tmp/junk_${user}_f55
fi

nullwhat

View Public Profile for nullwhat

Find all posts by nullwhat

6 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

network speed is slow

Hello, everyone: i encounter a problem these days , pls help me ,thanks in advance. my env: machine: ES40 A ES40 B os: true64 Unix 4.0f note: src.tar 8M network card speed 100M my problem: ...

2. News, Links, Events and Announcements

Intel Benchmark Test: Linux Goes to 600,000

For story: http://story.news.yahoo.com/news?tmpl=story&cid=75&ncid=738&e=9&u=/nf/20030606/tc_nf/21680

3. Shell Programming and Scripting

Test File Existance Remotely?

Thanks in advance to anyone that can help me answer this: I'm trying to write an if statement that will run test -f on whether a file exists on another server and if it does not then report that negative outcome to a log file. I'm thinking it should look something like this: if ; then rcp...

4. UNIX for Dummies Questions & Answers

Test existance of a file

Hi, I need to find out if a particular file exists and i am using if with -e option. Scenarion is like There is a possibility of two files having nomaincluture like below First file = abc20101028.somthing Second File = abc20101028.somthing.done I need to check abc20101028.somthing...

5. Shell Programming and Scripting

Slow Perl script: how to speed up?

I had written a perl script to compare two files: new and master and get the output of the first file i.e. the first file: words that are not in the master file STRUCTURE OF THE TWO FILES The first file is a series of names ramesh sushil jonga sudesh lugdi whereas the second file (could be...

6. Solaris

Rsync quite slow (using very little cpu): how to improve its speed?

I have "inherited" a OmniOS (illumos based) server. I noticed rsync is significantly slower in respect to my reference, FreeBSD 12-CURRENT, running on exactly same hardware. Using same hardware, same command with same source and target disks, OmniOS r151026 gives: test@omniosce:~# time...

LEARN ABOUT MOJAVE

jrunscript

jrunscript(1)						      General Commands Manual						     jrunscript(1)

NAME

       jrunscript - command line script shell

SYNOPSIS

       jrunscript [ options ] [ arguments... ]

PARAMETERS

       options
	  Options, if used, should follow immediately after the command name.

       arguments
	  Arguments, if used, should follow immediately after options or command name.

DESCRIPTION

       jrunscript  is  a command line script shell. jrunscript supports both an interactive (read-eval-print) mode and a batch (-f option) mode of
       script execution. This is a scripting language independent shell. By default, JavaScript is the language used, but the  -l  option  can	be
       used  to  specify  a  different	language.  Through Java to scripting language communication, jrunscript supports "exploratory programming"
       style.

       NOTE: This tool is experimental and may not be available in future versions of the JDK.

OPTIONS

       -classpath path
	  Specify where to find the user's .class files that are accessed by the script.

       -cp path
	  This is a synonym for -classpath path

       -Dname=value
	  Set a Java system property.

       -Jflag
	  Pass flag directly to the Java virtual machine on which jrunscript is run.

       -l language
	  Use the specified scripting language. By default, JavaScript is used. Note that to use other scripting languages, you also need to spec-
	  ify the corresponding script engine's jar file using -cp or -classpath option.

       -e script
	  Evaluate the given script. This option can be used to run "one liner" scripts specified completely on the command line.

       -encoding encoding
	  Specify the character encoding used while reading script files.

       -f script-file
	  Evaluate the given script file (batch mode).

       -f -
	  Read and evaluate a script from standard input (interactive mode).

       -help
	  Output help message and exit.

       -?
	  Output help message and exit.

       -q
	  List all script engines available and exit.

ARGUMENTS

       If  [arguments...] are present and if no -e or -f option is used, then the first argument is the script file and the rest of the arguments,
       if any, are passed as script arguments. If [arguments..] and -e or -f option are used, then all [arguments..] are passed  as  script  argu-
       ments.  If [arguments..], -e and -f are missing, interactive mode is used. Script arguments are available to a script in an engine variable
       named "arguments" of type String array.

EXAMPLES

       Executing inline scripts

	  jrunscript -e "print('hello world')"
	  jrunscript -e "cat('http://java.sun.com')"

       Use specified language and evaluate given script file

	  jrunscript -l js -f test.js

       Interactive mode

	  jrunscript
	  js>print('hello world');
	  hello world
	  js>34 + 55
	  89
	  js> thread(function() { print('hello world'); }
	  hello world
	  js>

       Run script file with script arguments

	  jrunscript test.js arg1 arg2 arg3

       test.js is script file to execute and arg1, arg2 and arg3 are passed to script as script arguments. Script can access  these  using  "argu-
       ments" array.

SEE ALSO

       If JavaScript is used, then before evaluating any user defined script, jrunscript initializes certain built-in functions and objects. These
       JavaScript built-ins are documented in jsdocs.

								    06 Aug 2006 						     jrunscript(1)