speed test +20,000 file existance checks too slow Post: 302267641

Sponsored Content

Top Forums UNIX for Advanced & Expert Users speed test +20,000 file existance checks too slow Post 302267641 by nullwhat on Saturday 13th of December 2008 03:01:53 AM

12-13-2008

Registered User

speed test +20,000 file existance checks too slow

Need to make a very fast file existence checker. Passing in 20-50K num of files

In the code below ${file} is a file with a listing of +20,000 files. test_speed is the script. I am commenting out the results of <time test_speed try>.

The normal "test -f" is much much too slow when a system call inside awk or perl. basic grep on +20,000 files is super fast, why does doing a file existence test slow it down so much.

Yes i am on try 55, and still i can not get this thing to go faster. I think try 55 would be very fast but i can not actauyly pass a file listing of +20,000 into a for loop becuase i run out of memory. anyone have any ideas on how to speed up a file check inside awk or perl or chell?

This would be fast if it actually worked

how can i pipe into pram $1 ?

awk '{print $10}' ${file} | if [ -f $1 ];then echo 1; else echo 0; fi

how can you pipe into an if statement?

Quote:

#!/bin/ksh

file=spySD.Dec10_aha~
u=aha2231

user=$USER

## No file existance test.

## time test_speed 1
## real 0m3.32s
## user 0m0.68s
## sys 0m0.19s

if [[ $1 = 1 ]];then
awk -v u=${u} '$5~u {print}' ${file} > /tmp/junk_${user}_f1
fi

## With existence test: Try 22

## time test_speed 22
##
## real 3h13m25.76s
## user 1h14m20.86s
## sys 52m23.13s

if [[ $1 = 22 ]];then

awk -v u=${u} '
$5~u {
sysA="if [[ -f " $10 " ]] ;then echo 1;else echo 0;fi"
sysA | getline chk
close(sysA)
if(chk=="1") {print}
}
' ${file} > /tmp/junk_${user}_f2

fi

## With existance test: Try 3
## This is slow too....

if [[ $1 = 3 ]];then

awk -v u=${u} '
$5~u {
sysA="ls " $10 " | grep -c " $10 " 2>/dev/null"
sysA | getline chk
close(sysA)
if(chk=="1") {print}
}
' ${file} > /tmp/junk_${user}_f3

fi

## With existence test: Try 55
if [[ $1 = 55 ]];then
for i in `awk '{print $10}' ${file}`
do

[ -f $i ] && echo 1 || echo 0

done > /tmp/junk_${user}_f55
fi

nullwhat

View Public Profile for nullwhat

Find all posts by nullwhat

6 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

network speed is slow

Hello, everyone: i encounter a problem these days , pls help me ,thanks in advance. my env: machine: ES40 A ES40 B os: true64 Unix 4.0f note: src.tar 8M network card speed 100M my problem: ...

2. News, Links, Events and Announcements

Intel Benchmark Test: Linux Goes to 600,000

For story: http://story.news.yahoo.com/news?tmpl=story&cid=75&ncid=738&e=9&u=/nf/20030606/tc_nf/21680

3. Shell Programming and Scripting

Test File Existance Remotely?

Thanks in advance to anyone that can help me answer this: I'm trying to write an if statement that will run test -f on whether a file exists on another server and if it does not then report that negative outcome to a log file. I'm thinking it should look something like this: if ; then rcp...

4. UNIX for Dummies Questions & Answers

Test existance of a file

Hi, I need to find out if a particular file exists and i am using if with -e option. Scenarion is like There is a possibility of two files having nomaincluture like below First file = abc20101028.somthing Second File = abc20101028.somthing.done I need to check abc20101028.somthing...

5. Shell Programming and Scripting

Slow Perl script: how to speed up?

I had written a perl script to compare two files: new and master and get the output of the first file i.e. the first file: words that are not in the master file STRUCTURE OF THE TWO FILES The first file is a series of names ramesh sushil jonga sudesh lugdi whereas the second file (could be...

6. Solaris

Rsync quite slow (using very little cpu): how to improve its speed?

I have "inherited" a OmniOS (illumos based) server. I noticed rsync is significantly slower in respect to my reference, FreeBSD 12-CURRENT, running on exactly same hardware. Using same hardware, same command with same source and target disks, OmniOS r151026 gives: test@omniosce:~# time...

LEARN ABOUT DEBIAN

liquidsoap

LIQUIDSOAP(1)						      General Commands Manual						     LIQUIDSOAP(1)

NAME

       liquidsoap - a multimedia streaming language

SYNOPSIS

       liquidsoap [options] [script|expression]

DESCRIPTION

       Liquidsoap  is  a  programming  language for describing multimedia streaming systems.  It is very flexible, making simple things simple but
       giving a lot of control for advanced uses.  Liquidsoap supports audio, video and MIDI streams, and a wide range of  input/output  operators
       including  Icecast and various soundcard APIs.  It can perform a broad range of signal processing, combine streams in various ways, support
       custom transitions, generate sound procedurally...  and all this can be assembled as you wish.  Input files can be  accessed  remotely,	or
       even be synthesized on the fly using external scripts such as speech synthesis.	Finally, interaction with a running liquidsoap instance is
       possible via telnet or socket.

       Liquidsoap scripts passed on the command line will be evaluated: they shall be used to define the streaming system to be ran.  It is possi-
       ble  to pass multiple scripts; they will all be ran successively, and definitions from one script can be used in subsequent ones.  A script
       will be read from standard input if - is given as script filename.  Information about scripting liquidsoap is  available  on  our  website:
       http://savonet.sf.net/

       If  the	parameter is not a file it will be treated as an expression which will be executed. It is a convenient way to test simple one-line
       scripts. When running only one-liners, the default is to log messages directly on stdout rather than to a file.

OPTIONS

       -
	       Read script from standard input.

       --     Stop parsing the command-line and pass subsequent items to the script.

       --debug
	      Print debugging log messages.

       --dynamic-plugins-dir path
	      Directory where to look for plugins.

       --errors-as-warnings
	      Issue warnings instead of fatal errors for unused variables and ignored expressions. If you are not sure about it, it is	better	to
	      not use it.

       --interactive
	      Start an interactive interpreter.

       --list-plugins
	      List all plugins (builtin scripting values, supported formats and protocols).

       --list-plugins-xml
	      List all plugins (builtin scripting values, supported formats and protocols), output as XML.

       --no-pervasives
	      Do not load pervasive script libraries.

       --version
	      Display liquidsoap's version.

       -c, --check
	      Check and evaluate scripts but do not perform any streaming.

       -cl, --check-lib
	      Like --check but treats all scripts and expressions as libraries, so that unused toplevel variables are not reported.

       -d, --daemon
	      Run in daemon mode.

       -f, --force-start
	      For advanced dynamic uses: force liquidsoap to start even when no active source is initially defined.

       -h plugin
	      Print the description of a plugin, eg. a builtin scripting function.

       -i     Display infered types.

       -p, --parse-only
	      Parse scripts but do not type-check and run them.

       -q, --quiet
	      Do not print log messages on standard output.

       -r filename
	      Process a request.

       -T, --disable-telnet
	      Disable the telnet server.

       -U, --disable-unix-socket
	      Disable the unix socket.

       -t, --enable-telnet
	      Enable the telnet server.

       -u, --enable-unix-socket
	      Enable the unix socket.

       -v, --verbose
	      Print log messages on standard output.

       --conf-descr-key key
	      Describe a configuration key.

       --conf-descr
	      Show all configuration keys with their documentation.

       --conf-descr-liqi
	      Show all configuration keys with their documentation in liqi (documentation wiki) format.

       --conf-dump
	      Dump the configuration state

       -help, --help
	      Display this list of options

SEE ALSO

       Our website http://savonet.sf.net/ and the HTML documentation coming with your distribution of liquidsoap.

AUTHOR

       Savonet team <savonet-users@lists.sourceforge.net>

Liquidsoap 1.0.1						   June 25, 2012						     LIQUIDSOAP(1)