Find command when there exist many files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Find command when there exist many files
# 1  
Old 02-17-2018
Find command when there exist many files

I want to run find and wondering if it struggles when there are many files. I have tried
and does not seem to complain.

Is this correct?
# 2  
Old 02-17-2018
It won't 'struggle' when there are many files.. it just might take longer to finish (depending on where it's looking). There's no chance of an 'arg list too long'-type error, as with, for example ls *, or similar commands, where the shell tries to expand *. find's often the best command to use in many situations where there are very many files involved.
# 3  
Old 02-17-2018
I'm pretty sure it would complain when running into difficulties. What "struggles" do you expect?
# 4  
Old 02-17-2018
Struggle

For example I cannot put all the files in a string when running a script as the amount of files is enormous. Also I cannot use a file display program such as Nautilus, the screen takes too long to load all the files.

I then need to run a program with each file as input to a program called mseed2sac and
may create multiple files for each file passed.

Now I want the script to give me a percentage so that user will know the progress. Ideally
I do not want that every calculation is printed on a new line. The following does not seen to
work well.

Code:
# Counts the number of files to process
flcnt=$(find . -type f | wc -l)
echo "flcnt: $flcnt" 

i=0
for f in $(find . -type f); do
  #echo "+ $f"
  i=$((i++))
  percent=$(awk "BEGIN { pc=100*${i}/${flcnt}; j=int(pc); print (pc-j<0.5)?j:j+1 }")
  echo "$percent"
  #${dir_mseed2sac}/mseed2sac $f
done


Last edited by kristinu; 02-17-2018 at 04:51 PM..
# 5  
Old 02-18-2018
The command:
Code:
  i=$((i++))

does the following things, in sequence:
  1. $((i++)) part 1: save the initial value of i,
  2. $((i++)) part 2: increment the value of i,
  3. $((i++)) part 1: return the initial value of i, and
  4. i=...: assign the returned value to i.
These four steps together assign the initial value of i to i discarding the increment that was performed in step 2.

To get what I think you're trying to do, try changing that line to one of the following:
Code:
  i=$((i+1))
  i=$((++i))
  ((i++))
  ((++i))
  ((i=i+1))

The first two of these will work with any POSIX-conforming shell. The last three will work in ksh, some versions of bash, and might also work in some other shells.

Furthermore, anytime you use find to get a list of files, you have to assume that any filename allowed by the underlying filesystem might pop us in find's output. You should write your code to assume that a pathname returned by find might contain whitespace or other problematic characters. So, a command like:
Code:
  #${dir_mseed2sac}/mseed2sac $f

(assuming that you intend to remove the # making it a comment at some later time, should instead be written as one of the following:
Code:
  #"${dir_mseed2sac}"/mseed2sac "$f"
  #"${dir_mseed2sac}/mseed2sac" "$f"

This User Gave Thanks to Don Cragun For This Post:
# 6  
Old 02-18-2018
Displaying percentage when it changes

I want to display the percentage when it changes by 3% and only print
then keeping the cursor on the same line.

Some suggestions would help me.

---------- Post updated at 07:03 AM ---------- Previous update was at 06:56 AM ----------

I have done it this way, which looks good

Code:
i=0
for f in $(find . -type f); do
  #echo "+ $f"
  i=$((i+1))
  percent=$(awk "BEGIN { pc=100*${i}/${flcnt}; j=int(pc); print (pc-j<0.5)?j:j+1 }")
  #echo "$percent"
  echo -e "\r${percent}%\c"
  #${dir_mseed2sac}/mseed2sac $f
done

---------- Post updated at 07:13 AM ---------- Previous update was at 07:03 AM ----------

The final thing I would like to do is have a progress bar enclosed by [ ]
and using stars to show the completion. The total number of stars should
be 60 for completion

Example:

Code:
[**********                                    ] 10%

Code:
i=0
nstars=0
for f in $(find . -type f); do
  #echo "+ $f"
  i=$((i+1))
  percent=$(awk "BEGIN { pc=100*${i}/${flcnt}; j=int(pc); print (pc-j<0.5)?j:j+1 }")
  #echo "$percent"
  echo -e "\r${percent}%\c"
  nstars=$((percent*0.6))
  echo -e "[*****                         ]"
  #${dir_mseed2sac}/mseed2sac $f
done

# 7  
Old 02-18-2018
How about - given you're using a recent bourne type shell (which you fail to mention) -
Code:
TOTFC=$(find . -type f | tee /tmp/WRK | wc -l)
while read FN
  do    printf "\r%4d %%" $((100 * ++i / TOTFC)) 
        #${dir_mseed2sac}/mseed2sac $FN
  done < /tmp/WRK
printf "\n"

EDIT: for the star bar, try
Code:
        printf -v XXX "%0*d" $((60 * ++i / TOTFC))
        printf "\r[%-60s]" "${XXX//0/*}"


Last edited by RudiC; 02-18-2018 at 11:54 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find command to find a word from list of files

I need to find a word '% Retail by State' in the folder /usr/sas/reports/RetailSalesTaxallocation. When I tried like below, -bash-4.1$ cd /usr/sas/reports/RetailSalesTaxallocation -bash-4.1$ find ./ -name % Retail by State find: paths must precede expression: Retail Usage: find ... (10 Replies)
Discussion started by: Ram Kumar_BE
10 Replies

2. Shell Programming and Scripting

Find if a directory exist from a list of the path

Hello, i want to script on sh to check from a path if the directory exist and isn't empty. I explain: path is : /aaa/bbb/ccc/ccc_name/ddd/ Where the cccc_name is in a list, so i think it's $1 My command find -name /aaa/bbb/ccc/$1/ddd/ didn't work because my $1 is the same and not... (5 Replies)
Discussion started by: steiner
5 Replies

3. UNIX for Dummies Questions & Answers

Command to find files

Hi All, Can anyone give me the command to copy files from 03-Mar-2013 to 07-Mar-2013 in folder. there are nearly 40+ thousand files in directory , so I just need files from Mar 3rd to Mar 7th and copy them to a location . Need quick help pls (2 Replies)
Discussion started by: rockingvj
2 Replies

4. Shell Programming and Scripting

find + move if destination path does not exist

hi frnds, please help ... what will happen with below command if destination path does not exist on the system.... find /var/adm/cft* -mtime +1 -exec mv {} /global/ \ in unix its remove all my files from the system from soruce file ... how is it possbile (1 Reply)
Discussion started by: dodasajan
1 Replies

5. Shell Programming and Scripting

what is the find to command to find the files created last 30 days

what is the find to command to find the files created last 30 days (5 Replies)
Discussion started by: rajkumar_g
5 Replies

6. Shell Programming and Scripting

Find out whether files exist.

I have the following data stored in a file. 1 /home/file13 /home/file2 2 /home/file41 /home/file654 3 /home/file61 /home/file45 4 /home/file81 /home/file43 ... I want to print the first column provided the files represented by the second and third column exist. How to do that? (3 Replies)
Discussion started by: kevintse
3 Replies

7. UNIX for Dummies Questions & Answers

How to find if file exist in directory using *

Hi, I know how to use the test command ( ...) to find a single given name file. However, I have a case in which I have a directory with one file and one sub-directory. I know that the file starts with "fub". The command doesn't work if i call the file "fub*" as it doesn't understand I meant a... (2 Replies)
Discussion started by: buj
2 Replies

8. Shell Programming and Scripting

Little bit weired : Find files in UNIX w/o using find or where command

Yes , I have to find a file in unix without using any find or where commands.Any pointers for the same would be very helpful as i am beginner in shell scritping and need a solution for the same. Thanks in advance. Regards Jatin Jain (10 Replies)
Discussion started by: jatin.jain
10 Replies

9. UNIX for Dummies Questions & Answers

Find in file. Does this exist under UNIX?

Hello, I have the following problem. I know there is a file somewhere on a UNIX machine that contains a string, but I don't know where. With the "grep" command, I can look into a file but only if I'm located in the correct directory. With the "find" command, I can search across directories... (2 Replies)
Discussion started by: scampsd
2 Replies

10. Shell Programming and Scripting

Can I find wether a particular file exist and size greater than zero...help please

Can I find wether a particular file exist and size greater than zero in one line command. similar to this if && something in one if test .... e.g. if 1.) is it possible ? ... if yes how 2.) what would be the return type in case there is success or failure. I mean if both are... (4 Replies)
Discussion started by: guhas
4 Replies
Login or Register to Ask a Question