Using UNIX Commands with Larger number of Files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Using UNIX Commands with Larger number of Files
# 8  
Old 11-09-2012
Thanks Scrutinizer and Don for the solutions. Also i am curious for each of these command ie mv , rm and cp can i use the FIND equivalent like this :
Code:
find  Inputpath -name '.pdf' -exec mv -f {} Outputpath   \;

Will this avoid the Large Number of files error too?

Thanks again.

Moderator's Comments:
Mod Comment Please use code tags next time for your code and data.
# 9  
Old 11-09-2012
Quote:
Originally Posted by pchegoor
Thanks Scrutinizer and Don for the solutions. Also i am curious for each of these command ie mv , rm and cp can i use the FIND equivalent like this :

find Inputpath -name '.pdf' -exec mv -f {} Outputpath \;

Will this avoid the Large Number of files error too?

Thanks again.
Almost. You need -name '*.pdf'; not -name '.pdf'. If you use:
Code:
find  Inputpath -name '*.pdf' -exec mv -f {} Outputpath +

(note the + instead of the \;), find will pass multiple file operands to each invocation of mv, which will reduce overhead and make it run faster.

Note also that if there are any subdirectories in Inputpath that contain PDF files, you'll need to recreate the directory structure under Outputpath before moving file PDF files themselves.

Last edited by Don Cragun; 11-09-2012 at 09:18 PM.. Reason: -name '*.pdf' NOT -name '.pdf'
# 10  
Old 11-09-2012
Thanks Don,
I can safely rewrite all the Unix Commands in my initial post using the FIND Command as follows to avoid the Large number of files error:

Code:
1) find  Inputpath -name '*.pdf' -exec mv -f {} Outputpath +

2 ) find  Inputpath -name '*.pdf' -exec cp  {}  Outputpath +

3) find Inputpath -name  '*.[Pp][Dd][Ff]' 

4) find Inputpath/* -exec  rm -rf  {}  +

In the 4th commnd above my aim is to delete all subdirectories and files within the Inputpath Directory but not delete the Inputpath directory itself.

Am I correct on all the above?

Moderator's Comments:
Mod Comment Please use code tags next time for your code and data.
# 11  
Old 11-09-2012
Quote:
Originally Posted by pchegoor
Thanks Don,
I can safely rewrite all the Unix Commands in my initial post using the FIND Command as follows to avoid the Large number of files error:

1) find Inputpath -name '*.pdf' -exec mv -f {} Outputpath +

2 ) find Inputpath -name '*.pdf' -exec cp {} Outputpath +

3) find Inputpath -name '*.[Pp][Dd][Ff]'

4) find Inputpath/* -exec rm -rf {} +

In the 4th commnd above my aim is to delete all subdirectories and files within the Inputpath Directory but not delete the Inputpath directory itself.

Am I correct on all the above?
No. #3 prints a lit of PDF files in Inputpath and its subdirectories when with mixed case spellings of PDF (PDF, PDf, PdF, Pdf, pDF, pDf, pdF, and pdf); the cp and mv in #1 and #2 will only act on PDF files ending in all lowercase letters (pdf). You need to decide whether you need case insensitive spellings of PDF and choose the appropriate pattern consistently based on your data.

It seems strange to me that you want to print the list of files you're working on after you have done #1 or #2.

This whole thread was started because you already know that #4 won't work; expanding Inputpath/* will create an argument list to long for the exec family of system calls to work. If there are no files in and under Inputpath other than PDF files, the easiest thing to do would be:
Code:
rm -rf Inputpath
mkdir Inputpath

# 12  
Old 11-09-2012
Hi Don when i use the Below i get the following Error:

Code:
find Inputpath -name '*.pdf' -exec mv -f {} Outputpath  + 

find: 0652-018 An expression term lacks a required parameter

Any reason why this is so?

Last edited by Scott; 11-12-2012 at 04:14 AM.. Reason: Code tags, please.
# 13  
Old 11-09-2012
Quote:
Originally Posted by pchegoor
Hi Don when i use the Below i get the following Error:

find: 0652-018 An expression term lacks a required parameter

find Inputpath -name '*.pdf' -exec mv -f {} Outputpath +

Any reason why this is so?
Yes. Sorry. You can only use "+" as the -exec primary terminator when the {} is the last argument being passed to the command to be executed. Doing this with find will require you to mv one file at a time (using \; instead of +) unless you create an intermediate script and exec it.
# 14  
Old 11-10-2012
May I advise against executing a script within a find command? This would include, for every file found, creating a shell to run the script, implying a huge overhead, esp. in this case with "Large Number of Files". Why not collect all filenames in a working file and then work on that?
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Help with Expect script for pulling log files size larger than 500Mb;

I am new at developing EXPECT scripts. I'm trying to create a script that will automatically connect to a several UNIX (sun solaris and HPUX) database server via FTP and pull the sizes of the listener/alert log files from specified server directory on the remote machines. 1. I want the script... (7 Replies)
Discussion started by: mikebantor
7 Replies

2. UNIX for Beginners Questions & Answers

Need to select files larger than 500Mb from servers

I need help modifying these two scripts to do the following: - print files in (MB) instead of (KB) - only select files larger than 500MB -> these will be mailed out daily - Select all files regardless of size all in (MB) -> these will be mailed out once a week this is what i have so far and... (5 Replies)
Discussion started by: donpasscal
5 Replies

3. UNIX for Dummies Questions & Answers

Split larger files into smaller ones with Column names

Hi, I have one large files of 100000 rows with header column. Eg: Emp Code, Emp Name 101,xxx 102,YYY 103,zzz ... ... I want to split the files into smaller files with only 30000 rows each..File 1,2 and 3 must have 30000 rows and file 4 must contain 10000 rows. But the column... (1 Reply)
Discussion started by: Nivas
1 Replies

4. Shell Programming and Scripting

Backingup larger files with TAR command

I need to backup my database but the files are very large and the TAR command will not let me. I searched aids and found that I could do something with the mknod, COMPRESS and TAR command using them together. I appreciate your help. (10 Replies)
Discussion started by: frizcala
10 Replies

5. UNIX for Dummies Questions & Answers

7z command for files larger than 4GB ( unzip doesn't work)

My unzip command doesn't work for files that are greater than 4GB. Consider my file name is unzip -p -a filename.zip, the command doesn't work since the size of the file is larger. I need to know the corresponding 7z command for the same. This is my Unix shell script program: if then ... (14 Replies)
Discussion started by: chandraprakash
14 Replies

6. Programming

Using basic UNIX commands to make/compile JAVA files

Hello! This is my first post, and I just learned what UNIX was this week. For a JAVA programming class I am taking, I must be able to create a directory in UNIX, use the nano command to create a JAVA program, compile it, and then run it on the command prompt using the java command. For some... (5 Replies)
Discussion started by: UNdvoItX
5 Replies

7. UNIX for Dummies Questions & Answers

unix commands related to ftp of files..

Hi, I am new to unix , I was planning to write a script that will FTP files to destination folder. , Please guide me what are the various networking commands that unix will help in this ftp process..?:confused: (1 Reply)
Discussion started by: rahul125
1 Replies

8. AIX

Tar files larger than 2GB

Hi, Does anyone know if it is possible to tar files larger than 2GB? The reason being is they want me to dump a single file (which is around 20GB) to a tape drive and they will restore it on a Solaris box. I know the tar have a limitation of 2GB so I am thinking of a way how to overcome this.... (11 Replies)
Discussion started by: depam
11 Replies

9. Shell Programming and Scripting

How to initialize array with a larger number?

Language: ksh OS: SunOS I have been getting the 'subscript out of range' error when the below array variable gets elements greater that 1024. I understand that 1024 is the default size for 'set -A' dynamic array, but is there a way to initialize it with a larger number? set -A arr `grep... (6 Replies)
Discussion started by: ChicagoBlues
6 Replies

10. UNIX for Advanced & Expert Users

sending larger files via ftp

hi all, i am looking for ways to make ftp efficient by tuning the parameters currently, tcp_max_buf is 1 MB tcp_xmit_hiwat is 48 KB say to transmit multiple 2 gb files from unix server to mainframe sys, will increasing the window size or the send buffer size of the current TCP/IP... (6 Replies)
Discussion started by: matrixmadhan
6 Replies
Login or Register to Ask a Question