Using UNIX Commands with Larger number of Files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Using UNIX Commands with Larger number of Files
# 1  
Old 11-09-2012
Hammer & Screwdriver Using UNIX Commands with Larger number of Files

Hello Unix Gurus,
I am new to Unix so need some help on this.

I am using the following commands:

1) mv -f Inputpath/*.[Pp][Dd][Ff] outputpath

2) cp Inputpath/*.[Pp][Dd][Ff] outputpath


3) rm -rf somepath/*

4) Find Inputpath/*.[Pp][Dd][Ff]

Now I get the following error with the Find command above :

Code:
/bin/ksh: /usr/bin/find: 0403-027 The parameter list is too long

This is happening because there are large number of files which the Find command has to process.

I am also thinking i may get the same type of error with other unix commands above.

So my question is How do i rewrite each of these commands to overcome this issue with large amount of files.


Any Help on this is Appreciated.

Thanks,

Last edited by Scrutinizer; 11-09-2012 at 04:41 PM.. Reason: code tags
# 2  
Old 11-09-2012
This can be avoided with a for loop. For example:
Code:
for file in Inputpath/*.[Pp][Dd][Ff]
do
  cp "$file" outputpath
done

# 3  
Old 11-09-2012
Hi Scrutinizer ,
Thanks for your Reply.Actually is ther a way do rewrite each of the commands i mentioned without using a FOR LOOP.I use these commands in a SAS Program so i require to use them as a single line command.Is it possible to use FIND command with some options to rewrite each of the commands and also overcome the issue with large number of files.

Thanks,
# 4  
Old 11-09-2012
Single line? Like so?
Code:
for file in Inputpath/*.[Pp][Dd][Ff]; do cp "$file" outputpath; done

# 5  
Old 11-09-2012
Quote:
Originally Posted by Scrutinizer
This can be avoided with a for loop. For example:
Code:
for file in Inputpath/*.[Pp][Dd][Ff]
do
  cp "$file" outputpath
done

Given the large number of files being copied, moved, ..., I would try the following first:
Code:
cd Inputpath;cp *.[Pp][Dd][Ff] ../outputpath;cd ..

Reducing the length of the filename arguments may be enough to avoid the {ARG_MAX} limits while keeping the number of invocations of cp (or the other utilities) to a minimum.

Last edited by Don Cragun; 11-09-2012 at 05:17 PM.. Reason: Converted to single-line command...
# 6  
Old 11-09-2012
Hi, yes that was my thought as well, but I figured I'd provide a solution that should work in any case...
# 7  
Old 11-09-2012
Actually, for find the solution is much simpler. Rather than using:
Code:
find Inputpath/*.[Pp][Dd][Ff] ...

use:
Code:
find Inputpath -name '*.[Pp][Dd][Ff]' ...

Rooting a find search in a very long list of regular files (instead of in a list of a small number of directories) frequently defeats the purpose of using find in the first place.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Help with Expect script for pulling log files size larger than 500Mb;

I am new at developing EXPECT scripts. I'm trying to create a script that will automatically connect to a several UNIX (sun solaris and HPUX) database server via FTP and pull the sizes of the listener/alert log files from specified server directory on the remote machines. 1. I want the script... (7 Replies)
Discussion started by: mikebantor
7 Replies

2. UNIX for Beginners Questions & Answers

Need to select files larger than 500Mb from servers

I need help modifying these two scripts to do the following: - print files in (MB) instead of (KB) - only select files larger than 500MB -> these will be mailed out daily - Select all files regardless of size all in (MB) -> these will be mailed out once a week this is what i have so far and... (5 Replies)
Discussion started by: donpasscal
5 Replies

3. UNIX for Dummies Questions & Answers

Split larger files into smaller ones with Column names

Hi, I have one large files of 100000 rows with header column. Eg: Emp Code, Emp Name 101,xxx 102,YYY 103,zzz ... ... I want to split the files into smaller files with only 30000 rows each..File 1,2 and 3 must have 30000 rows and file 4 must contain 10000 rows. But the column... (1 Reply)
Discussion started by: Nivas
1 Replies

4. Shell Programming and Scripting

Backingup larger files with TAR command

I need to backup my database but the files are very large and the TAR command will not let me. I searched aids and found that I could do something with the mknod, COMPRESS and TAR command using them together. I appreciate your help. (10 Replies)
Discussion started by: frizcala
10 Replies

5. UNIX for Dummies Questions & Answers

7z command for files larger than 4GB ( unzip doesn't work)

My unzip command doesn't work for files that are greater than 4GB. Consider my file name is unzip -p -a filename.zip, the command doesn't work since the size of the file is larger. I need to know the corresponding 7z command for the same. This is my Unix shell script program: if then ... (14 Replies)
Discussion started by: chandraprakash
14 Replies

6. Programming

Using basic UNIX commands to make/compile JAVA files

Hello! This is my first post, and I just learned what UNIX was this week. For a JAVA programming class I am taking, I must be able to create a directory in UNIX, use the nano command to create a JAVA program, compile it, and then run it on the command prompt using the java command. For some... (5 Replies)
Discussion started by: UNdvoItX
5 Replies

7. UNIX for Dummies Questions & Answers

unix commands related to ftp of files..

Hi, I am new to unix , I was planning to write a script that will FTP files to destination folder. , Please guide me what are the various networking commands that unix will help in this ftp process..?:confused: (1 Reply)
Discussion started by: rahul125
1 Replies

8. AIX

Tar files larger than 2GB

Hi, Does anyone know if it is possible to tar files larger than 2GB? The reason being is they want me to dump a single file (which is around 20GB) to a tape drive and they will restore it on a Solaris box. I know the tar have a limitation of 2GB so I am thinking of a way how to overcome this.... (11 Replies)
Discussion started by: depam
11 Replies

9. Shell Programming and Scripting

How to initialize array with a larger number?

Language: ksh OS: SunOS I have been getting the 'subscript out of range' error when the below array variable gets elements greater that 1024. I understand that 1024 is the default size for 'set -A' dynamic array, but is there a way to initialize it with a larger number? set -A arr `grep... (6 Replies)
Discussion started by: ChicagoBlues
6 Replies

10. UNIX for Advanced & Expert Users

sending larger files via ftp

hi all, i am looking for ways to make ftp efficient by tuning the parameters currently, tcp_max_buf is 1 MB tcp_xmit_hiwat is 48 KB say to transmit multiple 2 gb files from unix server to mainframe sys, will increasing the window size or the send buffer size of the current TCP/IP... (6 Replies)
Discussion started by: matrixmadhan
6 Replies
Login or Register to Ask a Question