Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

arg list too long error

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting arg list too long error
# 1  
Old 12-07-2010
arg list too long error
Hello,

I'm trying to search through 30,000 files in 1 directory, and am getting the "arg list too long" error. I've searched this forum and have been playing around with xargs and can't get that to work either. I'm using ksh on Solaris.

Here's my original code:
Code:
nawk "/Nov 21/{_=2}_&&_--" $LOGDIR/*.log > $NEWLOGFILE

Error:
Code:
ksh: /usr/bin/nawk: arg list too long

Any help is appreciated!
Last edited by Scott; 12-13-2010 at 03:48 PM.. Reason: Code tags
# 2  
Old 12-07-2010
One way (untested). I assume you want each hit to appear in ${NEWLOGFILE}.
The brackets "()" are important.

Code:
(
find "${LOGDIR}" -type f -name \*\.log -print | while read FILENAME
do
           nawk "/Nov 21/{_=2}_&&_--" "${FILENAME}"
done
) 2>&1 > "${NEWLOGFILE}"

# 3  
Old 12-07-2010
Code:
ls $LOGDIR/*.log |xargs nawk "/Nov 21/{_=2}_&&_--" > $NEWLOGFILE

# 4  
Old 12-09-2010
Thanks for your responses!

rdcwayx - I got the same error:
Code:
/usr/bin/ls: arg list too long

methyl - I got this to work, however it runs very slow. Is there a way this would work as a one liner and not in a loop? I don't know if that would speed it up though since it's looking through 30,000 files.
Code:
(
find "${LOGDIR}" -type f -name \*\.log -print | while read FILENAME
do
           nawk "/Nov 21/{_=2}_&&_--" "${FILENAME}"
done
) 2>&1 >> $NEWLOGFILE

Last edited by Scott; 12-13-2010 at 03:48 PM.. Reason: Code tags
# 5  
Old 12-09-2010
No.

It is serial - one file at a time

Code:
cnt=0
find "${LOGDIR}" -type f -name \*\.log -print | 
while read FILENAME
do
  nawk "/Nov 21/{_=2}_&&_--" "${FILENAME}" >> $NEWLOGFILE &
  cnt=$(( cnt + 1 ))
  if [ $(( $CNT % 10  )) -eq 0 ] ; then
   wait
  fi
done

This runs 10 files at one time.
# 6  
Old 12-09-2010
Quote:
Originally Posted by Kristin_in_CO
Thanks for your responses!

rdcwayx - I got the same error:
/usr/bin/ls: arg list too long
Try this:

Code:
echo "" > $NEWLOGFILE
find $LOGDIR -name "*.log" -type f |xargs nawk "/Nov 21/{_=2}_&&_--" >>$NEWLOGFILE

# 7  
Old 12-09-2010
If you have GNU coreutils and your are worried about any of these situations:

files with space in their names
find going into sub-diretories
awk still run once with no files on command line, when no .csv files exist

You can also do:

Code:
( find . -maxdepth 1 -name "*.csv" -print0 | xargs -r0 nawk "/Nov 21/{_=2}_&&_--"  ) > $NEWFILE

Last edited by Chubler_XL; 12-09-2010 at 10:07 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Arg list too long error while performing tar and zip operation

hi all i am trying to tar and then zip files present dir by using the below command tar -cvf ${abc}/xyz_backup_date_`date +%d%m%y%H%M%S`.tar xyz* when the files are in less number the above command executes perfectly but when there are large number of files i am getting "arg list too... (5 Replies)
Discussion started by: manoj
5 Replies

2. UNIX for Dummies Questions & Answers

Arg list too long

Hello All, I am trying to find a file name with .sh exention from a list of .dat files inside a directory. find /app/folder1/* -name '*.dat'| xargs grep '.sh' ksh: /usr/local/bin/find: arg list too long Please help me finding the command. Thanks (3 Replies)
Discussion started by: tkhan9
3 Replies

3. Shell Programming and Scripting

ls command - strange error - arg list too long

I am running a shell script which has the following command ls *.pdf | wc -l error: arg list too long Please post your thoughts on this.. (4 Replies)
Discussion started by: techmoris
4 Replies

4. Shell Programming and Scripting

arg list too long

Hi, Help. I have a file that contains a list of users in a file. I want to cat the content of the file and feed it into sed to a preformated report. The error I got is "ksh: /usr/bin/sed: arg list too long" My method below. A=`cat FILE1.txt` B=`echo $A` sed "s#USERLIST#$B#" FILE2 >... (2 Replies)
Discussion started by: Zenwork
2 Replies

5. UNIX for Dummies Questions & Answers

Arg List too Long in SCP

Hey guys. I have a program written in which i am trying to get the files from one remote machine and transferring the files to another remote machine using SCP. It works fine for 50 or 60 files but when the files grows to 250 then i get an error message stating "Arg list too long". #scp -p... (5 Replies)
Discussion started by: chris1234
5 Replies

6. UNIX for Dummies Questions & Answers

ls -t arg list too long

echo dirname/filename* | xargs ls -t As a substitute doesn't give the results desired when I exceed the buffer size. I still want the files listed in chronological order, unfortunately xargs releases the names piecemeal...does anyone have any ideas? :( (4 Replies)
Discussion started by: CSU_Ram
4 Replies

7. UNIX for Advanced & Expert Users

arg list too long

Does anyone have a solution for arg list too long error. I have got this from the web but I fail to make any sense out of it Thanks enc (8 Replies)
Discussion started by: encrypted
8 Replies

8. UNIX for Dummies Questions & Answers

zcat --> Arg list too long

Hi all I have more than 1000 files in a folder and when ever i use a "compress" or "zcat" command it give error /bin/zcat: Arg list too long. . any solution for this :o (3 Replies)
Discussion started by: muneebr
3 Replies

9. UNIX for Dummies Questions & Answers

egrep and Arg list too long

hi everyone, We have a heck of a lot of files in a particular directory and I need to search through all of them to find a list of all files containing particular text strings...one being a date and the other being the name of the report that is printed on the files..... I've tried the... (6 Replies)
Discussion started by: kingo
6 Replies

10. UNIX for Dummies Questions & Answers

arg list too long

I do ls -l ABC*, I get arg list too long message. This will not happen if ABC* has small no of files I believe 4000 files is limit. Any way of avoiding this. I even tried like this for i in `ls -l ABC*` do echo $i done Same problem. Any solution would be great. I am on HP-UX... (5 Replies)
Discussion started by: vingupta
5 Replies
Login or Register to Ask a Question