Shell Programming and Scripting

I use the "cat" command to cancaterate files into a new file.

Sometimes the cat command failed because there were too many files in the directory for cancateration.

I also tried to use the "ls" command to put the list of files into a text file and then cancaterate each file one by one by reading the text file. However when there are too many files, it failed the "ls" command as well.

What other method can I use?

Regards
Lena

The way to avoid too many arguments is not cramming the too many arguments into ls in backticks, or brackets, or anything else. Too many arguments is too many arguments no matter how you do it. It's an operating system limit.

The way to avoid too many arguments is to not use too many arguments. Often this means putting them through a pipe instead of putting them directly on the commandline. You had the right idea using ls, it is able to list them all, but just putting the output of ls into arguments gets you right back where you started. It's too many arguments no matter where they come from.

The xargs command is also useful, as it will convert arguments from a pipe into arguments on a commandline, running the command you ask it to with arguments limited to the maximum number the OS allows. If there are too many, it will run the program several times.

Code:

ls | xargs -I{} cat > /path/to/destination

If for example, the maximum number of arguments was 4 and it had 10 files, it'd run "cat file1 file2 file3 file4", then "cat file5 file6 file7 file8", then "cat file9 file10". All the output of those would go into the same destination file, as if you'd run "cat file1 file2 file3 file4 file5 file6 file7 file8 file9 file10".

The real limit is much higher than 10 but hopefully that gets the idea across.

You can also try something like this:

Code:

find . -type f -name "*" -exec cat >> newfile {} \;

-name "*" is redundant. It matches everything, which find does by default anyway.

You should also put >> newfile at the end of the line, after {} \;

On Linux, you can do '+' instead of \; which will cause find to act like xargs, putting as many arguments into one cat as possible.

Thanks for all the help. I shall certainly try them and see if it works.

The only really safe way is to run cat once for every file. Also by avoiding "ls" we can deal with filenames containing space characters.

Code:

# Initialise concatonated file (which is not in this directory)
concatonated="/tmp/concatonated_file"
> "${concatonated}"
# Append each file one-by-one
find . -type f -print | while read filename
do
         cat "${filename}" >> "${concatonated}"
done

How is this better than the the solution with 'find', which also runs cat for each individual file?