Shell Programming and Scripting

I have over 900 files that have the same name except for a unique numeric assignment. For all files I would like to cut the 2nd column and paste all into one new file. All in bash.
sample input format for each file:
1 2 3
1 2 3
1 2 3

sample command for what I want to do:
cut -d' ' -f2 file_in >> file_out

Ultimately I would like process the files through a loop.

Thanks in advance!!!
Edit/Delete Message

With that many files * might overflow the exec limits on command length...

Code:

cd /path/to/files
find . -type f  -name 'common_name_fragment*' | \
while read file
do
     cut -d' ' -f2 $file
done > ./tmp
mv ./tmp file_out

What's the temporary file for? Couldn't you just as well redirect to file_out directly?

And of course, if your shell can expand all files, the simple thing might work just fine:

Code:

cut -d ' ' -f2 * >file_out

If you get "argument list too long", you need a workaround like the find command which jim posted.

#!/bin/bash
for file in /home/epi/tmurray/reich_stuff/21
do cut -d' ' -f2 *.21 >> out.txt ; done

The above loop does the extraction but appends the column from each file vertically. How can I get it to append horizontally? e.g:
input:
file 1
1 2
1 2
1 2

file 2
1 2
1 2
1 2

current output
2
2
2
2
2
2

desired output:
2 2
2 2
2 2

thanks.

And Jim --that code you gave me doesn't work. I tweaked it and it still doesn't work. I keep getting errors about synthax


Thanks

Your "loop" only loops over the directory, once, then ignores the directory when actually doing the cut, but never mind.

cut has a friend paste which places stuff next to each other. For a large number of files, it might not be workable, though; you'd need a temporary file, or a temporary stream, for each file in the set.

Sounds to me like at this point you would be best served by a simple awk or perl script.

I don't see anything wrong with the syntax in jim's command; can you post the actual error message?

For a small number of files, this works, at least roughly, but it causes my Bash to dump core with a malloc error when I try it on a large number of files. I'm posting it mainly for its curiosity value.

Code:

eval paste "<("cut -d\" \" -f2 `echo *  | sed 's/ /) <(cut -d " " -f2 /g'`")"

This constructs a command line consisting of paste <(cut file1) <(cut file2) <(cut file3) via some rather black magic. The key is really the use of eval and the quoting and backslashes required to pull it off.

And here finally is a simple Perl one-liner (or to be really honest, two-liner) which collects an array for each input line number, and at the end prints out each array in line number order.

Code:

perl -ane 'push @{$L->[$.]}, $F[1]; close ARGV if eof;
END { shift @{$L}; for $l (@{$L}) { print join (" ", @{$l}), "\n"; } }' *

It's regrettable that you can't do this with just the basic Unix tools.

The close stuff is to reset the line numbering for each file (see perldoc -f eof) and the shift is to get rid of line number zero, which should be empty anyway. $F[1] is the second, space-padded input field, in case you want to change it to something else (Perl arrays are zero-based, so the first field is $F[0]). It's easy enough to make it split on a different separator if you like; see the -a and -F options. This uses references for efficiency (or simply because I'm past bedtime), so it's not particularly elegant or readable.