I have over 900 files that have the same name except for a unique numeric assignment. For all files I would like to cut the 2nd column and paste all into one new file. All in bash.
sample input format for each file:
1 2 3
1 2 3
1 2 3

sample command for what I want to do:
cut -d' ' -f2 file_in >> file_out

Ultimately I would like process the files through a loop.

Thanks in advance!!!
Edit/Delete Message

With that many files * might overflow the exec limits on command length...

What's the temporary file for? Couldn't you just as well redirect to file_out directly?

And of course, if your shell can expand all files, the simple thing might work just fine:

If you get "argument list too long", you need a workaround like the find command which jim posted.

#!/bin/bash
for file in /home/epi/tmurray/reich_stuff/21
do cut -d' ' -f2 *.21 >> out.txt ; done

The above loop does the extraction but appends the column from each file vertically. How can I get it to append horizontally? e.g:
input:
file 1
1 2
1 2
1 2

file 2
1 2
1 2
1 2

current output
2
2
2
2
2
2

desired output:
2 2
2 2
2 2

thanks.

And Jim --that code you gave me doesn't work. I tweaked it and it still doesn't work. I keep getting errors about synthax


Thanks

Your "loop" only loops over the directory, once, then ignores the directory when actually doing the cut, but never mind.

cut has a friend paste which places stuff next to each other. For a large number of files, it might not be workable, though; you'd need a temporary file, or a temporary stream, for each file in the set.

Sounds to me like at this point you would be best served by a simple awk or perl script.

I don't see anything wrong with the syntax in jim's command; can you post the actual error message?

For a small number of files, this works, at least roughly, but it causes my Bash to dump core with a malloc error when I try it on a large number of files. I'm posting it mainly for its curiosity value.

This constructs a command line consisting of paste <(cut file1) <(cut file2) <(cut file3) via some rather black magic. The key is really the use of eval and the quoting and backslashes required to pull it off.

And here finally is a simple Perl one-liner (or to be really honest, two-liner) which collects an array for each input line number, and at the end prints out each array in line number order.

It's regrettable that you can't do this with just the basic Unix tools.

The close stuff is to reset the line numbering for each file (see perldoc -f eof) and the shift is to get rid of line number zero, which should be empty anyway. $F[1] is the second, space-padded input field, in case you want to change it to something else (Perl arrays are zero-based, so the first field is $F[0]). It's easy enough to make it split on a different separator if you like; see the -a and -F options. This uses references for efficiency (or simply because I'm past bedtime), so it's not particularly elegant or readable.