I have over 900 files that have the same name except for a unique numeric assignment. For all files I would like to cut the 2nd column and paste all into one new file. All in bash.
sample input format for each file:
1 2 3
1 2 3
1 2 3
sample command for what I want to do:
cut -d' ' -f2 file_in >> file_out
Ultimately I would like process the files through a loop.
#!/bin/bash
for file in /home/epi/tmurray/reich_stuff/21
do cut -d' ' -f2 *.21 >> out.txt ; done
The above loop does the extraction but appends the column from each file vertically. How can I get it to append horizontally? e.g:
input:
file 1
1 2
1 2
1 2
file 2
1 2
1 2
1 2
current output
2
2
2
2
2
2
desired output:
2 2
2 2
2 2
thanks.
And Jim --that code you gave me doesn't work. I tweaked it and it still doesn't work. I keep getting errors about synthax
Your "loop" only loops over the directory, once, then ignores the directory when actually doing the cut, but never mind.
cut has a friend paste which places stuff next to each other. For a large number of files, it might not be workable, though; you'd need a temporary file, or a temporary stream, for each file in the set.
Sounds to me like at this point you would be best served by a simple awk or perl script.
I don't see anything wrong with the syntax in jim's command; can you post the actual error message?
For a small number of files, this works, at least roughly, but it causes my Bash to dump core with a malloc error when I try it on a large number of files. I'm posting it mainly for its curiosity value.
This constructs a command line consisting of paste <(cut file1) <(cut file2) <(cut file3) via some rather black magic. The key is really the use of eval and the quoting and backslashes required to pull it off.
Last edited by era; 05-12-2008 at 05:25 PM..
Reason: Belated typo fix
And here finally is a simple Perl one-liner (or to be really honest, two-liner) which collects an array for each input line number, and at the end prints out each array in line number order.
Code:
perl -ane 'push @{$L->[$.]}, $F[1]; close ARGV if eof;
END { shift @{$L}; for $l (@{$L}) { print join (" ", @{$l}), "\n"; } }' *
It's regrettable that you can't do this with just the basic Unix tools.
The close stuff is to reset the line numbering for each file (see perldoc -f eof) and the shift is to get rid of line number zero, which should be empty anyway. $F[1] is the second, space-padded input field, in case you want to change it to something else (Perl arrays are zero-based, so the first field is $F[0]). It's easy enough to make it split on a different separator if you like; see the -a and -F options. This uses references for efficiency (or simply because I'm past bedtime), so it's not particularly elegant or readable.
Given a directory containing say a few thousand files,
please output a list of all the names of the files in the directory that are exactly the same, i.e. have the same contents.
func(a_directory_name) output -> {“matches”: , ... ]}
e.g. func(“/home/my/files”) where the directory... (7 Replies)
Hi All,
I get ~1000 files everyday and the current unix script process one file at a time... like it unzips the file, loads into table . This process will repeat how many ever files were present in the directory. I was wondering can we process multiple files (like unzipping and loading X amount... (1 Reply)
Hi,
I am trying to print multiple patterns in a line using sed. But it is printing only the last occurance of a pattern.
If the line is
the the output should be
Lookup Procedure|Stored proc
But the output I am getting is
Stored proc
The code I am using is
echo... (9 Replies)
I try to trim a FTP log using awk, but the escape sequences does not work the way I want it.
Input data Wed 3Oct10 21:48:00 - (002117) Sent file d:\ftp\home\tools\htmake.rar successfully (48.2 kB/sec - 40997 Bytes)
Wed 3Oct10 22:25:46 - (002118) Sent file d:\ftp\files\main\oct\camera1... (4 Replies)
Hi
I always do a ftp in Unix to bring in a file from Mainframes. Then run a set of commands say 8 commands one by one to execute the brought in file. Upon completion of this commands, the data from the file gets transfered to Mainframe database. The problem is i can execute only 1 file at a time.... (2 Replies)
Hi I'm working on a file which looks like:
00001| |XXX|ABCD|27-SEP-1983|31-DEC-2099|Name|00001|N| |31|L| | | | |25-AUG-1990
I need to change the date format from DD-MMM-YYYY to MM/DD/YYYY
So i using
cat <filename> | sed... (2 Replies)
Hi all,
I have mistkanely gzipped twice an entire folder and sub folders, and also renamed the files during that process.
I am trying to undo this, and I need help to create the batch to work on it.
All folders are under my images directory, I have a output.txt file that holds all the... (1 Reply)
I have over 900 files that have the same name except for a unique numeric assignment. For all files I would like to cut the 2nd column and paste all into one new file. All in bash.
sample input format for each file:
1 2 3
1 2 3
1 2 3
sample command for what I want to do:
cut -d' ' -f2... (1 Reply)
Hi Gurus,
I am new to UNIX(HP). my requirmnet is File needs to needs to be split into multiple files dependa on one key value.
example
1 abc 333 us
2 bbc 444 ch
5 nnn 333 kk
7 hhh 555 ll
3 hhh 333 hh
now the requirment is line with 333 should be copied into test1.txt and... (14 Replies)