Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

loop? print max column in each line for 800 files and merge

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers loop? print max column in each line for 800 files and merge
# 1  
Old 08-02-2011
Power loop? print max column in each line for 800 files and merge
Hello,

I have 800 or so files with 3 columns each and >10000 lines each.
For each file and each line I would like to print the maximum column number for each line. Then I would like to 'paste' each of these files together (column-wise) so that the file with expression in label '_1' is the first field and file labeled '_800' is the last field.
e.g. input files
Code:
file_1        file_2     file_3 ... file_800
1 2 3       2 1 9       4 7 8     2 8 6
4 2 1       3 6 7       2 1 6     9 4 3
.             .            .           .
.             .            .           .
.             .            .           .

final file
Code:
3 9 8 ... 8
4 7 6 ... 9
.
.
.

Is there an efficient way to do this?
ThanksSmilie
# 2  
Old 08-02-2011
Code:
#!/usr/bin/ksh
rm -f xyz.File_*.xyz
ls -1 File_* | while read mFName; do
  mOtherFN='xyz.'${mFName}'.xyz'
  while read mLine; do
    mMax=$(echo ${mLine} | tr ' ' '\n' | sort -r | head -1)
    echo ${mMax} >> ${mOtherFN}
  done < ${mFName}
done
paste xyz.File_*.xyz
rm -f xyz.File_*.xyz

This User Gave Thanks to Shell_Life For This Post:
peanuts48
# 3  
Old 08-03-2011
Thanks, I will give this a try!

---------- Post updated at 09:52 AM ---------- Previous update was at 09:35 AM ----------

Hello, thanks for the script, the only issue is the paste file_*. This does not paste in numeric order. For example if I have 5 files labeled file_1 file_2 file_3 file_10 file_100, paste_* will paste them in this order: file_1 file_10 file_100 file_2 file_3. Is there a way to order the paste numerically? I have looked over the forum for an answer to this, but paste* is all I've seen.

Thanks.
# 4  
Old 08-03-2011
To ensure files are in order:
Code:
paste $(ls -1 xyz.File_*.xyz | sort)

This User Gave Thanks to Shell_Life For This Post:
peanuts48
# 5  
Old 08-03-2011
Quote:
Originally Posted by Shell_Life
To ensure files are in order:
Code:
paste $(ls -1 xyz.File_*.xyz | sort)

That won't sort the sample files properly. What's needed is to identify where the number begins and define a numeric key beginning at that character position (of course, this assumes a fixed-width prefix).

For files named file_1, file_100.ext, file_1000_abc.ext, sort -k1.6n would do the job.

Regards,
Alister
This User Gave Thanks to alister For This Post:
peanuts48
# 6  
Old 08-03-2011
fixed it!
Thanks! The order still wan't quite right with the above, but I ended up using a variation, which did work:

Code:
 
 paste $(ls -1 temp* | sort -t'_' -n -k2,2)

# 7  
Old 08-08-2011
What if, instead of printing the maximum value for each line, I wanted to print the column number of the maximum value for each line?

eg, the output file is now:
Code:
333...2
133...1
.
.
.

Thanks in advance!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Print a row with the max number in a column

Hello, I have this table: chr1_16857_17742 - chr1 17369 17436 "ENST00000619216.1"; "MIR6859-1"; - 67 chr1_16857_17742 - chr1 14404 29570 "ENST00000488147.1"; "WASH7P"; - 885 chr1_16857_18061 - chr1 ... (5 Replies)
Discussion started by: coppuca
5 Replies

2. Shell Programming and Scripting

Find Max value of line and print

I need to find the max value of all columns except the 1st column and print the answer along with the 1st column. Input 123xyz 0 0 1 2 0 0 0 0 0 0 0 234xyz 0 0 0 0 0 0 0 0 0 0 0 345xyz 0 0 1 0 0 0 ... (8 Replies)
Discussion started by: ncwxpanther
8 Replies

3. Shell Programming and Scripting

Merge two files line by line and column by column

Hi All, I have two files having oracle query result. I want to merge to files line by line and also with column File1 23577|SYNC TYPE 23578|Order Number|ConnectionState 23585|Service State|Service NameFile2 23577|AR Alarm Sync 23578|A5499|9 23585|7|test_nov7Result... (18 Replies)
Discussion started by: Harshal22
18 Replies

4. Shell Programming and Scripting

Print min and max value from two column

Dear All, I have data like this, input: 1254 10125 1254 10126 1254 10127 1254 10128 1254 10129 1255 10130 1255 10131 1255 10132 1255 10133 1256 10134 1256 10135 1256 10137... (3 Replies)
Discussion started by: aksin
3 Replies

5. Shell Programming and Scripting

merge same pattern of same column in one line

Hello, I have some problem in the modified shell script. I would like to merge the same word in column 1 in one line. Example : A1 B1 2 A2 B1 4 A3 B1 7 A1 B2 1 A2 B2 10 A3 B2 8 Expected output : A1 B1 B2 2 1 A2 B1 B2 4 10 A3 ... (6 Replies)
Discussion started by: awil
6 Replies

6. UNIX for Dummies Questions & Answers

[Solved] Print a line using a max and a min values of different columns

Hi guys, I already search on the forum but i can't solve this on my own. I have a lot of files like this: And i need to print the line with the maximum value in last column but if the value is the same (2 in this exemple for the 3 last lines) i need get the line with the minimum value in... (4 Replies)
Discussion started by: MetaBolic0
4 Replies

7. UNIX for Dummies Questions & Answers

Writing a loop to merge multiple files by common column

I have 100 data files labelled 250.1.txt through 250.100.txt. The second column of the data files partially match (there is about %90 overlap). Each data file has 4 columns. I want the merge all these text files by the matching values in the second column. In the output, the first column should... (1 Reply)
Discussion started by: evelibertine
1 Replies

8. Shell Programming and Scripting

loop in awk - column max for each column

Hello all, this should really be easy for you... I need AWK to print column maxima for each column of such input: Input: 1 2 3 1 2 1 1 3 2 1 1 2 Output should be: 2 2 3 3 This does the sum, but i need max instead: { for(i=1; i<=NF; i++) sum +=$i } END {for(i=1; i in sum;... (3 Replies)
Discussion started by: irrevocabile
3 Replies

9. Shell Programming and Scripting

Help: Find line where column has max vlaue

Hi all, Ive searched the forum but with no luck... I have a file: ID Name Val 1 bob 45 2 joe 89 3 sue 11 4 steve 89 ... etc I want to find a way to print... (6 Replies)
Discussion started by: muay_tb
6 Replies

10. Shell Programming and Scripting

compare the column from 3 files and merge that line

I have 3 file, each of has got 80000 records. file1.txt ----------------------- ABC001;active;modify;accept; ABC002;notactive;modify;accept; ABC003;notactive;no-modify;accept; ABC004;active;modify;accept; ABC005;active;no-modify;accept; file2.txt ---------------------------... (8 Replies)
Discussion started by: ganesh_mak
8 Replies
Login or Register to Ask a Question