remove lines based on score criteria


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting remove lines based on score criteria
# 1  
Old 12-01-2008
Power remove lines based on score criteria

Hi guys,

Please guide for Solution.

PART-I

INPUT FILE (has 2 columns ID and score)
TC5584_1 93.9
DV161411_2 79.5
BP132435_5 46.8
EB682112_1 34.7
BP132435_4 29.5
TC13860_2 10.1

OUTPUT FILE (It shudn't contain the line ' BP132435_4 29.5 ' as BP132435 is repeated and it has lower score. If an ID is repeated more than twice, one with highest score should remain)
TC5584_1 93.9
DV161411_2 79.5
BP132435_5 46.8
EB682112_1 34.7
TC13860_2 10.1

PART-II

====FILE1======
TC5584_1 93.9
DV161411_2 79.5
BP132435_5 46.8
EB682112_1 34.7
TC13860_2 10.1

=====FILE2======
EB681299_3 129 269
EB425502_1 71 182
TC5584_1 66 188
BP132435_5 37 106
EB682112_1 22 150
BP132435_4 117 175
TC13860_2 16 93
DV161411_2 36 239

===OUTPUT_FILE===== (It contains column1 from FILE1 and its corresponding row from FILE2)
TC5584_1 66 188
DV161411_2 36 239
BP132435_5 37 106
EB682112_1 22 150
TC13860_2 16 93

your help is highly appreciated.

Thanks in advance. Smilie

Last edited by smriti_shridhar; 12-01-2008 at 07:22 AM.. Reason: formating
# 2  
Old 12-01-2008
Try:
Part 1:

Code:
for each in $(awk -F"_" '{ print $1; }' input_file | sort -u); do sort -k1,1 -k2n input_file | grep $each | tail -1 >>output; done

For Part 2:

Code:
for each in $(awk -F"_" '{ print $1; }' file1 | sort -u); do sort -k1,1 -k2n file2 | grep $each | tail -1 >>output; done

# 3  
Old 12-02-2008
Thanks

I tried the code but its giving an error saying -
Illegal variable name. Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Remove lines from File.A based on criteria in File.B

Hello, I have two files of the following form. I would like to remove from File.A where the first three colum matches values in File.B to give the output in File.C File.A 121 54321 PQR CAT 122 765431 ABC DOG 124 98765 ZXY TIGER 125 86432 GEF LION File.B 122 765431 ABC 125 86432 GEF... (4 Replies)
Discussion started by: Gussifinknottle
4 Replies

2. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

3. Shell Programming and Scripting

Split a file in more files based on score content

Dear All, I have the following file tabulated: ID distanceTSS score 8434 571269 10 10122 393912 9 7652 6 10 4863 1451 9 8419 39 2 9363 564 21 9333 7714 22 9638 8334 9 1638 1231 11 10701 918 1000 6587 32056 111 What I would like to do is the following, create 100 new files based... (5 Replies)
Discussion started by: paolo.kunder
5 Replies

4. Shell Programming and Scripting

Need To Delete Lines Based On Search Criteria

Hi All, I have following input file. I wish to retain those lines which match multiple search criteria. The search criteria is stored in a variable seperated from each other by comma(,). SEARCH_CRITERIA = "REJECT, DUPLICATE" Input File: ERROR,MYFILE_20130214_11387,9,37.75... (3 Replies)
Discussion started by: angshuman
3 Replies

5. Shell Programming and Scripting

Select lines from a file based on a criteria

Hi I need to select lines from a txt file, I have got a line starting with ZMIO:MSISDN= and after a few line I have another line starting with 'MOBILE STATION ISDN NUMBER' and another one starting with 'VLR-ADDRESS' I need to copy these three lines as three different columns in a separate... (3 Replies)
Discussion started by: Tlcm sam
3 Replies

6. Shell Programming and Scripting

Merging Lines based on criteria

Hello, Need help with following scenario. A file contains following text: {beginning of file} New: This is a new record and it is not on same line. Since I have lost touch with script take this challenge and bring all this in one line. New: Hello losttouch. You seem to be struggling... (4 Replies)
Discussion started by: losttouch
4 Replies

7. UNIX for Dummies Questions & Answers

remove duplicates based on a field and criteria

Hi, I have a file with fields like below: A;XYZ;102345;222 B;XYZ;123243;333 C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 desired output: C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 ie, if the 4rth field is a duplicate.. i need only those records where... (5 Replies)
Discussion started by: wanderingmind16
5 Replies

8. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

9. Shell Programming and Scripting

Delete new lines based on search criteria

Hi all! A bit of background: I am trying to create a script that formats SQL statements. I have gotten so far as to add new lines based on certain match criteria like commas, keywords etc. In the process, I end up adding newlines where I don't want. For example: substr(colName, 1, 10)... (3 Replies)
Discussion started by: jayarkay
3 Replies

10. Shell Programming and Scripting

Append specific lines to a previous line based on sequential search criteria

I'll try explain this as best I can. Let me know if it is not clear. I have large text files that contain data as such: 143593502 09-08-20 09:02:13 xxxxxxxxxxx xxxxxxxxxxx 09-08-20 09:02:11 N line 1 test line 2 test line 3 test 143593503 09-08-20 09:02:13... (3 Replies)
Discussion started by: jesse
3 Replies
Login or Register to Ask a Question