Sponsored Content
Top Forums Shell Programming and Scripting Filtering duplicates based on lookup table and rules Post 302920598 by ritakadm on Friday 10th of October 2014 11:23:00 AM
Old 10-10-2014
Let me go through each category in the original sample input.

Category 1 (cat1 in column 2)

First set of duplicates; 1 and 2
Code:
 
 
1;cat1;val1
2;cat1;val2

Duplicate ids 1 and 2 have different values val1 and val2 in cat1. So we output only the first (1) and call the value ambiguous.

Code:
1;cat1;ambiguous

Moving on to the second set of duplicates,; 3,4 and 5

Code:
 
 
3;cat1;val3
4;cat1;val3
5;cat1;val3

For duplicates 3,4 and 5 in category 1 , they have the same value val3. So we just output the first.

Code:
 
3;cat1;val3

Category 2 (cat2 in column 2)
First set of duplicates; 1 and 2
Code:
 
2;cat2;val2

Only 2 is present, no 1.So we output as it is.
Code:
 
2;cat2;val2

Moving on to second set of duplicates; 3, 4 and 5

Code:
3;cat2;val3
5;cat2;val3

Only 3 and 5 are present, no 4 and they have the same value val3.

So we output the first (3) with value val3.

Code:
 
3;cat2;val3

Moving on to third set of replicates; 6 and 7


Code:
6;cat2;val3

Code:
7;cat2;val4

Both 6 and 7 are present with different values , so output the first (6) and the report the value ambiguous.

Code:
 
6;cat2;ambiguous

Lastly the id 8 is not duplicated in the lookup file.

Code:
 
8;cat2;val4

So report as it is


Code:
 
8;cat2;val4

So the sample input file (combined from the above steps)

Code:
 
varid;category;value
1;cat1;val1
2;cat1;val2
3;cat1;val3
4;cat1;val3
5;cat1;val3
2;cat2;val2
3;cat2;val3
5;cat2;val3
6;cat2;val3
7;cat2;val4
8;cat2;val4

Desired output (combined the red rows from each step)

Code:
 
1;cat1;ambiguous
3;cat1;val3
2;cat2;val2
3;cat2;val3
6;cat2;ambiguous
8;cat2;val4

---------- Post updated at 10:23 AM ---------- Previous update was at 10:10 AM ----------

Quote:
Originally Posted by Aia
According to what you are saying and using your first post input:
Code:
varid;category;value
1;cat1;val1
2;cat1;val2
3;cat1;val3
4;cat1;val3
5;cat1;val3
 
2;cat2;val2
3;cat2;val3
5;cat2;val3
6;cat2;val3
7;cat2;val4
8;cat2;val4

Only the following should be kept because they are the first of their distinctive categories:
Code:
1;cat1;val1
2;cat2;val2

Further, it should have their value changed to `ambiguous' because their group contain different values among them.
Code:
1;cat1;ambiguous
2;cat2;ambiguous

Unfortunately, that doesn't match your original example output:

Code:
varid;category;value
1;cat1;ambiguous
3;cat1;val3
2;cat2;val2
3;cat2;val3
6;cat2;ambiguous
8;cat2;val4

Aia, this will not be the case because they are different categories.
The first row belongs to cat1 and the second row belongs to cat2, so they must be treated independently.
Code:
1;cat1;ambiguous
2;cat2;ambiguous

It should be
Code:
1;cat1;ambiguous
2;cat2;val2


Last edited by ritakadm; 10-10-2014 at 12:16 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

lookup table in perl??

hi, i am very much new in perl and have this very basic question in the same:( the requirement is as below: i have an input file (txt file) in which i have fields invoice number and customer number. Now i have to take input this combination of invoice n customer number and check in a... (2 Replies)
Discussion started by: Bhups
2 Replies

2. UNIX for Dummies Questions & Answers

HELP with using a lookup table

Using AIX 5.2, Bourne and Korn Shell. I have two flat text files. One is a main file and one is a lookup table that contains a number of letter codes and membership numbers as follows: 316707965EGM01 315672908ANM92 Whenever one of these records from the lookup appears in the main file... (6 Replies)
Discussion started by: Dolph
6 Replies

3. Programming

64-bit CRC Transition To Bytewise Lookup-Table

Good Evening, I started working on the 17x17 4-colouring challenge, and I ran into a bit of an I/O snag. It was an enormous headache to detect the differences in very similar 289-char strings. Eventually, it made more sense to associate a CRC-Digest with each colouring. After learning... (0 Replies)
Discussion started by: HeavyJ
0 Replies

4. Shell Programming and Scripting

Sed variable from lookup table

I have a file with the following format --TABLEA_START-- field1=data1;field2=data2;field3=data3 --TABLEA_END-- --TABLEB_START-- field1=data1;field2=data2;field3=data3 --TABLEB_END-- --TABLEA_START-- field1=data1;field2=data2;field3=data3 ... (0 Replies)
Discussion started by: milo7
0 Replies

5. UNIX for Dummies Questions & Answers

string replacement using a lookup table

Dear all thanks for helping in advance.. Know this should be fairly simple but I failed in searching for an answer. I have a file (replacement table) containing two columns, e.g.: ACICJ ACIDIPHILIUM ACIF2 ACIDITHIOBACILLUS ACIF5 ACIDITHIOBACILLUS ACIC5 ACIDOBACTERIUM ACIC1 ACIDOTHERMUS... (10 Replies)
Discussion started by: roussine
10 Replies

6. UNIX for Dummies Questions & Answers

Filtering the duplicates

Hello, I want to filter all the duplicates of a record to one place. Sample input and output will give you better idea. I am new to unix. Can some one help me on this? Input: 7488 7389 chr1.fa chr1.fa 3546 9887 chr5.fa chr9.fa 7387 7898 chrX.fa chr3.fa 7488 7389 chr1.fa chr1.fa... (2 Replies)
Discussion started by: koneru_18
2 Replies

7. Shell Programming and Scripting

Filtering out duplicates with the highest version number

Hi, I have a huge text file with filenames which which looks like the following ie uniquenumber_version_filename: e.g. 1234_1_xxxx 1234_2_vfvfdbb 343333_1_vfvfdvd 2222222_1_ggggg 55555_1_xxxxxx 55555_2_vrbgbgg 55555_3_grgrbr What I need to do is examine the file, look for... (4 Replies)
Discussion started by: mantis
4 Replies

8. Web Development

Help on filtering the table in HTML

1. how to get the filter option on table so that user can enter the fields which ever they want to print only according to the need ? 2.how to print the full fledge table if there is no value in the rows of the table but it should print the whole rows and column in proper tabular form? (2 Replies)
Discussion started by: sidhi
2 Replies

9. Shell Programming and Scripting

PERL "filtering the log file removing the duplicates

Hi folks, I have a log file in the below format and trying to get the output of the unique ones based on mnemonic IN PERL. Could any one please let me know with the code and the logic ? Severity Mnemonic Log Message 7 CLI_SCHEDULER Logfile for scheduled CLI... (3 Replies)
Discussion started by: scriptscript
3 Replies

10. Shell Programming and Scripting

Korn shell - lookup table

Hi All I need to pass country code into a pipe delimited file for lookup. It will search country code (column 3) in the file, if the country code matched, it will return value from other columns. Here is my mapping file. #CountryName|CountryRegion|CountryCode-3|CountryCode-2... (5 Replies)
Discussion started by: lafrance
5 Replies
All times are GMT -4. The time now is 03:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy