Remove duplicates separated by delimiter


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove duplicates separated by delimiter
# 8  
Old 05-22-2018
When RudiC says he doesn't understand clearly what you are trying to do, that is an indication that your specification is not clear.

I also think we need a better specification of what is to be removed when $3 is not an empty field. If $3 is A, should every subfield in $6 be removed or will $3 always contain a letter and a number? If a number is always present, should subfields matching that string be removed, or just strings that start with the same letter and the same number? (For example, if $3 contains A1 should it remove a subfield that starts with A12 or just subfields that are A1 or start with A1-? If $3 contains A5 should all subfields of $6 containing A5 be removed (such as A1-A5 and A49-A51, or should it just remove subfields that are A5 or start with A5- or end with -A5?)
# 9  
Old 05-23-2018
True, it was not clear at all, A1 can stands for A12 [learn that while using sed lol ], which in this case is a false positive., just like selecting A5; removing A51. I saw the post of RudiC, and it is in fact what I was trying to achieve.
@RudiC
what does
Code:
(t ~ $3)

stands for, I know that its a reference to field $3, but why the "~" ?
Quote:
will $3 always contain a letter and a number?
Yes indeed.
Thank you both for helping me.
# 10  
Old 05-23-2018
Quote:
Originally Posted by enrikS
. . . @RudiC
what does
Code:
(t ~ $3)

stands for, I know that its a reference to field $3, but why the "~" ?
. . .

That's awk's matching operator (man awk!). read: (t matches / contains $3), e.g. "A1-A4" matches / contains "A1" but wouldn't "A2". You could "sharpen" (narrow, restrict) the match with e.g. word boundaries like \b, \<, or \> (these may not be available in ALL regex engines): "A12-A14" ~ "\bA1\b" will yield FALSE.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Remove duplicates from comma separated list

Hi, I have following input file: niki niki niki1 niki niki2 niki,niki2 niki3 niki,niki3,niki niki4 niki4,blabla niki5 jkjkl niki6 niki60,niki6 I would like to delete lines with identical matches completely and remove the selfmatches in the other lines. ... (2 Replies)
Discussion started by: niki0211
2 Replies

2. Shell Programming and Scripting

Remove leading zeros separated by pipe

I have a below file and I wanted to remove the leading zeros in each field separated by pipe File: 01/09/2017|2017/09/06|2017/02/06|02/06/2017|02/06/2017 06:50:06 AM|2017/02/06|02/06/2017|02/07/2017 05:45:06 AM| 02/08/2017|2017/08/06|2017/09/06|02/05/2017|02/07/2017 05:40:06... (4 Replies)
Discussion started by: Joselouis
4 Replies

3. Shell Programming and Scripting

How to remove duplicates using for loop?

values=(1 2 3 5 4 2 3 1 6 8 3 5 ) #i need the output like this by removing the duplicates 1 2 3 5 4 6 8 #i dont need sorting in my program #plz explain me as simple using for loop #os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
5 Replies

4. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

5. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

6. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

7. Shell Programming and Scripting

need help extracting values from string separated by a delimiter

hi guys, basically what i'm trying to do is fetching a set of columns from an oracle database like so... my_row=`sqlplus -s user/pwd << EOF set head off select user_id, username from all_users where rownum = 1; EOF` echo $my_row the code above returns... 1 ADSHOCKER so then i... (3 Replies)
Discussion started by: adshocker
3 Replies

8. Shell Programming and Scripting

Script to remove duplicates

Hi I need a script that removes the duplicate records and write it to a new file for example I have a file named test.txt and it looks like abcd.23 abcd.24 abcd.25 qwer.25 qwer.26 qwer.98 I want to pick only $1 and compare with the next record and the output should be abcd.23... (6 Replies)
Discussion started by: antointoronto
6 Replies

9. Shell Programming and Scripting

Extract semicolon separated delimiter

The log reads as follows. fname1;lname1;eid1;addr;pincode1; fname2;lname2;eid2;addr2;pincode2; fname3;lname3;eid3;addr3;pincode3; fname4;lname4;eid;addr4;pincode4; how do i extract only fname and save it in an array similarly for lname and so on i tried reading a file and cutting each... (5 Replies)
Discussion started by: vkca
5 Replies

10. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies
Login or Register to Ask a Question