Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove duplicates separated by delimiter
Top Forums Shell Programming and Scripting Remove duplicates separated by delimiter Post 303017785 by RudiC on Tuesday 22nd of May 2018 10:17:50 AM
Old 05-22-2018
Not sure I understand correctly, but if you want to remove all elements from the array that match / contain $3, try (with your new sample code):

Code:
awk '
function RMDUP(P1, T, TX, DL)   {for (n = split (P1, TMP, ","); n; n--)  T[TMP[n]]
                                 for (t in T) if (!(t ~ $3))    {TX = TX DL t
                                                                 DL = ","
                                                                }
                                 return TX
                                }
        {$6 = RMDUP($6)
         $7 = RMDUP($7)
        }
 1
' file
M3 C2 A1 D5 HH:FF A5-A6 B1-B2,B2-B4,B4-B

Last edited by RudiC; 05-22-2018 at 12:06 PM..
This User Gave Thanks to RudiC For This Post:
enrikS
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

2. Shell Programming and Scripting

Extract semicolon separated delimiter

The log reads as follows. fname1;lname1;eid1;addr;pincode1; fname2;lname2;eid2;addr2;pincode2; fname3;lname3;eid3;addr3;pincode3; fname4;lname4;eid;addr4;pincode4; how do i extract only fname and save it in an array similarly for lname and so on i tried reading a file and cutting each... (5 Replies)
Discussion started by: vkca
5 Replies

3. Shell Programming and Scripting

Script to remove duplicates

Hi I need a script that removes the duplicate records and write it to a new file for example I have a file named test.txt and it looks like abcd.23 abcd.24 abcd.25 qwer.25 qwer.26 qwer.98 I want to pick only $1 and compare with the next record and the output should be abcd.23... (6 Replies)
Discussion started by: antointoronto
6 Replies

4. Shell Programming and Scripting

need help extracting values from string separated by a delimiter

hi guys, basically what i'm trying to do is fetching a set of columns from an oracle database like so... my_row=`sqlplus -s user/pwd << EOF set head off select user_id, username from all_users where rownum = 1; EOF` echo $my_row the code above returns... 1 ADSHOCKER so then i... (3 Replies)
Discussion started by: adshocker
3 Replies

5. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

6. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

7. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

8. Shell Programming and Scripting

How to remove duplicates using for loop?

values=(1 2 3 5 4 2 3 1 6 8 3 5 ) #i need the output like this by removing the duplicates 1 2 3 5 4 6 8 #i dont need sorting in my program #plz explain me as simple using for loop #os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
5 Replies

9. Shell Programming and Scripting

Remove leading zeros separated by pipe

I have a below file and I wanted to remove the leading zeros in each field separated by pipe File: 01/09/2017|2017/09/06|2017/02/06|02/06/2017|02/06/2017 06:50:06 AM|2017/02/06|02/06/2017|02/07/2017 05:45:06 AM| 02/08/2017|2017/08/06|2017/09/06|02/05/2017|02/07/2017 05:40:06... (4 Replies)
Discussion started by: Joselouis
4 Replies

10. UNIX for Beginners Questions & Answers

Remove duplicates from comma separated list

Hi, I have following input file: niki niki niki1 niki niki2 niki,niki2 niki3 niki,niki3,niki niki4 niki4,blabla niki5 jkjkl niki6 niki60,niki6 I would like to delete lines with identical matches completely and remove the selfmatches in the other lines. ... (2 Replies)
Discussion started by: niki0211
2 Replies

LEARN ABOUT DEBIAN

shtool-mkshadow

SHTOOL-MKSHADOW.TMP(1)					      GNU Portable Shell Tool					    SHTOOL-MKSHADOW.TMP(1)

NAME

       shtool-mkshadow - GNU shtool create shadow tree using symlinks

SYNOPSIS

       shtool mkshadow [-v|--verbose] [-t|--trace] [-a|--all] src-dir dst-dir

DESCRIPTION

       This command creates a shadow tree of src-dir under dst-dir by recreating the directory hierarchy of src-dir under dst-dir and by creating
       the files of src-dir by linking them into the corresponding directories under dst-dir via symbolic links. When src-dir can be reached via
       relative paths from dst-dir, relative symbolic links are used, too. This high-level functionality is originally designed for developers to
       create copies of source trees.

OPTIONS

       The following command line options are available.

       -v, --verbose
	   Display some processing information.

       -t, --trace
	   Enable the output of the essential shell commands which are executed.

       -a, --all
	   Really shadow all files and directories in src-dir. Default is to skip CVS related files and directories, backup files, object files,
	   etc.

EXAMPLE

	#   shell script
	shtool mkshadow -v -a . /tmp/shadow

HISTORY

       The GNU shtool mkshadow command was originally written by Ralf S.  Engelschall <rse@engelschall.com> in 1998 for Apache. It was later
       revised and taken over into GNU shtool.

SEE ALSO

       shtool(1), ln(1).

18-Jul-2008							   shtool 2.0.8 					    SHTOOL-MKSHADOW.TMP(1)
All times are GMT -4. The time now is 07:36 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy