Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove duplicates separated by delimiter
Top Forums Shell Programming and Scripting Remove duplicates separated by delimiter Post 303017775 by Don Cragun on Tuesday 22nd of May 2018 06:07:14 AM
Old 05-22-2018
Hi enrikS,
If keeping the input order of elements in fields 6 and 7 is important and you want <tab> as your output field separator (as shown in your second code snippet), you could also try:
Code:
awk '
BEGIN {	OFS = "\t"
}
function RMDUP(input,	i, n, NoDupArray, output, ValueArray) {
	n = split(input, ValueArray, /,/)
	NoDupArray[output = ValueArray[1]]
	for(i = 2; i <= n; i++)
		if(!(ValueArray[i] in NoDupArray)) {
			output = output "," ValueArray[i]
			NoDupArray[ValueArray[i]]
		}
	return output
}
{	$6 = RMDUP($6)
	$7 = RMDUP($7)
}
1' data

In addition to the change you have already made to your original post, note also that if you want the field 6 output to be:
Code:
A1-A2,A5-A6,A1-A4

you can't have the input be:
Code:
A1-A2,A5-A6,A1-A2,A1-4

The above code produces the output:
Code:
M3	C2	V5	D5	HH:FF	A1-A2,A5-A6,A1-4	B4-B6,B2-B4,B1-B2

from the sample input you provided in post #1.

For some hints as to why your second code snippet didn't work, note that your awk code is specifying that the input field separator (FS) is a <tab> character, but there are no <tab>s in your sample input (just <space>s; no <tab>s). Therefore, your awk script is only seeing one input field; not eight. And split()ing an empty field (e.g., $6) produces an array with zero elements.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

2. Shell Programming and Scripting

Extract semicolon separated delimiter

The log reads as follows. fname1;lname1;eid1;addr;pincode1; fname2;lname2;eid2;addr2;pincode2; fname3;lname3;eid3;addr3;pincode3; fname4;lname4;eid;addr4;pincode4; how do i extract only fname and save it in an array similarly for lname and so on i tried reading a file and cutting each... (5 Replies)
Discussion started by: vkca
5 Replies

3. Shell Programming and Scripting

Script to remove duplicates

Hi I need a script that removes the duplicate records and write it to a new file for example I have a file named test.txt and it looks like abcd.23 abcd.24 abcd.25 qwer.25 qwer.26 qwer.98 I want to pick only $1 and compare with the next record and the output should be abcd.23... (6 Replies)
Discussion started by: antointoronto
6 Replies

4. Shell Programming and Scripting

need help extracting values from string separated by a delimiter

hi guys, basically what i'm trying to do is fetching a set of columns from an oracle database like so... my_row=`sqlplus -s user/pwd << EOF set head off select user_id, username from all_users where rownum = 1; EOF` echo $my_row the code above returns... 1 ADSHOCKER so then i... (3 Replies)
Discussion started by: adshocker
3 Replies

5. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

6. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

7. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

8. Shell Programming and Scripting

How to remove duplicates using for loop?

values=(1 2 3 5 4 2 3 1 6 8 3 5 ) #i need the output like this by removing the duplicates 1 2 3 5 4 6 8 #i dont need sorting in my program #plz explain me as simple using for loop #os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
5 Replies

9. Shell Programming and Scripting

Remove leading zeros separated by pipe

I have a below file and I wanted to remove the leading zeros in each field separated by pipe File: 01/09/2017|2017/09/06|2017/02/06|02/06/2017|02/06/2017 06:50:06 AM|2017/02/06|02/06/2017|02/07/2017 05:45:06 AM| 02/08/2017|2017/08/06|2017/09/06|02/05/2017|02/07/2017 05:40:06... (4 Replies)
Discussion started by: Joselouis
4 Replies

10. UNIX for Beginners Questions & Answers

Remove duplicates from comma separated list

Hi, I have following input file: niki niki niki1 niki niki2 niki,niki2 niki3 niki,niki3,niki niki4 niki4,blabla niki5 jkjkl niki6 niki60,niki6 I would like to delete lines with identical matches completely and remove the selfmatches in the other lines. ... (2 Replies)
Discussion started by: niki0211
2 Replies

LEARN ABOUT SUSE

gtk2::pagesetup

Gtk2::PageSetup(3)					User Contributed Perl Documentation					Gtk2::PageSetup(3)

NAME

       Gtk2::PageSetup

HIERARCHY

	 Glib::Object
	 +----Gtk2::PageSetup

METHODS

   pagesetup = Gtk2::PageSetup->new
   pagesetup = Gtk2::PageSetup->new_from_file ($file_name)
       o   $file_name (localized file name)

       May croak with a Glib::Error in $@ on failure.

       Since: gtk+ 2.12

   pagesetup = Gtk2::PageSetup->new_from_key_file ($key_file, $group_name)
       o   $key_file (Glib::KeyFile)

       o   $group_name (string or undef)

       May croak with a Glib::Error in $@ on failure.

       Since: gtk+ 2.12

   double = $setup->get_bottom_margin ($unit)
       o   $unit (Gtk2::Unit)

   $setup->set_bottom_margin ($margin, $unit)
       o   $margin (double)

       o   $unit (Gtk2::Unit)

   double = $setup->get_left_margin ($unit)
       o   $unit (Gtk2::Unit)

   $setup->set_left_margin ($margin, $unit)
       o   $margin (double)

       o   $unit (Gtk2::Unit)

   $setup->load_file ($file_name)
       o   $file_name (string)

       May croak with a Glib::Error in $@ on failure.

       Since: gtk+ 2.14

   $setup->load_key_file ($key_file, $group_name)
       o   $key_file (Glib::KeyFile)

       o   $group_name (string or undef)

       May croak with a Glib::Error in $@ on failure.

       Since: gtk+ 2.14

   pageorientation = $setup->get_orientation
   $setup->set_orientation ($orientation)
       o   $orientation (Gtk2::PageOrientation)

   double = $setup->get_page_height ($unit)
       o   $unit (Gtk2::Unit)

   double = $setup->get_page_width ($unit)
       o   $unit (Gtk2::Unit)

   double = $setup->get_paper_height ($unit)
       o   $unit (Gtk2::Unit)

   $setup->set_paper_size_and_default_margins ($size)
       o   $size (Gtk2::PaperSize)

   papersize = $setup->get_paper_size
   $setup->set_paper_size ($size)
       o   $size (Gtk2::PaperSize)

   double = $setup->get_paper_width ($unit)
       o   $unit (Gtk2::Unit)

   double = $setup->get_right_margin ($unit)
       o   $unit (Gtk2::Unit)

   $setup->set_right_margin ($margin, $unit)
       o   $margin (double)

       o   $unit (Gtk2::Unit)

   $setup->to_file ($file_name)
       o   $file_name (localized file name)

       May croak with a Glib::Error in $@ on failure.

       Since: gtk+ 2.12

   $setup->to_key_file ($key_file, $group_name)
       o   $key_file (Glib::KeyFile)

       o   $group_name (string or undef)

       Since: gtk+ 2.12

   double = $setup->get_top_margin ($unit)
       o   $unit (Gtk2::Unit)

   $setup->set_top_margin ($margin, $unit)
       o   $margin (double)

       o   $unit (Gtk2::Unit)

ENUMS AND FLAGS

   enum Gtk2::PageOrientation
       o   'portrait' / 'GTK_PAGE_ORIENTATION_PORTRAIT'

       o   'landscape' / 'GTK_PAGE_ORIENTATION_LANDSCAPE'

       o   'reverse-portrait' / 'GTK_PAGE_ORIENTATION_REVERSE_PORTRAIT'

       o   'reverse-landscape' / 'GTK_PAGE_ORIENTATION_REVERSE_LANDSCAPE'

   enum Gtk2::Unit
       o   'pixel' / 'GTK_UNIT_PIXEL'

       o   'points' / 'GTK_UNIT_POINTS'

       o   'inch' / 'GTK_UNIT_INCH'

       o   'mm' / 'GTK_UNIT_MM'

SEE ALSO

       Gtk2, Glib::Object

COPYRIGHT

       Copyright (C) 2003-2008 by the gtk2-perl team.

       This software is licensed under the LGPL.  See Gtk2 for a full notice.

perl v5.12.1							    2010-07-05							Gtk2::PageSetup(3)
All times are GMT -4. The time now is 01:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy