How to remove duplicate ID's?

Login or Register to Ask a Question and Join Our Community

How to remove duplicate ID's?

Tags

Login to Discuss or Reply to this Discussion in Our Community

Top Forums Shell Programming and Scripting How to remove duplicate ID's?

01-24-2013

Registered User

57, 0

Join Date: Jan 2013

Last Activity: 14 November 2015, 1:17 PM EST

Posts: 57

Thanks Given: 15

Thanked 0 Times in 0 Posts

How to remove duplicate ID's?

HI

I have file contains 1000'f of duplicate id's with (upper and lower first character) as below
i/p:

Code:

a411532A411532a508661A508661c411532C411532

Requirement: But i need to ignore lowercase id's and need only below id's

o/p:

Code:

A411532
A508661
C411532

Last edited by Scrutinizer; 01-25-2013 at 02:27 PM.. Reason: code tags

buzzme

View Public Profile for buzzme

Find all posts by buzzme

01-24-2013

Registered User

4,673, 588

Join Date: Oct 2010

Last Activity: 1 February 2016, 3:35 PM EST

Location: Southern NJ, USA (Nord)

Posts: 4,673

Thanks Given: 8

Thanked 588 Times in 561 Posts

Code:

egrep -i 'A411532|A508661|C411532' in_file | sort -fut: -k1,1 > out_file

Last edited by DGPickett; 01-24-2013 at 05:51 PM.. Reason: lower all case for unique in sort on id.

DGPickett

View Public Profile for DGPickett

Find all posts by DGPickett

01-24-2013

Registered User

15,129, 5,008

Join Date: Jul 2012

Last Activity: 4 May 2020, 4:31 PM EDT

Location: Aachen, Germany

Posts: 15,129

Thanks Given: 735

Thanked 5,008 Times in 4,483 Posts

PLEASE use code tags as advised!

Your specification is a bit faint. Will your input file be one line only, will your IDs always be seven alphanumeric chars, and the like. Nevertheless, try

Code:

$ sed 's:.\{7\}:&\n:g' file | tr 'a-z' 'A-Z' | sort -u
A411532
A508661
C411532

RudiC

View Public Profile for RudiC

Find all posts by RudiC

01-24-2013

Registered User

57, 0

Join Date: Jan 2013

Last Activity: 14 November 2015, 1:17 PM EST

Posts: 57

Thanks Given: 15

Thanked 0 Times in 0 Posts

Thanks to Rudic and DGPickett its working.

Buzzme-

buzzme

View Public Profile for buzzme

Find all posts by buzzme

01-24-2013

Read Only

1,278, 486

Join Date: Sep 2012

Last Activity: 27 February 2020, 8:59 PM EST

Location: Houston, Texas, USA

Posts: 1,278

Thanks Given: 0

Thanked 486 Times in 451 Posts

try also (Using RudiC's solution):

Code:

sed 's:.\{7\}:\U&\n:g; s/\n$//' file | sort -u

although some sed versions might not support this solution.

rdrtx1

View Public Profile for rdrtx1

Find all posts by rdrtx1

01-25-2013

Registered User

4,673, 588

Join Date: Oct 2010

Last Activity: 1 February 2016, 3:35 PM EST

Location: Southern NJ, USA (Nord)

Posts: 4,673

Thanks Given: 8

Thanked 588 Times in 561 Posts

Yes, the layer for chopping the 7 byre sections into lines:

Code:

sed '
  s/.\{7\}/&\
/g
  s/\n$//
 ' in_file | . . . .

DGPickett

View Public Profile for DGPickett

Find all posts by DGPickett

01-25-2013

Moderator

12,296, 3,792

Join Date: Nov 2008

Last Activity: 1 January 2021, 1:47 AM EST

Location: Amsterdam

Posts: 12,296

Thanks Given: 679

Thanked 3,792 Times in 3,282 Posts

Ignore lowercase id's:

Code:

sed 's/[a-z]....../\
/g' file

Code:

awk -F'[a-z]......' '{$1=$1}1' OFS='\n' file

Last edited by Scrutinizer; 01-25-2013 at 03:12 PM..

Scrutinizer

View Public Profile for Scrutinizer

Find all posts by Scrutinizer

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ...

2. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR

3. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance

4. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common...

5. Shell Programming and Scripting

Help with remove duplicate content

Input file data_1 10 US data_1 2 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA data_3 4 CHINA . . data_60 123 US data_60 23 UK data_60 45 US Desired output file data_1 10 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA

6. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC

7. UNIX for Dummies Questions & Answers

Remove duplicate in array

Hi, I have a list of numbers stored in an array as below. 5 7 10 30 30 40 50 Please advise how could I remove the duplicate value in the array ? Thanks in advance.

8. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006...

9. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006...

10. Shell Programming and Scripting

remove duplicate

i have a text its contain many record, but its written in one line, i want to remove from that line the duplicate record, not record have fixed width ex: width = 4 inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc i want the outputfile =abc cdf fgh only those records can any one help...

Login or Register to Ask a Question