Help with remove duplicated content


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with remove duplicated content
# 1  
Old 08-21-2011
Help with remove duplicated content

Input file:
Code:
hcmv-US25-2-3p	hsa-3160-5
hcmv-US33 hsa-47
hcmv-UL70-3p hsa-4508
hcmv-UL70-3p hsa-4486
hcms-US25 hsa-360-5
hcms-US25 hsa-4
hcms-US25 hsa-458
hcms-US25 hsa-44812
.
.

Desired Output file:
Code:
hcmv-US25-2-3p	hsa-3160-5
hcmv-US33 hsa-47
hcmv-UL70-3p hsa-4508
	     hsa-4486
hcms-US25 hsa-360-5
	  hsa-4
	  hsa-458
	  hsa-44812
.
.

I would like to remove those duplicate content based on column 1.
Many thanks for any advice.
# 2  
Old 08-22-2011
Code:
awk '{if ($1==x){gsub(/./," ",$1)}else{x=$1;}}1' file

Guru.
This User Gave Thanks to guruprasadpr For This Post:
# 3  
Old 08-22-2011
Hi,

Althought you already got the answer, here a solution using 'sed':
Code:
$ cat infile
hcmv-US25-2-3p  hsa-3160-5
hcmv-US33 hsa-47
hcmv-UL70-3p hsa-4508
hcmv-UL70-3p hsa-4486
hcms-US25 hsa-360-5
hcms-US25 hsa-4
hcms-US25 hsa-458
hcms-US25 hsa-44812
$ cat script.sed
:a

## Append next line to pattern space but last.
$! N

## If characters after '\n' are the same that characters after '^' ...
/^\([^ \t]\+[ \t]\+\).*\n\1.*$/ {
        :b
        ## Substitute each char with a space and repeat for all of them (instruction 'tb'). After that, read next line.
        s/\(\n[^ \t]*\)[^ \t][ \t]/\1  /
        tb
        ba
}

:c
## If there are two (or more) lines in pattern space... go to ':d' and read next line.
s/\n/\n/
td

## If last line, print it and exit.
$ { 
        P
        D
}

ba

## Print one line (until first '\n') and delete it.
:d
P
s/^[^\n]*\n//
tc
$ sed -f script.sed infile
hcmv-US25-2-3p  hsa-3160-5
hcmv-US33 hsa-47
hcmv-UL70-3p hsa-4508
             hsa-4486
hcms-US25 hsa-360-5
          hsa-4
          hsa-458
          hsa-44812

Regards,
Birei
# 4  
Old 08-22-2011
Here's a Perl solution -

Code:
$
$
$ cat f9
hcmv-US25-2-3p  hsa-3160-5
hcmv-US33 hsa-47
hcmv-UL70-3p hsa-4508
hcmv-UL70-3p hsa-4486
hcms-US25 hsa-360-5
hcms-US25 hsa-4
hcms-US25 hsa-458
hcms-US25 hsa-44812
$
$
$ perl -lane 'print $F[0] eq $prev ? " " x length($prev) : $F[0], " ", $F[1]; $prev=$F[0]' f9
hcmv-US25-2-3p hsa-3160-5
hcmv-US33 hsa-47
hcmv-UL70-3p hsa-4508
             hsa-4486
hcms-US25 hsa-360-5
          hsa-4
          hsa-458
          hsa-44812
$
$
$

tyler_durden
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicated records and update last line record counts

Hi Gurus, I need to remove duplicate line in file and update TRAILER (last line) record count. the file is comma delimited, field 2 is key to identify duplicated record. I can use below command to remove duplicated. but don't know how to replace last line 2nd field to new count. awk -F","... (11 Replies)
Discussion started by: green_k
11 Replies

2. AIX

Remove duplicated bootlist entries

Hello. I have a server with 2 boot disk but in the bootlist there are 5 paths of one disk but no path of the other. How can I remove paths from one disk to insert paths from the other disk? Thanks in advance. server074:root:/# bootlist -om normal hdisk0 blv=hd5 pathid=0 hdisk0... (7 Replies)
Discussion started by: Gabriander
7 Replies

3. Shell Programming and Scripting

How to remove duplicated column in a text file?

Dear all, How can I remove duplicated column in a text file? Input: LG10_PM_map_19_LEnd 1000560 G AA AA AA AA AA GG LG10_PM_map_19_LEnd 1005621 G GG GG GG AA AA GG LG10_PM_map_19_LEnd 1011214 A AA AA AA AA GG GG LG10_PM_map_19_LEnd 1011673 T TT TT TT TT CC CC... (1 Reply)
Discussion started by: huiyee1
1 Replies

4. Shell Programming and Scripting

Merge files and remove duplicated rows

In a folder I'll several times daily receive new files that I want to combine into one big file, without any duplicate rows. The file name in the folder will look like e.q: MissingData_2014-08-25_09-30-18.txt MissingData_2014-08-25_09-30-14.txt MissingData_2014-08-26_09-30-12.txt The content... (9 Replies)
Discussion started by: Bergans
9 Replies

5. Shell Programming and Scripting

How to remove duplicated lines?

Hi, if i have a file like this: Query=1 a a b c c c d Query=2 b b b c c e . . . (7 Replies)
Discussion started by: the_simpsons
7 Replies

6. Shell Programming and Scripting

Remove rows with first 4 fields duplicated in awk

Hi, I am trying to use awk to remove all rows where the first 4 fields are duplicates. e.g. in the following data lines 6-9 would be removed, leaving one copy of the duplicated row (row 5) Borgarhraun FH9822 ol24 FH9822_ol24_m20 ol Deformed c Borgarhraun FH9822 ol24 ... (3 Replies)
Discussion started by: tomahawk
3 Replies

7. UNIX for Dummies Questions & Answers

How to remove duplicated based on longest row & largest value in a column

Hii i have a file with data as shown below. Here i need to remove duplicates of the rows in such a way that it just checks for 2,3,4,5 column for duplicates.When deleting duplicates,retain largest row i.e with many columns with values should be selected.Then it must remove duplicates such that by... (11 Replies)
Discussion started by: reva
11 Replies

8. Shell Programming and Scripting

remove duplicated columns

hi all, i have a file contain multicolumns, this file is sorted by col2 and col3. i want to remove the duplicated columns if the col2 and col3 are the same in another line. example fileA AA BB CC DD CC XX CC DD BB CC ZZ FF DD FF HH HH the output is AA BB CC DD BB CC ZZ FF... (6 Replies)
Discussion started by: kamel.seg
6 Replies

9. Shell Programming and Scripting

remove duplicated lines without sort

Hi Just wondering whether or not I can remove duplicated lines without sort For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal. Normally, I would do who | cut -d" " -f1 |... (6 Replies)
Discussion started by: lalelle
6 Replies

10. Shell Programming and Scripting

remove duplicated xml record in a file under unix

Hi, If i have a file with xml format, i would like to remove duplicated records and save to a new file. Is it possible...to write script to do it? (8 Replies)
Discussion started by: happyv
8 Replies
Login or Register to Ask a Question