Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

remove duplicated columns

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting remove duplicated columns
# 1  
Old 02-21-2008
remove duplicated columns
hi all,

i have a file contain multicolumns, this file is sorted by col2 and col3.

i want to remove the duplicated columns if the col2 and col3 are the same in another line.

example
fileA

AA BB CC DD
CC XX CC DD
BB CC ZZ FF
DD FF HH HH


the output is
AA BB CC DD
BB CC ZZ FF
DD FF HH HH

thanks
# 2  
Old 02-21-2008
uniq -f1 -f2 -c file1
# 3  
Old 02-21-2008
it works but i have also in the file i get the occurence number of the line
can i remove it ??
# 4  
Old 02-21-2008
just remove '-c' option
# 5  
Old 02-21-2008
yes it work but the file a have have many columns and i want just to compare the colx and coly and ignor the rest what can i do, because f1 and f2 ignore the two first columns??
# 6  
Old 02-21-2008
yes give the columns to be avoided with -f option.
# 7  
Old 02-21-2008
i try sort -u -kx,y

with x and y are the columns numbers it works well
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

Remove duplicated bootlist entries

Hello. I have a server with 2 boot disk but in the bootlist there are 5 paths of one disk but no path of the other. How can I remove paths from one disk to insert paths from the other disk? Thanks in advance. server074:root:/# bootlist -om normal hdisk0 blv=hd5 pathid=0 hdisk0... (7 Replies)
Discussion started by: Gabriander
7 Replies

2. Shell Programming and Scripting

How to remove duplicated column in a text file?

Dear all, How can I remove duplicated column in a text file? Input: LG10_PM_map_19_LEnd 1000560 G AA AA AA AA AA GG LG10_PM_map_19_LEnd 1005621 G GG GG GG AA AA GG LG10_PM_map_19_LEnd 1011214 A AA AA AA AA GG GG LG10_PM_map_19_LEnd 1011673 T TT TT TT TT CC CC... (1 Reply)
Discussion started by: huiyee1
1 Replies

3. Shell Programming and Scripting

Merge files and remove duplicated rows

In a folder I'll several times daily receive new files that I want to combine into one big file, without any duplicate rows. The file name in the folder will look like e.q: MissingData_2014-08-25_09-30-18.txt MissingData_2014-08-25_09-30-14.txt MissingData_2014-08-26_09-30-12.txt The content... (9 Replies)
Discussion started by: Bergans
9 Replies

4. Shell Programming and Scripting

How to remove duplicated lines?

Hi, if i have a file like this: Query=1 a a b c c c d Query=2 b b b c c e . . . (7 Replies)
Discussion started by: the_simpsons
7 Replies

5. UNIX for Dummies Questions & Answers

Find duplicated values in two columns out of three

hi! could u help in the following? I have the data (long list!) that looks like (three coumns white space separated): rs3094315 0.0665173 742429 rs12562034 0.0738998 758311 rs3934834 0.396449 995669 rs9442372 0.402693 1008567 rs3737728 0.406271 1011278 rs6687776 0.435429 1020428 rs9651273... (4 Replies)
Discussion started by: kush
4 Replies

6. Shell Programming and Scripting

Remove rows with first 4 fields duplicated in awk

Hi, I am trying to use awk to remove all rows where the first 4 fields are duplicates. e.g. in the following data lines 6-9 would be removed, leaving one copy of the duplicated row (row 5) Borgarhraun FH9822 ol24 FH9822_ol24_m20 ol Deformed c Borgarhraun FH9822 ol24 ... (3 Replies)
Discussion started by: tomahawk
3 Replies

7. Shell Programming and Scripting

Help with remove duplicated content

Input file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hcmv-UL70-3p hsa-4486 hcms-US25 hsa-360-5 hcms-US25 hsa-4 hcms-US25 hsa-458 hcms-US25 hsa-44812 . . Desired Output file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hsa-4486... (3 Replies)
Discussion started by: perl_beginner
3 Replies

8. Shell Programming and Scripting

using sed to get rid of duplicated columns...

I can not figure out this one, so I turn to unix.com for help, I have a file, in which there are some lines containing continuously duplicate columns, like the following adb abc abc asd adfj 123 123 123 345 234 444 444 444 444 444 23 and the output I want is adb abc asd adfj 123 345... (5 Replies)
Discussion started by: fedora
5 Replies

9. Shell Programming and Scripting

Help removing lines with duplicated columns

Hi Guys... Please Could you help me with the following ? aaaa bbbb cccc sdsd aaaa bbbb cccc qwer as you can see, the 2 lines are matched in three fields... how can I delete this pupicate ? I mean to delete the second one if 3 fields were duplicated ? Thanks (14 Replies)
Discussion started by: yahyaaa
14 Replies

10. Shell Programming and Scripting

remove duplicated lines without sort

Hi Just wondering whether or not I can remove duplicated lines without sort For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal. Normally, I would do who | cut -d" " -f1 |... (6 Replies)
Discussion started by: lalelle
6 Replies
Login or Register to Ask a Question