Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Duplicates to be removed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Duplicates to be removed
# 1  
Old 07-10-2008
Duplicates to be removed
Hi,

I have a text file with 2000 rows and 2000 columns (number of columns might vary from row to row) and "comma" is the delimiter.

In every row, there maybe few duplicates and we need to remove those duplicates and "shift left" the consequent values.

ex:

111 222 111 555
444 999 666 999 777

o/p must be like below:
111 222 555
444 999 666 777


TIA
Prvn
# 2  
Old 07-10-2008
Use nawk or /usr/xpg4/bin/awk on Solaris:

Code:
awk -F, '{
  for (f=1; f<=NF; f++)
    if (!_[$f]++)
      printf $f (f != NF ? FS : RS)
  split("", _)  
      }' input

With GNU Awk you can use delete _ instead of split.
# 3  
Old 07-10-2008
nawk -f prv.awk myFile.txt

prv.awk:
Code:
{
  for(i=1; i<=NF; i++) {
    if ($i in arr) continue
    printf("%s%s", $i, OFS)
    arr[$i]
  }
  printf ORS
  split("", arr)
}

# 4  
Old 07-10-2008
Thanks for your replies.

Vgersh - Your solution worked (as space is the delimiter). I'm sorry that i did not use "comma" in the example. Actually the delimiter is "comma" as mentioned in the post.

Please advise.

Prvn
# 5  
Old 07-10-2008
With Perl:

Code:
perl -F, -lane'$, = ","; 
  print grep !$_{$_}++, @F;   
    undef %_' input

# 6  
Old 07-10-2008
Quote:
Originally Posted by prvnrk
Thanks for your replies.

Vgersh - Your solution worked (as space is the delimiter). I'm sorry that i did not use "comma" in the example. Actually the delimiter is "comma" as mentioned in the post.

Please advise.

Prvn
use radoulov's code
# 7  
Old 07-10-2008
Thanks radoulov,

Your awk solution worked great!


Prvn
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merging few files into one, duplicates are removed

Hello, I have few file such as below: abc.txt def.txt ghi.txt jkl.txt n.txt I would like to merge all these files together into one file. At the same time, any duplicates will be removed. (5 Replies)
Discussion started by: alegnagrp
5 Replies

2. Linux

file removed

Hi Team, I have deleted a file accidentally by using rm command. I am not the root(admin) user. Can you please let me know how to get that .tex file? (2 Replies)
Discussion started by: darling
2 Replies

3. Shell Programming and Scripting

Removed Lines

Hi Guys, I am using SunOS 5.9 running Oracle Databases on it... I have log files that I suspect that some lines within the logs where removed. How do I tell if indeed some lines within a particular file where removed and by whom? Thanks in advance (2 Replies)
Discussion started by: Phuti
2 Replies

4. AIX

Default route removed - what to do ?

Hi, After accidental default route removal serial connection doesn't work. What should I do to connect to my machine ? thanks Vilius (7 Replies)
Discussion started by: vilius
7 Replies

5. UNIX for Advanced & Expert Users

Removed ^M from Libraries

I used the following to remove ^M in all files - I guess i did it in haste :mad: find / -name "*" | xargs perl -p -i -e 's/^M//g' * It changed all my LIBRABRIES since i used -- perl -p -i -e 's/^M//g' * Is there some way to revert this from my libraries . Does any revert command... (10 Replies)
Discussion started by: telecomics
10 Replies

6. UNIX for Dummies Questions & Answers

a way to tell what was removed after rm -rf ?

Hello all! I ran rm -rf on a wrong directory, noticed it and hit ctrl-c. Is there any way on a debian machine to tell what actually got deleted? As there were many dirs and files in this directory that I don't care for, I'd like to see if anything important was removed. Or do you know in... (4 Replies)
Discussion started by: thosch
4 Replies

7. Shell Programming and Scripting

directories are not getting removed

hello Everyone. I'm having the following problem: I have number of installation in the directory. each installation consists of executable file and directory. when I do the new installation I move old one to File_name-Time_stamp. this is done for executable and for directory. Everything is done... (6 Replies)
Discussion started by: slavam
6 Replies

8. UNIX for Dummies Questions & Answers

Will Old Files Be Removed

I have windows Xp installed, and decided to install Solaris Sun Unix 10. The hard disk was previousely partitioned into 5 partition. C: = Win98 D = WinXP and e,f,g,h are applications and so on. When istalling Sun Unix, will all the drives be removed, or I will specify where to install it. Thanks... (5 Replies)
Discussion started by: sunsation
5 Replies
Login or Register to Ask a Question