Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove duplicates by keeping the order intact

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove duplicates by keeping the order intact
# 1  
Old 11-22-2013
Remove duplicates by keeping the order intact
Hello friends,

I have a file with duplicate lines. I could eliminate duplicate lines by running

Code:
sort <file> |uniq >uniq_file

and it works fine BUT it changes the order of the entries as it we did "sort".

I need to remove duplicates and also need to keep the order/sequence of entries. I think i can do this by looping the contents and check in the uniq_file and create new file but it doesn't look optimal.

Please advise!
TIA
# 2  
Old 11-22-2013
Using awk:

Code:
awk '!a[$0]++' file

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Remove duplicates in a dataframe (table) keeping all the different cells of just one of the columns

Hello all, I need to filter a dataframe composed of several columns of data to remove the duplicates according to one of the columns. I did it with pandas. In the main time, I need that the last column that contains all different data ( not redundant) is conserved in the output like this: A ... (5 Replies)
Discussion started by: pedro88
5 Replies

2. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

3. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

4. Shell Programming and Scripting

Remove last few characters in a file but keeping Header and trailer intact

Hi All, I am trying write a simple command using AWK and SED to this but without any success. Here is what I am using: head -1 test1.txt>test2.txt|sed '1d;$d' test1.txt|awk '{print substr($0,0,(length($0)-2))}' >>test2.txt|tail -1 test1.txt>>test2.txt Input: Header 1234567 abcdefgh... (2 Replies)
Discussion started by: nvuradi
2 Replies

5. Shell Programming and Scripting

Keeping the number intact

Currently I have the following to separate the numeric values. However the decimal point get separated. ls -lrt *smp*.cmd | awk '{print $NF}' | sed 's/^.*\///' | sed 's/\(*\)/ & /g' As an example on the files n02-z30-dsr65-terr0.50-dc0.05-4x3smp.cmd... (8 Replies)
Discussion started by: kristinu
8 Replies

6. UNIX for Dummies Questions & Answers

sort by keeping the headings intact?

Hi all, I have a file with 3 columns separated by space. Each column has a heading. I want to sort according to the values in the 2nd column (ascending order). Ex. Name rank direction goory 0.05 --+ laby 0.0006 --- namy 0.31 -+- ....etc. Output should be Name rank direction laby... (3 Replies)
Discussion started by: Unilearn
3 Replies

7. Shell Programming and Scripting

Help with file editing while keeping file format intact

Hi, I am having a file which is fix length and comma seperated. And I want to replace values for one column. I am reading file line by line in variable $LINE and then replacing the string. Problem is after changing value and writing new file temp5.txt, formating of original file is getting... (8 Replies)
Discussion started by: Mruda
8 Replies

8. Solaris

Remove the zfs snapshot keeping the original volume and clone

I created a snapshot and subsequent clone of a zfs volume. But now i 'm not able to remove the snapshot it gives me following error zfs destroy newpool/ldom2/zdisk4@bootimg cannot destroy 'newpool/ldom2/zdisk4@bootimg': snapshot has dependent clones use '-R' to destroy the following... (7 Replies)
Discussion started by: fugitive
7 Replies

9. Shell Programming and Scripting

Remove text between headers while leaving headers intact

Hi, I'm trying to strip all lines between two headers in a file: ### BEGIN ### Text to remove, contains all kinds of characters ... Antispyware-Downloadserver.com (Germany)=http://www.antispyware-downloadserver.c om/updates/ Antispyware-Downloadserver.com #2... (3 Replies)
Discussion started by: Trones
3 Replies

10. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies
Login or Register to Ask a Question