Remove duplicates lines in a files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Remove duplicates lines in a files
# 1  
Old 11-29-2012
Remove duplicates lines in a files

Quote:
Hi Am using unix ksh..
I have a file called FILE
Code:
cat FILE
11/11/2012
11/11/2012
12/11/2012
15/11/2012

need to remove the duplicates dates ( ie 11/11/2012 is present two times i need remove one duplicates date )

Code:
Need outputs like this 
11/11/2012
12/11/2012
15/11/2012

I have tried using awk command and uniq command its not working
Code:
sort -n -k 1.7,1.10 -k 1.4,1.5 -k 1.1,1.2 FILE
awk !x[$0]++ FILE

Can any one help me....
# 2  
Old 11-29-2012
What's not working with the uniq or the awk command? They work fine for me.

Show error message or the output you get.
# 3  
Old 11-29-2012
Code:
+ awk $0 != "//" FILE2
+ 1>> FILE4
+ sort -n -k 1.7,1.10 -k 1.4,1.5 -k 1.1,1.2 FILE4
19/11/2012
19/11/2012 
21/11/2012
22/11/2012
27/11/2012
28/11/2012 
+ cat FILE4
19/11/2012 
19/11/2012
21/11/2012
22/11/2012
27/11/2012
28/11/2012 
+ uniq FILE4 FILE3
+ cat FILE3
19/11/2012 
19/11/2012
21/11/2012
22/11/2012
27/11/2012
28/11/2012 


For me its not working ?
getting the duplicates when am using uniq command ?

Last edited by Venkatesh1; 11-29-2012 at 07:58 AM..
# 4  
Old 11-29-2012
Post the output of od -c < FILE4 using code tags.
# 5  
Old 11-30-2012
sort and uniq

I have an input file like so:
Code:
apples
peaches
pears
apples
peaches
pears
apples
peaches
pears

I sort it like so:
Code:
sort oldfruit > myfruit

I then use the uniq command like so:
Code:
uniq myfruit newfruit

and the result is:
Code:
apples
peaches
pears

This was done in AIX 5.2 from the command line, but should work the same in ksh.

Last edited by zaxxon; 11-30-2012 at 10:30 AM.. Reason: misunderstood intentions completely. reversed everything, sorry.
# 6  
Old 11-30-2012
I guess you are having a space in file4 after the first occurrence of 19/11/2012 - that why it is different from the second line.
# 7  
Old 11-30-2012
Again this topic has nothing to do in advanced... moving to "dummies"
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove lines from output in files using awk

I have two large files (~250GB) that I am trying to remove the where GT: 0/0 or 1/1 or 2/2 for both files. I was going to use a bash with the below awk, which I think will find each line but how do I remove that line is that condition is found? Thank you :). Input 20 60055 . A ... (4 Replies)
Discussion started by: cmccabe
4 Replies

2. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

3. Shell Programming and Scripting

Remove duplicates from two files using n characters for comparison

Hi All, I have two files file1 123456CRTGHG 125437CRNDGF 126537CRDDGF file2 123456CRTZHC 124567CJHGHG 125987CJHGDF I need to compare the two files and any records in file 1 and 2 based on initial n characters (6 in example) need to be ignored. string separated by unprintable... (2 Replies)
Discussion started by: Bruble
2 Replies

4. Shell Programming and Scripting

How to remove the last 3 lines from many files?

Hello, I need to run a command or shell script that will remove the last 3 lines from every .js file that is under the directory /var/ww/vhost/ Can you please help ? thank you. (22 Replies)
Discussion started by: MaRiOsGR
22 Replies

5. Shell Programming and Scripting

Compare one files with strings from another + remove lines

Have two files and want to compare the content of file1 with file2. When matched remove the line. awk 'NR==FNR {b; next} !(b in $0)' file1 file2file1 1. if match 2. removefile2 1. this line has to be removed if match 2. this line has a match, remove 3. this line has no match, no removingThe... (3 Replies)
Discussion started by: sdf
3 Replies

6. Shell Programming and Scripting

Remove the files that have less than certain lines

Hi all, I'm a newbie and I'm sorry if my question is too simple. I'm having problem to delete the files that have less than certain lines, say 16. #!/bin/tcsh set filen = `sh -c 'ls *csv 2> /dev/null'` foreach fil (${filen}) if ]; then rm -f ${filen} fi end exit ... (2 Replies)
Discussion started by: GoldenFire
2 Replies

7. Shell Programming and Scripting

compare files and then remove some lines

Hi everyone I have a dilemma and I'm hoping someone has an answer for me. I have two files: # cat masterfile line3 line4 line5 line6 line7 # cat tempfile line1 line2 line3 line4 I want to compare tempfile with masterfile. (3 Replies)
Discussion started by: soliberus
3 Replies

8. Shell Programming and Scripting

How to delete lines in a file that have duplicates or derive the lines that aper once

Input: a b b c d d I need: a c I know how to get this (the lines that have duplicates) : b d sort file | uniq -d But i need opossite of this. I have searched the forum and other places as well, but have found solution for everything except this variant of the problem. (3 Replies)
Discussion started by: necroman08
3 Replies

9. Shell Programming and Scripting

compare two files and to remove the matching lines on both the files

I have two files and need to compare the two files and to remove the matching lines from both the files (4 Replies)
Discussion started by: shellscripter
4 Replies

10. Shell Programming and Scripting

Shell script to remove duplicates lines in a file

Hi, I am writing a shell script that needs to remove duplicate lines within a file by category. example: section a a c b a section b a b a c I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section... (1 Reply)
Discussion started by: RichElks
1 Replies
Login or Register to Ask a Question