Sponsored Content
Top Forums UNIX for Dummies Questions & Answers How to delete or remove duplicate lines in a file Post 302335611 by reva on Monday 20th of July 2009 05:06:41 AM
Old 07-20-2009
Question

hi uniq is working but perl command is giving an error. I hav a dhought if i need to remove duplicates & keep just one of it in file then i can use uniq. but if i want to remove duplicate lines based on a criteria
for example
Code:
SIG    765   0   0   0   0   0.00  35.2000N   60.4000E   25      39 
SSR   765   0   0   0   0   0.00  34.5600N   65.4000E   25      67       89    
SSR 1390   5   0   0   0   0.00  39.8000N   64.4000E   20      56
LEE  1458   8   0   0   0   0.00  25.1000N   99.2000E    9                 56

now i want my output file to look lik
Code:
SSR   765   0   0   0   0   0.00  34.5600N   65.4000E   25      67       89    
SSR 1390   5   0   0   0   0.00  39.8000N   64.4000E   20      56
LEE  1458   8   0   0   0   0.00  25.1000N   99.2000E    9                 56

i mean to say only few specific colums should be checked if its same for example in the first file 2nd,3rd,4th,5th,6th,7th colums were same for 1st & 2nd row so i must remove the duplicate lines & retain a line which has maximum fields or colums in it.
Help me out if thier any command to check

Last edited by Yogesh Sawant; 07-20-2009 at 07:37 AM.. Reason: added code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

delete semi-duplicate lines from file?

Ok here's what I'm trying to do. I need to get a listing of all the mountpoints on a system into a file, which is easy enough, just using something like "mount | awk '{print $1}'" However, on a couple of systems, they have some mount points looking like this: /stage /stand /usr /MFPIS... (2 Replies)
Discussion started by: paqman
2 Replies

4. UNIX for Dummies Questions & Answers

Delete duplicate lines and print to file

OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script. So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup ... (2 Replies)
Discussion started by: bfurlong
2 Replies

5. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

6. Shell Programming and Scripting

remove duplicate lines from file linux/sh

greetings, i'm hoping there is a way to cat a file, remove duplicate lines and send that output to a new file. the file will always vary but be something similar to this: please keep in mind that the above could be eight occurrences of each hostname or it might simply have another four of an... (2 Replies)
Discussion started by: crimso
2 Replies

7. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a 50 MB file size

hi, Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file. (6 Replies)
Discussion started by: vsachan
6 Replies

9. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

10. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies
GIT-STRIPSPACE(1)						    Git Manual							 GIT-STRIPSPACE(1)

NAME
git-stripspace - Remove unnecessary whitespace SYNOPSIS
git stripspace [-s | --strip-comments] < input DESCRIPTION
Clean the input in the manner used by git for text such as commit messages, notes, tags and branch descriptions. With no arguments, this will: o remove trailing whitespace from all lines o collapse multiple consecutive empty lines into one empty line o remove empty lines from the beginning and end of the input o add a missing to the last line if necessary. In the case where the input consists entirely of whitespace characters, no output will be produced. NOTE: This is intended for cleaning metadata, prefer the --whitespace=fix mode of git-apply(1) for correcting whitespace of patches or files in the repository. OPTIONS
-s, --strip-comments Skip and remove all lines starting with #. EXAMPLES
Given the following noisy input with $ indicating the end of a line: |A brief introduction $ | $ |$ |A new paragraph$ |# with a commented-out line $ |explaining lots of stuff.$ |$ |# An old paragraph, also commented-out. $ | $ |The end.$ | $ Use git stripspace with no arguments to obtain: |A brief introduction$ |$ |A new paragraph$ |# with a commented-out line$ |explaining lots of stuff.$ |$ |# An old paragraph, also commented-out.$ |$ |The end.$ Use git stripspace --strip-comments to obtain: |A brief introduction$ |$ |A new paragraph$ |explaining lots of stuff.$ |$ |The end.$ GIT
Part of the git(1) suite Git 1.7.10.4 11/24/2012 GIT-STRIPSPACE(1)
All times are GMT -4. The time now is 08:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy