How to delete or remove duplicate lines in a file

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still...

2. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some...

3. Shell Programming and Scripting

Remove duplicate lines from a 50 MB file size

hi, Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file.

4. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later!

5. Shell Programming and Scripting

remove duplicate lines from file linux/sh

greetings, i'm hoping there is a way to cat a file, remove duplicate lines and send that output to a new file. the file will always vary but be something similar to this: please keep in mind that the above could be eight occurrences of each hostname or it might simply have another four of an...

6. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to...

7. UNIX for Dummies Questions & Answers

Delete duplicate lines and print to file

OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script. So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup ...

8. Shell Programming and Scripting

delete semi-duplicate lines from file?

Ok here's what I'm trying to do. I need to get a listing of all the mountpoints on a system into a file, which is easy enough, just using something like "mount | awk '{print $1}'" However, on a couple of systems, they have some mount points looking like this: /stage /stand /usr /MFPIS...

9. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but...

10. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file...

LEARN ABOUT LINUX

git-stripspace

GIT-STRIPSPACE(1)						    Git Manual							 GIT-STRIPSPACE(1)

NAME

       git-stripspace - Remove unnecessary whitespace

SYNOPSIS

       git stripspace [-s | --strip-comments] < input

DESCRIPTION

       Clean the input in the manner used by Git for text such as commit messages, notes, tags and branch descriptions.

       With no arguments, this will:

       o   remove trailing whitespace from all lines

       o   collapse multiple consecutive empty lines into one empty line

       o   remove empty lines from the beginning and end of the input

       o   add a missing 
 to the last line if necessary.

       In the case where the input consists entirely of whitespace characters, no output will be produced.

       NOTE: This is intended for cleaning metadata, prefer the --whitespace=fix mode of git-apply(1) for correcting whitespace of patches or
       files in the repository.

OPTIONS

       -s, --strip-comments
	   Skip and remove all lines starting with comment character (default #).

       -c, --comment-lines
	   Prepend comment character and blank to each line. Lines will automatically be terminated with a newline. On empty lines, only the
	   comment character will be prepended.

EXAMPLES

       Given the following noisy input with $ indicating the end of a line:

	   |A brief introduction   $
	   |   $
	   |$
	   |A new paragraph$
	   |# with a commented-out line    $
	   |explaining lots of stuff.$
	   |$
	   |# An old paragraph, also commented-out. $
	   |	  $
	   |The end.$
	   |  $

       Use git stripspace with no arguments to obtain:

	   |A brief introduction$
	   |$
	   |A new paragraph$
	   |# with a commented-out line$
	   |explaining lots of stuff.$
	   |$
	   |# An old paragraph, also commented-out.$
	   |$
	   |The end.$

       Use git stripspace --strip-comments to obtain:

	   |A brief introduction$
	   |$
	   |A new paragraph$
	   |explaining lots of stuff.$
	   |$
	   |The end.$

GIT

       Part of the git(1) suite

Git 1.8.5.3							    01/14/2014							 GIT-STRIPSPACE(1)

UNIX for Dummies Questions & Answers