Remove duplicate lines from a file

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still...

2. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323...

3. Shell Programming and Scripting

Remove duplicate lines from a 50 MB file size

hi, Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file.

4. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later!

5. Shell Programming and Scripting

Remove duplicate lines from first file comparing second file

Hi, I have two files with below data:: file1:- 123|aaa|ppp 445|fff|yyy 999|ttt|jjj 555|hhh|hhh file2:- 445|fff|yyy 555|hhh|hhh The records present in file1, not present in file 2 should be writtent to the out put file. output:- 123|aaa|ppp 999|ttt|jjj Is there any one line...

6. Shell Programming and Scripting

remove duplicate lines from file linux/sh

greetings, i'm hoping there is a way to cat a file, remove duplicate lines and send that output to a new file. the file will always vary but be something similar to this: please keep in mind that the above could be eight occurrences of each hostname or it might simply have another four of an...

7. UNIX for Dummies Questions & Answers

How to delete or remove duplicate lines in a file

Hi please help me how to remove duplicate lines in any file. I have a file having huge number of lines. i want to remove selected lines in it. And also if there exists duplicate lines, I want to delete the rest & just keep one of them. Please help me with any unix commands or even fortran...

8. Shell Programming and Scripting

Command/Script to remove duplicate lines from the file?

Hello, Can anyone tell Command/Script to remove duplicate lines from the file?

9. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but...

10. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file...

LEARN ABOUT CENTOS

peekfd

PEEKFD(1)							   User Commands							 PEEKFD(1)

NAME

       peekfd - peek at file descriptors of running processes

SYNOPSIS

       peekfd [-8,--eight-bit-clean] [-n,--no-headers] [-f,--follow] [-d,--duplicates-removed] [-V,--version] [-h,--help] pid [fd] [fd] ...

DESCRIPTION

       peekfd  attaches to a running process and intercepts all reads and writes to file descriptors.  You can specify the desired file descriptor
       numbers or dump all of them.

OPTIONS

       -8     Do no post-processing on the bytes being read or written.

       -n     Do not display headers indicating the source of the bytes dumped.

       -c     Also dump the requested file descriptor activity in any new child processes that are created.

       -d     Remove duplicate read/writes from the output.  If you're looking at a tty with echo, you might want this.

       -V     Display a version string.

       -h     Display a help message.

FILES

       /proc/*/fd
	      Not used but useful for the user to look at to get good file descriptor numbers.

ENVIRONMENT

       None.

DIAGNOSTICS

       The following diagnostics may be issued on stderr:

       Error attaching to pid ...
	      An unknown error occurred while attempted to attach to a process..  you may need to be root.

BUGS

       Probably lots.  Don't be surprised if the process you are monitoring dies.

AUTHOR

       Trent Waddington <trent.waddington@gmail.com>

SEE ALSO

       ttysnoop(8)

psmisc								    2012-07-28								 PEEKFD(1)

Shell Programming and Scripting