Sponsored Content
Top Forums UNIX for Dummies Questions & Answers How to delete or remove duplicate lines in a file Post 302335695 by reva on Monday 20th of July 2009 08:16:40 AM
Old 07-20-2009
I dont know to use perl and i am not understanding the code which you have given..so if u tell me any other simple command like awk or anything else it will be helpful.I am new to linux .
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

delete semi-duplicate lines from file?

Ok here's what I'm trying to do. I need to get a listing of all the mountpoints on a system into a file, which is easy enough, just using something like "mount | awk '{print $1}'" However, on a couple of systems, they have some mount points looking like this: /stage /stand /usr /MFPIS... (2 Replies)
Discussion started by: paqman
2 Replies

4. UNIX for Dummies Questions & Answers

Delete duplicate lines and print to file

OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script. So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup ... (2 Replies)
Discussion started by: bfurlong
2 Replies

5. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

6. Shell Programming and Scripting

remove duplicate lines from file linux/sh

greetings, i'm hoping there is a way to cat a file, remove duplicate lines and send that output to a new file. the file will always vary but be something similar to this: please keep in mind that the above could be eight occurrences of each hostname or it might simply have another four of an... (2 Replies)
Discussion started by: crimso
2 Replies

7. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a 50 MB file size

hi, Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file. (6 Replies)
Discussion started by: vsachan
6 Replies

9. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

10. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies
SHTOOL-PATH.TMP(1)					      GNU Portable Shell Tool						SHTOOL-PATH.TMP(1)

NAME
shtool-path - GNU shtool command dealing with shell path variables SYNOPSIS
shtool path [-s|--suppress] [-r|--reverse] [-d|--dirname] [-b|--basename] [-m|--magic] [-p|--path path] str [str ...] DESCRIPTION
This command deals with shell $PATH variables. It can find a program through one or more filenames given by one or more str arguments. It prints the absolute filesystem path to the program displayed on "stdout" plus an exit code of 0 if it was really found. OPTIONS
The following command line options are available. -s, --suppress Supress output. Useful to only test whether a program exists with the help of the return code. -r, --reverse Transform a forward path to a subdirectory into a reverse path. -d, --dirname Output the directory name of str. -b, --basename Output the base name of str. -m, --magic Enable advanced magic search for ""perl"" and ""cpp"". -p, --path path Search in path. Default is to search in $PATH. EXAMPLE
# shell script awk=`shtool path -p "${PATH}:." gawk nawk awk` perl=`shtool path -m perl` cpp=`shtool path -m cpp` revpath=`shtool path -r path/to/subdir` HISTORY
The GNU shtool path command was originally written by Ralf S. Engelschall <rse@engelschall.com> in 1998 for Apache. It was later taken over into GNU shtool. SEE ALSO
shtool(1), which(1). 18-Jul-2008 shtool 2.0.8 SHTOOL-PATH.TMP(1)
All times are GMT -4. The time now is 02:58 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy