Sponsored Content
Top Forums UNIX for Dummies Questions & Answers How to delete or remove duplicate lines in a file Post 302335641 by radoulov on Monday 20th of July 2009 06:25:45 AM
Old 07-20-2009
Code:
perl -lane'  
    $k = join " ", @F[ 1 .. 6 ];
    $m{$k} = @F if $u{$k}++ and @F > $m{$k};
    push @r, $_;

    END {
        for (@r) {
            $k = join " ", (split)[ 1 .. 6 ];
            print if $u{$k} == 1 or split == $m{$k};
        }
    }' infile

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

delete semi-duplicate lines from file?

Ok here's what I'm trying to do. I need to get a listing of all the mountpoints on a system into a file, which is easy enough, just using something like "mount | awk '{print $1}'" However, on a couple of systems, they have some mount points looking like this: /stage /stand /usr /MFPIS... (2 Replies)
Discussion started by: paqman
2 Replies

4. UNIX for Dummies Questions & Answers

Delete duplicate lines and print to file

OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script. So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup ... (2 Replies)
Discussion started by: bfurlong
2 Replies

5. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

6. Shell Programming and Scripting

remove duplicate lines from file linux/sh

greetings, i'm hoping there is a way to cat a file, remove duplicate lines and send that output to a new file. the file will always vary but be something similar to this: please keep in mind that the above could be eight occurrences of each hostname or it might simply have another four of an... (2 Replies)
Discussion started by: crimso
2 Replies

7. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a 50 MB file size

hi, Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file. (6 Replies)
Discussion started by: vsachan
6 Replies

9. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

10. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies
Env(3pm)						 Perl Programmers Reference Guide						  Env(3pm)

NAME
Env - perl module that imports environment variables as scalars or arrays SYNOPSIS
use Env; use Env qw(PATH HOME TERM); use Env qw($SHELL @LD_LIBRARY_PATH); DESCRIPTION
Perl maintains environment variables in a special hash named %ENV. For when this access method is inconvenient, the Perl module "Env" allows environment variables to be treated as scalar or array variables. The "Env::import()" function ties environment variables with suitable names to global Perl variables with the same names. By default it ties all existing environment variables ("keys %ENV") to scalars. If the "import" function receives arguments, it takes them to be a list of variables to tie; it's okay if they don't yet exist. The scalar type prefix '$' is inferred for any element of this list not prefixed by '$' or '@'. Arrays are implemented in terms of "split" and "join", using $Config::Config{path_sep} as the delimiter. After an environment variable is tied, merely use it like a normal variable. You may access its value @path = split(/:/, $PATH); print join(" ", @LD_LIBRARY_PATH), " "; or modify it $PATH .= ":."; push @LD_LIBRARY_PATH, $dir; however you'd like. Bear in mind, however, that each access to a tied array variable requires splitting the environment variable's string anew. The code: use Env qw(@PATH); push @PATH, '.'; is equivalent to: use Env qw(PATH); $PATH .= ":."; except that if $ENV{PATH} started out empty, the second approach leaves it with the (odd) value "":."", but the first approach leaves it with ""."". To remove a tied environment variable from the environment, assign it the undefined value undef $PATH; undef @LD_LIBRARY_PATH; LIMITATIONS
On VMS systems, arrays tied to environment variables are read-only. Attempting to change anything will cause a warning. AUTHOR
Chip Salzenberg <chip@fin.uucp> and Gregor N. Purdy <gregor@focusresearch.com> perl v5.18.2 2013-11-04 Env(3pm)
All times are GMT -4. The time now is 06:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy