Sponsored Content
Top Forums Shell Programming and Scripting Command to remove duplicate lines with perl,sed,awk Post 302463127 by cola on Friday 15th of October 2010 07:53:04 PM
Old 10-15-2010
Command to remove duplicate lines with perl,sed,awk

Input:
Code:
hello hello
hello hello
monkey
donkey
hello hello
drink
dance
drink

Output should be:
Code:
hello hello
monkey
donkey
drink
dance

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Command/Script to remove duplicate lines from the file?

Hello, Can anyone tell Command/Script to remove duplicate lines from the file? (2 Replies)
Discussion started by: Rahulpict
2 Replies

2. Shell Programming and Scripting

Sed or Awk to remove specific lines

I have searched the forum for this - forgive me if I missed a previous post. I have the following file: blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah alter table "informix".esc_acct add constraint (foreign key (fi_id) references "informix".fi ... (5 Replies)
Discussion started by: Shoeless_Mike
5 Replies

3. Shell Programming and Scripting

How to remove lines before and after with awk / sed ?

Hi guys, I need to remove the pattern (ID=180), one line before and four lines after. Thanks. (5 Replies)
Discussion started by: ashimada
5 Replies

4. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

5. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

6. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

7. Shell Programming and Scripting

AWK Command to duplicate lines in a file?

Hi, I have a file with date in it like: UserString1 UserString2 UserString3 UserString4 UserString5 I need two entries for each line so it reads like UserString1 UserString1 UserString2 UserString2 etc. Can someone help me with the awk command please? Thanks (4 Replies)
Discussion started by: Grueben
4 Replies

8. Shell Programming and Scripting

Sed/awk/perl command to replace pattern in multiple lines

Hi I know sed and awk has options to give range of line numbers, but I need to replace pattern in specific lines Something like sed -e '1s,14s,26s/pattern/new pattern/' file name Can somebody help me in this.... I am fine with see/awk/perl Thank you in advance (9 Replies)
Discussion started by: dani777
9 Replies

9. Shell Programming and Scripting

Using sed, awk or perl to remove substring of all lines except the first

Greetings All, I would like to find all occurences of a pattern and delete a substring from the all matching lines EXCEPT the first. For example: 1234::group:user1,user2,user3,blah1,blah2,blah3 2222::othergroup:user9,user8 4444::othergroup2:user3,blah,blah,user1 1234::group3:user5,user1 ... (11 Replies)
Discussion started by: jacksolm
11 Replies

10. Shell Programming and Scripting

How to put the command to remove duplicate lines in my awk script?

I create a CGI in bash/html. My awk script looks like : echo "<table>" for fn in /var/www/cgi-bin/LPAR_MAP/*; do echo "<td>" echo "<PRE>" awk -F',|;' -v test="$test" ' NR==1 { split(FILENAME ,a,""); } $0 ~ test { if(!header++){ ... (12 Replies)
Discussion started by: Tim2424
12 Replies
Pod::Coverage::TrustPod(3pm)				User Contributed Perl Documentation			      Pod::Coverage::TrustPod(3pm)

NAME
Pod::Coverage::TrustPod - allow a module's pod to contain Pod::Coverage hints VERSION
version 0.100002 DESCRIPTION
This is a Pod::Coverage subclass (actually, a subclass of Pod::Coverage::CountParents) that allows the POD itself to declare certain symbol names trusted. Here is a sample Perl module: package Foo::Bar; =head1 NAME Foo::Bar - a bar at which fooes like to drink =head1 METHODS =head2 fee returns the bar tab =cut sub fee { ... } =head2 fie scoffs at bar tab =cut sub fie { ... } sub foo { ... } =begin Pod::Coverage foo =end Pod::Coverage =cut This file would report full coverage, because any non-empty lines inside a block of POD targeted to Pod::Coverage are treated as "trustme" patterns. Leading and trailing whitespace is stripped and the remainder is treated as a regular expression anchored at both ends. Remember, anywhere you could use "=begin" and "=end" as above, you could instead write: =for Pod::Coverage foo AUTHOR
Ricardo SIGNES <rjbs@cpan.org> COPYRIGHT AND LICENSE
This software is copyright (c) 2012 by Ricardo SIGNES. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. perl v5.14.2 2012-04-21 Pod::Coverage::TrustPod(3pm)
All times are GMT -4. The time now is 07:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy