Sponsored Content
Top Forums Shell Programming and Scripting Command to remove duplicate lines with perl,sed,awk Post 302463127 by cola on Friday 15th of October 2010 07:53:04 PM
Old 10-15-2010
Command to remove duplicate lines with perl,sed,awk

Input:
Code:
hello hello
hello hello
monkey
donkey
hello hello
drink
dance
drink

Output should be:
Code:
hello hello
monkey
donkey
drink
dance

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Command/Script to remove duplicate lines from the file?

Hello, Can anyone tell Command/Script to remove duplicate lines from the file? (2 Replies)
Discussion started by: Rahulpict
2 Replies

2. Shell Programming and Scripting

Sed or Awk to remove specific lines

I have searched the forum for this - forgive me if I missed a previous post. I have the following file: blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah alter table "informix".esc_acct add constraint (foreign key (fi_id) references "informix".fi ... (5 Replies)
Discussion started by: Shoeless_Mike
5 Replies

3. Shell Programming and Scripting

How to remove lines before and after with awk / sed ?

Hi guys, I need to remove the pattern (ID=180), one line before and four lines after. Thanks. (5 Replies)
Discussion started by: ashimada
5 Replies

4. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

5. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

6. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

7. Shell Programming and Scripting

AWK Command to duplicate lines in a file?

Hi, I have a file with date in it like: UserString1 UserString2 UserString3 UserString4 UserString5 I need two entries for each line so it reads like UserString1 UserString1 UserString2 UserString2 etc. Can someone help me with the awk command please? Thanks (4 Replies)
Discussion started by: Grueben
4 Replies

8. Shell Programming and Scripting

Sed/awk/perl command to replace pattern in multiple lines

Hi I know sed and awk has options to give range of line numbers, but I need to replace pattern in specific lines Something like sed -e '1s,14s,26s/pattern/new pattern/' file name Can somebody help me in this.... I am fine with see/awk/perl Thank you in advance (9 Replies)
Discussion started by: dani777
9 Replies

9. Shell Programming and Scripting

Using sed, awk or perl to remove substring of all lines except the first

Greetings All, I would like to find all occurences of a pattern and delete a substring from the all matching lines EXCEPT the first. For example: 1234::group:user1,user2,user3,blah1,blah2,blah3 2222::othergroup:user9,user8 4444::othergroup2:user3,blah,blah,user1 1234::group3:user5,user1 ... (11 Replies)
Discussion started by: jacksolm
11 Replies

10. Shell Programming and Scripting

How to put the command to remove duplicate lines in my awk script?

I create a CGI in bash/html. My awk script looks like : echo "<table>" for fn in /var/www/cgi-bin/LPAR_MAP/*; do echo "<td>" echo "<PRE>" awk -F',|;' -v test="$test" ' NR==1 { split(FILENAME ,a,""); } $0 ~ test { if(!header++){ ... (12 Replies)
Discussion started by: Tim2424
12 Replies
Sub::Install(3) 					User Contributed Perl Documentation					   Sub::Install(3)

NAME
Sub::Install - install subroutines into packages easily VERSION
version 0.927 SYNOPSIS
use Sub::Install; Sub::Install::install_sub({ code => sub { ... }, into => $package, as => $subname }); DESCRIPTION
This module makes it easy to install subroutines into packages without the unsightly mess of "no strict" or typeglobs lying about where just anyone can see them. FUNCTIONS
install_sub Sub::Install::install_sub({ code => &subroutine, into => "Finance::Shady", as => 'launder', }); This routine installs a given code reference into a package as a normal subroutine. The above is equivalent to: no strict 'refs'; *{"Finance::Shady" . '::' . "launder"} = &subroutine; If "into" is not given, the sub is installed into the calling package. If "code" is not a code reference, it is looked for as an existing sub in the package named in the "from" parameter. If "from" is not given, it will look in the calling package. If "as" is not given, and if "code" is a name, "as" will default to "code". If "as" is not given, but if "code" is a code ref, Sub::Install will try to find the name of the given code ref and use that as "as". That means that this code: Sub::Install::install_sub({ code => 'twitch', from => 'Person::InPain', into => 'Person::Teenager', as => 'dance', }); is the same as: package Person::Teenager; Sub::Install::install_sub({ code => Person::InPain->can('twitch'), as => 'dance', }); reinstall_sub This routine behaves exactly like "install_sub", but does not emit a warning if warnings are on and the destination is already defined. install_installers This routine is provided to allow Sub::Install compatibility with Sub::Installer. It installs "install_sub" and "reinstall_sub" methods into the package named by its argument. Sub::Install::install_installers('Code::Builder'); # just for us, please Code::Builder->install_sub({ name => $code_ref }); Sub::Install::install_installers('UNIVERSAL'); # feeling lucky, punk? Anything::At::All->install_sub({ name => $code_ref }); The installed installers are similar, but not identical, to those provided by Sub::Installer. They accept a single hash as an argument. The key/value pairs are used as the "as" and "code" parameters to the "install_sub" routine detailed above. The package name on which the method is called is used as the "into" parameter. Unlike Sub::Installer's "install_sub" will not eval strings into code, but will look for named code in the calling package. EXPORTS
Sub::Install exports "install_sub" and "reinstall_sub" only if they are requested. exporter Sub::Install has a never-exported subroutine called "exporter", which is used to implement its "import" routine. It takes a hashref of named arguments, only one of which is currently recognize: "exports". This must be an arrayref of subroutines to offer for export. This routine is mainly for Sub::Install's own consumption. Instead, consider Sub::Exporter. SEE ALSO
Sub::Installer This module is (obviously) a reaction to Damian Conway's Sub::Installer, which does the same thing, but does it by getting its greasy fingers all over UNIVERSAL. I was really happy about the idea of making the installation of coderefs less ugly, but I couldn't bring myself to replace the ugliness of typeglobs and loosened strictures with the ugliness of UNIVERSAL methods. Sub::Exporter This is a complete Exporter.pm replacement, built atop Sub::Install. EXTRA CREDITS
Several of the tests are adapted from tests that shipped with Damian Conway's Sub-Installer distribution. AUTHOR
Ricardo SIGNES <rjbs@cpan.org> COPYRIGHT AND LICENSE
This software is copyright (c) 2005 by Ricardo SIGNES. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. perl v5.18.2 2013-10-15 Sub::Install(3)
All times are GMT -4. The time now is 06:31 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy