Sponsored Content
Top Forums Shell Programming and Scripting Command to remove duplicate lines with perl,sed,awk Post 302463137 by kurumi on Friday 15th of October 2010 10:37:40 PM
Old 10-15-2010
Code:
$ ruby -00 -ne 'puts $_.split("\n").uniq' file
hello hello
monkey
donkey
drink
dance

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Command/Script to remove duplicate lines from the file?

Hello, Can anyone tell Command/Script to remove duplicate lines from the file? (2 Replies)
Discussion started by: Rahulpict
2 Replies

2. Shell Programming and Scripting

Sed or Awk to remove specific lines

I have searched the forum for this - forgive me if I missed a previous post. I have the following file: blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah alter table "informix".esc_acct add constraint (foreign key (fi_id) references "informix".fi ... (5 Replies)
Discussion started by: Shoeless_Mike
5 Replies

3. Shell Programming and Scripting

How to remove lines before and after with awk / sed ?

Hi guys, I need to remove the pattern (ID=180), one line before and four lines after. Thanks. (5 Replies)
Discussion started by: ashimada
5 Replies

4. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

5. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

6. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

7. Shell Programming and Scripting

AWK Command to duplicate lines in a file?

Hi, I have a file with date in it like: UserString1 UserString2 UserString3 UserString4 UserString5 I need two entries for each line so it reads like UserString1 UserString1 UserString2 UserString2 etc. Can someone help me with the awk command please? Thanks (4 Replies)
Discussion started by: Grueben
4 Replies

8. Shell Programming and Scripting

Sed/awk/perl command to replace pattern in multiple lines

Hi I know sed and awk has options to give range of line numbers, but I need to replace pattern in specific lines Something like sed -e '1s,14s,26s/pattern/new pattern/' file name Can somebody help me in this.... I am fine with see/awk/perl Thank you in advance (9 Replies)
Discussion started by: dani777
9 Replies

9. Shell Programming and Scripting

Using sed, awk or perl to remove substring of all lines except the first

Greetings All, I would like to find all occurences of a pattern and delete a substring from the all matching lines EXCEPT the first. For example: 1234::group:user1,user2,user3,blah1,blah2,blah3 2222::othergroup:user9,user8 4444::othergroup2:user3,blah,blah,user1 1234::group3:user5,user1 ... (11 Replies)
Discussion started by: jacksolm
11 Replies

10. Shell Programming and Scripting

How to put the command to remove duplicate lines in my awk script?

I create a CGI in bash/html. My awk script looks like : echo "<table>" for fn in /var/www/cgi-bin/LPAR_MAP/*; do echo "<td>" echo "<PRE>" awk -F',|;' -v test="$test" ' NR==1 { split(FILENAME ,a,""); } $0 ~ test { if(!header++){ ... (12 Replies)
Discussion started by: Tim2424
12 Replies
RUBY-SWITCH(1)															    RUBY-SWITCH(1)

NAME
ruby-switch - switch between different Ruby interpreters USAGE
ruby-switch --list ruby-switch --check ruby-switch --set RUBYVERSION ruby-switch --auto DESCRIPTION
ruby-switch can be used to easily switch to different Ruby interpreters as the default system-wide interpreter for your Debian system. When run with --list, all supported Ruby interpreters are listed. When --check is passed, ruby-switch will check which Ruby interpreter is currently being used. If the settings are inconsistent -- e.g. `ruby` is Ruby 1.8 and `gem` is using Ruby 1.9.1, ruby-switch will issue a big warning. When --set RUBYINTERPRETER is used ruby-switch will switch your system to the corresponding Ruby interpreter. This includes, for example, the default implementations for the following programs: ruby, gem, irb, erb, testrb, rdoc, ri. ruby-switch --set auto will make your system use the default Ruby interpreter currently suggested by Debian. OPTIONS
-h, --help Displays the help and exits. A NOTE ON RUBY 1.9.x Ruby uses two parallel versioning schemes: the `Ruby library compatibility version' (1.9.1 at the time of writing this), which is similar to a library SONAME, and the `Ruby version' (1.9.3 is about to be released at the time of writing). Ruby packages in Debian are named using the Ruby library compatibility version, which is sometimes confusing for users who do not follow Ruby development closely. ruby-switch also uses the Ruby library compatibility version, so specifying `ruby1.9.1' might give you Ruby with version 1.9.2, or with version 1.9.3, depending on the current Ruby version of the `ruby1.9.1' package. COPYRIGHT AND AUTHORS
Copyright (c) 2011, Antonio Terceiro <terceiro@debian.org> This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>. 2011-11-20 RUBY-SWITCH(1)
All times are GMT -4. The time now is 03:20 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy