Sponsored Content
Full Discussion: Removing duplicated lines??
Top Forums UNIX for Dummies Questions & Answers Removing duplicated lines?? Post 302600988 by dinesh1178 on Wednesday 22nd of February 2012 03:32:18 PM
Old 02-22-2012
Quote:
Originally Posted by Scrutinizer
@dinesh: I don't see how sort could work, wouldn't that mess up the order of the lines?
I thought order of line is not important, as requester also tried it. By seeing the reply, it seems sort worked fine for him.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicated lines without sort

Hi Just wondering whether or not I can remove duplicated lines without sort For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal. Normally, I would do who | cut -d" " -f1 |... (6 Replies)
Discussion started by: lalelle
6 Replies

2. Shell Programming and Scripting

Help removing lines with duplicated columns

Hi Guys... Please Could you help me with the following ? aaaa bbbb cccc sdsd aaaa bbbb cccc qwer as you can see, the 2 lines are matched in three fields... how can I delete this pupicate ? I mean to delete the second one if 3 fields were duplicated ? Thanks (14 Replies)
Discussion started by: yahyaaa
14 Replies

3. UNIX for Dummies Questions & Answers

duplicated lines not recognized by sort and uniq

Hello all, I've got a strange behaviour of sort and uniq commands: they do not recognise apparently duplicated lines in a file (already sorted). The lines are identical by eye, but they must differ in smth, because when they are put in two files, those have slightly different size. What can make... (8 Replies)
Discussion started by: roussine
8 Replies

4. Shell Programming and Scripting

awk to count duplicated lines

We have an input file as follows: 2010-09-15-12.41.15 2010-09-15-12.41.15 2010-09-15-12.41.24 2010-09-15-12.41.24 2010-09-15-12.41.24 2010-09-15-12.41.24 2010-09-15-12.41.25 2010-09-15-12.41.26 2010-09-15-12.41.26 2010-09-15-12.41.26 2010-09-15-12.41.26 2010-09-15-12.41.26... (3 Replies)
Discussion started by: ux4me
3 Replies

5. Shell Programming and Scripting

awk to insert duplicated lines

Dear All, Suppose I have a file: 1 1 1 1 2 2 2 2 3 3 3 3I want to insert new line under each old line so that the file would become: 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3How can this be accomplished using awk (or sed)? (5 Replies)
Discussion started by: littlewenwen
5 Replies

6. Shell Programming and Scripting

How to remove duplicated lines?

Hi, if i have a file like this: Query=1 a a b c c c d Query=2 b b b c c e . . . (7 Replies)
Discussion started by: the_simpsons
7 Replies

7. Shell Programming and Scripting

Removing duplicated first field rows

Hello, I am trying to eliminate rows where the first field is duplicated, leaving the row where the last field is "NET". Data file: 345234|22.34|LST 546543|55.33|LST 793929|98.23|LST 793929|64.69|NET 149593|49.22|LST Desired output: 345234|22.34|LST 546543|55.33|LST... (2 Replies)
Discussion started by: palex
2 Replies

8. UNIX for Dummies Questions & Answers

Removing PATTERN from txt without removing lines and general text formatting

Hi Everybody! First post! Totally noobie. I'm using the terminal to read a poorly formatted book. The text file contains, in the middle of paragraphs, hyphenation to split words that are supposed to be on multiple pages. It looks ve -- ry much like this. I was hoping to use grep -v " -- "... (5 Replies)
Discussion started by: AxeHandle
5 Replies

9. Shell Programming and Scripting

Deleting lines containing duplicated strings

Dear all, I always appreciate your help. I would like to delete lines containing duplicated strings in the second column. test.txt 658 invert_d2e_q_reg_0_/Qalu_ecl_zlow_e 0.825692 659 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692 660 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692... (1 Reply)
Discussion started by: jypark22
1 Replies
sort(3pm)						 Perl Programmers Reference Guide						 sort(3pm)

NAME
sort - perl pragma to control sort() behaviour SYNOPSIS
use sort 'stable'; # guarantee stability use sort '_quicksort'; # use a quicksort algorithm use sort '_mergesort'; # use a mergesort algorithm use sort 'defaults'; # revert to default behavior no sort 'stable'; # stability not important use sort '_qsort'; # alias for quicksort my $current = sort::current(); # identify prevailing algorithm DESCRIPTION
With the "sort" pragma you can control the behaviour of the builtin "sort()" function. In Perl versions 5.6 and earlier the quicksort algorithm was used to implement "sort()", but in Perl 5.8 a mergesort algorithm was also made available, mainly to guarantee worst case O(N log N) behaviour: the worst case of quicksort is O(N**2). In Perl 5.8 and later, quick- sort defends against quadratic behaviour by shuffling large arrays before sorting. A stable sort means that for records that compare equal, the original input ordering is preserved. Mergesort is stable, quicksort is not. Stability will matter only if elements that compare equal can be distinguished in some other way. That means that simple numerical and lexical sorts do not profit from stability, since equal elements are indistinguishable. However, with a comparison such as { substr($a, 0, 3) cmp substr($b, 0, 3) } stability might matter because elements that compare equal on the first 3 characters may be distinguished based on subsequent characters. In Perl 5.8 and later, quicksort can be stabilized, but doing so will add overhead, so it should only be done if it matters. The best algorithm depends on many things. On average, mergesort does fewer comparisons than quicksort, so it may be better when compli- cated comparison routines are used. Mergesort also takes advantage of pre-existing order, so it would be favored for using "sort()" to merge several sorted arrays. On the other hand, quicksort is often faster for small arrays, and on arrays of a few distinct values, repeated many times. You can force the choice of algorithm with this pragma, but this feels heavy-handed, so the subpragmas beginning with a "_" may not persist beyond Perl 5.8. The default algorithm is mergesort, which will be stable even if you do not explicitly demand it. But the stability of the default sort is a side-effect that could change in later versions. If stability is important, be sure to say so with a use sort 'stable'; The "no sort" pragma doesn't forbid what follows, it just leaves the choice open. Thus, after no sort qw(_mergesort stable); a mergesort, which happens to be stable, will be employed anyway. Note that no sort "_quicksort"; no sort "_mergesort"; have exactly the same effect, leaving the choice of sort algorithm open. CAVEATS
This pragma is not lexically scoped: its effect is global to the program it appears in. That means the following will probably not do what you expect, because both pragmas take effect at compile time, before either "sort()" happens. { use sort "_quicksort"; print sort::current . " "; @a = sort @b; } { use sort "stable"; print sort::current . " "; @c = sort @d; } # prints: # quicksort stable # quicksort stable You can achieve the effect you probably wanted by using "eval()" to defer the pragmas until run time. Use the quoted argument form of "eval()", not the BLOCK form, as in eval { use sort "_quicksort" }; # WRONG or the effect will still be at compile time. Reset to default options before selecting other subpragmas (in case somebody carelessly left them on) and after sorting, as a courtesy to others. { eval 'use sort qw(defaults _quicksort)'; # force quicksort eval 'no sort "stable"'; # stability not wanted print sort::current . " "; @a = sort @b; eval 'use sort "defaults"'; # clean up, for others } { eval 'use sort qw(defaults stable)'; # force stability print sort::current . " "; @c = sort @d; eval 'use sort "defaults"'; # clean up, for others } # prints: # quicksort # stable Scoping for this pragma may change in future versions. perl v5.8.0 2002-06-01 sort(3pm)
All times are GMT -4. The time now is 09:42 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy