Remove duplicates by keeping the order intact Post: 302876315

Login or Register to Ask a Question and Join Our Community

Sponsored Content

Top Forums Shell Programming and Scripting Remove duplicates by keeping the order intact Post 302876315 by magnus29 on Friday 22nd of November 2013 09:55:48 PM

Old

11-22-2013

Registered User

Remove duplicates by keeping the order intact

Hello friends,

I have a file with duplicate lines. I could eliminate duplicate lines by running

Code:

sort <file> |uniq >uniq_file

and it works fine BUT it changes the order of the entries as it we did "sort".

I need to remove duplicates and also need to keep the order/sequence of entries. I think i can do this by looping the contents and check in the uniq_file and create new file but it doesn't look optimal.

Please advise!
TIA

magnus29

View Public Profile for magnus29

Find all posts by magnus29

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra...

2. Shell Programming and Scripting

Remove text between headers while leaving headers intact

Hi, I'm trying to strip all lines between two headers in a file: ### BEGIN ### Text to remove, contains all kinds of characters ... Antispyware-Downloadserver.com (Germany)=http://www.antispyware-downloadserver.c om/updates/ Antispyware-Downloadserver.com #2...

3. Solaris

Remove the zfs snapshot keeping the original volume and clone

I created a snapshot and subsequent clone of a zfs volume. But now i 'm not able to remove the snapshot it gives me following error zfs destroy newpool/ldom2/zdisk4@bootimg cannot destroy 'newpool/ldom2/zdisk4@bootimg': snapshot has dependent clones use '-R' to destroy the following...

4. Shell Programming and Scripting

Help with file editing while keeping file format intact

Hi, I am having a file which is fix length and comma seperated. And I want to replace values for one column. I am reading file line by line in variable $LINE and then replacing the string. Problem is after changing value and writing new file temp5.txt, formating of original file is getting...

5. UNIX for Dummies Questions & Answers

sort by keeping the headings intact?

Hi all, I have a file with 3 columns separated by space. Each column has a heading. I want to sort according to the values in the 2nd column (ascending order). Ex. Name rank direction goory 0.05 --+ laby 0.0006 --- namy 0.31 -+- ....etc. Output should be Name rank direction laby...

6. Shell Programming and Scripting

Keeping the number intact

Currently I have the following to separate the numeric values. However the decimal point get separated. ls -lrt *smp*.cmd | awk '{print $NF}' | sed 's/^.*\///' | sed 's/\(*\)/ & /g' As an example on the files n02-z30-dsr65-terr0.50-dc0.05-4x3smp.cmd...

7. Shell Programming and Scripting

Remove last few characters in a file but keeping Header and trailer intact

Hi All, I am trying write a simple command using AWK and SED to this but without any success. Here is what I am using: head -1 test1.txt>test2.txt|sed '1d;$d' test1.txt|awk '{print substr($0,0,(length($0)-2))}' >>test2.txt|tail -1 test1.txt>>test2.txt Input: Header 1234567 abcdefgh...

8. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the...

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001...

10. UNIX for Beginners Questions & Answers

Remove duplicates in a dataframe (table) keeping all the different cells of just one of the columns

Hello all, I need to filter a dataframe composed of several columns of data to remove the duplicates according to one of the columns. I did it with pandas. In the main time, I need that the last column that contains all different data ( not redundant) is conserved in the output like this: A ...

LEARN ABOUT REDHAT

msguniq

MSGUNIQ(1)								GNU								MSGUNIQ(1)

NAME

       msguniq - unify duplicate translations in message catalog

SYNOPSIS

       msguniq [OPTION] [INPUTFILE]

DESCRIPTION

       Unifies duplicate translations in a translation catalog.  Finds duplicate translations of the same message ID.  Such duplicates are invalid
       input for other programs like msgfmt, msgmerge or msgcat.  By default, duplicates are merged together.  When using the  --repeated  option,
       only  duplicates  are  output,  and  all  other	messages are discarded.  Comments and extracted comments will be cumulated, except that if
       --use-first is specified, they will be taken from the first translation.  File positions  will  be  cumulated.	When  using  the  --unique
       option, duplicates are discarded.

       Mandatory arguments to long options are mandatory for short options too.

   Input file location:
       INPUTFILE
	      input PO file

       -D, --directory=DIRECTORY
	      add DIRECTORY to list for input files search

       If no input file is given or if it is -, standard input is read.

   Output file location:
       -o, --output-file=FILE
	      write output to specified file

       The results are written to standard output if no output file is specified or if it is -.

   Message selection:
       -d, --repeated
	      print only duplicates

       -u, --unique
	      print only unique messages, discard duplicates

   Output details:
       -t, --to-code=NAME
	      encoding for output

       --use-first
	      use first available translation for each message, don't merge several translations

       -e, --no-escape
	      do not use C escapes in output (default)

       -E, --escape
	      use C escapes in output, no extended chars

       --force-po
	      write PO file even if empty

       -i, --indent
	      write the .po file using indented style

       --no-location
	      do not write '#: filename:line' lines

       -n, --add-location
	      generate '#: filename:line' lines (default)

       --strict
	      write out strict Uniforum conforming .po file

       -w, --width=NUMBER
	      set output page width

       --no-wrap
	      do not break long message lines, longer than the output page width, into several lines

       -s, --sort-output
	      generate sorted output

       -F, --sort-by-file
	      sort output by file location

   Informative output:
       -h, --help
	      display this help and exit

       -V, --version
	      output version information and exit

AUTHOR

       Written by Bruno Haible.

REPORTING BUGS

       Report bugs to <bug-gnu-gettext@gnu.org>.

COPYRIGHT

       Copyright (C) 2001-2002 Free Software Foundation, Inc.
       This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICU-
       LAR PURPOSE.

SEE ALSO

       The full documentation for msguniq is maintained as a Texinfo manual.  If the info and msguniq programs	are  properly  installed  at  your
       site, the command

	      info msguniq

       should give you access to the complete manual.

GNU gettext 0.11.4						     July 2002								MSGUNIQ(1)