Duplicate lines in the file

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some...

2. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A

3. Shell Programming and Scripting

bash keep only duplicate lines in file

hello all in my bash script I have a file and I only want to keep the lines that appear twice in the file.Is there a way to do this? thanks in advance!

4. UNIX for Advanced & Expert Users

Inserting duplicate lines in a file

Hi, I copied the contents of a binary file into a .text file using hd (hexdump) command. The data in binary file is such that I get in many places like following 00000250 00 00 00 00 3f 2d 91 68 3f 69 fb e7 00 00 00 00 |....?-.h?i......| 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00...

5. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later!

6. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to...

7. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12...

8. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but...

9. UNIX for Dummies Questions & Answers

How to redirect duplicate lines from a file????

Hi, I am having a file which contains many duplicate lines. I wanted to redirect these duplicate lines into another file. Suppose I have a file called file_dup.txt which contains some line as file_dup.txt A100-R1 ACCOUNTING-CONTROL ACTONA-ACTASTOR ADMIN-AUTH-STATS ACTONA-ACTASTOR...

10. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file...

LEARN ABOUT SUSE

msguniq

MSGUNIQ(1)								GNU								MSGUNIQ(1)

NAME

       msguniq - unify duplicate translations in message catalog

SYNOPSIS

       msguniq [OPTION] [INPUTFILE]

DESCRIPTION

       Unifies duplicate translations in a translation catalog.  Finds duplicate translations of the same message ID.  Such duplicates are invalid
       input for other programs like msgfmt, msgmerge or msgcat.  By default, duplicates are merged together.  When using the  --repeated  option,
       only  duplicates  are  output,  and  all  other	messages are discarded.  Comments and extracted comments will be cumulated, except that if
       --use-first is specified, they will be taken from the first translation.  File positions  will  be  cumulated.	When  using  the  --unique
       option, duplicates are discarded.

       Mandatory arguments to long options are mandatory for short options too.

   Input file location:
       INPUTFILE
	      input PO file

       -D, --directory=DIRECTORY
	      add DIRECTORY to list for input files search

       If no input file is given or if it is -, standard input is read.

   Output file location:
       -o, --output-file=FILE
	      write output to specified file

       The results are written to standard output if no output file is specified or if it is -.

   Message selection:
       -d, --repeated
	      print only duplicates

       -u, --unique
	      print only unique messages, discard duplicates

   Input file syntax:
       -P, --properties-input
	      input file is in Java .properties syntax

       --stringtable-input
	      input file is in NeXTstep/GNUstep .strings syntax

   Output details:
       -t, --to-code=NAME
	      encoding for output

       --use-first
	      use first available translation for each message, don't merge several translations

       -e, --no-escape
	      do not use C escapes in output (default)

       -E, --escape
	      use C escapes in output, no extended chars

       --force-po
	      write PO file even if empty

       -i, --indent
	      write the .po file using indented style

       --no-location
	      do not write '#: filename:line' lines

       -n, --add-location
	      generate '#: filename:line' lines (default)

       --strict
	      write out strict Uniforum conforming .po file

       -p, --properties-output
	      write out a Java .properties file

       --stringtable-output
	      write out a NeXTstep/GNUstep .strings file

       -w, --width=NUMBER
	      set output page width

       --no-wrap
	      do not break long message lines, longer than the output page width, into several lines

       -s, --sort-output
	      generate sorted output

       -F, --sort-by-file
	      sort output by file location

   Informative output:
       -h, --help
	      display this help and exit

       -V, --version
	      output version information and exit

AUTHOR

       Written by Bruno Haible.

REPORTING BUGS

       Report bugs to <bug-gnu-gettext@gnu.org>.

COPYRIGHT

       Copyright (C) 2001-2007 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

SEE ALSO

       The  full  documentation  for  msguniq  is maintained as a Texinfo manual.  If the info and msguniq programs are properly installed at your
       site, the command

	      info msguniq

       should give you access to the complete manual.

GNU gettext-tools 0.17						   November 2007							MSGUNIQ(1)

UNIX for Advanced & Expert Users