Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: bash keep only duplicate lines in file
Top Forums Shell Programming and Scripting bash keep only duplicate lines in file Post 302653585 by agama on Saturday 9th of June 2012 02:15:58 PM
Old 06-09-2012
Quote:
Originally Posted by vlm
I found this:

Code:
comm -3 file1 file2 > file3

but it stores to file3 the lines that appear only in the one file and not in the other,the exact opposite from what I wantSmilie
Comm can output lines that are common to both files, but from your initial post you suggest that you only have one file to work with and comm won't help with that.

Given that your file is already sorted this is the easy case and something like this will probably do what you need:
Code:
awk ' p == $0; { p = $0 }' input-file >output-file

It will write each line that is duplicated once onto standard output.

---------- Post updated at 14:15 ---------- Previous update was at 14:13 ----------

Ok, looks like we crossed posts and you have two files, not one as suggested in your initial question.

Using comm -12 is the easiest, and probably most efficient method.
This User Gave Thanks to agama For This Post:
vlm
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies

3. UNIX for Dummies Questions & Answers

removing duplicate lines from a file

Hi, I am trying to remove duplicate lines from a file. For example the contents of example.txt is: this is a test 2342 this is a test 34343 this is a test 43434 and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with: 2342... (4 Replies)
Discussion started by: ocelot
4 Replies

4. UNIX for Dummies Questions & Answers

How to redirect duplicate lines from a file????

Hi, I am having a file which contains many duplicate lines. I wanted to redirect these duplicate lines into another file. Suppose I have a file called file_dup.txt which contains some line as file_dup.txt A100-R1 ACCOUNTING-CONTROL ACTONA-ACTASTOR ADMIN-AUTH-STATS ACTONA-ACTASTOR... (3 Replies)
Discussion started by: zing_foru
3 Replies

5. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

6. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies

7. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

8. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

9. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A (7 Replies)
Discussion started by: nsuresh316
7 Replies

10. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

LEARN ABOUT SUNOS

gst-complete

gst-complete-0.8(1)						   User Commands					       gst-complete-0.8(1)

NAME

       gst-complete-0.8, gst-complete - perform bash completion for gst-launch command lines

SYNOPSIS

       complete -Cgst-complete-0.8 gst-launch

       gst-complete [--gst-list-mm] [--gst-mm=string] [--print]

DESCRIPTION

       gst-complete-0.8 enables bash to provide context-sensitive tab completion for gst-launch command lines. See gst-launch(1) for the syntax of
       gst-launch command lines.

       You must have a version of bash that supports programmable completion. Such versions of bash provide the complete built-in command.

       gst-complete is a wrapper script that runs the latest installed version of gst-complete-X.X. For example, if both gst-complete-0.7 and gst-
       complete-0.8 are installed on your system, gst-complete runs gst-complete-0.8.

OPTIONS

       The following options are supported by gst-complete-0.8 and gst-complete:

       --help	       Display detailed usage message. You can also use -? to specify this option.

       The following options are supported by gst-complete only:

       --gst-list-mm   List found major/minor versions. This option displays the versions that are available.

       --gst-mm=string Force  major/minor  version.  This  option  enables you to specify a specific version to run, if you do not want to run the
		       default version.

       --print	       Print wrapped command line. This option displays the command that will be run, and then runs the command.

EXAMPLES

       Example 1: Generating the Registry of Available Plugins by Running gst-compprep

       Run the following command in your .bash_profile or other appropriate place:

       example% complete -C gst-complete-0.8 gst-launch

       Example 2: Running the Wrapper Script and Displaying the Command Name

       example% gst-complete --print
       /usr/bin/gst-complete-0.8

FILES

       The following files are used by this application:

       /usr/bin/gst-complete-0.Executable for bash completion for gst-launch

       /usr/bin/gst-complete   Wrapper script that runs the latest installed version of gst-complete-X.X

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWgnome-media		   |
       +-----------------------------+-----------------------------+
       |Interface stability	     |External			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       gst-compprep(1), gst-feedback(1), gst-inspect(1), gst-launch(1), gst-launch-ext(1), gst-md5sum(1), gst-register(1), gst-thumbnail(1),  gst-
       typefind(1), gst-xmlinspect(1), gst-xmllaunch(1), gstreamer-properties(1), libgstreamer-0.8(3), libgstgetbits(3)

NOTES

       Original man page written by the GStreamer team at http://gstreamer.net/.

       Updated by Brian Cameron, Sun Microsystems Inc., 2004.

SunOS 5.10							    14 Oct 2004 					       gst-complete-0.8(1)
All times are GMT -4. The time now is 09:38 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy