Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to find duplicates contents in a files by comparing other files?
Top Forums UNIX for Advanced & Expert Users How to find duplicates contents in a files by comparing other files? Post 302678045 by Ygor on Friday 27th of July 2012 06:37:03 AM
Old 07-27-2012
Perhaps try md5sum, e.g...
Code:
md5sum -t files* | awk 'a[$1]{printf "%s and %s are the same\n",a[$1],$2;next}{a[$1]=$2}'

This User Gave Thanks to Ygor For This Post:
Venkatesh1
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

comparing files to contents of a file

Hi I have a problem trying to run a while statement. I have files under one directory that i need to compare to a value in filex and update that file with the result files in the directory are DFC1. DFC5. DFC345. DFC344. DFC9. The program i am trying to run will take the number... (3 Replies)
Discussion started by: SummitElse
3 Replies

2. UNIX for Dummies Questions & Answers

Help comparing 2 files to find deleted records

Hi, I need to compare todays file to yesterdays file to find deletes. I cannot use comm -23 file.old file.new. Because each record may have a small change in it but is not really a delete. I have two delimited files. the first field in each file is static. All other fields may change. I... (2 Replies)
Discussion started by: eja
2 Replies

3. Shell Programming and Scripting

Comparing contents of files

Hi, I hav two files a1.txt and a2.txt, a1.txt contains: --------------- asdev ebcdev .... a2.txt contains: --------------- asdev ebcdev prod .... a1.txt will be updated by a process,.. now i want to compare two files and i want to see data which is not in a1.txt am i clear....?? ... (3 Replies)
Discussion started by: rrs
3 Replies

4. Shell Programming and Scripting

Find duplicates from multuple files with 2 diff types of files

I need to compare 2 diff type of files and find out the duplicate after comparing each types of files: Type 1 file name is like: file1.abc (the extension abc could any 3 characters but I can narrow it down or hardcode for 10/15 combinations). The other file is file1.bcd01abc (the extension... (2 Replies)
Discussion started by: ricky007
2 Replies

5. Shell Programming and Scripting

comparing two files and find mismatch

hi i have two files and i want to compare both the files and find out mismatch in 3rd file file1 00354|1|0|1|1|0|0|0|1|2 52424|1|0|1|1|0|0|0|1|2 43236|1|0|1|1|0|0|0|1|2 41404|1|0|1|1|0|0|0|1|2 79968|1|0|1|1|0|0|0|1|2 file2 00354|1|0|1|1|0|0|0|1|2 52424|1|0|1|1|0|0|0|0|2... (9 Replies)
Discussion started by: dodasajan
9 Replies

6. UNIX for Dummies Questions & Answers

sort and find duplicates for files with no white space

example data 5666700842511TAfmoham03151008075205999900000001000001000++ 5666700843130MAfmoham03151008142606056667008390315100005001 6666666663130MAfmoham03151008142606056667008390315100005001 I'd like to sort on position 10-14 where the characters are eq "130MA". Then based on positions... (0 Replies)
Discussion started by: mmarshall
0 Replies

7. Shell Programming and Scripting

Script to find files and delete it by comparing

I have a directory where lot of "gzip" files are dropped in every 5 minutes. There is an application which will process these gzip and move it to another directory but will leave a gzip.out file with lot of output data. I need to remove all the outfiles except for the one which is being... (1 Reply)
Discussion started by: gubbu
1 Replies

8. Shell Programming and Scripting

how to check the duplicates files contents

Hi all , am using unix aix .. how to check the duplicates files contents and moving to someother folders ex .. i have files called qqq cat qqq 1 2 3 i have another file qqq02 cat qqq02 1 2 3 here qqq02 has having same contents as qqq file.. i need to check the duplicates files... (1 Reply)
Discussion started by: Venkatesh1
1 Replies

9. Shell Programming and Scripting

Need Help with Bash - comparing directory contents with list of files

Hey guys, need help with a script I'm trying to write. Basically I need to compare the contents of a folder called "profiles" with a list of files called "template". when the file matches the contents of the folder it needs to set a variable called "checked" to "1" Cookies to anyone... (4 Replies)
Discussion started by: Scriporium
4 Replies

10. UNIX for Dummies Questions & Answers

Find a list of files in directory, move to new, allow duplicates

Greetings. I know enough Unix to be dangerous (!) and know that there is a clever way to do the following and it will save me about a day of agony (this time) and I will use it forever after! (many days of agony saved in the future)! Basically I need to find any image files (JPGs, PSDs etc)... (5 Replies)
Discussion started by: Clyde Lovett
5 Replies

LEARN ABOUT CENTOS

gst-md5sum

gst-md5sum-0.8(1)						   User Commands						 gst-md5sum-0.8(1)

NAME

       gst-md5sum-0.8, gst-md5sum - get an md5sum of a GStreamer pipeline through md5sink

SYNOPSIS

       gst-md5sum-0.8 [--verbose] [gst-std-options] [partial-pipeline-description]

       gst-md5sum [--verbose] [gst-std-options] [partial-pipeline-description] [--gst-list-mm] [--gst-mm=string] [--print]

DESCRIPTION

       gst-md5sum-0.8 generates MD5 checksums of the data generated by a GStreamer pipeline.

       In theory, both of the following commands should print out the same checksum:

       gst-md5sum-0.8 filesrc location=music.mp3

       md5sum music.mp3

       If  the	pipeline  contains  an md5sink element, gst-md5sum-0.8 queries it for the md5sum at the end of pipeline iteration. If the pipeline
       does not contain an md5sink element, gst-md5sum-0.8 automatically connects an md5sink to the right-hand side of the given pipeline.

       See gst-launch(1) or the GStreamer documentation for more information on how to create a partial-pipeline-description.

       gst-md5sum is a wrapper script that runs the latest installed version of gst-md5sum-X.X. For  example,  if  both  gst-md5sum-0.7  and  gst-
       md5sum-0.8 are installed on your system, gst-md5sum runs gst-md5sum-0.8.

OPTIONS

       The following options are supported by gst-md5sum-0.8 and gst-md5sum:

       --verbose       Output verbose information. You can also use -v to specify this option.

       gst-std-options Standard options available for use with most GStreamer applications.  See gst-std-options(5) for more information.

       The following options are supported by gst-md5sum only:

       --gst-list-mm   List found major/minor versions. This option displays the versions that are available.

       --gst-mm=string Force  major/minor  version.  This  option  enables you to specify a specific version to run, if you do not want to run the
		       default version.

       --print	       Print wrapped command line. This option displays the command that will be run, and then runs the command.

OPERANDS

       The following operands are supported:

       partial-pipelinePartialppipeline description.

EXAMPLES

       Example 1: Running the Wrapper Script and Displaying the Command Name

       example% gst-md5sum --print
       /usr/bin/gst-md5sum-0.8

FILES

       The following files are used by this application:

       /usr/bin/gst-md5sum-0.8 Executable for GNOME help browser

       /usr/bin/gst-md5sum     Wrapper script that runs the latest installed version of gst-md5sum-X.X

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWgnome-media		   |
       +-----------------------------+-----------------------------+
       |Interface stability	     |External			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       gst-complete(1), gst-compprep(1), gst-feedback(1), gst-inspect(1),  gst-launch(1),  gst-launch-ext(1),  gst-register(1),  gst-thumbnail(1),
       gst-typefind(1), gst-xmlinspect(1), gst-xmllaunch(1), gstreamer-properties(1), libgstreamer-0.8(3), libgstgetbits(3), gst-std-options(5)

NOTES

       Original man page written by the GStreamer team at http://gstreamer.net/.

       Updated by Brian Cameron, Sun Microsystems Inc., 2004.

SunOS 5.10							    14 Oct 2004 						 gst-md5sum-0.8(1)
All times are GMT -4. The time now is 11:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy