Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

How to find duplicates contents in a files by comparing other files?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users How to find duplicates contents in a files by comparing other files?
# 8  
Old 07-27-2012
Try this - can be lengthy (about 100 times slower than md5sum solution). Works on linux.

Code:
for i in *; do for j in *; do if [ $i != $j ]; then diff -sq $i $j 2>/dev/null; fi; done; done|grep identical

Last edited by RudiC; 07-27-2012 at 10:22 AM.. Reason: removed diff on same filename
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Find a list of files in directory, move to new, allow duplicates

Greetings. I know enough Unix to be dangerous (!) and know that there is a clever way to do the following and it will save me about a day of agony (this time) and I will use it forever after! (many days of agony saved in the future)! Basically I need to find any image files (JPGs, PSDs etc)... (5 Replies)
Discussion started by: Clyde Lovett
5 Replies

2. Shell Programming and Scripting

Need Help with Bash - comparing directory contents with list of files

Hey guys, need help with a script I'm trying to write. Basically I need to compare the contents of a folder called "profiles" with a list of files called "template". when the file matches the contents of the folder it needs to set a variable called "checked" to "1" Cookies to anyone... (4 Replies)
Discussion started by: Scriporium
4 Replies

3. Shell Programming and Scripting

how to check the duplicates files contents

Hi all , am using unix aix .. how to check the duplicates files contents and moving to someother folders ex .. i have files called qqq cat qqq 1 2 3 i have another file qqq02 cat qqq02 1 2 3 here qqq02 has having same contents as qqq file.. i need to check the duplicates files... (1 Reply)
Discussion started by: Venkatesh1
1 Replies

4. Shell Programming and Scripting

Script to find files and delete it by comparing

I have a directory where lot of "gzip" files are dropped in every 5 minutes. There is an application which will process these gzip and move it to another directory but will leave a gzip.out file with lot of output data. I need to remove all the outfiles except for the one which is being... (1 Reply)
Discussion started by: gubbu
1 Replies

5. UNIX for Dummies Questions & Answers

sort and find duplicates for files with no white space

example data 5666700842511TAfmoham03151008075205999900000001000001000++ 5666700843130MAfmoham03151008142606056667008390315100005001 6666666663130MAfmoham03151008142606056667008390315100005001 I'd like to sort on position 10-14 where the characters are eq "130MA". Then based on positions... (0 Replies)
Discussion started by: mmarshall
0 Replies

6. Shell Programming and Scripting

comparing two files and find mismatch

hi i have two files and i want to compare both the files and find out mismatch in 3rd file file1 00354|1|0|1|1|0|0|0|1|2 52424|1|0|1|1|0|0|0|1|2 43236|1|0|1|1|0|0|0|1|2 41404|1|0|1|1|0|0|0|1|2 79968|1|0|1|1|0|0|0|1|2 file2 00354|1|0|1|1|0|0|0|1|2 52424|1|0|1|1|0|0|0|0|2... (9 Replies)
Discussion started by: dodasajan
9 Replies

7. Shell Programming and Scripting

Find duplicates from multuple files with 2 diff types of files

I need to compare 2 diff type of files and find out the duplicate after comparing each types of files: Type 1 file name is like: file1.abc (the extension abc could any 3 characters but I can narrow it down or hardcode for 10/15 combinations). The other file is file1.bcd01abc (the extension... (2 Replies)
Discussion started by: ricky007
2 Replies

8. Shell Programming and Scripting

Comparing contents of files

Hi, I hav two files a1.txt and a2.txt, a1.txt contains: --------------- asdev ebcdev .... a2.txt contains: --------------- asdev ebcdev prod .... a1.txt will be updated by a process,.. now i want to compare two files and i want to see data which is not in a1.txt am i clear....?? ... (3 Replies)
Discussion started by: rrs
3 Replies

9. UNIX for Dummies Questions & Answers

Help comparing 2 files to find deleted records

Hi, I need to compare todays file to yesterdays file to find deletes. I cannot use comm -23 file.old file.new. Because each record may have a small change in it but is not really a delete. I have two delimited files. the first field in each file is static. All other fields may change. I... (2 Replies)
Discussion started by: eja
2 Replies

10. Shell Programming and Scripting

comparing files to contents of a file

Hi I have a problem trying to run a while statement. I have files under one directory that i need to compare to a value in filex and update that file with the result files in the directory are DFC1. DFC5. DFC345. DFC344. DFC9. The program i am trying to run will take the number... (3 Replies)
Discussion started by: SummitElse
3 Replies
Login or Register to Ask a Question

LEARN ABOUT ULTRIX

nl

nl(1)							      General Commands Manual							     nl(1)

Name
       nl - line numbering filter

Syntax
       nl [-h type] [-b type] [-f type] [-v start#] [-i incr] [-p ] [-l num] [-s sep] [-w width] [-n format] [-d delim] file

Description
       The  command reads lines from the named file or from the standard input, if no file is named, and reproduces the lines on the standard out-
       put.  Lines are numbered on the left in accordance with the command options in effect.

       The command views the text it reads in terms of logical pages.  Line numbering is reset at the start of each logical page.  A logical  page
       consists  of  a header, a body, and a footer section.  Empty sections are valid.  Different line numbering options are independently avail-
       able for header, body, and footer.  For example, you can elect not to number header and footer lines while numbering  blank  lines  in  the
       body.

       The start of logical page sections is signaled by input lines containing nothing but the following delimiter characters:

		 Line contents	Start of

		 ::: 	header

		 ::		body

		 :		footer

       Unless otherwise specified, assumes that the text it is reading is in the body of a single logical page.

Options
       Command options may appear in any order and may be intermingled with an optional file name.  Only one file may be named.

       -b type		   Specifies  which  logical page body lines are to be numbered.  The following are recognized types and their meaning: a,
			   number all lines; t, number lines with printable text only; n, no line numbering; pstring, number only lines that  con-
			   tain the regular expression specified in string.

			   The default type for logical page body is t (text lines numbered).

       -h type		   Same as -b type except for header.  Default type for logical page header is n (no lines numbered).

       -f type		   Same as -b type except for footer.  Default for logical page footer is n (no lines numbered).

       -p		   Do not restart numbering at logical page delimiters.

       -v start#	   The initial value used to number logical page lines.  Default is 1.

       -i incr		   The increment value used to number logical page lines.  Default is 1.

       -s sep		   The character used in separating the line number and the corresponding text line.  Default sep is a tab.

       -w width 	   The number of characters used for the line number.  Default width is 6.

       -n format	   The	line  numbering  format.   Recognized values are the following: ln, left justified, leading zeroes suppressed; rn,
			   right justified, leading zeroes suppressed; rz, right justified, leading zeroes kept.  Default format is rn (right jus-
			   tified).

       -l num		   The	number	of  blank lines to be considered as one.  For example, -l2 results in only the second adjacent blank being
			   numbered (if the appropriate -ha, -ba, or -fa option is set).  Default is 1.

       -d xx		   The delimiter characters specifying the start of a logical page section may be changed from the default characters (:)
			   to two user-specified characters.  If only one character is entered, the second character remains the default character
			   (:).  No space should appear between the -d and the delimiter characters.  To enter a  backslash,  you  must  type  two
			   backslashes (//).

Examples
       nl -v10 -i10 -d!+ file1
       This command numbers file1 starting at line number 10 with an increment of ten.	The logical page delimiters are !+.

See Also
       pr(1)

																	     nl(1)