Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: find duplicate string in many different files
Top Forums Shell Programming and Scripting find duplicate string in many different files Post 302713439 by rdrtx1 on Wednesday 10th of October 2012 04:45:24 PM
Old 10-10-2012
Code:
awk '
{if (length(fns[FILENAME])<1) {
  fn[fc++]=FILENAME;
  fns[FILENAME]=FILENAME;
 }
 wd[$1]=$1;
 ws[$1 FILENAME]=$2;
}
END{
 printf("%-20s", "string");
 for (i=0; i<fc; i++) {
  printf("%-20s", fn[i]);
 }
 print;
 for (i in wd) {
  printf("%-20s", i);
  for (j=0; j<fc; j++) {
   printf("%-20s", ws[i fn[j]]);
  }
  print
 }
}' file1 file2 file3 ...

Last edited by rdrtx1; 10-10-2012 at 06:02 PM..
This User Gave Thanks to rdrtx1 For This Post:
xshang
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to find duplicate files with find ?

hello all I like to make search on files , and the result need to be the files that are duplicated? (8 Replies)
Discussion started by: umen
8 Replies

2. Shell Programming and Scripting

Find Duplicate files, not by name

I have a directory with images: -rw-r--r-- 1 root root 26216 Mar 19 21:00 020109.210001.jpg -rw-r--r-- 1 root root 21760 Mar 19 21:15 020109.211502.jpg -rw-r--r-- 1 root root 23144 Mar 19 21:30 020109.213002.jpg -rw-r--r-- 1 root root 31350 Mar 20 00:45 020109.004501.jpg -rw-r--r-- 1 root... (2 Replies)
Discussion started by: Ikon
2 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

Find duplicate files

What utility do you recommend for simply finding all duplicate files among all files? (4 Replies)
Discussion started by: kiasas
4 Replies

5. Shell Programming and Scripting

Find duplicate files by file size

Hi! I want to find duplicate files (criteria: file size) in my download folder. I try it like this: find /Users/frodo/Downloads \! -type d -exec du {} \; | sort > /Users/frodo/Desktop/duplicates_1.txt; cut -f 1 /Users/frodo/Desktop/duplicates_1.txt | uniq -d | grep -hif -... (9 Replies)
Discussion started by: Dirk Einecke
9 Replies

6. Shell Programming and Scripting

Find duplicate files but with different extensions

Hi ! I wonder if anyone can help on this : I have a directory: /xyz that has the following files: chsLog.107.20130603.gz chsLog.115.20130603 chsLog.111.20130603.gz chsLog.107.20130603 chsLog.115.20130603.gz As you ca see there are two files that are the same but only with a minor... (10 Replies)
Discussion started by: fretagi
10 Replies

7. Shell Programming and Scripting

Find duplicate rows between files

Hi champs, I have one of the requirement, where I need to compare two files line by line and ignore duplicates. Note, I hav files in sorted order. I have tried using the comm command, but its not working for my scenario. Input file1 srv1..development..employee..empname,empid,empdesg... (1 Reply)
Discussion started by: Selva_2507
1 Replies

8. Shell Programming and Scripting

Find help in shell - that clears away duplicate files

I am so frustrated!!! I want a nice command that clears away duplicate files: find . -type f -regex '.*{1,3}\..*' | xargs -I## rm -v '##' should work in my opinion. But it finds nothing even though I have files that have the file name: Scooby-Doo-1.txt Himalaya-2.jpg Camping... (8 Replies)
Discussion started by: Mr.Glaurung
8 Replies

9. Shell Programming and Scripting

To Find Duplicate files using latest in Linux

I have tried the following code and with that i couldnt achieve what i want. #!/usr/bin/bash find ./ -type f \( -iname "*.xml" \) | sort -n > fileList sed -i '/\.\/fileList/d' fileList NAMEOFTHISFILE=$(echo $0|sed -e 's/\/()$*.^|/\\&/g') sed -i "/$NAMEOFTHISFILE/d"... (2 Replies)
Discussion started by: gold2k8
2 Replies

10. UNIX for Advanced & Expert Users

AIX find duplicate backup files

I would like find and delete old backup files in aix. How would I go about doing this? For example: server1_1-20-2020 server1_1-21-2020 server1_1-22-2020 server1_1-23-2020 server2_1-20-2020 server2_1-21-2020 server2_1-22-2020 server2_1-23-2020 How would I go about finding and... (3 Replies)
Discussion started by: cokedude
3 Replies

LEARN ABOUT DEBIAN

tm::coverage

TM::Coverage(3pm)					User Contributed Perl Documentation					 TM::Coverage(3pm)

NAME

       TM::Coverage - Topic Maps, Code Coverage

DESCRIPTION

       This auxiliary package keeps track of the code coverage. Probably quite irrelevant for a user.

       Automatically generated for TM (1.44).

	  ---------------------------- ------ ------ ------ ------ ------ ------ ------
	  File				 stmt	bran   cond    sub    pod   time  total
	  ---------------------------- ------ ------ ------ ------ ------ ------ ------
	  blib/lib/TM.pm		 96.4	84.5   58.0   93.7   90.6   27.5   89.6
	  blib/lib/TM/Analysis.pm	100.0	90.9	n/a  100.0  100.0    0.0   97.5
	  blib/lib/TM/AsTMa/Fact.pm	100.0	 n/a	n/a  100.0    n/a    0.0  100.0
	  blib/lib/TM/AsTMa/Fact2.pm	100.0	 n/a	n/a  100.0    n/a    0.0  100.0
	  blib/lib/TM/Bulk.pm		 76.8	71.4   62.5   85.7  100.0    0.0   75.3
	  blib/lib/TM/CTM/CParser.pm	 65.6	34.3   31.0   84.6    0.0    1.6   52.5
	  blib/lib/TM/CTM/Parser.pm	 90.9	50.0	n/a  100.0    0.0    0.0   79.4
	  blib/lib/TM/DM.pm		100.0	65.0   66.7  100.0    n/a    0.0   96.5
	  blib/lib/TM/Graph.pm		 99.1	88.5	n/a  100.0  100.0    0.0   97.4
	  blib/lib/TM/Index.pm		100.0	80.0   60.0  100.0   83.3    0.3   87.8
	  .../Index/Characteristics.pm	100.0  100.0	n/a  100.0    0.0    0.0   97.0
	  blib/lib/TM/Index/Match.pm	 98.5	87.5	n/a  100.0  100.0    0.5   97.6
	  blib/lib/TM/LTM/CParser.pm	 84.9	45.3   42.7   98.2    0.0    0.2   67.8
	  blib/lib/TM/LTM/Parser.pm	 95.7	62.5   50.0  100.0    0.0    0.0   87.1
	  blib/lib/TM/Literal.pm	 45.8	 0.0   11.8   30.0    0.0    0.4   24.2
	  blib/lib/TM/MapSphere.pm	 96.5	84.6   68.8  100.0  100.0    0.0   91.7
	  .../TM/Materialized/AsTMa.pm	100.0	 n/a	n/a  100.0    0.0    0.0   95.5
	  ...ib/TM/Materialized/LTM.pm	100.0	 n/a	n/a  100.0    0.0    0.0   95.5
	  .../TM/Materialized/MLDBM.pm	100.0	83.3	n/a  100.0    0.0    0.0   92.6
	  ...b/TM/Materialized/Null.pm	100.0	 n/a	n/a  100.0    n/a    0.0  100.0
	  ...TM/Materialized/Stream.pm	100.0  100.0  100.0  100.0    0.0    0.0   96.0
	  ...ib/TM/Materialized/XTM.pm	100.0	 n/a	n/a  100.0    0.0    0.0   96.7
	  blib/lib/TM/PSI.pm		100.0	 n/a	n/a  100.0    n/a    0.0  100.0
	  blib/lib/TM/QL.pm		 91.4	61.5   66.7  100.0   50.0    0.1   83.9
	  blib/lib/TM/QL/CParser.pm	 76.2	42.4   41.9   90.3    0.0    9.1   62.0
	  blib/lib/TM/QL/PE.pm		 83.8	77.3   80.0   91.7    0.0   52.5   80.5
	  blib/lib/TM/QL/TS.pm		 65.2	35.4   27.8   70.8   53.3    2.4   56.9
	  blib/lib/TM/ResourceAble.pm	 88.1	87.5	n/a  100.0  100.0    0.0   90.0
	  .../TM/ResourceAble/MLDBM.pm	100.0	75.0	n/a  100.0    0.0    0.0   94.2
	  blib/lib/TM/Serializable.pm	 94.9	65.6  100.0  100.0  100.0    0.0   84.1
	  .../TM/Serializable/AsTMa.pm	 99.1	87.9   66.7  100.0  100.0    0.0   94.2
	  ...TM/Serializable/Dumper.pm	100.0	 n/a	n/a  100.0    0.0    0.0   93.5
	  ...ib/TM/Serializable/LTM.pm	 94.1	 n/a	n/a   83.3  100.0    0.0   92.0
	  ...ib/TM/Serializable/XTM.pm	 96.3	85.2   72.9  100.0  100.0    0.2   91.5
	  .../lib/TM/Synchronizable.pm	100.0	75.0  100.0  100.0  100.0    0.0   97.2
	  ...M/Synchronizable/MLDBM.pm	100.0	50.0	n/a  100.0    0.0    0.0   89.3
	  ...nchronizable/MapSphere.pm	100.0	63.6   64.3  100.0  100.0    0.0   88.6
	  ...TM/Synchronizable/Null.pm	100.0	 n/a	n/a  100.0    0.0    0.0   87.5
	  blib/lib/TM/Tau.pm		100.0	91.7   80.0  100.0    0.0    0.0   95.6
	  blib/lib/TM/Tau/Federate.pm	 63.2	39.3	n/a   62.5   27.3    0.0   54.5
	  blib/lib/TM/Tau/Filter.pm	100.0	91.7   66.7  100.0   50.0    0.0   90.0
	  .../TM/Tau/Filter/Analyze.pm	100.0	 n/a	n/a  100.0  100.0    0.0  100.0
	  blib/lib/TM/Tree.pm		100.0	57.1   44.4  100.0   66.7    0.0   88.0
	  blib/lib/TM/Utils.pm		100.0	50.0	n/a  100.0    0.0    0.0   89.3
	  ...ib/TM/Utils/TreeWalker.pm	100.0  100.0	n/a  100.0    0.0    0.0   96.0
	  ...ib/TM/Workbench/Plugin.pm	100.0	 n/a	n/a  100.0    0.0    0.0   75.0
	  ...M/Workbench/Plugin/Tau.pm	 87.5	50.0	n/a   80.0    0.0    0.0   73.1
	  yapp/astma-fact.yp		 96.9	91.5   73.3   97.8    0.0    4.8   92.7
	  yapp/astma2-fact.yp		 97.4	91.6   58.3  100.0    0.0    0.2   91.6
	  Total 			 76.8	45.2   40.2   91.2   57.0  100.0   63.8
	  ---------------------------- ------ ------ ------ ------ ------ ------ ------

SEE ALSO

       TM

COPYRIGHT AND LICENSE

       Copyright 200[8] by Robert Barta, <drrho@cpan.org>

       This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.10.1							    2008-10-05							 TM::Coverage(3pm)
All times are GMT -4. The time now is 03:33 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy