To Find Duplicate files using latest in Linux Post: 303007616

Sponsored Content

Top Forums Shell Programming and Scripting To Find Duplicate files using latest in Linux Post 303007616 by gold2k8 on Saturday 18th of November 2017 11:40:56 PM

11-19-2017

Registered User

To Find Duplicate files using latest in Linux

I have tried the following code and with that i couldnt achieve what i want.

Code:

    #!/usr/bin/bash
    find ./ -type f \( -iname "*.xml" \) | sort -n > fileList
    sed -i '/\.\/fileList/d' fileList
    NAMEOFTHISFILE=$(echo $0|sed -e 's/[]\/()$*.^|[]/\\&/g')
    sed -i "/$NAMEOFTHISFILE/d" fileList
    cp fileList auxFileList
    while read FILENAME
    do
        sed -i '1d' auxFileList
        #echo "Comparing $FILENAME with :"
        #Read the aux file and compare current file with every other element in the file
        while read COMPFILENAME
        do
            RETURN=$(diff $FILENAME $COMPFILENAME)
            if [ "$RETURN" == "" ]
            then
            cat $FILENAME | awk ' BEGIN { FS="_" } { printf( "%03d\n",$2) }' | sort | awk ' { printf( "data_%d_box\n", $1)  }'
             #echo "$FILENAME AND $COMPFILENAME are identical"
             #rm -r $FILENAME
            fi
            #echo "  $COMPFILENAME"
        done<auxFileList
    done<fileList
    rm fileList auxFileList &>/dev/null
    printf '\n\n'

this code selecting all the files initially. I have to amend my code in such a way that only recent modified filename patterns for example

Code:

    File 1: AAA_555_0000 
    File 2: AAAA_123_123 
    File 3: AAAA_452_452 [latest]
    
    File 4: BBB_555_0000 
    File 5: BBB_555_555 
    File 6: BBB_999_999 [latest]
    
    File 7: CCC_555_0000 
    File 8: CCC_000_000 
    File 9: CCC_000_111 [latest]

Script has to pick latest file in all the filename patterns in the folder and it should compare and delete the duplicates.

Appreciate if you can help me with this logic.

Thanks much!

gold2k8

View Public Profile for gold2k8

Find all posts by gold2k8

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

shell script to find latest date and time of the files

Hi everyone, Please help:) I have a list of 1000 different files which comes daily to the directory.Some of the files are not coming to the directory now. I need to write a shell script to find the latest date and time of the files they came to the directory. The files should be unique....

2. UNIX for Dummies Questions & Answers

How to find the latest file on Unix or Linux

Please help me out how to identify the latest file in one directory by looking at file's timestamp or datestamp. You can say using system command. Thanks

3. Shell Programming and Scripting

Find Duplicate files, not by name

I have a directory with images: -rw-r--r-- 1 root root 26216 Mar 19 21:00 020109.210001.jpg -rw-r--r-- 1 root root 21760 Mar 19 21:15 020109.211502.jpg -rw-r--r-- 1 root root 23144 Mar 19 21:30 020109.213002.jpg -rw-r--r-- 1 root root 31350 Mar 20 00:45 020109.004501.jpg -rw-r--r-- 1 root...

4. Shell Programming and Scripting

find the latest files in multiple directory

I want to get the latest files from multiple directories, d1, d2,d3 and d4 under the parent dierectoy d. can anyone help out with this? thx

5. Shell Programming and Scripting

Find duplicate files

What utility do you recommend for simply finding all duplicate files among all files?

6. Shell Programming and Scripting

Find the latest directory and loop through the files and pick the error messages

Hi, I am new to unix and shell scripting,can anybody help me in sctipting a requirement. my requirement is to get the latest directory the name of the directory will be like CSB.monthdate_time stamp like CSB.Sep29_11:16 and CSB.Oct01_16:21. i need to pick the latest directory. in the...

7. Shell Programming and Scripting

How to find the latest file on Unix or Linux (recursive)

Hi all, I need to get the latest file. I have found this command "ls -lrt" that is great but not recursive. Can anyone help? Thanx by advance.

8. Shell Programming and Scripting

Find duplicate files but with different extensions

Hi ! I wonder if anyone can help on this : I have a directory: /xyz that has the following files: chsLog.107.20130603.gz chsLog.115.20130603 chsLog.111.20130603.gz chsLog.107.20130603 chsLog.115.20130603.gz As you ca see there are two files that are the same but only with a minor...

9. Shell Programming and Scripting

How to find duplicate line in Linux?

Hi, Gurus, I need find the duplicate record in unix file. what command I should use for this. Thanks in advance

10. UNIX for Beginners Questions & Answers

Modified or latest files copy from windows to Linux

To copy the file from windows to linux i use pscp command(pscp source user@destination). Know i want to copy the latest modified or created files from windows to linux. could any one please help me out with it. Thanks and Regards, Sourabh

LEARN ABOUT CENTOS

paps

PAPS(1) 						      General Commands Manual							   PAPS(1)

NAME

       paps - UTF-8 to PostScript converter using Pango

SYNOPSIS

       paps [options] files...

DESCRIPTION

       paps reads a UTF-8 encoded file and generates a PostScript language rendering of the file. The rendering is done by creating outline curves
       through the pango ft2 backend.

OPTIONS

       These programs follow the usual GNU command line syntax, with long options starting with  two  dashes  (`-').   A  summary  of  options	is
       included below.

       --landscape
	      Landscape output. Default is portrait.

       --columns=cl
	      Number of columns output. Default is 1.

       --font=desc
	      Set the font description. Default is Monospace 12.

       --rtl  Do rtl layout.

       --paper ps
	      Choose paper size. Known paper sizes are legal, letter, a4. Default is A4.

       --bottom-margin=bm
	      Set bottom margin in postscript points (1/72 inch). Default is 36.

       --top-margin=tm
	      Set top margin. Default is 36.

       --left-margin=lm
	      Set left margin. Default is 36.

       --right-margin=rm
	      Set right margin. Default is 36.

       --help Show summary of options.

       --header
	      Draw page header for each page.

       --markup
	      Interpret the text as pango markup.

       --encoding=ENCODING
	      Assume the documentation encoding is ENCODING.

       --lpi  Set the lines per inch. This determines the line spacing.

       --cpi  Set the characters per inch. This is an alternative method of specifying the font size.

       --stretch-chars
	      Indicates  that  characters  should be stretched in the y-direction to fill up their vertical space. This is similar to the texttops
	      behaviour.

AUTHOR

       paps was written by Dov Grobgeld <dov.grobgeld@gmail.com>.

       This manual page was written by Lior Kaplan <kaplan@debian.org>, for the Debian project (but may be used by others).

								  April  17, 2006							   PAPS(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

shell script to find latest date and time of the files

Discussion started by: karthicss

2. UNIX for Dummies Questions & Answers

How to find the latest file on Unix or Linux

Discussion started by: duke0001

3. Shell Programming and Scripting

Find Duplicate files, not by name

Discussion started by: Ikon

4. Shell Programming and Scripting

find the latest files in multiple directory

Discussion started by: shyork2001