Script to compare files in 2 folders and delete the large file Post: 302999907

Sponsored Content

Top Forums Shell Programming and Scripting Script to compare files in 2 folders and delete the large file Post 302999907 by jim mcnamara on Thursday 29th of June 2017 07:20:45 PM

06-29-2017

Registered User

Three typos made this a mess. Check for zero length files added. You can remove it.
Apologies. Thanks Rudi for spotting the problem.

Code:

# get all the filenames in one place -> /tmp/list
find /path/to/directory1 /path/to/directory2 -type f > /tmp/list
#  you now have all the file names
#
# rewrite /tmp/list to have the correct values
while read fname   # fname is the complete file name
do
      shortfile=$(basename $fname)
      shortfile=${shortfile%%.*}
      size=$(stat -c '%s' $fname)
      [ $size -eq 0 ] && continue # skip zero-length files 
      echo "$shortfile $size $fname"
done < /tmp/list > /tmp/next

# /tmp/next has the data, so let's sort and aggregate it -  assuming no spaces in the shortfile name
# sort by shortfile

sort -k1 -k2n -o /tmp/next /tmp/next

# aggregate
# awk fields are $1 - shortfile, $2 - size,  $3 - fullname
awk '{ 
         arr[$1]=$3 " " $2  # note that the last values to be stored for shortfile
                                   # come from  the last time shortfile is in the file
                                   
        }
         END { for (i in arr) {print arr[i]} }
        ' /tmp/next > /tmp/final
        
# removed the rm stuff for now

jim mcnamara

View Public Profile for jim mcnamara

Find all posts by jim mcnamara

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare data in 2 files and delete if file exist

Hi there, I have written a script called "compare" (see below) to make comparison between 2 files namely test_put.log and Output_A0.log #!/bin/ksh while read file do found="no" while read line do echo $line | grep $file > /dev/null if then echo $file found found="yes" break fi...

2. Shell Programming and Scripting

Looking for Large Files and Folders

Hello, On one of my UNIX boxes, there are many folders. I'm looking for a way / command that will search for and display folder names / location / size so I can do some cleanups. How can I do this?

3. Shell Programming and Scripting

Script to Compare a large number of files.

I have a large Filesystem on an AIX server and another one on a Red Hat box. I have syncd the two filesystems using rsysnc. What Im looking for is a script that would compare to the two filesystems to make sure the bits match up and the number of files match up. its around 2.8 million...

4. Shell Programming and Scripting

Compare files in two folders and delete missing ones

I do not know much about shell scripting so I am at a loss here. If someone can help me, that would be great! I have two directories /dir1 /dir2 I need to delete all files from /dir1 and that does not have a correspondent file in /dir2. It should NOT check file suffixes in /dir2 . Why?...

5. Shell Programming and Scripting

Shell Script to delete files within a particular time frame under multiple sub folders

Greetings! I'm looking for starting information for a shell script. Here's my scenario: I have multiple folders(100) for example: /www/test/applications/app1/logs /www/test/applications/app2/logs Within these folders there are log files files that need to be deleted after a month. ...

6. Shell Programming and Scripting

Need to delete large set of files (i.e) close to 100K from a directory based on the input file

Hi all, I need a script to delete a large set of files from a directory under / based on an input file and want to redirect errors into separate file. I have already prepared a list of files in the input file. Kndly help me. Thanks, Prash

7. Shell Programming and Scripting

Linux Script to compare two folders and copy missing files

Hi, I need help in shell scripting. If someone can help me, that would be great! Problem. I want Linux Script to compare two folders and copy missing files. Description. I have two directories /dir1 /dir2 I need to copy all distinct/new/unique/missing files from /dir1 and that...

8. Shell Programming and Scripting

Script to delete folders and files from a prompt

Hi Everyone, I work for GE Money IVR as a DB analyst and the environment on which I work is Solaris 5.0 server and Oracle 11g. I got a project in which I have to clean up the folders and files which are not used in DB. I copied an existing script and edited it, dont know this is the...

9. UNIX for Advanced & Expert Users

Help with creating script to delete log files/folders

Hi I am new to Linux / scripting language. I need to improve our Linux servers at work and looking to claim some space my deleting log files/ folders on a 5 day basis. Can someone help me with creating a script to do so. Any sample script will be helpful.:b: Regards

10. Shell Programming and Scripting

Script to compare partial filenames in two folders and delete duplicates

Background: I use a TV tuner card to capture OTA video files (.mpeg) and then my Plex Media Server automatically optimizes the files (transcodes for better playback) and places them in a new directory. I have another Plex Library pointing to the new location for the optimized .mp4 files. This...

LEARN ABOUT DEBIAN

i-ls

i-ls(1) 						      General Commands Manual							   i-ls(1)

NAME

       i-ls - display file stat and checksum information for files

SYNOPSIS

       i-ls [-s] filename

       i-ls [-s] dirname

       Specify a filename for i-ls to examine, or specify a directory with dirname to examine all the files in the directory.

       -s

	      Do not show/calculate checksums for all regular files.

DESCRIPTION

       The integrit system detects intrusion by detecting when trusted files have been altered.

       integrit records information about files in a database.	i-ls allows the user to see that information for live files.

       The output syntax is the same as that of integrit.

EXAMPLES

	    i-ls /tmp/foo.txt

       shows file stat attributes for the file, "/tmp/foo.txt".

	    i-ls /tmp

       shows file stat attributes for all files in the directory, "/tmp".

	    i-ls -s /tmp

       shows file stat attributes for all files in the directory, "/tmp", also including also checksums for all regular files.

AUTHOR

       Ed L Cashin <ecashin@users.sourceforge.net>

       See the integrit web site for up to date releases and information at

	   http://integrit.sourceforge.net/

SEE ALSO

       Integrit File Verification System Manual (info), lstat(2)

Version 4.1							   Jul 27, 2001 							   i-ls(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare data in 2 files and delete if file exist

Discussion started by: lweegp

2. Shell Programming and Scripting

Looking for Large Files and Folders

Discussion started by: bbbngowc

3. Shell Programming and Scripting

Script to Compare a large number of files.

Discussion started by: zippdawg2001

4. Shell Programming and Scripting

Compare files in two folders and delete missing ones

Discussion started by: kaah