Shell Programming and Scripting

Hi All,

I am stuck with a problem here. I have two directories with really huge number of files about 200000+. I did some file processing and in between my program crashed thereby creating some inconsistent files. Running the script over again is out of question now as it takes lot of time to process them.

I need to know which are the inconsistent files and which files are missing in the new directory?

Here's the scenario:
1. I have one directory named main_directory which has the main files and are error free. These are the files which my script was reading and doing some processing.

2. After doing processing my script was writing the files to another directory named "scores". It is here the inconsistencies might exist.

My files in main_directory look like these:

1.wcor

Code:

1234
43232
9483
2345
9484

All the files in main_directory have names like 1.wcor, 2.wcor, 5.wcor etc. I have the complete list of these files in another file file_list.txt, which I populated by

Code:

ls -1 *.wcor > file_list.txt command

The files in my scores directory have some processed files and have the same file name except the extension. For example. 1.wcor above will have 1.sco in scores, 2.wcor in main_directory will have 2.sco in scores directory. But kindly note the these are not named continuously. In between it might happen that after 2.wcor, 5.wcor might come (no 3.wcor or 4.wcor exists) and this goes for scores directory too.

My corresponding 1.sco looks like this:

Code:

3232 5443333 5454 3232 54343

One checking factor here is the number of lines in 1.wcor and number of spaces in 1.sco. If they match, then the file is consistent. This is applicable to all the files both in main_directory and scores.

My task is to print all those files which are "missing and inconsistent". Missing in the sense that "files which do not exist in scores directory but are there in main_directory". Since, my script write the files in write mode, so I do not need to delete inconsistent files, they are all overwritten.

This is to just let me program read only those files and complete the entire operation for all files.

I am using Linux with bash and I have tried some solutions but to no avail:

One of them being:

Code:

find /path/to/main_directory -name '*' -type f | \
while read file
do
      wc -c "$file"  | read size dummy
      echo "`basename $file` $size"
done > realfiles

find / -type ! -name '/path/to/main_directory/*' |\
while read file
do
       wc -c "$file"  | read size dummy
       echo "`basename $file` $size  $file"
done > score_files

# create a file badfiles that is a list of all the failures
awk '{
        FILENAME=="realfiles" {
                key[$1 $2]++
        }
        FILENAME=="shadowfiles" {
                if( !key[$1 $2]) { print $3 }
        }
      }'   main_directory score_files > badfiles

See if this helps you:

Code:

#!/usr/bin/ksh
typeset -i mChars1
typeset -i mChars2
ls -1 main_directory/*.wcor | while read mFName; do
  mBase=$(basename ${mFName} '.wcor')
  if [[ -f "scores_dir/${mBase}.sco" ]]; then
    mChars1=$(wc -c < b1)+1
    mChars2=$(wc -c < b2)
    if [[ ${mChars1} -ne ${mChars2} ]]; then
      echo "File <${mBase}> does not match." > No_Match.txt
    fi
  else
    echo "Not found <${mBase}>" > Not_Found.txt
  fi
done