I am stuck with a problem here. I have two directories with really huge number of files about 200000+. I did some file processing and in between my program crashed thereby creating some inconsistent files. Running the script over again is out of question now as it takes lot of time to process them.
I need to know which are the inconsistent files and which files are missing in the new directory?
Here's the scenario:
1. I have one directory named main_directory which has the main files and are error free. These are the files which my script was reading and doing some processing.
2. After doing processing my script was writing the files to another directory named "scores". It is here the inconsistencies might exist.
My files in main_directory look like these:
1.wcor
All the files in main_directory have names like 1.wcor, 2.wcor, 5.wcor etc. I have the complete list of these files in another file file_list.txt, which I populated by
The files in my scores directory have some processed files and have the same file name except the extension. For example. 1.wcor above will have 1.sco in scores, 2.wcor in main_directory will have 2.sco in scores directory. But kindly note the these are not named continuously. In between it might happen that after 2.wcor, 5.wcor might come (no 3.wcor or 4.wcor exists) and this goes for scores directory too.
My corresponding 1.sco looks like this:
One checking factor here is the number of lines in 1.wcor and number of spaces in 1.sco. If they match, then the file is consistent. This is applicable to all the files both in main_directory and scores.
My task is to print all those files which are "missing and inconsistent". Missing in the sense that "files which do not exist in scores directory but are there in main_directory". Since, my script write the files in write mode, so I do not need to delete inconsistent files, they are all overwritten.
This is to just let me program read only those files and complete the entire operation for all files.
I am using Linux with bash and I have tried some solutions but to no avail:
hello masters ,
please help here. I have 4 cols, I am looking for consistent 'geno' values within
'line', 'part' combinations. If the geno values are not consistent within a 'line', 'part' block, then we delete that block. One of the complications is that geno values are always 2 character, but... (7 Replies)
HI Guys,
I have some 8 files with different name and extensions. I need to check if they are present in a specific folder or not and also want that script to show me which all are not present. I can write if condition for each file but from a developer perspective , i feel that is not a good... (3 Replies)
Hi,
I have 9 files which are generated dynamically & if there is a some condition which doesn't meet the criteria then file is not created or is of zero size.
so further i am unable to consolidate the files based on following code 1
awk -F, -v ptime="201407" 'FNR==1... (3 Replies)
Hello Experts,
File contains 5 columns with | delimeter. 1,3,5 columns are required columns means it should contains values.
reset of the columns it will contain value or not.
test1.txt:
a@a.com|a|b|c|d
|a|b|c|d
output: test2.txt
a@a.com|a|b|c|d
I need the unix script, read the... (5 Replies)
Hello Experts,
File contains 5 columns with | delimeter. 1,3,5 columns are required columns means it should contains values.
reset of the columns it will contain value or not.
test1.txt:
Code:
a@a.com|a|b|c|d |a|b|c|d
output: test2.txt
Code:
a@a.com|a|b|c|d
I need the unix... (1 Reply)
Hi
I have 4 files, I need the check whether these 4 files are having header and Trailer records. header and trailer records are identified with 1,b. If any file is not having these we will not proceed with other process.
Output should be 1 if all files are having header and footer other... (4 Replies)
All,
Is there a way to keep checking for a file over and over again in the same script for an interval of time?
Ie
If {
mail -user
continue checking until file arrives
file arrives
tasks
exit
I don't want the script to run each time and email the user each time a file... (4 Replies)
Hi All,
I am very new to Shell scripting...
I got a requirement.
I will have few text files(data files) in a particular directory. they will be with .txt extension. With same name, but with a different extension control files also will be there. For example, Sample_20081001.txt is the data... (4 Replies)
I want to check the files in particular directory are more that 0 Bytes i.e, Non zero byte file. The script should print a msg if all the files in that directory are empty( 0 Byte). (2 Replies)
Hi,
I'm currently trying to write a script that checks a log file for certain errors. Once checked it then records the filesize in another file. All this is fine, my problem is that the next time I do my error check I only want to check from previously recorded filesize to the end of file. I'm... (2 Replies)