Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

How to find list of missing files based on the file format?

Shell Programming and Scripting


Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 1 Week Ago
nalu nalu is offline
Registered User
 
Join Date: Apr 2017
Last Activity: 25 May 2017, 1:31 AM EDT
Posts: 22
Thanks: 1
Thanked 0 Times in 0 Posts
How to find list of missing files based on the file format?

Hi All,

In the file names we have dates.
Based on the file format given by the user,
if any file is not existed for a particular date with in a given interval we should consider that file is missing.

I have the below files in the directory /bin/daily/voda_files.


Code:
asr_spir_2017-05-10-150325_2017-05-10-112227_2017-05-13-112227.txt
adb_voda_2017-05-11-150325_2017-05-10-112227_2017-05-13-112227.txt
adb_voda_2017-05-14-150325_2017-05-11-112227_2017-05-10-112227.txt
adb_voda_2017-05-12-150325_2017-05-12-112227_2017-05-11-112227
adb_voda_2017-05-16_2017-04-30_2017-05-01.txt
adb_voda_20170510.txt
adb_voda_2017-05-10.txt
2017-05-11
2017-05-10.txt
2017-05-12

If user entered


Code:
file_format=xxx_xxxx_YYYY-MM-DD-HHIISS_?????????????????_?????????????????.txt
prog_name="abd_voda_"
interval=10 (from current date -1 to 10 days back It should be from 2017-05-08 to 2017-05-17)

In this case it should consider the first date in the file.
The missing files output should be

Code:
adb_voda_2017-05-08
adb_voda_2017-05-09
adb_voda_2017-05-10
adb_voda_2017-05-12
adb_voda_2017-05-13
adb_voda_2017-05-15
adb_voda_2017-05-16
adb_voda_2017-05-17

If user entered


Code:
file_format=xxx_xxxx_?????????????????_?????????????????_YYYY-MM-DD-HHIISS.txt
prog_name="abd_voda_"
interval=10 (from current date -1 to 10 days back It should be from 2017-05-08 to 2017-05-17)

In this case it should consider the last date in the file.
The missing files output should be

Code:
adb_voda_2017-05-08
adb_voda_2017-05-09
adb_voda_2017-05-11
adb_voda_2017-05-12
adb_voda_2017-05-14
adb_voda_2017-05-15
adb_voda_2017-05-16
adb_voda_2017-05-17

If user entered


Code:
file_format=YYYY-MM-DD
prog_name=""
interval=10 (from current date -1 to 10 days back It should be from 2017-05-08 to 2017-05-17).

It should consider the files which are having only YYYY-MM-DD.
The missing files output should be

Code:
2017-05-08
2017-05-09
2017-05-10
2017-05-13
2017-05-14
2017-05-15
2017-05-16
2017-05-17

Please help me with the script.

Thanks in advance.
Sponsored Links
    #2  
Old Unix and Linux 1 Week Ago
vgersh99's Unix or Linux Image
vgersh99 vgersh99 is online now Forum Advisor  
Forum Advisor
 
Join Date: Feb 2005
Last Activity: 29 May 2017, 7:28 AM EDT
Location: Foxborough, MA
Posts: 8,152
Thanks: 220
Thanked 770 Times in 721 Posts
given your detailed description.... what have you tried and where exactly are you stuck?
Sponsored Links
    #3  
Old Unix and Linux 1 Week Ago
nalu nalu is offline
Registered User
 
Join Date: Apr 2017
Last Activity: 25 May 2017, 1:31 AM EDT
Posts: 22
Thanks: 1
Thanked 0 Times in 0 Posts
Hi,

I have tried this but its not working as expected.


Code:
file_format=xxx_xxxx_?????????????????_?????????????????_YYYY-MM-DD-HHIISS.txt
prog_name="abd_voda_"
interval=10

 #checking before . extension
YearFormat=$(echo $file_format | sed 's/X//g;s/x//g;s/_*$//g;s/^_*//g' | awk 'BEGIN{FS=".";}{for (i = 1; i <= NF; i++){if ( $i ~ /YY/ ){print $i;}}}' | sed 's/X//g;s/x//g;s/_*$//g;s/^_*//g' | head -1)
YearFormat_count=$(echo $YearFormat | wc -l)
echo "YearFormat-$YearFormat"
if [[ $YearFormat_count -lt 1 ]]; then
echo "enter proper year format"
exit 0
fi
 #exit is missing here
month_count=0;y_count=0;d_count=0;hour_count=0;min_count=0;sec_count=0
for (( i=0; i<${#YearFormat}; i++ )); do
year=$(echo "${YearFormat:$i:1}")
if [[ $year == 'Y' ]];then
y_count=`expr $y_count + 1`
fi
if [[ $year == 'M' ]];then
month_count=`expr $month_count + 1`
fi
if [[ $year == 'D' ]];then
d_count=`expr $d_count + 1`
fi
if [[ $year == 'H' ]];then
hour_count=`expr $hour_count + 1`
fi
if [[ $year == 'I' ]];then
min_count=`expr $min_count + 1`
fi
if [[ $year == 'S' ]];then
sec_count=`expr $sec_count + 1`
fi
done

 #Remove the duplicates
cleandateFormat=$(echo $YearFormat | tr -s 'A-Z') #YmD_HMS
echo $cleandateFormat
if [[ $y_count -eq 2 ]]; then
cleandateFormat=$(echo $cleandateFormat | sed 's/Y/y/g;s/M/m/g;s/D/d/g;s/I/M/g')
elif [[ $y_count -eq 4 ]]; then
cleandateFormat=$(echo $cleandateFormat | sed 's/M/m/g;s/D/d/g;s/I/M/g')
else
echo "enter correct year format"
exit 0
fi
 #maintain format including special character as seperator
finalFormat=""
for (( i=0; i<${#cleandateFormat}; i++ )); do
f_year=$(echo "${cleandateFormat:$i:1}")
echo "char : $f_year"
if [[ $f_year == [a-zA-Z] ]];then
temp=$(echo $f_year | sed 's/^/%/g')
finalFormat="$finalFormat$temp"
echo "IF : $finalFormat"
else
finalFormat="$finalFormat$f_year"
fi
done

if [[ $check_mode -eq 1 ]]
then
missing_count=0
day_count=0
while [[ $interval -ne 0 ]]; do
finalFormat=$(echo "$finalFormat" | sed -r 's/[HMS]+//g;s/%*$//g;s/-*$//g;s/_*$//g')
start=`date +${finalFormat} -d "$interval day ago"`
IFS=$','
for path in ${file_path}; do
count=$(ls -l /bin/daily/voda_files/${prog_name}*${start}* 2>/dev/null | wc -l)
if [[ $count -gt 0 ]]
then
break
fi
done
unset IFS
if [[ $count -eq 0 ]]
then
missing_count=`expr $missing_count + 1`
file_name="${start}"
printf "$file_name\n"
fi
interval=`expr $interval - 1`
done >missingfiles.txt
fi

Please help me.
Thanks in advance.
    #4  
Old Unix and Linux 1 Week Ago
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 29 May 2017, 6:37 AM EDT
Location: Aachen, Germany
Posts: 10,830
Thanks: 272
Thanked 3,324 Times in 3,061 Posts
"its not working as expected" doesn't really answer vgersh99's Q: "where exactly are you stuck".
And, in recent threads you received hints and examples of data & date manipulation which I don't find back in above. Were all those posts in vain?
Sponsored Links
    #5  
Old Unix and Linux 1 Week Ago
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 29 May 2017, 4:14 AM EDT
Location: San Jose, CA, USA
Posts: 10,309
Thanks: 521
Thanked 3,587 Times in 3,055 Posts
Your requirements here (as in your previous three threads on this topic) are confusing and incomplete.

I have no idea how you expect to match a filename to the second (as required by the format strings you are using that specify not only year, month, and day but also hour minute and second) which you then compare to the year, month, day (for the previous 10 days) and the hour, minute, and second at the time at which you run your script. How will you guarantee that you are running your script at exactly 15:03:25 when you are looking for matches for the 1st dates in your filenames and at exactly 11:22:27 when you are looking for matches for the last dates in your filenames?

If your input filename samples:

Code:
asr_spir_2017-05-10-150325_2017-05-10-112227_2017-05-13-112227.txt
adb_voda_2017-05-11-150325_2017-05-10-112227_2017-05-13-112227.txt
adb_voda_2017-05-14-150325_2017-05-11-112227_2017-05-10-112227.txt
adb_voda_2017-05-12-150325_2017-05-12-112227_2017-05-11-112227
adb_voda_2017-05-16_2017-04-30_2017-05-01.txt
adb_voda_20170510.txt
adb_voda_2017-05-10.txt
2017-05-11
2017-05-10.txt
2017-05-12

are correct, and you want to match filenames starting with adb_voda_ with the dates marked in red, and ending with .txt, it would seem that the format you feed into your script should be:

Code:
adv_voda_YYYY-MM-DD-??????_????-??-??-??????_????-??-??-??????.txt

which your code would then convert to the date format string:

Code:
adv_voda_%Y-%m-%d-??????_????-??-??-??????_????-??-??-??????.txt

and date would then create a pathname matching pattern from that that would match the file(s) you want to select for a given date without a prefix pattern and without asterisks that have cause you problems in all of your previous threads (as well as in this thread).

All of the code you have that is stripping off _s, and Xs and ?s seems to be fighting against matching only the filenames you want to match.

Similarly, if you wanted to match the last date in those files (marked in blue), it would seem that you want the input format string to be:

Code:
adv_voda_????-??-??-??????_????-??-??-??????_YYYY-DD-MM-??????.txt

which your code would then convert to the date format string:

Code:
adv_voda_????-??-??-??????_????-??-??-??????_%Y-%m-%d-??????.txt

Sponsored Links
    #6  
Old Unix and Linux 1 Week Ago
nalu nalu is offline
Registered User
 
Join Date: Apr 2017
Last Activity: 25 May 2017, 1:31 AM EDT
Posts: 22
Thanks: 1
Thanked 0 Times in 0 Posts
Hi don,

Thanks a lot for your response.


could you lease help me how to convert this
Code:
tadv_voda_YYYY-MM-DD-??????_????-??-??-??????_????-??-??-??????.txt

to date format string as you said.

which your code would then convert to the date format string:



Code:
adv_voda_%Y-%m-%d-??????_????-??-??-??????_????-??-??-??????.txt

Thanks in advance.

---------- Post updated at 05:55 AM ---------- Previous update was at 04:30 AM ----------

Hi don,

I have done the code to convert to the date format string as below.


Code:
adv_voda_%Y-%m-%d-??????_????-??-??-??????_????-??-??-??????.txt

Now how can I search for missing dates.
Could you lease help me.

What changes I have to do in the below code.


Code:
missing_count=0
day_count=0
while [[ $interval -ne 0 ]]; do
finalFormat=$(echo "$finalFormat" | sed -r 's/[HMS]+//g;s/%*$//g;s/-*$//g;s/_*$//g')
start=`date +${finalFormat} -d "$interval day ago"`
IFS=$','
for path in ${file_path}; do
count=$(ls -l /bin/daily/voda_files/${prog_name}*${start}* 2>/dev/null | wc -l)
if [[ $count -gt 0 ]]
then
break
fi
done
unset IFS
if [[ $count -eq 0 ]]
then
missing_count=`expr $missing_count + 1`
file_name="${start}"
printf "$file_name\n"
fi
interval=`expr $interval - 1`
done >missingfiles.txt
fi

Please help me.
Thanks in advance.

Last edited by nalu; 1 Week Ago at 07:08 AM..
Sponsored Links
    #7  
Old Unix and Linux 1 Week Ago
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 29 May 2017, 4:14 AM EDT
Location: San Jose, CA, USA
Posts: 10,309
Thanks: 521
Thanked 3,587 Times in 3,055 Posts
What operating system (including release number) are you using?

What shell (including version number) are you using?

Do you have a ksh (version 93u+ or later) that I can use instead of whatever shell you're using to provide an example?
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Find list of files missing read & execute permission Catullus Shell Programming and Scripting 2 12-09-2015 01:52 PM
How to list the files based on the modification time using the find command? Kesavan Shell Programming and Scripting 6 03-06-2012 10:59 AM
Find files older than X with a weird file format MickAAA Shell Programming and Scripting 2 12-16-2010 10:07 PM
Find missing files from a list ultimatix Shell Programming and Scripting 5 08-10-2010 06:05 AM
Help with find command and list in a long format each found file alexcol Shell Programming and Scripting 5 12-10-2007 07:59 AM



All times are GMT -4. The time now is 07:32 AM.