A bit new to bash and am having an issue with a for loop. I look for filenames in a specified directory and pull the date string from each meeting a certain criteria, and then would like to make a directory for each date found, like this:
Code:
search 20180101.gz 20180102.gz 20180103.gz
store 20180101 20180102 20180103 in an array
mkdir for each value found
The problem is that the mkdir part is only making a directory for the first value found.
This is the actual code:
Code:
for line in $(echo ${DPPATH}/${DBNAME}*.dmp.gz ${MTIMECMD} -type f |grep -Eo '[[:digit:]]{8}')
do
find ${DPPATH}/${DBNAME}*.dmp.gz ${MTIMECMD} -type f -exec /usr/local/bin/aws s3 mv {} "s3://"${S3BUCKET}/export_area/${DBNAME}/${line}/ \;
done
Any input would be greatly appreciated.
Last edited by jim mcnamara; 03-13-2018 at 04:16 PM..
search 20180101.gz 20180102.gz 20180103.gz
store 20180101 20180102 20180103 in an array
mkdir for each value found
The problem is that the mkdir part is only making a directory for the first value found.
First off: you won't need an array at all, so is there any other reason (outside of you thinking you'd need one) for it?
If not: a for-loop gets a list of values and runs the loops body with each of it. Here is an example:
Code:
for LOOPVAR in first second third fourth ; do
echo "LOOPVAR is: $LOOPVAR"
done
The loops body is the echo-statement here and as you can see the variable LOOPVAR is assigned one value after the other. You can not only use a static list to fill this variable but also a (so-called) "fileglob": a pattern with wildcards, which will be expanded to a list of filenames fitting this pattern:
Code:
for LOOPVAR in *dmp.gz ; do
echo "LOOPVAR is: $LOOPVAR"
done
Run this in the right directory (i have ommitted pathes for readability) and you will see that LOOPVAR is assigned a list of filenames consecutively.
Now you are left with two tasks: first, create the directory name from the filename (if i have read your problem sttement correctly you want to create a directory "foo" for a file named "foo.dmp.gz" found, yes?) and secondly, replace the "echo"-command with the "mkdir"-command. It is always a good practice to first try such constructs with "echo" and only as the last step replace it with the real command.
For the change of the filename to the directory name we use "variable expansion" and you may want to read up upon it. It is a versatile device and you should know about it:
Code:
for FILENAME in *dmp.gz ; do
echo "FILENAME is: $FILENAME, DIRNAME is: ${FILENAME%.dmp.gz}"
done
${variable%pattern} means: print the content of variable but omit an eventual pattern from the end of it: "foopattern" -> "foo".
We have all in place now, so we put in our mkdir command:
Code:
for FILENAME in *dmp.gz ; do
mkdir "${FILENAME%.dmp.gz}"
done
That's it. In case you want to create the directory somewhere else:
Code:
for FILENAME in *dmp.gz ; do
mkdir "/some/where/else/${FILENAME%.dmp.gz}"
done
I am still having trouble getting the loop to work for each date pulled from the filename using this code:
Code:
for line in $(echo ${DPPATH}/${DBNAME}*.dmp.gz ${MTIMECMD} -type f |grep -Eo '[[:digit:]]{8}')
do
find ${DPPATH}/${DBNAME}*.dmp.gz ${MTIMECMD} -type f -exec /usr/local/bin/aws s3 mv {} "s3://"${S3BUCKET}/export_area/${DBNAME}/${line%}/ \;
done
My expectation is:
1. For each file (*.dmp.gz), pull the YYYYMMDD date older than MTIMECMD - this works.
2. For each date matching the criteria, execute /usr/local/bin/aws s3 mv command - this down not work. It does not 'break' from one filename date to the next:
Code:
+ echo /backup01/export_area/d1bebo/d1bebo_metadata_exp_schema_exp_01_20180312_1824.dmp.gz /backup01/export_area/d1bebo/d1bebo_metadata_exp_schema_exp_01_20180314_1130.dmp.gz -type f
+ find /backup01/export_area/d1bebo/d1bebo_metadata_exp_schema_exp_01_20180312_1824.dmp.gz /backup01/export_area/d1bebo/d1bebo_metadata_exp_schema_exp_01_20180314_1130.dmp.gz -type f -exec /usr/local/bin/aws s3 mv '{}' <path_removed>/20180312/ ';'
Completed 1 of 2 part(s) with 1 file(s) remaining
Completed 2 of 2 part(s) with 1 file(s) remaining
move: ./d1bebo_metadata_exp_schema_exp_01_20180312_1824.dmp.gz to <path_removed>/20180312/d1bebo_metadata_exp_schema_exp_01_20180312_1824.dmp.gz
Completed 1 of 2 part(s) with 1 file(s) remaining
Completed 2 of 2 part(s) with 1 file(s) remaining
move: ./d1bebo_metadata_exp_schema_exp_01_20180314_1130.dmp.gz to <path_removed>/20180312/d1bebo_metadata_exp_schema_exp_01_20180314_1130.dmp.gz
+ find '/backup01/export_area/d1bebo/d1bebo*.dmp.gz' -type f -exec /usr/local/bin/aws s3 mv '{}' <path_removed>/20180314/ ';'
find: ‘/backup01/export_area/d1bebo/d1bebo*.dmp.gz': No such file or directory
Here, I find 2 files. One dated 20180312 and one 20180314. The first file is moved to the 20180312 directory as expected, but then, the second file is also moved to that directory. The piece of code then tries to move the file dated 20180314 to the 20180314 directory, but can't find it since it has already been moved.
Any suggestions would be greatly appreciated. Thanks again for the help.
I am still having trouble getting the loop to work for each date pulled from the filename using this code:
Code:
for line in $(echo ${DPPATH}/${DBNAME}*.dmp.gz ${MTIMECMD} -type f |grep -Eo '[[:digit:]]{8}')
do
find ${DPPATH}/${DBNAME}*.dmp.gz ${MTIMECMD} -type f -exec /usr/local/bin/aws s3 mv {} "s3://"${S3BUCKET}/export_area/${DBNAME}/${line%}/ \;
done
My expectation is:
1. For each file (*.dmp.gz), pull the YYYYMMDD date older than MTIMECMD - this works.
2. For each date matching the criteria, execute /usr/local/bin/aws s3 mv command - this down not work. It does not 'break' from one filename date to the next:
Code:
+ echo /backup01/export_area/d1bebo/d1bebo_metadata_exp_schema_exp_01_20180312_1824.dmp.gz /backup01/export_area/d1bebo/d1bebo_metadata_exp_schema_exp_01_20180314_1130.dmp.gz -type f
+ find /backup01/export_area/d1bebo/d1bebo_metadata_exp_schema_exp_01_20180312_1824.dmp.gz /backup01/export_area/d1bebo/d1bebo_metadata_exp_schema_exp_01_20180314_1130.dmp.gz -type f -exec /usr/local/bin/aws s3 mv '{}' <path_removed>/20180312/ ';'
Completed 1 of 2 part(s) with 1 file(s) remaining
Completed 2 of 2 part(s) with 1 file(s) remaining
move: ./d1bebo_metadata_exp_schema_exp_01_20180312_1824.dmp.gz to <path_removed>/20180312/d1bebo_metadata_exp_schema_exp_01_20180312_1824.dmp.gz
Completed 1 of 2 part(s) with 1 file(s) remaining
Completed 2 of 2 part(s) with 1 file(s) remaining
move: ./d1bebo_metadata_exp_schema_exp_01_20180314_1130.dmp.gz to <path_removed>/20180312/d1bebo_metadata_exp_schema_exp_01_20180314_1130.dmp.gz
+ find '/backup01/export_area/d1bebo/d1bebo*.dmp.gz' -type f -exec /usr/local/bin/aws s3 mv '{}' <path_removed>/20180314/ ';'
find: ‘/backup01/export_area/d1bebo/d1bebo*.dmp.gz': No such file or directory
Here, I find 2 files. One dated 20180312 and one 20180314. The first file is moved to the 20180312 directory as expected, but then, the second file is also moved to that directory. The piece of code then tries to move the file dated 20180314 to the 20180314 directory, but can't find it since it has already been moved.
Any suggestions would be greatly appreciated. Thanks again for the help.
From your trace output we can see that the variable MTIMECMD is either unset or set to an empty string. We, therefore, know that your statement:
Quote:
For each file (*.dmp.gz), pull the YYYYMMDD date older than MTIMECMD - this works.
is wrong. The values extracted by your for loop are in no way related to any date (since no date is specified by that variable). If MTIMECMD had been set to something like:
Code:
MTIMECMD="! -newer pathname"
where pathname is the pathname of a file you wanted to use to select gzipped dump files that are not newer than you want to select, you would still have the problem that the echo command in your for loop doesn't understand find primitives as a means to select parameters to be echoed. And, even if it did, as you have already found, there is nothing in your find command inside your for loop that makes any attempt to select files destined for a particular date from being selected to be moved to a different directory.
Maybe something more like the following would work better. Note, however, that the following is totally untested.
Code:
IAm=${0##*/}
tmpfile=$IAm.$$
DBNAME="something"
DPPATH="/what/ever"
MTIMECMD="! -newer /some/pathname"
S3BUCKET="who_knows"
trap 'rm -f "$tmpfile"' EXIT
for datestamp in $(find "$DPPATH" -name "/${DBNAME}*.dmp.gz" $MTIMECMD -type f | tee "$tmpfile" | grep -Eo '[[:digit:]]{8}' | sort -u)
do # The following assumes that your aws command creates the needed datestamp directories for you.
grep "$datestamp" "$tmpfile" | while IFS="" read -r path
do /usr/local/bin/aws s3 mv "$path" "s3://$S3BUCKET/export_area/$DBNAME/$datestamp/"
done
done
This User Gave Thanks to Don Cragun For This Post:
Hello,
i have a script that i need account_number to match a name.
for exsample :
ACCOUNT_ID=(IatHG8DC7mZbdymSoOr11w KbnlG2j-KRQ0-1_Xk356s8)
and i run a loop curl requst with this the issue is that i want to know on
which account were talking about so bash will know this :
... (4 Replies)
I am trying to check whether particular host and port are responding or not. I am using below script to check. but node_port array that i am using in loop is getting replaced with previous iteration value. Script and output is given.
Please help me to understanding why node_port values are... (5 Replies)
hello,
i need a bit of help on how to do this effectively in bash without a lot of extra looping or massive switch/case
i have a long array of M elements and a short array of N elements, so M > N always. M is not a multiple of N.
for case 1, I want to stretch N to fit M
arrayHuge
H = (... (2 Replies)
Hi,
another little question...
"sn" is an array whose elements can vary from about 55,000 to about 150,000 elements. Each element consists of an integer between 0-255, eg: ${sn} contain the value: 103 . For a decrypt-procedure I need scroll all the elements 4 or 5 times. Here is an example of... (15 Replies)
Trying to do some control flow parsing based on the index postion of an array member. Here is the pseudo code I am trying to write in (preferably in pure bash) where possible. I am thinking regex with do the trick, but need a little help.
pesudo code
if == ENDSINFIVEINTS ]]; then
do... (4 Replies)
Hello All,
Maybe I'm Missing something here but I have NOOO idea what the heck is going on with this....?
I have a Variable that contains a PATTERN of what I'm considering "Illegal Characters". So what I'm doing is looping
through a string containing some of these "Illegal Characters". Now... (5 Replies)
Below is a test script I was trying to use so that I could understand why the logic was not working in a larger script. While accessing and printing array data inside the while loop, everything is fine. Outside the loop, i guess everything is null?? The for loop that is meant to cycle... (4 Replies)
I'm trying to understand if it's possible to create a set of variables that are numbered based on another variable (using eval) in a loop, and then call on it before the loop ends.
As an example I've written a script called question (The fist command is to show what is the contents of the... (2 Replies)
Hi,
I use AIX (ksh) and Linux (bash) servers. I'm trying to do scripts to will run in both ksh and bash, and most of the time it works. But this time I don't get it in bash (I'm more familar in ksh).
The goal of my script if to read a "config file" (like "ini" file), and make various report.... (2 Replies)
03-13-2018
