Script to tar/rsync/rm multiple folder names


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to tar/rsync/rm multiple folder names
# 36  
Old 04-25-2016
so its this

Code:
if ! tar cf "${DIR}".tar "${DIR}"
then    mail -s "tar failed $DIR" $MailAddress <<< "creating of tar $DIR failed due to error, removing $DIR.tar"
rm -f "$DIR".tar
continue
fi

# 37  
Old 04-25-2016
Quote:
Originally Posted by robertkwild
so its this

Code:
if ! tar cf "${DIR}".tar "${DIR}"
then    mail -s "tar failed $DIR" $MailAddress <<< "creating of tar $DIR failed due to error, removing $DIR.tar"
rm -f "$DIR".tar
continue
fi

We realize that the tab key is broken on your keyboard and you don't care about being able to see the structure of your code by properly indenting it. So, yes, the above code will be accepted by your shell.

But, some day, perhaps years from now when you have to look at your code again when something needs to be changed, you'll realize that programming style does make a difference and if statements should be indented when they appear inside a loop and statements in then and else clauses in if statements should be indented further.

Code:
	if ! tar cf "${DIR}".tar "${DIR}"
	then	mail -s "tar failed $DIR" $MailAddress <<< "creating of tar $DIR failed due to error, removing $DIR.tar"
		rm -f "$DIR".tar
		continue
	fi

# 38  
Old 04-25-2016
ok so the ! (explanation mark) means if the command doesnt return a 0 or if it fails

sorry to bang on about this just learning so excuse me if i am annoying you guys

---------- Post updated at 08:43 AM ---------- Previous update was at 07:39 AM ----------

also i want to add to my script that if the directory stays the same size then it is safe to start of the script but if the directory changes size ie someones is still moving the directory in "to_be_archived" dont do it and wait till the directoey doesnt increase in size and then it can start off the script

Last edited by robertkwild; 04-25-2016 at 11:11 AM..
# 39  
Old 04-25-2016
Quote:
Originally Posted by robertkwild
so the ! means if its not equal to 0/doesnt succeed/fails in doing the tar command it will send a mail and remove the tar file?
Yes. Whenever the tar-command returns a non-zero value the part between "then" and "fi" (or, in case an else-branch is there, the "then and the "else") is executed. In your case, since there is a "mail"-command and an "rm"-command it will do exactly that.

Quote:
Originally Posted by robertkwild
sorry to bang on about this just learning so excuse me if i am annoying you guys
You are not annoying us because of your questions. What is really annoying is the way you constantly ignore what we say about indenting. Believe me, i have written several hundred thousands lines of code in my life (and Don, i believe, something in the same order of magnitude) and i can tell you that it is absolutely necessary! I wrote programs in Assembler, C, FORTRAN, Clipper, and some interpreted languages (mostly shells) and this was constant in all of them!

If you do not understand how to indent properly: say so. We will explain it to you. But to simply ignore our many suggestions, THAT is really annoying - because it creates a feeling that what we do is in vain.

Here is a simple procedure how to indent properly (in every language):

Indentation is for making stand out two sorts of statements: conditionally executed ones and loop contents. Here is how a loop should look like:

Code:
loop-command (while, for, until, repeat, ....)
     command to execute every iteration of the loop 1
     command to execute every iteration of the loop 2
     command to execute every iteration of the loop 3
     [...]
end-loop command

What you see instantly is: 1) where the loop starts and 2) where it ends and 3) what it consists of (the "loops body").

Here is a real-world example in shell:

Code:
(( i = 1 ))
while [ $i -le 100 ] ; do
     echo "i is now: $1"
     (( i = i + 1 ))
done

Now, this is my style of doing it, Don Cragun does it a bit different (but equally correct - that is a matter of personal taste):

Code:
(( i = 1 ))
while [ $i -le 100 ]
do
     echo "i is now: $1"
     (( i = i + 1 ))
done

Now the conditionals: the same principle applies. Whatever is causing the condition (the "if"-part) is put into a line, then everything between that and the else-line is indented, the "else" itself is at the same level as the "if" and then everything else is indented again until the end of the "if":

Code:
if <some-condition>
     command executed if condition is true 1
     command executed if condition is true 2
     command executed if condition is true 3
     [...]
else
     command executed if condition is false 1
     command executed if condition is false 2
     command executed if condition is false 3
     [...]
end-if

Here is a real-world example, this time not in shell but in C:

Code:
if (a < b ) {
     printf( "a is smaller than b\n" );
     a += 1;
} else {
     printf( "a is NOT smaller than b\n" );
     b += 1;
}

Notice again how the important parts - which condition everything is based upon ("a < b"), which part is executed when and where each part begins and ends - stand out.
Most programming languages allow to "chain" if-statements. Altough it is not necessary i always align the various conditions horizontally so that they start at the same column. I find that easier to keep track of:

Code:
if   [ $x -eq 1 ] ; then
     echo "x is 1"
elif [ $x -eq 2 ] ; then
     echo "x is 2"
else
     echo "x is neither 1 nor 2"
fi

Finally there is another conditional in most programming languages: "case" (or, in C, "switch"). It allows for several branches depending on various values of a single variable or expression. The syntax differs from language to language, but the sctructure is always the same:

Code:
case <expression>
     first-possible-value)
          command 1 to execute if expression is "first-possible-value"
          command 2 to execute if expression is "first-possible-value"
          command 3 to execute if expression is "first-possible-value"
          [...]

     second-possible-value)
          command 1 to execute if expression is "second-possible-value"
          command 2 to execute if expression is "second-possible-value"
          command 3 to execute if expression is "second-possible-value"
          [...]

     third-possible-value)
          [...]

end-of-case

Here is the chained if-else-construct from above, this time with a "case":

Code:
case $x in
     1)
          echo "x has the value 1"
          ;;

     2)
          echo "x has the value 2"
          ;;

     *)
          echo "x has neither the value 1 nor 2"
          ;;

esac

Notice how the important parts spring right into your eyes: the beginning and end of the construct, the various possible values (1, 2, *) and what part of the code is to be executed if these were to be the case (again, with the beginning and the ending of each).

Finally: you define some "indenting step" for yourself. Pick anything you are comfortable with. I prefer 5 spaces because i like the clear picture it gives. On the other hand you start pretty soon to get far right if you have many indentation steps. Some prefer only 3 spaces, some even two. One can nest more levels with less indentation but at the same time different parts of the code are not separated as clearly as with more indentation. Find your own optimum but FIND one. Do NOT write code the way you do because you will hate yourself once you have to correct or change your own programs after a while.

I hope this helps.

bakunin
These 3 Users Gave Thanks to bakunin For This Post:
# 40  
Old 04-25-2016
Quote:
Originally Posted by robertkwild
ok so the ! (explanation mark) means if the command doesnt return a 0 or if it fails

sorry to bang on about this just learning so excuse me if i am annoying you guys

---------- Post updated at 08:43 AM ---------- Previous update was at 07:39 AM ----------

also i want to add to my script that if the directory stays the same size then it is safe to start of the script but if the directory changes size i.e. someones is still moving the directory in "to_be_archived" dont do it and wait till the directoey doesnt increase in size and then it can start off the script
It isn't that you're annoying us. We know you're learning. As bakunin said, what bothers us is that you ask questions, we make suggestions, and you ignore our suggestions. (Did you try the script I suggested in post #16 in this thread to see what it would do? Did you even read that post?)

The size of a directory tells you very little about whether or not files in that directory are growing, and (if any files have ever been removed from a directory) even if a new file is created in a directory, its size might not change.

It would be much better if you would create a hidden directory under /to_be_archived (such as /to_be_archived/.in_progress/project), move everything into that directory, and then after that project directory contains everything you want to archive, issue the command:
Code:
mv /to_be_archived/.in_progress/project /to_be_archived/project

which will be an atomic operation. Your existing script won't see the hidden directory so it won't see .in_progress/project until it is ready to process after it has been moved up.

Last edited by Don Cragun; 04-25-2016 at 07:47 PM..
# 41  
Old 04-26-2016
Hi.

I see that there are similar recent questions posted in
wait and continue if directory stays same size
and in
wait and continue if directory stays same size

My take is that directory space is allocated not just one entry at a time, but by blocks. So that there may be no noticeable change when objects are added to a directory.

Similarly, on some systems (perhaps older), directory space was not reduced when object are deleted.

A search provides these links, describing such properties as directory size:
linux - Shrink/reset directory size? - Server Fault
inode - Monotonic growth of Linux directory size/block count - Server Fault
https://bbs.archlinux.org/viewtopic.php?id=135252

So trying to detect a change in directory content by monitoring the size seems to be fruitless.

However, there are, as can be seen from the responses at LQ, that there may be ways to monitor events within a directory. (I thought I had posted a demonstration of that, but I cannot find it just now.)

Best wishes ... cheers, drl

---------- Post updated at 06:17 ---------- Previous update was at 06:08 ----------

Hi.

Quote:
Originally Posted by bakunin
Finally: you define some "indenting step" for yourself. Pick anything you are comfortable with. I prefer 5 spaces because i like the clear picture it gives. On the other hand you start pretty soon to get far right if you have many indentation steps. Some prefer only 3 spaces, some even two. One can nest more levels with less indentation but at the same time different parts of the code are not separated as clearly as with more indentation. Find your own optimum but FIND one. Do NOT write code the way you do because you will hate yourself once you have to correct or change your own programs after a while.
Exactly.

In my situation, I almost never format/indent manually because I'm too fallible and there are tools to do this. I apply these continually while I am developing. Here are some:
Code:
Tidy, format, indent, beautify, pretty-print source code text files

        0) pretty-printing on postscript printer
           trueprint
           a2ps

        1) perltidy

        2) awk-pretty
           ftp://ftp.armory.com/pub/scripts/fmtawksh (verified 2016.04.25)
           http://www.armory.com/~ftp/

        3) f90tidy
           http://www.ifremer.fr/ditigo/molagnon/fortran90/ (verified 2016.04.25)

        4) f77tidy
           http://www.pdas.com/tidy.html (verified 2016.04.25)

        5) bash, shell
           http://arachnoid.com/python/beautify_bash_program.html
           (verified 2016.04.25)

        6) tclfrink
           http://wiki.tcl.tk/2611 (verified 2016.04.25)
           tcltidy
           http://wiki.tcl.tk/15731 (verified 2016.04.25)

        7) ruby
           http://arachnoid.com/ruby/rbeautify.rb.txt (verified 2016.04.25)
           ruby-beautify

        8) C, C++, Objective-C, C#, and Java
           astyle
           C
           indent

Best wishes ... cheers, drl
# 42  
Old 04-26-2016
dont believe it, on the last hurdle i get this error while running my script

bash: /archive_script.sh: /bin/bash^M: bad interpreter: No such file or directory

this is my code

Code:
#!/bin/bash
cd /to_be_archived/
for DIR in * ;

        do
        MailAddress="robertw@molinare.co.uk"

        if ! inotifywait -rq -e modify,create,delete -t 60 /to_be_archived/"$DIR" ; then
        mail -s "Modified $DIR" $MailAddress <<< "$DIR failed has been modified within the last minute"
        continue
        fi

        if ! tar cf "$DIR".tar "$DIR" ; then
        mail -s "tar failed $DIR" $MailAddress <<< "creating of tar $DIR failed due to error, removing $DIR.tar"
        rm -f "$DIR".tar
        continue
        fi

        if [ -f /archived_projects/"$DIR".tar ]; then
        mail -s "duplicate exists $DIR" $MailAddress <<< "$DIR.tar already exists"
        rm -f "$DIR".tar
        continue
        fi

        if ! rsync -a "$DIR".tar /archived_projects/ ; then
        mail -s "rsync failed $DIR" $MailAddress <<< "rsync of $DIR failed due to error, removing $DIR.tar"
        rm -f "$DIR".tar
        continue
        fi

        if ! rm -f "$DIR".tar ; then
        mail -s "remove tar failed $DIR" $MailAddress <<< "removing of tar $DIR failed due to error"
        continue
        fi

        if ! rm -rf "$DIR" ; then
        mail -s "remove folder failed $DIR" $MailAddress <<< "removing of $DIR failed due to error"
        continue
                else
                mail -s "success $DIR" $MailAddress <<< "successfully completed archiving $DIR"
        fi

        done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Checking Multiple File existance in a UNIX folder(Note: File names are all different)

HI Guys, I have some 8 files with different name and extensions. I need to check if they are present in a specific folder or not and also want that script to show me which all are not present. I can write if condition for each file but from a developer perspective , i feel that is not a good... (3 Replies)
Discussion started by: shankarpanda003
3 Replies

2. Homework & Coursework Questions

Bash script for printing folder names and their sizes

1. The problem statement, all variables and given/known data: The task is to create a script that would reproduce the output of 'du' command, but in a different way: what 'du' does is: <size> <folder name>and what is needed is <folder name> <size>We need to show only 10 folders which are the... (3 Replies)
Discussion started by: UncleIS
3 Replies

3. Shell Programming and Scripting

Bash script for printing folder names and their sizes

Good day, everyone! I'm very new to bash scripting. Our teacher gave us a task to create a script that basically does the same job the 'du' command does, with the difference that 'du' command gives an output in the form of <size> <folder name>and what we need is <folder name> <size>As for... (1 Reply)
Discussion started by: UncleIS
1 Replies

4. Shell Programming and Scripting

Script to move one folder to multiple folder...

Hi All, I have to requirement to write a shell script to move file from one folder (A) to another five folder (B,C,D,E,F) and destination folder should be blank. In not blank just skip. This script will run as a scheduler every 2 minutes. It will check number of files in folder A and move 1 to... (9 Replies)
Discussion started by: peekuabc
9 Replies

5. UNIX for Dummies Questions & Answers

Do I need to extract the entire tar file to confirm the tar folder is fine?

I would like to confirm my file.tar is been tar-ed correctly before I remove them. But I have very limited disc space to untar it. Can I just do the listing instead of actual extract it? Can I say confirm folder integrity if the listing is sucessful without problem? tar tvf file1.tar ... (1 Reply)
Discussion started by: vivien_chu
1 Replies

6. Shell Programming and Scripting

tar command to explore multiple layers of tar and tar.gz files

Hi all, I have a tar file and inside that tar file is a folder with additional tar.gz files. What I want to do is look inside the first tar file and then find the second tar file I'm looking for, look inside that tar.gz file to find a certain directory. I'm encountering issues by trying to... (1 Reply)
Discussion started by: bashnewbee
1 Replies

7. Shell Programming and Scripting

editing names of files in multiple folder

I have 1000's of directories which is named as numbers. Each directory contains multiple files. Each of these directories have a file named "att". I need to rename all the att files by adding the directory name followed by "_" then att for each of the directories. Directories 120 att... (2 Replies)
Discussion started by: Lucky Ali
2 Replies

8. Shell Programming and Scripting

Script to move files with similar names to folder

I have in directory /media/AUDIO/WAVE many .mp3 files with names like: my filename_01of02.mp3 my filename_02of02.mp3 Your File_01of06.mp3 Your File_02of06.mp3 etc.... In the same directory, /media/AUDIO/WAVE, I have many folders with names like 9780743579490 9780743579491 etc.. Inside... (7 Replies)
Discussion started by: glev2005
7 Replies

9. Shell Programming and Scripting

lvm/tar/rsync backup script feedback/criticism

I have written a shell script to perform backups using tar, rsync and optionally utilise lvm snapshots. The script is not finished but is in a working state and comments/descriptions are poor. I would greatly appreciate any criticism and suggestions of the script to help improve my own learning... (0 Replies)
Discussion started by: jelloir
0 Replies

10. UNIX for Dummies Questions & Answers

Copying multiple folders to local machine (don't know folder names)

Hi. I'm trying to copy multiple folders from the remote machine to the local machine. I wrote a batch file to run an ftp window. The problem I am having is that the only command to copy files is mget *, and this copies only files, not folders. For example, ftp ts555 cd ts555/test ' test... (5 Replies)
Discussion started by: leenyburger
5 Replies
Login or Register to Ask a Question