Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to tar/rsync/rm multiple folder names

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to tar/rsync/rm multiple folder names
# 1  
Old 04-12-2016
Script to tar/rsync/rm multiple folder names
hi all,

i attach a link with what im trying to do automatically via script but i have some questions i need answering please, bear in mind i am really new to bash scripting, the only thing i know how to do is commands in scripts like cd rm tar rsync cp stuff like that

i have mutiple project folders in the "to_be_archived" folder ie

batman
superman
hulk
spiderman
iron_man
etc etc...

so it makes a tar file of the folder, rsyncs tar file to another folder "archived_projects" and deletes the tar file and folder in "to_be_archived"

my example i just did a test with a untitled folder to check to see if it works and it does

i want it to -

do all folders in the list and not just one specific folder, so once it does the 1st one it does the 2nd one so on so forth

i want to check before it deletes the folder and tar file in the (to_be_archived) i want to double check if it has rsynced the whole file across to (archived_projects)

cheers,

rob
Script to tar/rsync/rm multiple folder names-archivejpg
Last edited by robertkwild; 04-12-2016 at 07:49 AM..
# 2  
Old 04-13-2016
Quote:
Originally Posted by robertkwild
bear in mind i am really new to bash scripting, the only thing i know how to do is commands in scripts like cd rm tar rsync cp stuff like that
Welcome to the forum. There is no problem with being a beginner. We are here to help you (learn). How about i give you some pointers and you try to fill in the rest?

Quote:
Originally Posted by robertkwild
i have mutiple project folders in the "to_be_archived" folder ie

batman
superman
hulk
spiderman
iron_man
etc etc...

so it makes a tar file of the folder, rsyncs tar file to another folder "archived_projects" and deletes the tar file and folder in "to_be_archived"
I take it, you want a separate tar-file for each folder, i.e. a batman.tar, a superman.tar, and so on, yes?

If so: this is really easy to set up by using a loop. Here is how to do it:

Code:
for DIR in /some/dir/to_be_archived/* ; do
    echo $DIR
done

This will cycle through all directory entries of /some/dir/to_be_archived/* (in scripts it is always better to use absolute paths instead of relative ones) and set the variable "$DIR" to each value for every run through the loop. As it is there is only one command - echo $DIR - but it shows the mechanism.

Because you perhaps want to use the last part of the name for the naming you need to extract this first into another variable and then use this:

Code:
for DIR in /some/dir/to_be_archived/* ; do
    echo full DIR is: $DIR
    echo dir name : ${DIR##*/}
done
full DIR is: /some/dir/to_be_archived/batman
dir name : batman
full DIR is: /some/dir/to_be_archived/superman
dir name : superman
full DIR is: /some/dir/to_be_archived/hulk
dir name : hulk
[....]

Now let us construct some commands around this:
Code:
cd /some/dir/to_be_archived
for DIR in /some/dir/to_be_archived/* ; do
    fSaveDir="${DIR##*/}"
    echo tar cvf /some/dir/to_be_archived/${fSaveDir}.tar ./${fSaveDir}
done
tar /some/dir/to_be_archived/batman.tar ./batman
tar /some/dir/to_be_archived/superman.tar ./superman
tar /some/dir/to_be_archived/hulk.tar ./hulk
[....]

If this is what you want, remove the "echo" so that the command - instead of being displayed - is executed. The same way you can add more commands inside the loop and use "${fSaveDir}" whereever you would enter the directories name.

Quote:
Originally Posted by robertkwild
i want to check before it deletes the folder and tar file in the (to_be_archived) i want to double check if it has rsynced the whole file across to (archived_projects)
This is understandable, but there is a better way to do this: UNIX-commands always set a "return code" upon their exit. This return code is 0 (zero) when the command was successful, something else when not. You can simply check this return code after each command and do something (write an error message, exit the script, ....) if it isn't 0. There is a special variable "$?" for this, but you can use an "if"-statement as well (i suggest you try it out with a few commands). First the command version, then the same within an if-statement:

Code:
# ls /etc/hosts ; echo $?
/etc/hosts
0

# ls /some/file/which/doesnot/exist ; echo $?
/some/file/which/doesnot/exist not found
2

Code:
if <some command> ; then
     echo "this command returned 0."
else
     echo "this command returned non-zero."
fi

This will make your script look like this, with a possible outcome below:

Code:
cd /some/dir/to_be_archived
for DIR in /some/dir/to_be_archived/* ; do
    fSaveDir="${DIR##*/}"
    if tar cvf /some/dir/to_be_archived/${fSaveDir}.tar ./${fSaveDir} ; then
          echo "tar-command for ${fSaveDir} successful"
    else
          echo "tar-command for ${fSaveDir} failed, aborting."
          exit 2
    fi
    if <next-command> ${fSaveDir} ; then
          echo "<next-command> for ${fSaveDir} successful"
    else
          echo "<next-command> for ${fSaveDir} failed, aborting."
          exit 3
    fi
    if
        [....]
    fi
done
tar-command for batman successful
<next-command> for batman successful
<other-command> for batman successful
[...]
tar-command for superman successful
<next-command> for superman successful
<other-command> for superman failed, aborting.

This would not only tell you what exactly worked but also where the script failed.

If you still have questions feel free to ask.

I hope this helps.

bakunin
# 3  
Old 04-13-2016
thank you so much for your help, this is a great template to learn of

i need to figure out what each syntax you put does, i know what some mean

yes i do want a seperate tar file for each folder (ie whatever project folder is in the to_be_archived i want it to have that folder name.tar)

what about the rsync and rm commands in my screenshots?

cheers,

rob

---------- Post updated at 10:51 AM ---------- Previous update was at 09:30 AM ----------

getting there

Code:
#!/bin/bash
cd /to_be_archived/
for DIR in * ; do
fSaveDir="${DIR##*/}"
tar -cf "${fSaveDir}".tar "${fSaveDir}"
rsync -a "${fSaveDir}".tar /archived_projects/
rm -f "${fSaveDir}".tar
rm -rf "${fSaveDir}"
done

how did you know this command

fSaveDir="${DIR##*/}"?


Moderator's Comments:
Mod Comment Please use code tags as required by forum rules!
Last edited by RudiC; 04-13-2016 at 12:53 PM.. Reason: Added code tags.
# 4  
Old 04-13-2016
Quote:
Originally Posted by robertkwild
getting there

Code:
#!/bin/bash
cd /to_be_archived/
for DIR in * ; do
fSaveDir="${DIR##*/}"
tar -cf "${fSaveDir}".tar "${fSaveDir}"
rsync -a "${fSaveDir}".tar /archived_projects/
rm -f "${fSaveDir}".tar
rm -rf "${fSaveDir}"
done

Excellent!

Just a few points you might want to observe as you go along:

First, it is common style to indent code: every conditional statement and every looping statement triggers one (more) level of indentation. The reason is that it lets stand out the loop body and the conditionally executed statements. Software (any software - even 5-line-scripts) is written to be maintained easily and this helps getting faster what code does. Suppose you write your script, don't look at it for some months and then want to change something: you won't have its "inner organisation" as present as you have it now.

Another thing is: like you organize a longer text into paragraphs to make it easier to read you can put empty lines into the code to group parts of the commands. I would have written your script this way:

Code:
#!/bin/bash

cd /to_be_archived/
for DIR in * ; do
     fSaveDir="${DIR##*/}"

     tar -cf "${fSaveDir}".tar "${fSaveDir}"
     rsync -a "${fSaveDir}".tar /archived_projects/
     rm -f "${fSaveDir}".tar
     rm -rf "${fSaveDir}"
done

Second: "tar" is a command which takes subcommands. "c" and "f" are such subcommands, not options. They are therefore NOT introduced with a dash. Yes, this is inconsequent and, yes, most tar-versions tolerate the dashes anyway because the misuse is so common, but still: correct is this:

Code:
tar cf /target/file.tar /directory/to/process

and not this:

Code:
tar -cf /target/file.tar /directory/to/process

Finally: your script relies on a certain environment being set. enter env at the commandline and you will see many variables being set to certain values. All these variables are set also inside your script when it runs. You rely, for instance, on a certain value of "PATH", because tar is usually /usr/bin/tar. This implicitly set environment may not always be there, especially when you put this script into cron to have it executed automatically from time to time. Do yourself a favour and set the environment you need explicitly:

Code:
#!/bin/bash

PATH="/usr/bin:/usr/local/bin:<or whatever else you may need>"
export PATH
<other variables you may need inside your script>

cd /to_be_archived/
for DIR in * ; do
     fSaveDir="${DIR##*/}"

     tar -cf "${fSaveDir}".tar "${fSaveDir}"
     rsync -a "${fSaveDir}".tar /archived_projects/
     rm -f "${fSaveDir}".tar
     rm -rf "${fSaveDir}"
done

Quote:
Originally Posted by robertkwild
how did you know this command

fSaveDir="${DIR##*/}"?
It is called "parameter expansion" or "variable expansion" and you will find it (like most other things i touched upon) in the man page of your shell. Enter man bash at the command prompt or do a google search.

This specific expansion is "##" which means: take the pattern following after that and remove the left part of the variables content matching the pattern: because "*/" matches "everything up to the last "/" - which is everything save for the directories name. There is also a method for cutting from the rightmost part of the content, see below:

Try the following at the commandline:

Code:
x="a/b/c/d/e"            # our content to play with
echo "${x##*/}"         # gives you "e" - the longest possible match for "*/" is cut off leftside
echo "${x#*/}"         # gives you "b/c/d/e", the shortest possible match
echo "${x%/*}"         # gives you "a/b/c/d", the shortest possible match for "/*" is cut off rightside
echo "${x%%/*}"        # will give you "a", the longest possible match for "/*"

I hope this helps.

bakunin
# 5  
Old 04-14-2016
thank you bakunin

i understand why you put the executable paths in but what is this line for -

Code:
export PATH

also when i do this command i get a text file of the project folder name and size of each folder but how do i make it do file counts of each directory?

Code:
du -h /to_be_archived/* >> /archive

and i will add this code just before and after the lines

Code:
cd /to_be_archived/
i will put it here
for DIR in * ; do

many thanks,

rob
Last edited by robertkwild; 04-14-2016 at 07:40 AM..
# 6  
Old 04-14-2016
Quote:
Originally Posted by robertkwild
i understand why you put the executable paths in but what is this line for -
Code:
export PATH

It means: if this shell happens to open another shell (for instance by calling another script) the current value of the exported variable will be known there. Right now there is no such other shell called so this is not necessary. But in case you do call another shell at some time in the future it will be good to be there so you can't forget to add it. Again: maintainability at work.


Quote:
Originally Posted by robertkwild
also when i do this command i get a text file of the project folder name and size of each folder but how do i make it do file counts of each directory?

Code:
du -h /to_be_archived/* >> /archive

The same way you cycle through the subdirectories of "/to_be_archived": using the for-loop for tarring, rsyncing and everything else. Like this (i put a header line in for every directory to make it easier to read - it serves no other purpose so feel free to modify the format or drop it alltogether if you don't like it):

Code:
cd /to_be_archived/
for DIR in * ; do
     fSaveDir="${DIR##*/}"

     echo "------------- files from ${fSaveDir}" >> /archive
     du -h /to_be_archived/"${fSaveDir}"/* >> /archive

     tar [....rest of your code....]
done

I hope this helps.

bakunin
Last edited by bakunin; 04-14-2016 at 08:18 AM..
# 7  
Old 04-15-2016
aahhh this programming bash script language is over my head, can you please advise of any good how tos for complete dummys as this is all going over my head

sorry

---------- Post updated at 11:50 AM ---------- Previous update was at 08:08 AM ----------

im messing about with the error code now, i want to see if its possible to email me if a certain command fails

---------- Post updated 04-15-16 at 05:37 AM ---------- Previous update was 04-14-16 at 11:50 AM ----------

Code:
rsync -a "${fSaveDir}".tar /archived_projects/

new code goes here

Code:
rm -f "${fSaveDir}".tar


inbetween these two lines of code i want it to either continue the script if the last command returned a code 0 (aka successful) but if it returns any over number, quit the script and email


i know this is possible but i dont know how to implement it

Moderator's Comments:
Mod Comment edit by bakunin: PLEASE! you have been told every time now that you should use CODE-tags. Could you please stop ignoring this request and actually do it?
Last edited by bakunin; 04-15-2016 at 10:06 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Checking Multiple File existance in a UNIX folder(Note: File names are all different)

HI Guys, I have some 8 files with different name and extensions. I need to check if they are present in a specific folder or not and also want that script to show me which all are not present. I can write if condition for each file but from a developer perspective , i feel that is not a good... (3 Replies)
Discussion started by: shankarpanda003
3 Replies

2. Homework & Coursework Questions

Bash script for printing folder names and their sizes

1. The problem statement, all variables and given/known data: The task is to create a script that would reproduce the output of 'du' command, but in a different way: what 'du' does is: <size> <folder name>and what is needed is <folder name> <size>We need to show only 10 folders which are the... (3 Replies)
Discussion started by: UncleIS
3 Replies

3. Shell Programming and Scripting

Bash script for printing folder names and their sizes

Good day, everyone! I'm very new to bash scripting. Our teacher gave us a task to create a script that basically does the same job the 'du' command does, with the difference that 'du' command gives an output in the form of <size> <folder name>and what we need is <folder name> <size>As for... (1 Reply)
Discussion started by: UncleIS
1 Replies

4. Shell Programming and Scripting

Script to move one folder to multiple folder...

Hi All, I have to requirement to write a shell script to move file from one folder (A) to another five folder (B,C,D,E,F) and destination folder should be blank. In not blank just skip. This script will run as a scheduler every 2 minutes. It will check number of files in folder A and move 1 to... (9 Replies)
Discussion started by: peekuabc
9 Replies

5. UNIX for Dummies Questions & Answers

Do I need to extract the entire tar file to confirm the tar folder is fine?

I would like to confirm my file.tar is been tar-ed correctly before I remove them. But I have very limited disc space to untar it. Can I just do the listing instead of actual extract it? Can I say confirm folder integrity if the listing is sucessful without problem? tar tvf file1.tar ... (1 Reply)
Discussion started by: vivien_chu
1 Replies

6. Shell Programming and Scripting

tar command to explore multiple layers of tar and tar.gz files

Hi all, I have a tar file and inside that tar file is a folder with additional tar.gz files. What I want to do is look inside the first tar file and then find the second tar file I'm looking for, look inside that tar.gz file to find a certain directory. I'm encountering issues by trying to... (1 Reply)
Discussion started by: bashnewbee
1 Replies

7. Shell Programming and Scripting

editing names of files in multiple folder

I have 1000's of directories which is named as numbers. Each directory contains multiple files. Each of these directories have a file named "att". I need to rename all the att files by adding the directory name followed by "_" then att for each of the directories. Directories 120 att... (2 Replies)
Discussion started by: Lucky Ali
2 Replies

8. Shell Programming and Scripting

Script to move files with similar names to folder

I have in directory /media/AUDIO/WAVE many .mp3 files with names like: my filename_01of02.mp3 my filename_02of02.mp3 Your File_01of06.mp3 Your File_02of06.mp3 etc.... In the same directory, /media/AUDIO/WAVE, I have many folders with names like 9780743579490 9780743579491 etc.. Inside... (7 Replies)
Discussion started by: glev2005
7 Replies

9. Shell Programming and Scripting

lvm/tar/rsync backup script feedback/criticism

I have written a shell script to perform backups using tar, rsync and optionally utilise lvm snapshots. The script is not finished but is in a working state and comments/descriptions are poor. I would greatly appreciate any criticism and suggestions of the script to help improve my own learning... (0 Replies)
Discussion started by: jelloir
0 Replies

10. UNIX for Dummies Questions & Answers

Copying multiple folders to local machine (don't know folder names)

Hi. I'm trying to copy multiple folders from the remote machine to the local machine. I wrote a batch file to run an ftp window. The problem I am having is that the only command to copy files is mget *, and this copies only files, not folders. For example, ftp ts555 cd ts555/test ' test... (5 Replies)
Discussion started by: leenyburger
5 Replies
Login or Register to Ask a Question