Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Script to tar/rsync/rm multiple folder names
Top Forums Shell Programming and Scripting Script to tar/rsync/rm multiple folder names Post 302972070 by Don Cragun on Friday 29th of April 2016 02:36:04 PM
Old 04-29-2016
Quote:
Originally Posted by robertkwild
Hi Don

i dont really understand

so the operators, once they complete their projects, will move their project folders in the dir "to_be_archived" and once in there the script will take over/handle it and dump the output tar to dir "archived_projects"

lets say the operators have just started moving the project folder into the dir "to_be_archived" and my script is running and it intercepts the new project folder but the project folder is still growing in size, i dont want it to start of the process as they are still moving the project folder to the dir "to_be_archived"

... ... ...
Hi Robert,
You didn't pay close enough attention to what I suggested in post #40: I said it would be much better if you would create a hidden directory under /to_be_archived (such as /to_be_archived/.in_progress), have your operators move everything into a sub-directory under that directory (such as /to_be_archived/.in_progress/project), and then after that project directory contains everything your operators want to archive, have them issue the command:
Code:
mv /to_be_archived/.in_progress/project /to_be_archived/project

Your existing script WILL NOT SEE /to_be_archived/.in_progress nor any directories under it. (That is why it is called a hidden directory.) After your operators have completed copying files into /to_be_archived/.in_progress/project, then, and only then, as the final step they issue the mv command that will make the completed project directory visible to your existing script.

If you don't believe that this works, run the following simple script to see that hidden directories will not be visible to your script:
Code:
#!/bin/ksh
# Create a file hierarchy with one hidden project and one visible project:
mkdir junk junk/.hidden junk/.hidden/project2 junk/project1
cd junk
touch project0.tar project1/file1 project1/file2
touch .hidden/project2/hfile1 .hidden/project2/hfile2

echo 'File hierarchy with project2 directory hidden:'
ls -laR	# Note that if the -a is left out, ls will not see hidden files either.
echo

# The following is similar to the code you are using, but skips over artifacts
# left over from failed attempts to archive arvhive0.
for DIR in *
do	if [ ! -d "$DIR" ]
	then	printf 'Skipping non-directory file "%s".\n' "$DIR"
		continue
	fi
	printf 'Found file "%s" before moving hidden directory.\n' "$DIR"
	# Perform the rest of your script to archive this project...
done
printf '\nNote that .hidden and project2 should not appear above.\n\n'

# Add some more files to project2 (This could be done in parallel with running
# your current script since these files are still hidden from your script.
touch .hidden/project2/hfile3 .hidden/project2/hfile4

# Now make the hidden project2 visible 
mv .hidden/project2 project2
echo 'File hierarchy with project2 directory moved out from under hidden directory.'
ls -laR
echo
for DIR in *
do	if [ ! -d "$DIR" ]
	then	printf 'Skipping non-directory file "%s".\n' "$DIR"
		continue
	fi
	printf 'Found file "%s" after moving hidden sub-directory.\n' "$DIR"
done
printf '\nNote that project2 should now appear above but .hidden is still not seen.\n'

# Now clean up after ourselves...
cd -
rm -rf junk

which produces output similar to:
Code:
File hierarchy with project2 directory hidden:
total 0
drwxr-xr-x  5 dwc  staff  170 Apr 29 11:18 .
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 ..
drwxr-xr-x  3 dwc  staff  102 Apr 29 11:18 .hidden
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 project0.tar
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 project1

./.hidden:
total 0
drwxr-xr-x  3 dwc  staff  102 Apr 29 11:18 .
drwxr-xr-x  5 dwc  staff  170 Apr 29 11:18 ..
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 project2

./.hidden/project2:
total 0
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 .
drwxr-xr-x  3 dwc  staff  102 Apr 29 11:18 ..
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile1
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile2

./project1:
total 0
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 .
drwxr-xr-x  5 dwc  staff  170 Apr 29 11:18 ..
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 file1
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 file2

Skipping non-directory file "project0.tar".
Found file "project1" before moving hidden directory.

Note that .hidden and project2 should not appear above.

File hierarchy with project2 directory moved out from under hidden directory.
total 0
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 .
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 ..
drwxr-xr-x  2 dwc  staff   68 Apr 29 11:18 .hidden
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 project0.tar
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 project1
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 project2

./.hidden:
total 0
drwxr-xr-x  2 dwc  staff   68 Apr 29 11:18 .
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 ..

./project1:
total 0
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 .
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 ..
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 file1
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 file2

./project2:
total 0
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 .
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 ..
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile1
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile2
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile3
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile4

Skipping non-directory file "project0.tar".
Found file "project1" after moving hidden sub-directory.
Found file "project2" after moving hidden sub-directory.

Note that project2 should now appear above but .hidden is still not seen.
/Users/dwc/test/unix.com/shell/Script_to_tar,_rsync,_rm_multiple_folder_names

This User Gave Thanks to Don Cragun For This Post:
robertkwild
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Copying multiple folders to local machine (don't know folder names)

Hi. I'm trying to copy multiple folders from the remote machine to the local machine. I wrote a batch file to run an ftp window. The problem I am having is that the only command to copy files is mget *, and this copies only files, not folders. For example, ftp ts555 cd ts555/test ' test... (5 Replies)
Discussion started by: leenyburger
5 Replies

2. Shell Programming and Scripting

lvm/tar/rsync backup script feedback/criticism

I have written a shell script to perform backups using tar, rsync and optionally utilise lvm snapshots. The script is not finished but is in a working state and comments/descriptions are poor. I would greatly appreciate any criticism and suggestions of the script to help improve my own learning... (0 Replies)
Discussion started by: jelloir
0 Replies

3. Shell Programming and Scripting

Script to move files with similar names to folder

I have in directory /media/AUDIO/WAVE many .mp3 files with names like: my filename_01of02.mp3 my filename_02of02.mp3 Your File_01of06.mp3 Your File_02of06.mp3 etc.... In the same directory, /media/AUDIO/WAVE, I have many folders with names like 9780743579490 9780743579491 etc.. Inside... (7 Replies)
Discussion started by: glev2005
7 Replies

4. Shell Programming and Scripting

editing names of files in multiple folder

I have 1000's of directories which is named as numbers. Each directory contains multiple files. Each of these directories have a file named "att". I need to rename all the att files by adding the directory name followed by "_" then att for each of the directories. Directories 120 att... (2 Replies)
Discussion started by: Lucky Ali
2 Replies

5. Shell Programming and Scripting

tar command to explore multiple layers of tar and tar.gz files

Hi all, I have a tar file and inside that tar file is a folder with additional tar.gz files. What I want to do is look inside the first tar file and then find the second tar file I'm looking for, look inside that tar.gz file to find a certain directory. I'm encountering issues by trying to... (1 Reply)
Discussion started by: bashnewbee
1 Replies

6. UNIX for Dummies Questions & Answers

Do I need to extract the entire tar file to confirm the tar folder is fine?

I would like to confirm my file.tar is been tar-ed correctly before I remove them. But I have very limited disc space to untar it. Can I just do the listing instead of actual extract it? Can I say confirm folder integrity if the listing is sucessful without problem? tar tvf file1.tar ... (1 Reply)
Discussion started by: vivien_chu
1 Replies

7. Shell Programming and Scripting

Script to move one folder to multiple folder...

Hi All, I have to requirement to write a shell script to move file from one folder (A) to another five folder (B,C,D,E,F) and destination folder should be blank. In not blank just skip. This script will run as a scheduler every 2 minutes. It will check number of files in folder A and move 1 to... (9 Replies)
Discussion started by: peekuabc
9 Replies

8. Shell Programming and Scripting

Bash script for printing folder names and their sizes

Good day, everyone! I'm very new to bash scripting. Our teacher gave us a task to create a script that basically does the same job the 'du' command does, with the difference that 'du' command gives an output in the form of <size> <folder name>and what we need is <folder name> <size>As for... (1 Reply)
Discussion started by: UncleIS
1 Replies

9. Homework & Coursework Questions

Bash script for printing folder names and their sizes

1. The problem statement, all variables and given/known data: The task is to create a script that would reproduce the output of 'du' command, but in a different way: what 'du' does is: <size> <folder name>and what is needed is <folder name> <size>We need to show only 10 folders which are the... (3 Replies)
Discussion started by: UncleIS
3 Replies

10. Shell Programming and Scripting

Checking Multiple File existance in a UNIX folder(Note: File names are all different)

HI Guys, I have some 8 files with different name and extensions. I need to check if they are present in a specific folder or not and also want that script to show me which all are not present. I can write if condition for each file but from a developer perspective , i feel that is not a good... (3 Replies)
Discussion started by: shankarpanda003
3 Replies
All times are GMT -4. The time now is 04:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy