Moving files from parent path to multiple child path using bash in efficient way


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Moving files from parent path to multiple child path using bash in efficient way
# 1  
Old 05-03-2015
Moving files from parent path to multiple child path using bash in efficient way

Hi All,

Can you please provide some pointers to move files from Base path to multiple paths in efficient way.Folder Structure is already created.

Code:
/Path/AdminUser/User1/1111/Reports/aaa.txt to /Path/User1/1111/Reports/aaa.txt
/Path/AdminUser/User1/2222/Reports/bbb.txt to /Path/User1/2222/Reports/bbb.txt
/Path/AdminUser/User2/3333/Reports/ccc.txt to /Path/User2/3333/Reports/ccc.txt
/Path/AdminUser/User2/4444/Reports/ddd.txt to /Path/User2/4444/Reports/ddd.txt

I don't want to move directory structure. I want to move files from that source path to destination path on daily basis. I would need pointer to write a bash script to make sure aaa.txt would exactly go and sit only from /Path/AdminUser/User1/1111/Reports/ to /Path/User1/1111/Reports/........


One of geek folk helped provided with inputs to start as below, It still threw error, can we also write it in an efficient way?
Code:
#!/bin/bash
dir1=/Path
for i in "$dir1"/AdminUser/*; do
if [[ -d $i && ! -L $i ]]; then 
dir2="${i##*/}"
for j in "$i"/*; do 
if [[ -d $j && ! -L $j ]]; then 
j="${j##*/}"
mv "$i"/"$j"/Reports "$dir1"/"$dir2"/"$j"/
fi
done
fi
done


Last edited by Don Cragun; 05-03-2015 at 08:10 PM.. Reason: Add CODE and ICODE tags.
# 2  
Old 05-03-2015
Your requirement isn't clear.
  1. Does the file hierarchy you're modifying contain symbolic links matched by the patterns /Path/AdminUser/* and /Path/AdminUser/*/*? Your textual description didn't say anything about symbolic links, but your (non-working) code explicitly ignores them. If symbolic links are found, what do want to have done with them?
  2. Are you trying to move all files in the file hierarchy rooted in /Path/AdminUser/ up one level in that hierarchy to /Path/; or are you just trying to move regular files with pathnames matching the pattern /Path/AdminUser/*/*/Reports/*.txt to the corresponding pathname after removing AdminUser/ from the source pathname?
  3. If a directory in the target pathname does not already exist, should your script report an error and move on, should it create the missing directories, or should it silently ignore that source pathname?
  4. Do you want to leave the (empty) source directories in place after files are moved out of them, or should emptied directories be removed?
  5. Is the entire file hierarchy rooted in /Path in a single filesystem?
Instead of saying "It still threw error, can we also write it in an efficient way?", show us exactly what "error(s) it produces" (in CODE tags).

What do you believe it is doing inefficiently?
# 3  
Old 05-03-2015
Hi Cragun,

Please see answers for questions, which helps provide complete picture:


1. We dont have symbolic links in path, so we can ignore it.
2. I am trying to move regular files with pathnames matching the pattern /Path/AdminUser/*/*/Reports/*.txt to the corresponding pathname after removing AdminUser/ from the source pathname.

3. It can ignore that source pathname.

4. Empty source directories should be in place after files are moved out of them.

5. Yes the entire file hierarchy rooted in /Path in a single filesystem.

To improve performance can any thing like xargs play a major role
# 4  
Old 05-04-2015
You man have some more success using rsync perhaps like this:

Code:
$ rsync -a --remove-source-files /Path/AdminUser/ /Path

# 5  
Old 05-04-2015
@ Chubler
Thank you. rsync works, but since in my scenario, its not just folder structure, I have to make sure security & ownership of file system isn't affected so I was inclined to use script to move files in loop from parent path to child path, with out touching directories.
# 6  
Old 05-04-2015
You have said that a "geek" gave you a non-working script that uses bash, but you haven't said what shell(s) are available for you to use nor what operating system(s) are present on systems that will be running this script.

The following script will work with any POSIX-conforming shell on any system that has a shell that conforms to the 1992 or any later version of the POSIX Shell and Utilities standard (and many shells that support a fairly common subset of POSIX requirements) such as any version of ksh or bash:
Code:
#!/bin/ksh
# This script works with any POSIX conforming shell.
dir1="/Path"
cd "$dir1"
for srcd in AdminUser/*/*/Reports
do	targetd=${srcd#*/}
	if [ ! -d "$srcd" ] || [ ! -d "$targetd" ]
	then	continue	# Source or target directory does not exist.
	fi
	cd "$srcd" > /dev/null
	for file in *.txt
	do	if [ "$file" = '*.txt' ]
		then	break	# No *.txt files to move in this directory
		else	echo mv *.txt "$dir1/$targetd"	# Move the files
			break
		fi
	done
	cd - > /dev/null
done

If you have a recent ksh that expands ~(N)*.txt to an empty list if there are no files in the current directory ending with .txt or if you have a recent bash that expands [ICODE]*.txt[/CODE] to an empty list after the command shopt -s nullglob, then the 2nd for loop in the above script can be simplified as in:
Code:
#!/bin/ksh
# This script works with a recent Korn shell that recognizes: ~(N)*.txt
# to expand to nothing if there are no files in the current directory ending
# with ".txt"
dir1="/Path"
cd "$dir1"
for srcd in AdminUser/*/*/Reports
do	targetd=${srcd#*/}
	if [ ! -d "$srcd" ] || [ ! -d "$targetd" ]
	then	continue	# Source or target directory does not exist.
	fi
	cd "$srcd" > /dev/null
	for file in ~(N)*.txt
	do	echo mv *.txt "$dir1/$targetd"	# Move the files
		break
	done
	cd - > /dev/null
done

or:
Code:
#!/bin/bash
# This script works with a recent bash shell that recognizes: shopt -s nullglob
# followed by: *.txt
# to expand to nothing if there are no files in the current directory ending
# with ".txt"
shopt -s nullglob
dir1="/Path"
cd "$dir1"
for srcd in AdminUser/*/*/Reports
do	targetd=${srcd#*/}
	if [ ! -d "$srcd" ] || [ ! -d "$targetd" ]
	then	continue	# Source or target directory does not exist.
	fi
	cd "$srcd" > /dev/null
	for file in *.txt
	do	echo mv *.txt "$dir1/$targetd"	# Move the files
		break
	done
	cd - > /dev/null
done

If the mv commands echoed by the above scripts do what you want, remove the echo shown in red to actually execute the mv commands.

All of these scripts assume that (since they move into the various source directories before expanding the list of files to be moved) that list won't exceed ARG_MAX limits (so xargs or similar utilities to combine groups of files into single invocations of mv) won't be needed. They all produce one mv command for each source directory that contains files to be moved that also have an existing target directory to receive those files. If some directories have a huge number of files to be moved with names long enough to exceed ARG_MAX, the 2nd for loop can be modified to feed the list of files to be moved to xargs, but it will run a little bit slower (and depending on what operating system you're using), may have problems with filenames containing whitespace characters and may also require special processing to specify the target directory in the appropriate place in an xargs-generated mv command.
This User Gave Thanks to Don Cragun For This Post:
# 7  
Old 05-05-2015
Thanks a lot Don for detail explanation and scripts for each scenario. It really helps !!!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash to trim folder and files within a path that share a common file extension

The bash will trim the folder to trim folder. Within each of the folders (there may be more than 1) and the format is always the same, are several .bam and matching .bam.bai files (file structure) and the bashunder that executes and trims the .bam as expected but repeats the.bam.bai extentions... (9 Replies)
Discussion started by: cmccabe
9 Replies

2. UNIX for Advanced & Expert Users

Command to see the logical volume path, device mapper path and its corresponding dm device path

Currently I am using this laborious command lvdisplay | awk '/LV Path/ {p=$3} /LV Name/ {n=$3} /VG Name/ {v=$3} /Block device/ {d=$3; sub(".*:", "/dev/dm-", d); printf "%s\t%s\t%s\n", p, "/dev/mapper/"v"-"n, d}' Would like to know if there is any shorter method to get this mapping of... (2 Replies)
Discussion started by: royalibrahim
2 Replies

3. Shell Programming and Scripting

How to list all Subdirectories and files with its full path in a parent directory?

How to list all Subdirectories and files with its full path in a parent directory? (1 Reply)
Discussion started by: johnveslin
1 Replies

4. Shell Programming and Scripting

forking a child process and kill its parent to show that child process has init() as its parent

Hi everyone i am very new to linux , working on bash shell. I am trying to solve the given problem 1. Create a process and then create children using fork 2. Check the Status of the application for successful running. 3. Kill all the process(threads) except parent and first child... (2 Replies)
Discussion started by: vizz_k
2 Replies

5. Homework & Coursework Questions

Problem with path and child shells

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: a) Some Unix tools are at $HOME/mytools directory. Make these tools accessible for use from any directory. b)... (2 Replies)
Discussion started by: justOne21
2 Replies

6. Emergency UNIX and Linux Support

Find, replace, file path in multiple files for Solaris 10

Guys I have a big issue that I need to get fixed ASAP however I can not seem to find a way to do it. We started to use zones with Solaris 10 at work and we moved a zone from a SIT box to a DEV box. Problem is the software we have installed is looking at a /lcl/sit/apps/ path and it needs to look... (5 Replies)
Discussion started by: LRoberts
5 Replies

7. Shell Programming and Scripting

moving multiple folders/files in subversion using bash script

Hi, I'm new here an dlearning a lot from this forum. i didnt find any solution for this in the forum. I have already checked in folders in subversion named HTT01,... HTT21.. and have files in each folder like below: HTT01/HTT01_00000.hex HTT01/HTT01_00000_fb_result.hex... (2 Replies)
Discussion started by: ravishan21
2 Replies

8. Shell Programming and Scripting

multiple child scripts running in backgroud, how to use grep on the parent?

Hi I have a shell script A which calls another 10 shell scripts which run in background. How do i make the parent script wait for the child scripts complete, or in other words, i must be able to do a grep of parent script to find out if the child scripts are still running. My Code: ... (5 Replies)
Discussion started by: albertashish
5 Replies

9. Shell Programming and Scripting

full path of a file situated either in parent's dir. or parent's parent dir. so on...

hi experts(novice people can stay away as it is no child's game), i am developing a script which works like recycle bin of windows. the problem i am facing is that when ever i am trying to delete a file which is situated in parent directory or parent's parent directory i am unable to... (1 Reply)
Discussion started by: yahoo!
1 Replies

10. UNIX for Dummies Questions & Answers

Moving files by splitting the path embedded in the filename

Hello All. I am having a directory /tmp/rahul which contains many files in the format @#home@#rahul@#programs@#script.pl where /home/rahul/programs is the directory where the script.pl file is to be placed. I have many files in this format. What i want is a script which read these... (7 Replies)
Discussion started by: rahulrathod
7 Replies
Login or Register to Ask a Question