Shell Programming and Scripting

Hi,

I use the below script that will find n record the entire directory structure for a given folder.

Code:

bash-3.2$ more mkdir.sh
find  . -type d |while read rec
do
  echo "mkdir $rec"
  echo "chmod -R 777 $rec"
#done
done >> moht.sh

The problem is the folder i m running this script under where i want it to record all the directory structure has millions of files. This causes the below command to run for 4 hrs now with no results.

Apart from the slowness issue I see that my script above does not record all but only some of the folders structure under the directory it runs

Code:

bash-3.2$ ls -ltrd ServiceV2
drwxrwx---   5 user1 admin      1024 Nov  8  2014 ServiceV2
bash-3.2$ grep -i "ServiceV2" moht.sh
No results Found !!

Can you help fix this script ?

If you want to record a list of the directories in the file hierarchy rooted in your current working directory, the command you want would be:

Code:

find . -type d > directory_list

If you are trying to change the mode of every file (including all of the directories and all of the regular files) in the file hierarch rooted in your current working directory, the command you want would be:

Code:

chmod -R 777 .

(but that would be a TERRIBLE thing to do from a security standpoint).

If you are planning to run the shell script created by you current script (moht.sh), then be aware that all of the mkdir commands will fail (since the directories already exist),
and each chmod -R directory after the first one (chmod -R .) is duplicating work done by the first one (and the 1st one will take a LONG time if there are millions of files and subsequent chmod commands (repeating the work already done by the first one) will waste time proportional to the number of files in the hierarchy rooted in that subdirectory. And if any of your existing directory names contain space, tab, or newline characters; your mkdir commands will create new (presumably unwanted) directories (because you aren't properly quoting the arguments you're passing to mkdir).

The requirement is to create a similar directory only structure on a different server and there are more than 1000 folders and million files.

So i run my script which gives me moht.sh. I then take to moht.sh to the second server and simply run it to create similar directory & sub-directory structure on the other box.

But like i said 1. my script takes unreasonable time to complete and 2. it is missing to record a few directories / sub-directories like i posted in the OP.

mohtashims,
Is this the same system you had problems with November last year (Find command takes too long [too] complete)?
Are there any symbolic links in the hierarchy? These can be found with:

Code:

find . -type l -print

Also, check the manpage for find on your system, the -print was not always enabled by default.

To "record" the directory structure, why not use:

Code:

find . -type d -depth -print | cpio -ov > file

To duplicate the structure use:

Code:

cpio -idvu < file

find . -type l -print does not yields any output meaning i have no soft links.

The below seems to have helped to an extent that i guess i can see more folders that I was not able to in my OP script.

Code:

find . -type d -depth -print |while read rec
do
  echo "mkdir $rec"
  echo "chmod -R 777 $rec"
#done
done >> tryx.sh

However, one problem with this script is that it records the mkdir sub-directory before the mkdir parent-directory in the tryx.sh making it to fail.

So basically it mkdir hello/customer before mkdir hello making it fail.

I did not try the cpio option as i m not familiar with it and not sure if it will really help in the problem mentioned in this post.

I have no means to try that with a really big filesystem but as a wild guess I'd say the reason why this is slow is because of the shell-loop to do the writing to the file.

I suggest you do it this way:

Code:

find . -type d -print | sed  's/.*/mkdir -p &/; $ s/$/; chmod -R 777 */' > script.sh

This writes a "mkdir -p $dir" for every dir found and then a "chmod -R 777 *" as the last command. You didn't say that but i supposed you start the directory recreation process in an otherwise empty directory.

I hope this helps.

bakunin

When piped to a text processor like sed then the file name should be in quotes:

Code:

find . -type d -print | sed "s/.*/mkdir -p '&'/"

Same for the shell processor:

Code:

echo "chmod -R 777 '$rec'"

Two others that might restore faster:

Code:

find . -type d -exec printf "mkdir '%s'\n" {} +

Code:

find . -depth -type d -print | awk -F/ 'NF>=prev {printf "mkdir -p '%s'\n",$0} {prev=NF}'

If you want to copy the permissions, then cpio is the efficient solution, as was proposed by derekludwig.