Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help - Find command to exclude sub-directories

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help - Find command to exclude sub-directories
# 1  
Old 08-13-2010
Help - Find command to exclude sub-directories
Hi Forum.

I'm trying to write a script that finds and deletes files that are older than 300 days. The script will read a table that contains the following 3 columns:

1st col: “Y” means sub-directory scan; "N" means no subdirectory scan
2nd col: sub-directory location
3rd col: File prefix (* indicates all files)

For ex:
Y|/data/informatica/ming/Logs|*
N|/data/informatica/ming/ScriptsLogs|*
N|/data/sw/apps/informatica8/JBoss403/bin/|heapdump*.phd, javacore*.txt

Reason I'm using a table is that the list of directories can grow and I'm trying to avoid hard-coding anything in the script.

I have searched the forum but I haven't been successful in finding any examples on what I am trying to accomplish.

I have tried the following with the find command so far:

-prune option (but this requires hard-coding the sub-directories)
-maxdepth 1 (option does not work in my environment)
ls -l | grep -v ^d (not sure how to use this in conjuction with the find command)

In addition, the 3rd example from my table:
N|/data/sw/apps/informatica8/JBoss403/bin/|heapdump*.phd, javacore*.txt

How can I dynamically parse the 3rd field to scan for these files since there could be more files added to the 3rd field column in the future?

Any help will be greatly appreciated.
# 2  
Old 08-14-2010
Code:
# ./justdoit mytablefile

Code:
 
### justdoit ###
#!/bin/bash
while read -r  i
  do
    if [[ $(echo $i | cut -d"|" -f3 |grep ",") ]] ; then
      ar=( `echo $i|cut -d"|" -f3|sed 's/^/"/;s/,/"/;s/ / "/;s/$/"/'` )
      namex=""
      for nm in ${ar[*]}
       do
        nm=$(echo $nm|sed 's/"//g')
        namex="$nm -o -name $namex"
       done
       namex=$(echo $namex |sed 's/-o -name$//')
    else
     namex=$(echo "$i" | cut -d"|" -f3)
    fi
     if [[ $(echo $i | cut -d"|" -f1) = "N" ]] ; then
         echo ""
         echo "\"Patterns\" -> $namex <-----> \"Path\" -> $(echo $i | cut -d"|" -f2) "
         echo "==========================================================="
          if [ ${#ar[@]} -gt 0 ] ; then
            find $(echo $i | cut -d"|" -f2) -maxdepth 1 \( -name $namex \) -type f  -mtime +300 -exec rm {} \;
            echo ""
          else
            find $(echo $i | cut -d"|" -f2) -maxdepth 1 -name "$namex" -type f  -mtime +300 -exec rm {} \;
            echo ""
          fi
     else
       echo ""
       echo "\"Patterns\" -> $namex <-----> \"Path\" -> $(echo $i | cut -d"|" -f2) "
       echo "==========================================================="
       find $(echo $i | cut -d"|" -f2) -name "$namex" -type f  -mtime +300 -exec rm {} \;
       echo ""
     fi
  done < $1

Last edited by ygemici; 08-14-2010 at 10:10 AM..
# 3  
Old 08-14-2010
Try and adapt the following script (it just print file names) :
Code:
while IFS='|' read subdir dir files filler
do
   echo
   echo "-----------------------------------"
   echo "Entry: <$subdir>|<$dir>|<$files>"
   echo "-----------------------------------"
   echo
   find_names=$(echo "$files" | sed "s/[[:space:]]*//g;s/\([^,]*\)\(,\|\$\)/-name '\1'\2/g;s/,/ -o /g")
   case "$subdir" in
     N|n) dir_name=`basename $dir`
          find_subdir="'(' -type d ! -name '$dir_name' -prune ')' -o" ;;
       *) find_subdir="" ;;
   esac
   eval find "$dir" ${find_subdir} -type f "'(' '('" ${find_names} " ')' -print ')'"
done < pchang.dat

Code:
$ cat pchang.dat
Y|ARCHIVE|*
Y|ARCHIVE|*.txt, *.sh
N|ARCHIVE|*
N|ARCHIVE|*.txt
$ dir -R ARCHIVE
ARCHIVE:
dir1  dir2  grep_with_context.txt  my_test.sh  tree  tree.sed

ARCHIVE/dir1:
readme.txt  subdir_1  subdir_2

ARCHIVE/dir1/subdir_1:
list_1.txt  scan_dir.opt  scan_dir.sh

ARCHIVE/dir1/subdir_2:
compile.sh

ARCHIVE/dir2:
$ sh pchang.sh

-----------------------------------
Entry: <Y>|<ARCHIVE>|<*>
-----------------------------------

ARCHIVE/dir1/readme.txt
ARCHIVE/dir1/subdir_1/list_1.txt
ARCHIVE/dir1/subdir_1/scan_dir.opt
ARCHIVE/dir1/subdir_1/scan_dir.sh
ARCHIVE/dir1/subdir_2/compile.sh
ARCHIVE/grep_with_context.txt
ARCHIVE/my_test.sh
ARCHIVE/tree
ARCHIVE/tree.sed

-----------------------------------
Entry: <Y>|<ARCHIVE>|<*.txt, *.sh>
-----------------------------------

ARCHIVE/dir1/readme.txt
ARCHIVE/dir1/subdir_1/list_1.txt
ARCHIVE/dir1/subdir_1/scan_dir.sh
ARCHIVE/dir1/subdir_2/compile.sh
ARCHIVE/grep_with_context.txt
ARCHIVE/my_test.sh

-----------------------------------
Entry: <N>|<ARCHIVE>|<*>
-----------------------------------

ARCHIVE/grep_with_context.txt
ARCHIVE/my_test.sh
ARCHIVE/tree
ARCHIVE/tree.sed

-----------------------------------
Entry: <N>|<ARCHIVE>|<*.txt>
-----------------------------------

ARCHIVE/grep_with_context.txt
$

For your purpose modify the find command like that :
Code:
eval find "$dir" ${find_subdir} -type f "'(' '('" ${find_names} " ')' -mtime +300 -print ')'" | xargs rm

Jean-Pierre.
Last edited by aigles; 08-15-2010 at 06:27 AM..
# 4  
Old 08-14-2010
I believe that the script offered by aigles will have issues because the OP indicates that the 'sub directory location' is a full path, and the -name option (according to the man page) takes a basename, not a path name.

Instad of the -name option the -path option should be used.

This script needs only one pass over the filesystem and defaults to listing the files. Supply 'rm' as the parameter to the script and the files will be deleted:

Code:
#!/usr/bin/env ksh

# read table and generate find commands
# {Y|N} | filesystem | filelist
awk -F "|" -v cmd="${1:-ls -l}" '
        {
                if( $1 == "Y" || $1 == "y" )            # if no-decend, add prune logic that skips directories
                        no_depth = sprintf( "\\( ! -path \"%s\"  -type d -prune \\) -o", $2 );
                else
                        no_depth = "";     # ok to examine directories -- nothing extra needed

                printf( "find %s %s ", $2, no_depth );   # output first part of cmd

                gsub( ",", " ", $3 );                # allow for pattern,pattern or pattern patern in the list
                n = split( $3, a, " " );            # process each filename pattern
                for( x = 1; x <= n; x++ )
                        printf( "-name \"%s\" ", a[x] );   # add to command

                printf( " -print | xargs %s\n", cmd );   # finish the command
        }
' | ksh           # finally pipe to ksh to execute the commands

# 5  
Old 08-16-2010
Thank you guys for your quick responses and proposed solutions.

I will have a look at each one and try them out.

---------- Post updated at 04:14 PM ---------- Previous update was at 10:37 AM ----------

Hi Guys.

I finally had a chance to test out the 3 proposed solutions and here are some results:

ygemici - maxdepth option does not work in my environment
aigles - script does not work when a full path is provided. My table will contain the full paths of the directories that I would like to cleanup. How can the script be modified to accept full path directories?
agama - How do I go about embedding your code into a script?
Copied entire code just prior to the last pipe into a new script member test.sh - how to execute the script to read my table pchang.dat?

Thank you very much for all your feedback/help.
# 6  
Old 08-16-2010
Quote:
Originally Posted by pchang
agama - How do I go about embedding your code into a script?
Copied entire code just prior to the last pipe into a new script member test.sh - how to execute the script to read my table pchang.dat?

Sorry -- I failed to realise that the testing I was doing was reading the table in from stdin and thus no filename was presented to awk and thus my info wasn't percise.

You can either redirect the table into your script like this:

Code:
test.sh <pchang.dat

or pass the table name on the command line, and use add a $1 following the last single quote of the awk programme.

Code:
awk '
# lines of awk code
' $1

My recommendation would be the first way. If you use the second, then you should add a test in your script and generate an error message if the user forgets to put the table name on the command line.

The output will be the find commands, they won't be executed unless you pipe them into Kshell or bash. You can put the pipe into the script, or pipe the output manually -- I do this sometimes so that I can verify the commands before I execute them.

Code:
# execute the commands generated by test.sh
test.sh <table.dat | ksh

Hope this makes sense.
# 7  
Old 08-17-2010
hi agama.

I tried your suggestion and was able to run the script.

But it seems that I cannot use the -path option with the find command.

This is the message that I'm getting.

"find: 0652-017 -path is not a valid option."

Any other ideas?

Thanks.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need Tar Ball command to exclude directories

Hi, uname -a SunOS mymac 5.11 11.2 sun4u sparc SUNW,SPARC-Enterprise I need to tar a folder /tmp/moht but do not want these three folders to be included in the tar file -> savejpg, bmpsave and imgsave I tried --exclude, -path, -not options but it says bad option Can you help me with... (3 Replies)
Discussion started by: mohtashims
3 Replies

2. Shell Programming and Scripting

Exclude directories in FIND command

Can you please help tweak the below command to exclude all directories with the name "logs" and "tmp" find . -type f \( ! -name "*.tar*" ! -name "*.bkp*" \) -exec /usr/xpg4/bin/grep -i "user_1" /dev/null {} + >result.out bash-3.2$ uname -a SunOS mymac 5.10 Generic_150400-26 sun4v sparc sun4v... (9 Replies)
Discussion started by: mohtashims
9 Replies

3. AIX

Exclude Directories in my tar command

Hi, im having some issues after i execute the next command: tar -cvf /varios/restore/test.tar -X /jfma/test1/excludefile /jfma | gzip -c > /varios/restore/test.tar.gz this creates the desired "test.tar.gz" file, but whe i try to open it it says "tar: 0511-164 There is a media read or write... (6 Replies)
Discussion started by: blacksteel1988
6 Replies

4. UNIX for Dummies Questions & Answers

Using grep command to find the pattern of text in all directories and sub-directories.

Hi all, Using grep command, i want to find the pattern of text in all directories and sub-directories. e.g: if i want to search for a pattern named "parmeter", i used the command grep -i "param" ../* is this correct? (1 Reply)
Discussion started by: vinothrajan55
1 Replies

5. UNIX for Dummies Questions & Answers

Find command to exclude directories and setup alias or script?

Hi, Firstly - sorry for the duplicate my other post looked like i was posting a how to for people. But i am wanting some help :P I want to search from / to find files and exclude my mounted ntfs drives. I have found this thread (Which I can't post the URL to until i have 5 posts) it's... (4 Replies)
Discussion started by: mightymouse2045
4 Replies

6. Shell Programming and Scripting

Perl exclude directories in command line

Hi, I use find command to list all the files in a directory and its sub-directories, but the problem is to exclude certain directories during search. Can i give the directory names in command line to skip them and search rest of the directories? For example i have directories: test ../test1... (1 Reply)
Discussion started by: nmattam
1 Replies

7. UNIX for Dummies Questions & Answers

How to Exclude multiple directories from find command?

Hi, Can some one help me how to exclude multiple directories using find command.. I have the directory structure below. /a/a1/b1 /a/c1/c2 /a/d1/d2/d3 I want to exlcude a1,c2and d3 from the above using find,can some one suggest pls.. thanks in advance... Use code tags... (1 Reply)
Discussion started by: jagadish_gaddam
1 Replies

8. Shell Programming and Scripting

Find but exclude directories

Hello, I have a line in my script to find the files changed in the last 24 hours. It is as below: find /home/hary -type f -mtime -1 I now want to exclude a directory named "/home/hary/temp/cache" from the above find command. How do I add it to my script? Any help is appreciated. ... (9 Replies)
Discussion started by: tadi18
9 Replies

9. UNIX for Dummies Questions & Answers

find command to exclude directories

Howdy I have this directory structure ... eep eepaptest eepfatest eepgltest eep.old eeppoptest ehf ehfaptest ehfgltest ehp ehpgltest I want to find files in these directories, but I want to exclude eep, ehf & ehp. Cany anyone help with the correct command ?? (1 Reply)
Discussion started by: SmurfGGM
1 Replies
Login or Register to Ask a Question