AIX : Find files ignoring certain file extensions

06-22-2015

Registered User

6, 0

Join Date: May 2013

Last Activity: 23 June 2015, 1:09 AM EDT

Posts: 6

Thanks Given: 1

Thanked 0 Times in 0 Posts

AIX : Find files ignoring certain file extensions

Hi All,

I am scripting a program to find and archive files. There are certain file types that I do not want to archive. Below is the scenario.

I have created a lookup file which has details on folders days and file extensions that needs to be ignored

Quote:

/Folder1,30,.txt;.csv
/Folder2,60,.sh,.ksh,.txt
/folder3,90,.param

I have separated the individual into fields and I am having trouble while excluding the file extensions in the find command. I tried to methods to ignore files.

Method 1: This method there are no errors, but the file extensions are still appearing.

Code:

IGNOREFILES="-o name *.txt* -o name *.csv*"
find $DIRNAME/* -prune -type f ! \( -name $IGNOREFILES \) -mtime +$TIME -exec ls -ltr {} \;

Method 2: This method I am using "grep -v" to ignore. It works when executed manually. But in the script is it throwing the following error.

Code:

IGNOREFILES="grep -v .txt | grep -v .csv"
find $DIRNAME/* -prune -type f ! \( -name $IGNOREFILES \) -mtime +$TIME -exec ls -ltr {} \;

Quote:

# echo $IGNORELIST
grep -v .xml | grep -v .sql | grep -v .dat | grep -v .csv
# find $SDIRNAME/* -prune -type f -mtime +$MTIME -exec ls -ltr {} \; | $IGNORELIST
grep: 0652-033 Cannot open |.
grep: 0652-033 Cannot open grep.
grep: 0652-033 Cannot open -v.
grep: 0652-033 Cannot open .sql.
grep: 0652-033 Cannot open |.
grep: 0652-033 Cannot open grep.
grep: 0652-033 Cannot open -v.
grep: 0652-033 Cannot open .dat.
grep: 0652-033 Cannot open |.
grep: 0652-033 Cannot open grep.
grep: 0652-033 Cannot open -v.
grep: 0652-033 Cannot open .csv.

Thanks,
Kev

kavinmjr

View Public Profile for kavinmjr

Find all posts by kavinmjr

06-23-2015

Moderator

3,105, 1,603

Join Date: May 2013

Last Activity: 31 August 2020, 1:46 AM EDT

Location: Chennai

Posts: 3,105

Thanks Given: 1,269

Thanked 1,603 Times in 1,369 Posts

Hello Kev,

Following may help you in same, I haven't tested it but you can take it as a startup and let us know if you have any queries.

Code:

awk -F, '{if(NR>1){printf "\n"};A="find " $1 "/*" " -prune -type f \\( "; {for(i=3;i<=NF;i++){O="! -name *" $i;if(i!=NF){O=O " -o"};P=P?P OFS O:O;print A OFS P;A=P="";};print " \\) -mtime +" $2 " -exec ls -ltr {} \\;"}}' ORS=" " Input_file

Output will be as follows.

Code:

find /Folder1/* -prune -type f \(  ! -name *.txt -o  ! -name *.csv  \) -mtime +30 -exec ls -ltr {} \;
find /Folder2/* -prune -type f \(  ! -name *.sh -o  ! -name *.ksh -o  ! -name *.txt  \) -mtime +60 -exec ls -ltr {} \;
find /folder3/* -prune -type f \(  ! -name *.param   \) -mtime +90 -exec ls -ltr {} \;

If happy with results then you can pass these output to sh as follows. But it is advisable to test it first and if it prints correct expected output then do pass output to sh.

Thanks,
R. Singh

Last edited by RavinderSingh13; 06-23-2015 at 02:29 AM..

This User Gave Thanks to RavinderSingh13 For This Post:

RavinderSingh13

View Public Profile for RavinderSingh13

Find all posts by RavinderSingh13

06-23-2015

Registered User

12,315, 4,560

Join Date: Jul 2012

Last Activity: 22 November 2019, 4:29 PM EST

Location: San Jose, CA, USA

Posts: 12,315

Thanks Given: 952

Thanked 4,560 Times in 3,818 Posts

I haven't tested it either, but the basic principles used by RavinderSingh13 look sound. But the logic is slightly off. If you are trying to exclude files with names ending in .txt and exclude files with names ending in .csv you want an AND of the ! -name clauses; not an OR. And, with all ANDs, the parentheses aren't needed. And, the -name primaries require a single argument, so an argument containing an asterisk has to be quoted. And, it will be much more efficient to pass ls multiple operands rather than invoking ls for each selected file individually. And, finally, note that the sample lookup file uses both commas and a semicolon as field separators (although the semicolon on the 1st line might be a typo). I think the commands you need are more like:

Code:

find "/Folder1"/* -prune -type f ! -name "*.txt" ! -name "*.csv" -mtime +30 -exec ls -ltr {} +
find "/Folder2"/* -prune -type f ! -name "*.sh" ! -name "*.ksh" ! -name "*.txt" -mtime +60 -exec ls -ltr {} +
find "/folder3"/* -prune -type f ! -name "*.param" -mtime +90 -exec ls -ltr {} +

So, try:

Code:

awk -F'[,;]' '
{	printf("find \"%s\"/* -prune -type f ", $1)
	for(i = 3; i <= NF; i++)
		printf("! -name \"*%s\" ", $i)
	printf("-mtime +%s -exec ls -ltr {} +\n", $2}
}' Input_file

This User Gave Thanks to Don Cragun For This Post:

Don Cragun

View Public Profile for Don Cragun

Find all posts by Don Cragun

06-23-2015

Registered User

6,384, 2,214

Join Date: May 2005

Last Activity: 28 October 2019, 4:59 PM EDT

Location: In the leftmost byte of /dev/kmem

Posts: 6,384

Thanks Given: 143

Thanked 2,214 Times in 1,548 Posts

Quote:

Originally Posted by kavinmjr

Method 2: This method I am using "grep -v" to ignore. It works when executed manually. But in the script is it throwing the following error.

Code:

IGNOREFILES="grep -v .txt | grep -v .csv"
find $DIRNAME/* -prune -type f ! \( -name $IGNOREFILES \) -mtime +$TIME -exec ls -ltr {} \;

# find $SDIRNAME/* -prune -type f -mtime +$MTIME -exec ls -ltr {} \; | $IGNORELIST
grep: 0652-033 Cannot open |.
grep: 0652-033 Cannot open grep.
grep: 0652-033 Cannot open -v.
grep: 0652-033 Cannot open .sql.
grep: 0652-033 Cannot open |.
grep: 0652-033 Cannot open grep.
grep: 0652-033 Cannot open -v.
grep: 0652-033 Cannot open .dat.
grep: 0652-033 Cannot open |.
grep: 0652-033 Cannot open grep.
grep: 0652-033 Cannot open -v.
grep: 0652-033 Cannot open .csv.

You already get very sound advice but you might probably want to know why this doesn't work in the form you wrote it:

The shell evaluates a commandline in certain steps. All the evaluation in a certain step is done at once. For instance:

Code:

A="foo"
B="bar"

# command1 $A | command2 $B

"$A" and "$B" are expanded to their respective values at the same time. This is why the following:

Code:

A='$B'
B="bar"

# command1 $A

will provide command1 not with "bar" but with a literal "$B". When the step "evaluate variables" is done and "$A" is replaced by its value "$B" there will be no further evaluation of "$B" to "bar".

Furthermore, when the shell evaluates a variable it surrounds it by so-called "IFS" (internal field separator) characters. This IFS character is per default a blank but for the next evaluation of a command the shell is aware that there is a difference between a simple blank inside a variables value and an IFS character separating two arguments. here is another example for exactly the same problem, this time created by the guys who develop the operating system.

If you need a solution to such problems (you won't need it here, because you got better procedures already) there is: the eval command. I said that the shell is aware of the difference between a simple space and the IFS character, but only so for ONE evaluation. Fortunately you can (re-)start the command evaluation with a certain keyword, which is "eval". Just prepend your command with it and the shell will evaluate your command line, then evaluate the result again, and only then execute it. This way you could do:

Code:

A='$B'
B="bar"

# eval command1 $A

and have "bar" as argument to command1. In the first evaluation "$A" will be replaced by "$B" and in the second pass "$B" will be replaced by "bar". This will also work for your construct:

Code:

IGNOREFILES="grep -v .txt | grep -v .csv"
# eval find $SDIRNAME/* -prune -type f -mtime +$MTIME -exec ls -ltr {} \; | $IGNOREFILES

I hope this helps.

bakunin

bakunin

View Public Profile for bakunin

Find all posts by bakunin

06-24-2015

Registered User

5,091, 1,931

Join Date: May 2012

Last Activity: 15 July 2020, 4:46 AM EDT

Location: Simplicity

Posts: 5,091

Thanks Given: 565

Thanked 1,931 Times in 1,668 Posts

Method 1 as a shell script. Does it print the desired files?

Code:

#!/bin/sh
IFS=" ,;"
while read dir days exts
do
  omit=""
  set -- $exts
  for ext do
    omit="$omit ! -name *$ext"
  done
  find $dir/* -prune -type f -mtime +$days $omit -print
done < lookupfile

Don, you had ( } in

Code:

  printf("-mtime +%s -exec ls -ltr {} +\n", $2)

Last edited by MadeInGermany; 06-24-2015 at 03:44 PM..

MadeInGermany

View Public Profile for MadeInGermany

Find all posts by MadeInGermany

AIX

AIX : Find files ignoring certain file extensions

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Find wild card directory and its files of some extensions

Discussion started by: sssccc

2. Shell Programming and Scripting

Add file extensions to files EXCEPT the script that is issuing

Discussion started by: jeffs42885

3. AIX

Loading AIX kernel extensions on reboot

Discussion started by: jbleistein

4. Shell Programming and Scripting

Find duplicate files but with different extensions

Discussion started by: fretagi

5. Shell Programming and Scripting

adding file extensions to split output files

Discussion started by: ljk

6. Shell Programming and Scripting

Deleting all files recursively from directories while ignoring one file type

Discussion started by: pasc

7. Shell Programming and Scripting

Ignoring certain extensions

Discussion started by: anushree.a

8. UNIX for Dummies Questions & Answers

Find all the unique file extensions

Discussion started by: reldb

9. Shell Programming and Scripting

Find files ignoring subdirectories

Discussion started by: osramos

10. Shell Programming and Scripting

Find files with 3 different extensions

Discussion started by: mahalakshmi