Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Chaining together exec within find

Shell Programming and Scripting


Reply    
 
Thread Tools Search this Thread Display Modes
    #8  
Old Unix and Linux 06-16-2017
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 18 August 2017, 6:48 PM EDT
Location: Saskatchewan
Posts: 22,336
Thanks: 1,116
Thanked 4,216 Times in 3,899 Posts
I take it this script would run automatically at intervals. Given that, I think you could make your approach work. I'd break the archiving and deleting into two steps, so you can bail in case of error before files are trashed, rather than after.

Also, Once a tarball is created and compressed, it's essentially uneditable, so you have to compress it after you're finished appending to it, not during.




Code:
# Logfile for errors, >&2 and any errors printed by tar/gzip/etc
exec 2> /path/to/errorlog
# Logfile for files, captures default stdout
exec 1> /path/to/filelog

TSTAMP=$(date +%Y-%m-%d)
TARBALL=/path/to/folder/$TSTAMP-one.tar

echo "$(date '+%Y-%m-%d %H:%M:%S') $0 Beginning execution" >&2

echo "# Archiving to $TARBALL"

if [ -e "$TARBALL" ] || [ -e "$TARBALL".gz ]
then
        echo "$(date '+%Y-%m-%d %H:%M:%S') $TARBALL already exists, refusing to overwrite" >&2
        exit 1
fi

tar -rf "$TARBALL" # Create empty tar file to append to

if ! find one -type f -exec echo ls -latr '{}' '+' -exec echo tar -rvf /absolute/path/to/archive.tar '{}' '+'
then
        echo "$(date '+%Y-%m-%d %H:%M:%S') Creating archive failed" >&2
        rm -f "$TARBALL"
        exit 1
fi

if ! gzip "$TARBALL"
then
        echo "$(date '+%Y-%m-%d %H:%M:%S') Couldn't compress $TARBALL" >&2
        exit 1
fi

if ! find one -type f -exec echo rm '{}' '+'
then
        echo "$(date '+%Y-%m-%d %H:%M:%S') Error removing files" >&2
        exit 1
fi

echo "$(date '+%Y-%m-%d %H:%M:%S') $0 completed successfully"


Last edited by Corona688; 06-16-2017 at 04:38 PM..
Sponsored Links
    #9  
Old Unix and Linux 06-16-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 20 August 2017, 1:07 AM EDT
Location: San Jose, CA, USA
Posts: 10,464
Thanks: 533
Thanked 3,660 Times in 3,123 Posts
Quote:
Originally Posted by Corona688 View Post
Since tar may be run multiple times, you need to use the append option, not the create option.

How about:


Code:
$ tar -rf archive.tar # Create empty tar file to append to

$ find testout -type f -exec echo ls -latrd '{}' ';' -exec echo tar -rvf archive.tar '{}' ';' -exec echo rm '{}' ';'

ls -latrd testout/testfile1
tar -rvf archive.tar testout/testfile1
rm testout/testfile1
ls -latrd testout/testfile2
tar -rvf archive.tar testout/testfile2
rm testout/testfile2
ls -latrd testout/testfile3
tar -rvf archive.tar testout/testfile3
rm testout/testfile3

# remove echos to actually run these commands instead of printing them

If you have GNU find, you can use + instead of ; for increased efficiency as it will bundle several files into each call:


Code:
$ find testout -type f -exec echo ls -latrd '{}' '+' -exec echo tar -rvf /absolute/path/to/archive.tar '{}' '+' -exec echo rm '{}' '+'

ls -latrd testout/testfile1 testout/testfile2 testout/testfile3
tar -rvf /absolute/path/to/archive.tar testout/testfile1 testout/testfile2 testout/testfile3
rm testout/testfile1 testout/testfile2 testout/testfile3

$

No. Do not use -exec ... + in cases like this. If there are enough files to trigger an invocation of one of these -exec primaries before the find has processed the entire file hierarchy, the list of files processed by each -exec primary is likely to have a different set of operands that the other -exec primaries. For example, the 1st invocation of ls might process 100 files, the 1st invocation of tar might process 95 files, and the 1st invocation ofrm might process 105 files. The 2nd invocations of ls and tar will then fail because the 1st invocation of rm will have removed some of the files before they were listed and archived.

If there aren't enough files in the file hierarchy being processed by find to trigger invocations of of those tree utilities until the entire file hierarchy has been traversed, all three utilities could be run in parallel again allowing rm to remove some or all of the files before they are listed and archived.
Sponsored Links
    #10  
Old Unix and Linux 06-21-2017
jeffs42885 jeffs42885 is offline
Registered User
 
Join Date: Nov 2010
Last Activity: 27 July 2017, 10:47 AM EDT
Location: PA
Posts: 95
Thanks: 14
Thanked 0 Times in 0 Posts
I would like to chain everything together using one find command. Here is what I am attempting, and my output.


Code:
 
 find /directory/toscan -type f -exec ls -latr '{}' '+' -exec tar -rvf /directory/foroutput/archive.tar '{}' '+' -exec rm '{}' '+'

I am getting this -


Code:
 
 ls: illegal option -- v
usage: ls [-1ACFHLNRSabcdefgiklmnopqrstuxEUX] [File...]

I am on aix -

Code:
 
  
 6100-09-06-1543

    #11  
Old Unix and Linux 06-21-2017
rdrtx1 rdrtx1 is offline
Registered User
 
Join Date: Sep 2012
Last Activity: 20 August 2017, 3:23 PM EDT
Location: Houston, Texas, USA
Posts: 950
Thanks: 0
Thanked 322 Times in 306 Posts

Code:
find /directory/toscan -type f -exec ls -latr "{}" \; -exec tar -rvf /directory/foroutput/archive.tar "{}" \; -exec rm "{}" \;

Sponsored Links
    #12  
Old Unix and Linux 06-21-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 20 August 2017, 1:07 AM EDT
Location: San Jose, CA, USA
Posts: 10,464
Thanks: 533
Thanked 3,660 Times in 3,123 Posts
Quote:
Originally Posted by jeffs42885 View Post
I would like to chain everything together using one find command. Here is what I am attempting, and my output.


Code:
 
 find /directory/toscan -type f -exec ls -latr '{}' '+' -exec tar -rvf /directory/foroutput/archive.tar '{}' '+' -exec rm '{}' '+'

I am getting this -


Code:
 
 ls: illegal option -- v
usage: ls [-1ACFHLNRSabcdefgiklmnopqrstuxEUX] [File...]

I am on aix -

Code:
 
  
 6100-09-06-1543

I see that you have chosen to ignore the problems I mentioned in post #9 in this thread. You do so at your own peril!

From the error you have shown us, we might guess that one or more of the files you are processing has a hyphen as the first character of the pathname that is being passed to ls by find. But, that can't be the case with the command line you have shown us since every pathname that find would pass to ls would have to start with /directory/toscan and the options you have find passing to ls do not include -v.

Are you absolutely positive that the diagnostic you have shown us from ls came from one of the invocations of ls in the find command above?
Sponsored Links
    #13  
Old Unix and Linux 06-22-2017
apmcd47 apmcd47 is online now
Registered User
 
Join Date: Feb 2011
Last Activity: 21 August 2017, 4:23 AM EDT
Posts: 204
Thanks: 5
Thanked 47 Times in 44 Posts
Quote:
Originally Posted by jeffs42885 View Post
I would like to chain everything together using one find command.
Then use one find command. But modify Corona's code as follows:

Code:
# Logfile for errors, >&2 and any errors printed by tar/gzip/etc
exec 2> /path/to/errorlog
# Logfile for files, captures default stdout
exec 1> /path/to/filelog

TSTAMP=$(date +%Y-%m-%d)
TARBALL=/path/to/folder/$TSTAMP-one.tar

echo "$(date '+%Y-%m-%d %H:%M:%S') $0 Beginning execution" >&2

echo "# Archiving to $TARBALL"

if [ -e "$TARBALL" ] || [ -e "$TARBALL".gz ]
then
        echo "$(date '+%Y-%m-%d %H:%M:%S') $TARBALL already exists, refusing to overwrite" >&2
        exit 1
fi

tar -rf "$TARBALL" # Create empty tar file to append to

echo ls -ltr "$@"

if ! echo tar -rvf /absolute/path/to/archive.tar "$@"
then
        echo "$(date '+%Y-%m-%d %H:%M:%S') Creating archive failed" >&2
        rm -f "$TARBALL"
        exit 1
fi

if ! gzip "$TARBALL"
then
        echo "$(date '+%Y-%m-%d %H:%M:%S') Couldn't compress $TARBALL" >&2
        exit 1
fi

if ! echo rm "$@"
then
        echo "$(date '+%Y-%m-%d %H:%M:%S') Error removing files" >&2
        exit 1
fi

echo "$(date '+%Y-%m-%d %H:%M:%S') $0 completed successfully"

amd call it something like archive_and_delete. If AIX has xargs use:

Code:
find .... -print | xargs archive_and_delete

Even better if you can use

Code:
find .... -print0 | xargs -0 archive_and_delete

If AIX does not have xargs use:

Code:
find .... -exec archive_and_delete {} '+'

CAVEAT: I haven't tried the above script and I may even have introduced bugs into it with my edit. I am also assuming that you will continue to use the -type f directive to pass filenames rather than directory names.

Andrew

Last edited by apmcd47; 06-22-2017 at 08:59 AM.. Reason: deleted as I think I've misunderstood something
Sponsored Links
    #14  
Old Unix and Linux 06-22-2017
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 18 August 2017, 6:48 PM EDT
Location: Saskatchewan
Posts: 22,336
Thanks: 1,116
Thanked 4,216 Times in 3,899 Posts
Quote:
Originally Posted by Don Cragun View Post
No. Do not use -exec ... + in cases like this. If there are enough files to trigger an invocation of one of these -exec primaries before the find has processed the entire file hierarchy, the list of files processed by each -exec primary is likely to have a different set of operands that the other -exec primaries.
And this is a problem why?

Quote:
For example, the 1st invocation of ls might process 100 files, the 1st invocation of tar might process 95 files, and the 1st invocation ofrm might process 105 files.
Doesn't seem to work that way, and I can't imagine why it would. Why wouldn't all three execs get the exact same files?
Quote:
If there aren't enough files in the file hierarchy being processed by find to trigger invocations of of those tree utilities until the entire file hierarchy has been traversed, all three utilities could be run in parallel again allowing rm to remove some or all of the files before they are listed and archived.
Does this actually happen? find doesn't run things in parallel to my understanding.
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
find: missing argument to `-exec' while redirecting using find in perl ramkumarselvam Shell Programming and Scripting 2 12-16-2011 06:48 PM
Find and exec disconnectus Ubuntu 1 10-12-2011 07:29 AM
Find Exec prasanna1157 UNIX for Dummies Questions & Answers 1 09-27-2010 07:33 AM
Using MV FIND and -EXEC degraff63 Shell Programming and Scripting 0 05-05-2009 03:54 AM
find and exec big123456 UNIX for Advanced & Expert Users 2 01-02-2006 11:10 AM



All times are GMT -4. The time now is 04:24 AM.