I take it this script would run automatically at intervals. Given that, I think you could make your approach work. I'd break the archiving and deleting into two steps, so you can bail in case of error before files are trashed, rather than after.
Also, Once a tarball is created and compressed, it's essentially uneditable, so you have to compress it after you're finished appending to it, not during.
Code:
# Logfile for errors, >&2 and any errors printed by tar/gzip/etc
exec 2> /path/to/errorlog
# Logfile for files, captures default stdout
exec 1> /path/to/filelog
TSTAMP=$(date +%Y-%m-%d)
TARBALL=/path/to/folder/$TSTAMP-one.tar
echo "$(date '+%Y-%m-%d %H:%M:%S') $0 Beginning execution" >&2
echo "# Archiving to $TARBALL"
if [ -e "$TARBALL" ] || [ -e "$TARBALL".gz ]
then
echo "$(date '+%Y-%m-%d %H:%M:%S') $TARBALL already exists, refusing to overwrite" >&2
exit 1
fi
tar -rf "$TARBALL" # Create empty tar file to append to
if ! find one -type f -exec echo ls -latr '{}' '+' -exec echo tar -rvf /absolute/path/to/archive.tar '{}' '+'
then
echo "$(date '+%Y-%m-%d %H:%M:%S') Creating archive failed" >&2
rm -f "$TARBALL"
exit 1
fi
if ! gzip "$TARBALL"
then
echo "$(date '+%Y-%m-%d %H:%M:%S') Couldn't compress $TARBALL" >&2
exit 1
fi
if ! find one -type f -exec echo rm '{}' '+'
then
echo "$(date '+%Y-%m-%d %H:%M:%S') Error removing files" >&2
exit 1
fi
echo "$(date '+%Y-%m-%d %H:%M:%S') $0 completed successfully"
Last edited by Corona688; 06-16-2017 at 05:38 PM..
Since tar may be run multiple times, you need to use the append option, not the create option.
How about:
Code:
$ tar -rf archive.tar # Create empty tar file to append to
$ find testout -type f -exec echo ls -latrd '{}' ';' -exec echo tar -rvf archive.tar '{}' ';' -exec echo rm '{}' ';'
ls -latrd testout/testfile1
tar -rvf archive.tar testout/testfile1
rm testout/testfile1
ls -latrd testout/testfile2
tar -rvf archive.tar testout/testfile2
rm testout/testfile2
ls -latrd testout/testfile3
tar -rvf archive.tar testout/testfile3
rm testout/testfile3
# remove echos to actually run these commands instead of printing them
If you have GNU find, you can use + instead of ; for increased efficiency as it will bundle several files into each call:
Code:
$ find testout -type f -exec echo ls -latrd '{}' '+' -exec echo tar -rvf /absolute/path/to/archive.tar '{}' '+' -exec echo rm '{}' '+'
ls -latrd testout/testfile1 testout/testfile2 testout/testfile3
tar -rvf /absolute/path/to/archive.tar testout/testfile1 testout/testfile2 testout/testfile3
rm testout/testfile1 testout/testfile2 testout/testfile3
$
No. Do not use -exec ... + in cases like this. If there are enough files to trigger an invocation of one of these -exec primaries before the find has processed the entire file hierarchy, the list of files processed by each -exec primary is likely to have a different set of operands that the other -exec primaries. For example, the 1st invocation of ls might process 100 files, the 1st invocation of tar might process 95 files, and the 1st invocation ofrm might process 105 files. The 2nd invocations of ls and tar will then fail because the 1st invocation of rm will have removed some of the files before they were listed and archived.
If there aren't enough files in the file hierarchy being processed by find to trigger invocations of of those tree utilities until the entire file hierarchy has been traversed, all three utilities could be run in parallel again allowing rm to remove some or all of the files before they are listed and archived.
I would like to chain everything together using one find command. Here is what I am attempting, and my output.
Code:
find /directory/toscan -type f -exec ls -latr '{}' '+' -exec tar -rvf /directory/foroutput/archive.tar '{}' '+' -exec rm '{}' '+'
I am getting this -
Code:
ls: illegal option -- v
usage: ls [-1ACFHLNRSabcdefgiklmnopqrstuxEUX] [File...]
I am on aix -
Code:
6100-09-06-1543
I see that you have chosen to ignore the problems I mentioned in post #9 in this thread. You do so at your own peril!
From the error you have shown us, we might guess that one or more of the files you are processing has a hyphen as the first character of the pathname that is being passed to ls by find. But, that can't be the case with the command line you have shown us since every pathname that find would pass to ls would have to start with /directory/toscan and the options you have find passing to ls do not include -v.
Are you absolutely positive that the diagnostic you have shown us from ls came from one of the invocations of ls in the find command above?
I would like to chain everything together using one find command.
Then use one find command. But modify Corona's code as follows:
Code:
# Logfile for errors, >&2 and any errors printed by tar/gzip/etc
exec 2> /path/to/errorlog
# Logfile for files, captures default stdout
exec 1> /path/to/filelog
TSTAMP=$(date +%Y-%m-%d)
TARBALL=/path/to/folder/$TSTAMP-one.tar
echo "$(date '+%Y-%m-%d %H:%M:%S') $0 Beginning execution" >&2
echo "# Archiving to $TARBALL"
if [ -e "$TARBALL" ] || [ -e "$TARBALL".gz ]
then
echo "$(date '+%Y-%m-%d %H:%M:%S') $TARBALL already exists, refusing to overwrite" >&2
exit 1
fi
tar -rf "$TARBALL" # Create empty tar file to append to
echo ls -ltr "$@"
if ! echo tar -rvf /absolute/path/to/archive.tar "$@"
then
echo "$(date '+%Y-%m-%d %H:%M:%S') Creating archive failed" >&2
rm -f "$TARBALL"
exit 1
fi
if ! gzip "$TARBALL"
then
echo "$(date '+%Y-%m-%d %H:%M:%S') Couldn't compress $TARBALL" >&2
exit 1
fi
if ! echo rm "$@"
then
echo "$(date '+%Y-%m-%d %H:%M:%S') Error removing files" >&2
exit 1
fi
echo "$(date '+%Y-%m-%d %H:%M:%S') $0 completed successfully"
amd call it something like archive_and_delete. If AIX has xargs use:
Code:
find .... -print | xargs archive_and_delete
Even better if you can use
Code:
find .... -print0 | xargs -0 archive_and_delete
If AIX does not have xargs use:
Code:
find .... -exec archive_and_delete {} '+'
CAVEAT: I haven't tried the above script and I may even have introduced bugs into it with my edit. I am also assuming that you will continue to use the -type f directive to pass filenames rather than directory names.
Andrew
Last edited by apmcd47; 06-22-2017 at 09:59 AM..
Reason: deleted as I think I've misunderstood something
No. Do not use -exec ... + in cases like this. If there are enough files to trigger an invocation of one of these -exec primaries before the find has processed the entire file hierarchy, the list of files processed by each -exec primary is likely to have a different set of operands that the other -exec primaries.
And this is a problem why?
Quote:
For example, the 1st invocation of ls might process 100 files, the 1st invocation of tar might process 95 files, and the 1st invocation ofrm might process 105 files.
Doesn't seem to work that way, and I can't imagine why it would. Why wouldn't all three execs get the exact same files?
Quote:
If there aren't enough files in the file hierarchy being processed by find to trigger invocations of of those tree utilities until the entire file hierarchy has been traversed, all three utilities could be run in parallel again allowing rm to remove some or all of the files before they are listed and archived.
Does this actually happen? find doesn't run things in parallel to my understanding.
Guys,
I want to find the log files greather than 23 days and i want to perform 2 things here.
one is to list the files and second is to gzip the files. hope this can be done using sh -c option. but not sure the exact command.
find . -name "*.log" -mtime +23 -exec ls -la {} \;
... (5 Replies)
Hi all,
Please could someone help with the following command requirement.
I basically need to find files NEWER than a given file and order the result on time.
My attempt so far is as follows:
find . -newer <file_name> -exec ls -lrt {} ;\
But I dont seem to get the right result... (12 Replies)
Hi Friends,
Please help me to sort out this problem, I am running this in centos o/s and whenever I run this script I am getting "find: missing argument to `-exec' " but when I run the same code in the command line I didn't find any problem. I am using perl script to run this ... (2 Replies)
This is a huge issue. and I need it fixed ASAP.
account-system gate-system race_traffic_sensor
achievement-system global race_voicepack
admin glue-system realdriveby
admin-system gps realism-system... (5 Replies)
Hello,
I am a linux newbe. I want to install a program. I can download it only with wget command from internet.
As far as i know this wget command does not transfer the exacutable flags.
Because of that i wanted to find all configure files and change their mod to 744.
I found this... (1 Reply)
Hello All,
Is there a way to make exec do a couple of operations on a single input from find?
For example,
find . -type d -exec ls -l "{}" ";"
I would like to give the result of each "ls -l" in the above to a wc. Is that possible?
I want to ls -l | wc -l inside... (1 Reply)
Hi,
i would like to rename files in directories and subdirs.
Files contains specific french or strange caracters.
I want to replace all non alpha-numerics by _ (underscore)
First, i made this, but i think the "for" is limited.
How can i do this directly by FIND ?
for file in $(find .... (0 Replies)
can we use |(pipe operator) with find -exec.....?
or can pipe the output of find command to another command...?
if not, why...?
pls explain (3 Replies)
Hi,
i have query about "find" command. Do I need to put the command after -exec in single quotes? Why?
For ex. see output of these three find commands.
Any explanations?
cheers,
-Ashish (2 Replies)
Hi,
Happy new year.
Would you be so kind to explain me what does this instruction :
find /rep/app -type l -exec ls -l {} \;> allink.lst
Many thanks. (2 Replies)
