I never imagined, I would face so many problems with a directory archived the wrong way In any case, I was able to convert a complete directory archive into a directory of archives. In any case, here's a solution for those interested:
Problem:
A directory was gzipped on the whole. So, it becomes almost impossible to extract data from some particular files efficiently.
Constraint:
The archive is 13G and expands into 250G but the disk capacity is 50G
Conventional Answer:
13G Directory Archive --> Expands to 250G --> Converted into 13G Directory of Archives
Answer:
Convert the directory archive into a directory of archives.
Solution:
Step 1:
Prepare a shell script and place it in the directory where the archive is to be extracted: checkAndGzip.sh
Code:
#!/bin/bash
for FILE in `find ./ -name "*.extension"`
do
temp=`lsof $FILE | awk '{if(NR>1) if($4 ~ "w") print $4}'`;
if [ "$temp" = "" ]; then
#Implies that the file is not in use
#Initiate gzip on file
gzip $FILE;
fi
done;
Note: Observe the usage of lsof which is a nice utility that tells if the file is in use.
Step 2:
Setup a cron as
Code:
*/2 * * * * /path/to/checkAndGzip.sh > output
Note: Cron runs every two minutes
Step 3:
Run this command in the directory:
Code:
gunzip -c archive.tar.gz | tar -xvf -
Logic:The logic is pretty simple. On the one hand, the extraction takes place and on the other, the cron executes a shell script that checks if a new file has been generated and then gzips it. The reason why we use lsof is to verify if the file is still being extracted (gzip doesn't seem to care about partial files) and if a file is in use, skip it during this run.
If anyone has a better solution, or have an improvement for the above solution, kindly suggest
Not to complain, but the large graphic at the top of the page ensures that I have to scroll down everytime I change pages. Maybe the problem is the 1024x768 of my laptop screen, but it does seem excessive.
Keith (2 Replies)
I'm writting a shell script to email customers invoices.
after each RCPT to: email@address.com, i've put in a sleep 1 command,
but with 2000 customers to email and about 5 or 6 of these sleep commands it can take a very long time.
Is there any smaller amount of time the sleeper can sleep for... (4 Replies)
Hi All,
I am trying to grab a term which is just smaller and larger than the assigned value using the below code. But there seems to be some problem. The value i assign is 25 so i would expect it to output a smaller value to be 20 instead of 10 and 20 and larger value to be 30 instead of 30 and... (3 Replies)
Hi there,
I have one huge archive (it's a system image).
I need sometime to create smaller archives with only one or two file from my big archive.
So I'm looking for a command that extracts files from an archive and pipe them to another one.
I tried the following :
tar -xzOf oldarchive.tgz... (5 Replies)
Hi All,
I have a problem to find number of lines per column smaller than the values given in a different file. In example, compare the 1st column of file1 with the 1st line of the file2, 2nd column of file1 with the 2nd line of the file2, etc
cat file1
0.2 0.9 0.8 0.5 ...
0.6 0.5... (9 Replies)
hello
I have browsed for the similar requirement i found this
https://www.unix.com/shell-programming-scripting/40163-xls-csv-conversion.html
but my problem is i have multiple tabs in xls file having same metadata
I want to convert it into single csv file
any ways to do it pls... (5 Replies)
I have a script and it works fine, but I am sure this can be shrunk down to something much better. I would appreciate someone taking a crack at it for me.
What it does is take the ip block submitted and breaks it out down to /24's.
#!/bin/ksh
ipadd=${1}
octet1=`echo $ipadd | nawk -F.... (3 Replies)
Requirement:
Under fuse application we have placeholders called containers;
Every container has their logs under:
<container1>/data/log/fuse.log
<container1>/data/log/fuse.log.1
<container1>/data/log/fuse.log.XX
<container2>/data/log/fuse.log... (6 Replies)
Discussion started by: Arjun Goswami
6 Replies
LEARN ABOUT CENTOS
bf_tar
BF_TAR(1) Bogofilter Reference Manual BF_TAR(1)NAME
bf_tar - shell script to write a tar file of a bogofilter directory to stdout
SYNOPSIS
bf_tar [-r] [-R] bogofilter_directory
DESCRIPTION
bf_tar bundles a bogofilter working directory in tar format and copies it to standard output (your console, or where you redirect it, see
EXAMPLES below).
OPTIONS
The -r option causes bf_tar to remove inactive log files after the archive has been written successfully. The default is to leave log
files.
The -R option causes bf_tar to remove inactive log files before the archive is written. This may reduce chances that the resulting archive
is recoverable should it become damaged. The archive may be smaller though. The default is to leave log files.
EXIT STATUS
The script exits with status code 0 if everything went well, and nonzero if it encountered trouble.
EXAMPLES
o bf_tar ~/.bogofilter > outfile.tar
Writes a standard .tar file containing the essential files from ~/.bogofilter to outfile.tar.
o bf_tar ~/.bogofilter | gzip -9 -c > outfile.tar.gz
Writes a gzipped .tar.gz file containing the essential files from ~/.bogofilter to outfile.tar.gz.
o bf_tar `pwd`/mydirectory > outfile.tar
Prepend $(pwd)/ or `pwd`/ if you want to specify an absolute path instead of a relative path.
NOTES
This script is meant for use with Berkeley DB based bogofilter versions.
This script requires a SUSv2 compliant pax utility.
This script expects a SUSv2 compliant shell. Solaris systems should have the SUNWxcu4 package installed (when bogofilter is configured) so
that /usr/xpg4/bin/sh can be used.
AUTHORS
Matthias Andree
Bogofilter 10/22/2012 BF_TAR(1)
10-16-2008
In any case, I was able to convert a complete directory archive into a directory of archives. In any case, here's a solution for those interested: