Converting Huge Archive into smaller ones


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Converting Huge Archive into smaller ones
# 8  
Old 10-17-2008
That's correct, a second loop is needed,

Code:
#!/bin/bash
#set -x

cd /full/path/to/zipped_files

gunzip -c archive.tar.gz | tar -xvf -   &


# Instead of an infinite loop, you can use your condition to break out of the loop, say a count of files, a certain disk size is reached,...
# while [ condition ] ; do ...

while :
 do
 find . -name '*.extension' | while read FILE      2>/dev/null
  do
        temp=`lsof "$FILE" | awk 'NR>1 && $4 ~ "w"{ print $4 }'` ;
        
        if [ "$temp" = "" ]; then
                #Implies that the file is not in use
                #Initiate gzip on file
                gzip "$FILE";
        fi
    
        
  done > output
        
 # Wait for 2 minutes.
 sleep 120
 
 # If you use an infinite loop, then test when condition becomes true, break out of the loop.
 if [ condition ]     
  then 
      break
  fi

done

Again , the code is more of a guideline, than an actual working script, so modifications might be needed, but the main logic is there.
# 9  
Old 10-17-2008
Yes... That is it... I used a while true loop instead. Thanks a lot... Smilie
# 10  
Old 10-18-2008
Wouldn't be better to skip already gzipped files in "find ... | while" construction ?

find ! -name "*.gz" finds files that don't have *.gz extension, so the following

find . ! -name "*.gz" | while read FILE 2>/dev/null
do
...stuff...
done

should attempt gzip on non gzipped files only.

PS
I like the solution you have developed for it's readability, but I still believe the whole task can be done easier, somehow with parenthesis maybe...
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to archive logs and sftp to another archive server

Requirement: Under fuse application we have placeholders called containers; Every container has their logs under: <container1>/data/log/fuse.log <container1>/data/log/fuse.log.1 <container1>/data/log/fuse.log.XX <container2>/data/log/fuse.log... (6 Replies)
Discussion started by: Arjun Goswami
6 Replies

2. Shell Programming and Scripting

Breaking out ip subnet to smaller subnets

I have a script and it works fine, but I am sure this can be shrunk down to something much better. I would appreciate someone taking a crack at it for me. What it does is take the ip block submitted and breaks it out down to /24's. #!/bin/ksh ipadd=${1} octet1=`echo $ipadd | nawk -F.... (3 Replies)
Discussion started by: numele
3 Replies

3. Shell Programming and Scripting

Converting huge xls(having multiple tabs) to csv

hello I have browsed for the similar requirement i found this https://www.unix.com/shell-programming-scripting/40163-xls-csv-conversion.html but my problem is i have multiple tabs in xls file having same metadata I want to convert it into single csv file any ways to do it pls... (5 Replies)
Discussion started by: joshiamit
5 Replies

4. Shell Programming and Scripting

Number of lines smaller than specified value

Hi All, I have a problem to find number of lines per column smaller than the values given in a different file. In example, compare the 1st column of file1 with the 1st line of the file2, 2nd column of file1 with the 2nd line of the file2, etc cat file1 0.2 0.9 0.8 0.5 ... 0.6 0.5... (9 Replies)
Discussion started by: senayasma
9 Replies

5. Shell Programming and Scripting

Extracting from archive | compressing to new archive

Hi there, I have one huge archive (it's a system image). I need sometime to create smaller archives with only one or two file from my big archive. So I'm looking for a command that extracts files from an archive and pipe them to another one. I tried the following : tar -xzOf oldarchive.tgz... (5 Replies)
Discussion started by: chebarbudo
5 Replies

6. Shell Programming and Scripting

Grab a smaller and larger value

Hi All, I am trying to grab a term which is just smaller and larger than the assigned value using the below code. But there seems to be some problem. The value i assign is 25 so i would expect it to output a smaller value to be 20 instead of 10 and 20 and larger value to be 30 instead of 30 and... (3 Replies)
Discussion started by: Raynon
3 Replies

7. UNIX for Dummies Questions & Answers

Anything smaller than sleep 1

I'm writting a shell script to email customers invoices. after each RCPT to: email@address.com, i've put in a sleep 1 command, but with 2000 customers to email and about 5 or 6 of these sleep commands it can take a very long time. Is there any smaller amount of time the sleeper can sleep for... (4 Replies)
Discussion started by: markms
4 Replies

8. Post Here to Contact Site Administrators and Moderators

Smaller splash graphic

Not to complain, but the large graphic at the top of the page ensures that I have to scroll down everytime I change pages. Maybe the problem is the 1024x768 of my laptop screen, but it does seem excessive. Keith (2 Replies)
Discussion started by: kduffin
2 Replies
Login or Register to Ask a Question