Converting Huge Archive into smaller ones Post: 302247832

Sponsored Content

Top Forums Shell Programming and Scripting Converting Huge Archive into smaller ones Post 302247832 by rubin on Thursday 16th of October 2008 12:03:22 PM

10-16-2008

Registered User

Nice solution ...

and a few suggestions:
- You can consolidate all the commands in one script,
- and you can use the sleep command within the script, instead of setting up a cron process, running every 2 min,

checkAndGzip.sh

Code:

#!/bin/bash

#set -x


# Provide full path, so you can run the script from every dir.
cd /full/path/to/zipped_files

# Start unzipping the files, run it in the background so the files checking can start.
gunzip -c archive.tar.gz | tar -xvf -   &

# Start checking for the files, while the unzipping is happening. 
# Use a find ...| while read ... construction , because it doesn't break if the file name has white space in it.

 find . -name '*.extension' | while read FILE      2>/dev/null
  do
        temp=`lsof "$FILE" | awk 'NR>1 && $4 ~ "w"{ print $4 }'` ;
        
        if [ "$temp" = "" ]; then
                #Implies that the file is not in use
                #Initiate gzip on file
                gzip "$FILE";
        fi
    
        # Wait for 2 minutes.
        sleep 120

  done > output

Modify the code to fit any other requirement.

rubin

View Public Profile for rubin

Find all posts by rubin

8 More Discussions You Might Find Interesting

1. Post Here to Contact Site Administrators and Moderators

Smaller splash graphic

Not to complain, but the large graphic at the top of the page ensures that I have to scroll down everytime I change pages. Maybe the problem is the 1024x768 of my laptop screen, but it does seem excessive. Keith

2. UNIX for Dummies Questions & Answers

Anything smaller than sleep 1

I'm writting a shell script to email customers invoices. after each RCPT to: email@address.com, i've put in a sleep 1 command, but with 2000 customers to email and about 5 or 6 of these sleep commands it can take a very long time. Is there any smaller amount of time the sleeper can sleep for...

3. Shell Programming and Scripting

Grab a smaller and larger value

Hi All, I am trying to grab a term which is just smaller and larger than the assigned value using the below code. But there seems to be some problem. The value i assign is 25 so i would expect it to output a smaller value to be 20 instead of 10 and 20 and larger value to be 30 instead of 30 and...

4. Shell Programming and Scripting

Extracting from archive | compressing to new archive

Hi there, I have one huge archive (it's a system image). I need sometime to create smaller archives with only one or two file from my big archive. So I'm looking for a command that extracts files from an archive and pipe them to another one. I tried the following : tar -xzOf oldarchive.tgz...

5. Shell Programming and Scripting

Number of lines smaller than specified value

Hi All, I have a problem to find number of lines per column smaller than the values given in a different file. In example, compare the 1st column of file1 with the 1st line of the file2, 2nd column of file1 with the 2nd line of the file2, etc cat file1 0.2 0.9 0.8 0.5 ... 0.6 0.5...

6. Shell Programming and Scripting

Converting huge xls(having multiple tabs) to csv

hello I have browsed for the similar requirement i found this https://www.unix.com/shell-programming-scripting/40163-xls-csv-conversion.html but my problem is i have multiple tabs in xls file having same metadata I want to convert it into single csv file any ways to do it pls...

7. Shell Programming and Scripting

Breaking out ip subnet to smaller subnets

I have a script and it works fine, but I am sure this can be shrunk down to something much better. I would appreciate someone taking a crack at it for me. What it does is take the ip block submitted and breaks it out down to /24's. #!/bin/ksh ipadd=${1} octet1=`echo $ipadd | nawk -F....

8. Shell Programming and Scripting

Script to archive logs and sftp to another archive server

Requirement: Under fuse application we have placeholders called containers; Every container has their logs under: <container1>/data/log/fuse.log <container1>/data/log/fuse.log.1 <container1>/data/log/fuse.log.XX <container2>/data/log/fuse.log...

LEARN ABOUT REDHAT

set_color

set_color(1)							       fish							      set_color(1)

NAME

       set_color - set_color - set the terminal color

set_color - set the terminal color
   Synopsis
       set_color [-v --version] [-h --help] [-b --background COLOR] [COLOR]

   Description
       Change the foreground and/or background color of the terminal. COLOR is one of black, red, green, brown, yellow, blue, magenta, purple,
       cyan, white and normal.

       o -b, --background Set the background color

       o -c, --print-colors Prints a list of all valid color names

       o -h, --help Display help message and exit

       o -o, --bold Set bold or extra bright mode

       o -u, --underline Set underlined mode

       o -v, --version Display version and exit

       Calling set_color normal will set the terminal color to whatever is the default color of the terminal.

       Some terminals use the --bold escape sequence to switch to a brighter color set. On such terminals, set_color white will result in a grey
       font color, while set_color --bold white will result in a white font color.

       Not all terminal emulators support all these features. This is not a bug in set_color but a missing feature in the terminal emulator.

       set_color uses the terminfo database to look up how to change terminal colors on whatever terminal is in use. Some systems have old and
       incomplete terminfo databases, and may lack color information for terminals that support it. Download and install the latest version of
       ncurses and recompile fish against it in order to fix this issue.

Version 1.23.1							  Sun Jan 8 2012						      set_color(1)