Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Problem in compressing and moving files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Problem in compressing and moving files
# 1  
Old 06-01-2015
Problem in compressing and moving files
Hi

I am writing a sample script (sample.ksh) to compress files in the source directory and move them to another directory.
The script takes a config file (files.config) as the paramter the contents of which are as given under:

Code:
/abc/src ${TSTENV}-xxx-yyy~1.log /abc/src/dest
/abc/src ${TSTENV}-yyy~1.log /abc/src/dest
/abc/src ${TSTENV}-zzz~rem_plg~1.log /abc/src/dest

This file consists of three parts:

First is the source directory where the files reside: /abc/src
Second is the files to be compressed: ${TSTENV}-xxx-yyy~1.log, ${TSTENV}-yyy~1.log, ${TSTENV}-zzz~rem_plg~1.log
Third is the destination directory where the files are to be moved post compression: /abc/src/dest

The sample.ksh written so far looks like:

Code:
#!/bin/ksh
INP_PARAM=files.config

[[ ! -f ${INP_PARAM} ]] && echo "File ${INP_PARAM} is not present" && exit 1

DTTM=`date "+%Y%m%d"`

echo "`basename $0` started on `date`"

cat ${INP_PARAM} | \
while read LINE
do
        FIRST_ARG=`echo ${LINE} | cut -d" " -f1`
        WORK_DIR=`eval "echo ${FIRST_ARG}"`
        SECOND_ARG=`echo ${LINE} | cut -d" " -f2`
        FILE_NAME=`eval "echo ${SECOND_ARG}"`
        THIRD_ARG=`echo ${LINE} | cut -d" " -f3`
        ARCHIVE_DIR=`eval "echo ${THIRD_ARG}"`

        [[ ! -d ${ARCHIVE_DIR} ]] && mkdir ${ARCHIVE_DIR}

        cd ${WORK_DIR}

        #Compress the log files
        cut -d " "

        [[cut : -f ${FILE_NAME} ]] && (gzip "${FILE_NAME}"; chmod 644 "${FILE_NAME}".gz) || echo "${FILE_NAME} does not exists"
        [[ $? != 0 ]] && echo "Error: Unable to compress the log file ${FILE_NAME}" && exit 1

        #Move the .gz file to ARCHIVE directory
        [[ -f ${FILE_NAME}.gz ]] && mv "${FILE_NAME}".gz ${ARCHIVE_DIR}/"${FILE_NAME}".${DTTM}.gz || echo "${FILE_NAME} doesn't exists"
        [[ $? != 0 ]] && echo "Error: Unable to move the file ${FILE_NAME}.${DTTM}.gz to ${ARCHIVE_DIR}" && exit 1

        #Create empty log files
        touch ${FILE_NAME}

done

echo "`basename $0` ended on `date`"

exit 0

However, when I try to execute, am getting the following error:
Code:
Usage: cut {-b <list> [-n] | -c <list> | -f <list> [-d <char>] [-s]} file ...
./sample.ksh[27]: [[cut:  not found
mach01-xxx-yyy~1.log does not exists
mach01-xxx-yyy~1.log doesn't exists
Usage: cut {-b <list> [-n] | -c <list> | -f <list> [-d <char>] [-s]} file ...
./sample.ksh[27]: [[cut:  not found
mach01-yyy~1.log mach01-yyy~1.log. mach01-yyy~1.log.15-05-25_20:41:05 mach01-yyy~1.log.15-05-26_20:29:40 mach01-yyy~1.log.15-05-27_20:28:32 mach01-yyy~1.log.15-05-28_20:28:27 does not exists
mach01-yyy~1.log mach01-yyy~1.log. mach01-yyy~1.log.15-05-25_20:41:05 mach01-yyy~1.log.15-05-26_20:29:40 mach01-yyy~1.log.15-05-27_20:28:32 mach01-yyy~1.log.15-05-28_20:28:27 doesn't exists
Usage: cut {-b <list> [-n] | -c <list> | -f <list> [-d <char>] [-s]} file ...
./sample.ksh[27]: [[cut:  not found
mach01-zzz~rem_plg~1.log does not exists
mach01-zzz~rem_plg~1.log doesn't exists

The script isn't recognizing any of the files under the source directory even though they are present there and some of the log files have a date and timestamp extension as seen in the output above which I want to move as well but is not happening.

Any help would be greatly appreciated.
Last edited by swasid; 06-01-2015 at 01:31 PM..
# 2  
Old 06-01-2015
Have a dry try with:
Code:
while read base name dest
do echo "base:$base name:$name dest:$dest"
done<files.config

This would save you quite a bit of the required parsing.

hth
# 3  
Old 06-01-2015
The main problem is that the files present aren't getting moved to the destination folder. Is there a simpler way of achieving the same?
# 4  
Old 06-01-2015
You have a syntax error as clearly indicated by the messages. [[ and ]] need spaces around them.

And, what is the meaning of
Code:
[[cut :

?
# 5  
Old 06-01-2015
Trying out a simpler version this time with:

Code:
#!/bin/ksh

SOURCE_DIR="/abc/src"
DEST_DIR="/abc/src/dest"

cd $SOURCE_DIR

for FILENAME in xxx yyy zzz
do
    SRC_FILE="$SOURCE_DIR/$TSTENV-${FILENAME}*"
    gzip ${SRC_FILE}
    chmod 644 ${SRC_FILE}.gz
    mv ${SRC_FILE}.gz ${DEST_DIR}
done

I am trying to build the file pattern here where $TSTENV will translate to mach01, the name of the machine after which the files can be in any of the below given formats:

Code:
mach01-zzz~rem_plg~1.log
mach01-xxx-yyy~1.log
mach01-yyy~1.log
mach01-yyy~1.log.15-05-26_20:29:40

Basically I want anything after mach01-xxx or mach01-yyy or mach01-zzz to be zipped and sent. From the code written above, it is doing the compression in the source folder but is there a way I can remove the below message when it is run the second time?

Code:
gzip: /abc/src/mach01-xxx-yyy~1.log.gz already has .gz suffix -- unchanged

# 6  
Old 06-01-2015
Dont do this:
Code:
SRC_FILE="$SOURCE_DIR/$TSTENV-${FILENAME}*"

Take the asterix out of the code, or it will be an asterix, rather than expanding to filenames.

Example:
Code:
echo "/boot/grub2/themes/*"
/boot/grub2/themes/*

echo "/boot/grub2/themes/"*
/boot/grub2/themes/circled-nasa-aurora-space /boot/grub2/themes/circled-nasa-horizon /boot/grub2/themes/circled-nasa-sombrero /boot/grub2/themes/circled-nasa-spirale /boot/grub2/themes/system

That is a compress format that can be compressed only once.
Write a filter to sort out files that already have that extension.

Try:
Code:
SRC_FILE=$(echo "$SOURCE_DIR/$TSTENV-${FILENAME}"*|grep -v gz$)

Or insert right after the loop beginning:
Code:
[ "${FILENAME:0:(-3)}" = ".gz" ] && continue

hth
# 7  
Old 06-03-2015
Thanks sea. It's working. Just to know, is there any way my first approach can be refactored? The problem is that when I echo $FILE_NAME, it's listing all the files that are read as the second parameter from the config file but it's not moving them to the destination directory as expected because it's considering all of them to be one single file. It should ideally loop through all of the files and transfer them to the destination folder. Can you suggest any tweaking?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Logrotate and Compressing only yesterdays files

Hello, I have a syslog server at home and am currently experiencing an issue where my logs will rotate and compress however it will rotate and compress yesterdays file and the newly created log file for the current day. When it does this however it will also create another new file for today... (9 Replies)
Discussion started by: MyUserName7000
9 Replies

2. Shell Programming and Scripting

Getting latest files and compressing from a textfile

I'm doing a cleanup script for a directory using KSH. I'm keeping the file name prefixes in a text file. In a KSH, I want to read the prefix from the file, and match the pattern of the file and keep and compress(.Z) the latest 4 versions of the matched files in the directory. And I want to delete... (1 Reply)
Discussion started by: manchimahesh
1 Replies

3. Shell Programming and Scripting

Compressing files

I need help to do a script that will compress a file that's bigger than 5000 octets and won't overwrite the previous compress file. lets say I have mylogfile.log and I would compress it I would become mylogfile. 1. log and if I would compress again mylogfile.log it would be mylogfile. 2.... (8 Replies)
Discussion started by: Froob
8 Replies

4. Shell Programming and Scripting

What is the code for compressing files using pkzip ?

hi everyone , Can someone provide me a shell program to compress and decompress files using gzip , i dont know anything in shell programming , this code is a part of my project. So can someone help with me the code ? (2 Replies)
Discussion started by: mig23
2 Replies

5. UNIX for Dummies Questions & Answers

Compressing of log files

Hello All My first post in the forum. :) I've this huge log files of size 20GB-30 GB in my unix server. I want to analyse the log file for some error messages. But because of the enormity of the size of these files i'm not able to grep/search the pattern in the file . Also, tried to gzip the... (1 Reply)
Discussion started by: sgbhat
1 Replies

6. UNIX for Advanced & Expert Users

Compressing files on NAS mount

Hello, I am having difficulty compressing the files using compress or GZIP utility on NAS share NFS mounted on my linux server. Any one have idea on how to do this ? I get the followign error but the trying to compress the files STRP2> compress STR_OUTBOUND_CDM_LOG_LOB.PRT2008_26.txt... (1 Reply)
Discussion started by: kamathg
1 Replies

7. UNIX for Advanced & Expert Users

Problem in moving files to .txt

Hello All Iam running one script to Split the file and rename it with .txt extension Script Prefix=Z_PRICE_NEW_`date "+%Y%m%d%H%M%S"` split -3000 -a 4 Z_PRICE_NEW.txt $Prefix find . -name "$Prefix*" -print | { while read FILE; do mv $FILE $FILE.tmp done } Since the Source file was... (2 Replies)
Discussion started by: mohdtausifsh
2 Replies

8. UNIX for Dummies Questions & Answers

compressing two files in a script

Hi, i have written a script in unix which produces two files(.csv file) at the end. Now i want to add these to files in a zip file and send the zip file across the network by FTP. Problem is that i dunno how to make a single zip file containing the two files that have been created by the script.... (1 Reply)
Discussion started by: nimish
1 Replies

9. UNIX for Dummies Questions & Answers

Compressing files

I have never used this command before. I need to use the "compress" command to compress all files located in the subdirectories under the following directories: /home/ftp/inbound/Fordin Please advise, I appreciate your help. Thanks, Syed (3 Replies)
Discussion started by: sh9492
3 Replies

10. UNIX for Dummies Questions & Answers

Compressing files from DOS to Unix

As the title sugests I need to compress files from DOS to Unix. The files should be in .Z format, as created using 'compress' in an Unix environment. Ligs (2 Replies)
Discussion started by: ligs
2 Replies
Login or Register to Ask a Question