Is better way copy list of multiple files, rename and gzip


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Is better way copy list of multiple files, rename and gzip
# 1  
Old 01-31-2015
Is better way copy list of multiple files, rename and gzip

Is better way to write the script loop through one by one "Old_File_1: & New_File_1:" to copy 100 files to /staging/test folder then re-name & gzip all those files? I wrote this code below and don't like that much. Thanks
I have a control_file under /tmp/test folder like below 100 files and {DATE_FILE} = YYYYMMDD

Code:
/tmp/test: cat control_file
Old_File_1: AB_P_Cdf_{DATE_FILE}.txt 
Old_File-2: DD_P_DAdf_{DATE_FILE}.txt
Old_File-3: dsf_P_DEMO_{DATE_FILE}.txt
Old_File-4: sdfd_P_Pd_{DATE_FILE}.txt
bla bla until Old_File-100
 
New_File_1: test1_sd_WW_{DATE_FILE}.txt
New_File_2: test2vd_WW_new_{DATE_FILE}.txt
New_File_3: test3cfd_dfP_dff_{DATE_FILE}.txt
New_File_4: test4gdd_WW_P_OdfsDUCT_{DATE_FILE}.txt
bla bla until New_File-100

Code:
 
#!/bin/ksh
DATE="$1"
Old_File=`cat /tmp/test/controlfile | grep Old_File | awk '{print $2}' | sed "s/{DATE_FILE}/${DATE}/g"`
#Old_File=`cat /tmp/test/controlfile | grep Old_File | awk '{print $2}' | sed "s/{DATE_FILE}/${1}/"`
New_File=`cat /tmp/test/controlfile | grep New_File | awk '{print $2}' | sed "s/{DATE_FILE}/${DATE}/g"`
#New_File=`cat /tmp/test/controlfile | grep New_File | awk '{print $2}' | sed "s/{DATE_FILE}/${1}/"`
 
###################################
# Copy file to /staging/test folder
###################################
cd /tmp/test
#Old_File=`cat /tmp/test/controlfile | grep Old_File | awk '{print $2}' | sed "s/{DATE_FILE}/${DATE}/g`
for i in $Old_File;
do
cp $i /staging/test;
done
 
###################################
# Rename all Old File to New File
###################################
cd /staging/test
cat /tmp/test/controlfile | grep Old_File | awk '{print $2}' | sed "s/{DATE_FILE}/${DATE}/g" > /staging/test/file1.txt
cat /tmp/test/controlfile | grep New_File | awk '{print $2}' | sed "s/{DATE_FILE}/${DATE}/g" > /staging/test/file2.txt
 
paste -d" " file1.txt file2.txt > file3.txt
sed 's/^/mv /' file3.txt > file4.txt
chmod 775 file4.txt
./file4.txt
rm -f file*.txt
###################################
# Gzip all New File
###################################
for i in $New_File;
do
gzip $i
done


Last edited by dotran; 01-31-2015 at 11:57 PM.. Reason: Add CODE and ICODE tags and change QUOTE tags to CODE tags.
# 2  
Old 02-01-2015
This seems to run a little bit faster than your script and should produce the same results:
Code:
#!/bin/ksh
date="${1:-$(date '+%Y%m%d')}"	# Date to process ($1 or today if no operands specified)
from='/tmp/test'		# Source directory
to='/staging/test'		# Target directory

cd "$to"
awk -v date="$date" -v from="$from" -v to="$to" '
{	sub(/[{]DATE_FILE[}]/, date)	# Replace "{DATE_FILE}" with desired date
}
/^Old_File/ {
	o[++oc] = $2	# Accumulate old file names.
	next
}
/^New_File/ {
	# Process new file names...
	++nc	# Increment # of new file names seen
	printf("cp %s/%s %s\n", from, o[nc], $2)	# Print cp command
	printf("gzip %s\n", $2)	# Print gzip command
}' "$from/controlfile" | ksh

I would suggest that you remove the | ksh at the end of the script to see the cp and gzip command that the script will produce. Then, if the commands look right, put the pipe through the shell back in to actually execute the command instead of just printing them.

If you want to run this on a Solaris/SunOS system, change awk in the script to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk.

Last edited by Don Cragun; 02-01-2015 at 03:55 AM.. Reason: Fix typo in alternative awk name "s|/iusr/xpg4/bin/awk|/usr/xpg4/bin/awk".
# 3  
Old 02-01-2015
Thank you for you new code Mr Don....but I could not make it work...I did tried all the option below and still can't pass that awk command.


Code:
 
#!/bin/ksh
#date="${1:-$(date '+%Y%m%d')}" # Date to process ($1 or today if no operands specified)
date="${1}"                     # Date to process ($1 or today if no operands specified)
from='/tmp/test'  # Source directory
to='/staging/test'  # Target directory
cd "$to"
#nawk -v date="$date" -v from="$from" -v to="$to" '
#/usr/bin/awk -v date="$date" -v from="$from" -v to="$to" '
#/iusr/xpg4/bin/awk -v date="$date" -v from="$from" -v to="$to" '
#/usr/xpg6/bin/awk -v date="$date" -v from="$from" -v to="$to" '
/usr/bin/awk -v date="$date" -v from="$from" -v to="$to" '
{       sub(/[{]DATE_FILE[}]/, date)    # Replace "{DATE_FILE}" with desired date
}
/^Old_File/ {
        o[++oc] = $2    # Accumulate old file names.
        next
}
/^New_File/ {
        # Process new file names...
        ++nc    # Increment # of new file names seen
        printf("cp %s/%s %s\n", from, o[nc], $2)        # Print cp command
        printf("gzip %s\n", $2) # Print gzip command
}' "$from/controlfile" | ksh


Code:
./test4.ksh 20150109
awk: syntax error near line 1
awk: bailing out near line 1

Code:
 
uname -a
SunOS test 5.10 Generic_147147-26 sun4v sparc sun4v

# 4  
Old 02-01-2015
Sorry,
I had a typo, it should have been /usr/xpg4/bin/awk instead of /iusr/xpg4/bin/awk. But, I'm very surprised that nawk didn't work for you. (What diagnostics did you get when you tried nawk and /usr/xpg6/bin/awk.)

Last edited by Don Cragun; 02-01-2015 at 03:59 AM.. Reason: awk and /us/xpg4/bin/awk should have worked???
# 5  
Old 02-01-2015
Thanks Mr. Don.....I did try nawk and worked great. Thanks very much !!!

Code:
/usr/bin/nawk -v date="$date" -v from="$from" -v to="$to"

# 6  
Old 02-01-2015
In lieu of relying on the correct sequence of filenames in the control file, using file numbering might be more reliable (an assumption that is not true looking at the control file in post#1). Try
Code:
awk -vDATE="20150201" -vfrom="/tmp/" -vto="/staging/"\    
        '       {sub(/[{]DATE_FILE[}]/, DATE)
                 ONW=("Old"==substr ($1,1,3))
                 FIX=substr($1,5)
                 CPARR[FIX] = CPARR[FIX] (ONW?"cp " from:" "to) $2 
                }
         END    {for (c in CPARR) print CPARR[c]}
        ' file
cp /tmp/AB_P_Cdf_20150201.txt /staging/test1_sd_WW_20150201.txt
cp /tmp/DD_P_DAdf_20150201.txt /staging/test2vd_WW_new_20150201.txt
cp /tmp/dsf_P_DEMO_20150201.txt /staging/test3cfd_dfP_dff_20150201.txt
cp /tmp/sdfd_P_Pd_20150201.txt /staging/test4gdd_WW_P_OdfsDUCT_20150201.txt

# 7  
Old 02-03-2015
Please please help with this code. I spend couple days do all kinda stuffs and can't figure out how change this code (copy, rename and pkzip all the files) instead (copy, rename and gzip all the files). This code below work really great.....but require pkzip instead gzip the files.

Code:
pkzip -add test1_sd_WW_{DATE_FILE}.zip AB_P_Cdf_{DATE_FILE}.txt


Code:
 
#!/bin/ksh
date="${1}"                     # Date to process ($1 or today if no operands specified)
from='/tmp/test'  # Source directory
to='/staging/test'  # Target directory
cd "$to"
/usr/bin/nawk -v date="$date" -v from="$from" -v to="$to" '
{       sub(/[{]DATE_FILE[}]/, date)    # Replace "{DATE_FILE}" with desired date
}
/^Old_File/ {
        o[++oc] = $2    # Accumulate old file names.
        next
}
/^New_File/ {
        # Process new file names...
        ++nc    # Increment # of new file names seen
        printf("cp %s/%s %s\n", from, o[nc], $2)        # Print cp command
        printf("gzip %s\n", $2) # Print gzip command
}' "$from/controlfile" | ksh

Out put:
Code:
 
test1_sd_WW_{DATE_FILE}.zip
test2vd_WW_new_{DATE_FILE}.zip
test3cfd_dfP_dff_{DATE_FILE}.zip
test4gdd_WW_P_OdfsDUCT_{DATE_FILE}.zip
bla bla until New_File-100


Last edited by dotran; 02-03-2015 at 01:37 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Compress multiple gzip files

Good afternoon friends. I wanted to make a query, how to compress several files and leave them all in 1, for example flat text files: filename_1.csv filename_2.csv filename_3.csv expected result filename_end.gzip = (filename_1.csv filename_2.csv filename_3.csv) please (2 Replies)
Discussion started by: tricampeon81
2 Replies

2. Shell Programming and Scripting

Oop to copy and rename files through SQL Statement in shell Script

#!/bin/sh sqlplus -s "/ as sysdba" << EOF SET HEADING OFF SET FEEDBACK OFF Select pt.user_concurrent_program_name , OUTFILE_NAME FROm apps.fnd_concurrent_programs_tl pt, apps.fnd_concurrent_requests f where pt.concurrent_program_id = f.concurrent_program_id and pt.application_id =... (1 Reply)
Discussion started by: usman_oracle
1 Replies

3. Shell Programming and Scripting

Linux Script to copy and rename files through SQL statement

Hi, I require help to complete below requirement through Linux Script. I have a SQL query which shows two columns as output. One is Report Name and other is report path. Query return multiple rows. below is the output. Report Name Cotton Stock Report (Net Weight)- Customized Output... (3 Replies)
Discussion started by: usman_oracle
3 Replies

4. Shell Programming and Scripting

Help with script to copy/rename files, then delete by date

Hi All, I am new to scripting and am looking for some assistance setting up a script. Basically I need the script to scan a folder for the newest files and make a copy of those files, adding a month to the date stamp. I also need this script to delete the previously copied files to save space.... (4 Replies)
Discussion started by: Lucid13
4 Replies

5. Shell Programming and Scripting

Files rename and copy

hello, I am write a Script and i would listing all Files from Path1 out with DSR*.txt and give a new name an copy to the Path2. I have problems with that to rename. Someone can help me? Sorry, for my english. My english is not gut. I hope you understand my. That is my Script. ... (2 Replies)
Discussion started by: efeijoo
2 Replies

6. Shell Programming and Scripting

Copy files from folder and rename them

hello, I need to build a shell script that receives the folder to copy by parameter and copy all files except thumb.db to another folder and rename them like, file.jpg renamed to file_bb1.jpg. can someone help me Thanks (4 Replies)
Discussion started by: zeker
4 Replies

7. UNIX for Dummies Questions & Answers

script to rename files with current date and copy it.

I have few webservers logs like access.log. which would be growing everyday. what i do everyday is, take the backup of access.log as access.log_(currentdate) and nullify the access.log. So thought of writing a script... but stuck up in middle. My requirement: to take the backup and nullify... (6 Replies)
Discussion started by: logic0
6 Replies

8. UNIX for Dummies Questions & Answers

copy and rename list of files

Hi all, I am a newbie in writng unix..I am using ksh shell..Does anyone know how to copy a list o files from directory A to directory B with differnt names? i.e in Dir A, I have RPT101.555.TXT RPT102.666.TXT and I want to copy those files to dir B with new naming convention.. in Dir B,... (7 Replies)
Discussion started by: kinmak
7 Replies

9. UNIX for Dummies Questions & Answers

gzip, multiple files

Hello Everyone, Here is what I am trying to do. I have four text files, I want to gzip them under unix and mail the zipped file via outlook. I am able to do this easily enough, but using winzip or pkunzip to unzip the file, there is only one file. (In essence, all four files were... (2 Replies)
Discussion started by: smbodnar
2 Replies
Login or Register to Ask a Question