Gunzip and edit many files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Gunzip and edit many files
# 1  
Old 07-06-2017
Gunzip and edit many files

Experts - I have an requirement to gunzip and edit many files in a pair of directories.

I have two scripts that work great when run separately, but I'm having problems
combining the two.

The goal is to gunzip the files found in the first script and pipe them to the
bash/sed script and output to a different directory.

Here is the 1st script:
Code:
#!/bin/sh

cd $HOME

find . -type f -mmin -40 -name '*.gz'  -print  | ( while read i
do
        FILE=$(basename "${i}" .gz)
        gunzip -c "${i}" | $HOME/insert.sh > tmp/"${FILE}"
done
)

Here is the ($HOME/insert.sh) bash/sed script:
Code:
#!/bin/bash

startTime="$(grep "<Setup" $1)</Setup>"
dn=$(grep -m1 "DN" $1)

sed  -e "/<Target/a\ $dn" -e "/<Target/a \ \ \ \ \ \ $startTime" $1 > $1.chg

Anyone have any ideas on this?



Thanks
# 2  
Old 07-07-2017
Not sure I fully understand what you're after, but two things jump to my eyes :
- the second script seems to require data from stdin to work (grep) upon, but the first redirects its stdout to a file which, then, never appears again.
- the second script seems to require a parameter which the first doesn't supply when calling.
# 3  
Old 07-07-2017
Thanks for the reply.
Sorry if I wasn't very clear.

I'm trying to gunzip many files and run them through a sed script for further processing before they get written to a file.

The 1st script attempts to do 2 things:'

1. Find gz files that are 40 minutes old
2. gunzip the files found in step one and run them through the sed script before they get written to a file

I worked on writing 2 scripts that did this.
Now I want to combine the two scripts so that the files are gunzipped and "edited" before they are written to a file.

Now I have to run the first script to gunzip all these files and then run the second script on those files to edit and rewrite.

I'm trying to only write the files once be combining the two scripts.

Does this make sense?

Thanks
# 4  
Old 07-07-2017
Quote:
Originally Posted by timj123
Thanks for the reply.
Sorry if I wasn't very clear.

I'm trying to gunzip many files and run them through a sed script for further processing before they get written to a file.
OK, we have understood that. Still, What RudiC has said holds:

Code:
gunzip -c "${i}" | $HOME/insert.sh > tmp/"${FILE}"

Here the script $HOME/insert.sh is called, but without any parameter, but here:

Code:
startTime="$(grep "<Setup" $1)</Setup>"
dn=$(grep -m1 "DN" $1)

It seems that one parameter to this script is required, no? So the line in script one should look like

Code:
gunzip -c "${i}" | $HOME/insert.sh "some-param-here" > tmp/"${FILE}"

and whatever yo put into "some-param-here" will end up where you use "$1" in the insert.sh-script.


Another thing is that obviously insert.sh expects the parameter to be a file(name), because otherwise this line:

Code:
sed  -e "/<Target/a\ $dn" -e "/<Target/a \ \ \ \ \ \ $startTime" $1 > $1.chg

wouldn't make any sense at all. But you do not pass a filename to insert.sh, instead you flood its stdin with input. insert.sh has no method to deal with input from stdin, though. It is as if you are writing someone a letter who is only expecting telephone calls. Because he watches the phone the whole day but doesn't check his mailbox you can write as many letters as you want, he won't react.

I hope this helps.

bakunin

Last edited by vbe; 07-07-2017 at 10:47 AM.. Reason: missing square bracket...
These 3 Users Gave Thanks to bakunin For This Post:
# 5  
Old 07-07-2017
I had a misconception in my post - of course insert.sh's stdin receives data, and its stdout is redirected to the temp file. Sorry!
# 6  
Old 07-07-2017
I think I really over thought this and wasn't very clear in what my requirements were.

I originally had two scripts that worked great, but I needed two cron jobs to do and used more disk space because I gunzipped the files into one location and then edited them with another script into yet another directory.

My goal was to gunzip and edit together to only one directory, combining the two scripts I was using previously.

I got stuck on passing a shell variable from one script and passing it into the sed script.

This is what I came up with, I would appreciate it if anyone has a better way to do this.
Code:
#!/bin/sh

cd $HOME

find . -type f -mmin -40 -name '*.gz'  -print  | ( while read i
do
        FILE=$(basename "${i}" .gz)
        startTime="$(zgrep "<Setup" ${i})</Setup>"
        dn=$(zgrep -m1 "DN" ${i})
        gunzip -c "${i}" | sed  -e "/<Target/a\ $dn" -e "/<Target/a \ \ \ \ \ \ $startTime"  > tmp/"${FILE}"
done
)

Thanks again for looking into this!
# 7  
Old 07-07-2017
If you told people what you want to achieve - not what and how you coded - solutions ("better ways") could be thought up. With yout above script, you unzip every .gz file THREE times! And, if there are more than one occurrence of e.g. "DN" in a file, the variable and thus replacement string might not be what you desired. And, if "DN" is missing in a file, the last one's will be used.
If you could make 100% sure "setup..." and "DN" will always precede "<Target" in your files, a very simple awk code could be offered.
This User Gave Thanks to RudiC For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to decompress files using gunzip?

I have compressed files under directory '/root/data' and i need the uncompressed files in another directory '/root/uncom'. I running a shell script below shell script from directory '/root/' gunzip /root/data/*.gz -d /root/uncom But this is failing with : gunzip: /root/uncom is a directory... (2 Replies)
Discussion started by: hoyanet
2 Replies

2. Shell Programming and Scripting

Edit names of files in a directory

Hi all, I have a directory with multiple (thousnads) of files, which are named this way ABCDEF.wo.im-1 OKRAME.ire.roi IOJEAFO01.irt.gfg IMNYBL05.REG.gkf I would like to keep the part of the name (everything before the first dot in the filename). The desired output: ABCDEF... (3 Replies)
Discussion started by: Error404
3 Replies

3. Shell Programming and Scripting

How to decompress files using gunzip?

I have compressed files under directory '/root/data' and i need the uncompressed files in another directory '/root/uncom'. I running a shell script below shell script from directory '/root/' gunzip /root/data/*.gz -d /root/uncom But this is failing with gunzip: /root/uncom is a directory... (2 Replies)
Discussion started by: vel4ever
2 Replies

4. UNIX for Dummies Questions & Answers

Edit files with cat

Hi, sometimes one wants to edit files while still seeing output of earlier commands in terminal. I've found out that cat test && cat - >> test does the trick for displaying file content and adding lines but I believe I saw a much cooler command that was also able to erase lines from files. I cannot... (6 Replies)
Discussion started by: scarleo
6 Replies

5. Shell Programming and Scripting

Gunzip files

Hi ALL, Am working with the gunzip command to zip all the old files having 10 days am using the command find . -name '*.log' -type f -mtime +10 -exec gunzip {} \; am facing two issues 1.)it displays the files which are all older than a year 2.)when am trying to gunzip all the... (2 Replies)
Discussion started by: thelakbe
2 Replies

6. Shell Programming and Scripting

Gunzip files

Hello Everyone, I have a few files in a directory such as : abc.xyz.txt1.gz abc.xyz.txt2.gz .... .... ... ... abd.xyz.txt100.gz And I want uncompressed files such as: abc.xyz.txt1 abc.xyz.txt2 .... ... ..... .... (1 Reply)
Discussion started by: ad23
1 Replies

7. UNIX for Dummies Questions & Answers

edit _config files

Hi, I am trying to edit sshd_config file through the vi editor. logged on as a root. when I try to write the file I get: Read-only file, not written; use ! to override when i type :w!, I get: Error: etc/ssh/sshd_config Permission denied. I want to change: #PermitRootLogin no to yes freeBDS... (6 Replies)
Discussion started by: emosms
6 Replies

8. Gentoo

how to edit linux system files?

i had heard that linux is open source.....which meant that i could edit it. so how do i start out? i've already downloaded it. the name's "puppy linux".....someone please reply quick!!! and by the way, may i know what shell scripting is? (15 Replies)
Discussion started by: Dragster93
15 Replies

9. UNIX for Dummies Questions & Answers

Edit Multiple Files in VI

Here's what I have... $ vi foo1 - open foo1 and work around for a while. I yank a few lines into a buffer and then :w to save. Next I :e foo2 to open foo2 and paste my buffer. I :w to save, but I would like to then be able to go directly back into foo1 where I was before I opened foo2. ... (4 Replies)
Discussion started by: djschmitt
4 Replies

10. UNIX for Dummies Questions & Answers

how to edit large files using vi

How to edit large file using vi where you can't increase /usr/var/tmp anymore? (3 Replies)
Discussion started by: nazri
3 Replies
Login or Register to Ask a Question