reading a file for processing


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting reading a file for processing
# 1  
Old 04-20-2012
reading a file for processing

i have file which looks like this:
Code:
file=/var/log/log1.log
file=/var/log/log2.log
cmd=ls -l
cmd=dir > /var/log/mylog.log

now i have to write a script which will read all "file" tag values and copy them and read all "cmd" values and run them etc..

what i was thinking to do is:
read file by line and:
1. use awk or sed to get file value process or run cmd if cmd value.

problem with this aproach is while i am copying file i am blocked and cant process further so if any file copy or cmd takes more time i cant process further.

so i thought of using another approach:
i will process above file and create a array of files and cmds and then i can process them individually.

I am new to shell scripting, so creating array and processing i have to read about.
please suggest me right way if you can define better.

i can also consider changing my above file sturcture if you have better way of copying files and processing cmds written there.

Thanks, Ajay

Last edited by Scrutinizer; 04-20-2012 at 04:50 PM.. Reason: Use code tags instead of quote tags
# 2  
Old 04-20-2012
It's somewhat strange to have a script file reading a file of lines to script -- rather the long way around. If you're storing commands in a file to run, why not put them in the script file in the first place?

What are you actually trying to do -- not the algorithm you're following, but the goal the algorithm is trying to achieve...
This User Gave Thanks to Corona688 For This Post:
# 3  
Old 04-20-2012
thanks for reply

thank you so much for feedback.
let me explain my problem.

My project consist of submodules and each module want me to do some kind of same work(copy some of their files , run few command and gather output from them)
the file which i have shown is kind of configuration file where each module will provide this file with their data to me and i will have single script which will read this file and process as i explained.

Does this make sense? or i should have some other approach?

waiting eagerly for the inputs.
# 4  
Old 04-20-2012
I suspect I'd be making individual scripts instead of configuration files, here, and passing in the parameters as arguments instead of storing them in a configuration file, but it's hard to be sure when we know so little about your project.

You've explained in more detail how your config file is used, but still haven't said what it's for. What this project is supposed to be doing. Why you're breaking it up into modules. How these modules are supposed to be used.
This User Gave Thanks to Corona688 For This Post:
# 5  
Old 04-20-2012
ok here is more

Quote:
Originally Posted by Corona688
I suspect I'd be making individual scripts instead of configuration files, here, and passing in the parameters as arguments instead of storing them in a configuration file, but it's hard to be sure when we know so little about your project.

You've explained in more detail how your config file is used, but still haven't said what it's for. What this project is supposed to be doing. Why you're breaking it up into modules. How these modules are supposed to be used.
ok let me give u details:

1. so our product consists of different sub projects which doesn't talk to each other they work independently.
2. Each sub project has its own directory structure and set of work it does.
3. Now requirement comes from these each sub projects: They want there log files and some data which can be gathered by running commands which only they know. each sub projects has same requirement.
4. So i want to write that script which will take a configuration file which will be given by each projects and i will process for them in this master script.
Note: configuration file contain "file" and "cmd" those are filled by them and it can be many more. so instead of each one doing same thing i was thinking i can go with this approach.

Does it help understanding requirement now? let me know if you have any more question.
# 6  
Old 04-20-2012
Make individual scripts for each of the projects, run these scripts.

Or alternatively, make functions inside these scripts, and source these scripts to set variables and load functions, so you can load code exactly as given instead of having to strip it out of variables and run it with eval.
# 7  
Old 04-20-2012
Perhaps something like this, if you have bash or ksh:

Code:
# Template script file

VAR1=asdf
VAR2=asdf

function do_something
{
        echo "do_something called"
}

function do_something_else
{
        echo "Wow, something else"
}

And you'd use it like
Code:
. /path/to/template ; do_something ; do_something_else

This User Gave Thanks to Corona688 For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

awk - Rename output file, after processing, same as input file

I have one input file ABC.txt and one output DEF.txt. After the ABC is processed and created output, I want to rename ABC.txt to ABC.orig and DEF to ABC.txt. Currently when I am doing this, it does not process the input file as it cannot read and write to the same file. How can I achieve this? ... (12 Replies)
Discussion started by: High-T
12 Replies

2. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

3. Shell Programming and Scripting

Recursive file processing from a path and printing output in a file

Hi All, The script below read the path and searches for the directories/subdirectories and for the files. If files are found in the sub directories then read the content of the all files and put the content in csv(comma delimted) format and the call the write to xml function to write the std... (1 Reply)
Discussion started by: Optimus81
1 Replies

4. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

5. Shell Programming and Scripting

reading a file inside awk and processing line by line

Hi Sorry to multipost. I am opening the new thread because the earlier threads head was misleading to my current doubt. and i am stuck. list=`cat /u/Test/programs`; psg "ServTest" | awk -v listawk=$list '{ cmd_name=($5 ~ /^/)? $9:$8 for(pgmname in listawk) ... (6 Replies)
Discussion started by: Anteus
6 Replies

6. Shell Programming and Scripting

Reading a file line by line and processing for each line

Hi, I am a beginner in shell scripting. I have written the following script, which is supposed to process the while loop for each line in the sid_home.txt file. But I'm getting the 'end of file' unexpected for the last line. The file sid_home.txt gets generated as expected, but the script... (6 Replies)
Discussion started by: sagarparadkar
6 Replies

7. Shell Programming and Scripting

how to change the current file processing to some other random file in awk ?

Hello, say suppose i am processing an file emp.dat the field of which are deptno empno empname etc now say suppose i want to change the file to emp.lst then how can i do it? Here i what i attempted but in vain BEGIN{ system("sort emp.dat > emp.lst") FILENAME="emp.lst" } { print... (2 Replies)
Discussion started by: salman4u
2 Replies

8. Shell Programming and Scripting

Checking for a control file before processing a data file

Hi All, I am very new to Shell scripting... I got a requirement. I will have few text files(data files) in a particular directory. they will be with .txt extension. With same name, but with a different extension control files also will be there. For example, Sample_20081001.txt is the data... (4 Replies)
Discussion started by: purna.cherukuri
4 Replies

9. Shell Programming and Scripting

Have a shell script check for a file to exist before processing another file

I have a shell script that runs all the time looking for a certain type of file and then it processes the file through a series of other scripts. The script is watching a directory that has files uploaded to it via SFTP. It already checks the size of the file to make sure that it is not still... (3 Replies)
Discussion started by: heprox
3 Replies
Login or Register to Ask a Question