Using Make to batch process files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using Make to batch process files
# 1  
Old 11-19-2010
Question Using Make to batch process files

Hello all,

I have a make question, and I was hoping somebody here might be able to point me in the right direction.

Here is my issue; I have a command-line tool that I use to run a conversion on an input XML file that results in an output binary file. However, this particular tool needs to load up an overarching project file when it starts, and the time involved in that initial parsing is significant. Therefore, if I have 100 XML files that need processing, I waste a lot of time starting the tool over and over.

I like to use make for this kind of processing, because the dependency handling is so good. However, in this instance I would like to send all of the XML files that need processing to the tool in a single batch. Typically my make rules look like this:

Code:
%.bin : %.xml
           tool -export %.xml

... but this of course will run "tool" once for each XML file encountered in a previous dependency. I had considered this:

Code:
$(ALL_BIN) : $(ALL_XML)
                 tool -export $(ALL_XML)

... which will batch up the job, but it will rebuild every binary file if just a single XML file changes. Does anybody know a way that I keep make's dependency checking on a per-file basis, but run the actual command only once? Thanks much for any advice!

Moderator's Comments:
Mod Comment
Please use code tags when posting data and code samples!

Last edited by vgersh99; 11-19-2010 at 11:51 AM.. Reason: code tags, please!
# 2  
Old 11-19-2010
Working in multiple jobs is generally preferred -- this lets make split the work across multiple cores, meaning, lets it do more at once. If you actually have multiple cores, you may find it faster to let it continue this way and run make -j2, or -j4 etc.

On the other hand there is a macro for "dependencies more recent than the target": $? So this could do it:
Code:
$(ALL_BIN) : $(ALL_XML)
        tool -export $?

Note that the eight spaces in front of tool there must actually be a tab for this rule to work.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Start process on X number of files and then wait for the next batch

Thanks for RudiC for his extraordinary help on organizing files in a batch of 10 using below code. FL=($(ls)); for ((i=0;i<=${#FL};i++)); do for j in ${FL:$i:10}; do $batch ${j} ${j}.txt done; echo "Pausing for next iteration"; echo... (6 Replies)
Discussion started by: busyboy
6 Replies

2. Shell Programming and Scripting

Make background process interact with fg process

Hi, I have written a menu driven shell script in which as per the choice, I run the another script on background. For eg: 1. get info 2)process info 3)modify info All the operations have different scripts which i schedule in background using &. However I wish to display the error... (0 Replies)
Discussion started by: ashima jain
0 Replies

3. Shell Programming and Scripting

Cannot make pipe for process substitution: Too many open files

Hi, I've came across an issue with a script I've been writing to check DHCP addresses on an Solaris system, the script has been running reasonably well, until it hit the following problem: ./sub_mon_v2: redirection error: cannot duplicate fd: Too many open files ./sub_mon_v2: cannot make... (3 Replies)
Discussion started by: CiCa
3 Replies

4. Shell Programming and Scripting

Executing a batch of files within a shell script with option to refire the individual files in batch

Hello everyone. I am new to shell scripting and i am required to create a shell script, the purpose of which i will explain below. I am on a solaris server btw. Before delving into the requirements, i will give youse an overview of what is currently in place and its purpose. ... (2 Replies)
Discussion started by: goddevil
2 Replies

5. Shell Programming and Scripting

Processing different jobs as a batch process

Hi All, I want to process consecutive jobs in a sequence but when I execute 1 job ,the control does not return to the command prompt to continue with the next job. Can anyone help me here? Thanks (3 Replies)
Discussion started by: Taranjeet Singh
3 Replies

6. Shell Programming and Scripting

How to make the parent process to wait for the child process

Hi All, I have two ksh script. 1st script calls the 2nd script and the second script calls an 'C' program. I want 1st script to wait until the 'C' program completes. I cant able to get the process id for the 'C' program (child process) to make the 1st script to wait for the second... (7 Replies)
Discussion started by: sennidurai
7 Replies

7. Shell Programming and Scripting

how to start a process and make it sleep for 5 mins and then kill that process

how to start a process and make it sleep for 5 mins and then kill that process (6 Replies)
Discussion started by: shrao
6 Replies

8. Shell Programming and Scripting

User Prompt during batch script process

The script below performs an incremental database backup. When the increment backup is out of sequence, the process prompts the user: "Incremental backup out of sequece. Do you wish to continue? Y or N". This script is set to run as a scheduled process in background mode. When the script... (2 Replies)
Discussion started by: bond007jlv
2 Replies

9. Shell Programming and Scripting

trying to read batch process but need some command help

I am trying to access batch process that take place each nite. I am using Solaris 5.8 (and i am used to redhat). however I am trying to access say a certain directory. The home/oradev , is the directory...in there i am trying to access say a batch file within this, how can see if they are in... (1 Reply)
Discussion started by: etravels
1 Replies
Login or Register to Ask a Question