Using Make to batch process files


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Question Using Make to batch process files

Hello all,

I have a make question, and I was hoping somebody here might be able to point me in the right direction.

Here is my issue; I have a command-line tool that I use to run a conversion on an input XML file that results in an output binary file. However, this particular tool needs to load up an overarching project file when it starts, and the time involved in that initial parsing is significant. Therefore, if I have 100 XML files that need processing, I waste a lot of time starting the tool over and over.

I like to use make for this kind of processing, because the dependency handling is so good. However, in this instance I would like to send all of the XML files that need processing to the tool in a single batch. Typically my make rules look like this:

Code:
%.bin : %.xml
           tool -export %.xml

... but this of course will run "tool" once for each XML file encountered in a previous dependency. I had considered this:

Code:
$(ALL_BIN) : $(ALL_XML)
                 tool -export $(ALL_XML)

... which will batch up the job, but it will rebuild every binary file if just a single XML file changes. Does anybody know a way that I keep make's dependency checking on a per-file basis, but run the actual command only once? Thanks much for any advice!

Moderator's Comments:
Mod Comment
Please use code tags when posting data and code samples!

Last edited by vgersh99; 11-19-2010 at 11:51 AM.. Reason: code tags, please!
# 2  
Working in multiple jobs is generally preferred -- this lets make split the work across multiple cores, meaning, lets it do more at once. If you actually have multiple cores, you may find it faster to let it continue this way and run make -j2, or -j4 etc.

On the other hand there is a macro for "dependencies more recent than the target": $? So this could do it:
Code:
$(ALL_BIN) : $(ALL_XML)
        tool -export $?

Note that the eight spaces in front of tool there must actually be a tab for this rule to work.
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Start process on X number of files and then wait for the next batch
busyboy
Thanks for RudiC for his extraordinary help on organizing files in a batch of 10 using below code. FL=($(ls)); for ((i=0;i<=${#FL};i++)); do for j in ${FL:$i:10}; do $batch ${j} ${j}.txt done; echo "Pausing for next iteration"; echo...... Shell Programming and Scripting
6
Shell Programming and Scripting
Cannot make pipe for process substitution: Too many open files
CiCa
Hi, I've came across an issue with a script I've been writing to check DHCP addresses on an Solaris system, the script has been running reasonably well, until it hit the following problem: ./sub_mon_v2: redirection error: cannot duplicate fd: Too many open files ./sub_mon_v2: cannot make...... Shell Programming and Scripting
3
Shell Programming and Scripting
Executing a batch of files within a shell script with option to refire the individual files in batch
goddevil
Hello everyone. I am new to shell scripting and i am required to create a shell script, the purpose of which i will explain below. I am on a solaris server btw. Before delving into the requirements, i will give youse an overview of what is currently in place and its purpose. ...... Shell Programming and Scripting
2
Shell Programming and Scripting
How to make the parent process to wait for the child process
sennidurai
Hi All, I have two ksh script. 1st script calls the 2nd script and the second script calls an 'C' program. I want 1st script to wait until the 'C' program completes. I cant able to get the process id for the 'C' program (child process) to make the 1st script to wait for the second...... Shell Programming and Scripting
7
Shell Programming and Scripting
how to start a process and make it sleep for 5 mins and then kill that process
shrao
how to start a process and make it sleep for 5 mins and then kill that process... Shell Programming and Scripting
6
Shell Programming and Scripting

Featured Tech Videos