Sponsored Content
Top Forums Shell Programming and Scripting Parallel processing for functions in xargs Post 302944218 by tange on Sunday 17th of May 2015 01:39:54 PM
Old 05-17-2015
Use GNU Parallel

Install GNU Parallel and do this:
Code:
$ foo() { echo $*; }                                          

$ export fun="`typeset -f`"; parallel 'eval "$fun";'foo ::: works
works

$ export fun="`typeset -f`"; parallel 'eval "$fun";'foo :::: param_file


Last edited by tange; 05-17-2015 at 04:59 PM.. Reason: works in zsh, too, with "
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to do parallel processing??

Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a... (1 Reply)
Discussion started by: zing
1 Replies

2. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

3. Shell Programming and Scripting

Need Help With Parallel Processing

Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or... (6 Replies)
Discussion started by: imnewtothis23
6 Replies

4. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

5. Shell Programming and Scripting

Using xargs for multiple functions

Hi Experts, I am trying to parse some syslog outputs into a separate file per node using the below syntax but am having issues when it comes to my Xargs statements. The command which I was intending on using was: cat syslogs | nawk '/From/ { print $3 }' | uniq | xargs -I {} grep {}... (5 Replies)
Discussion started by: krypton
5 Replies

6. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

7. Programming

Algorithms for Parallel Processing

Hey, I just wanted to know how many algorithms there are that cannot be accelerated by parallel processing. I know one such algorithm is Euclid's Algorithm (for GCF). does anyone know any other algorithms that cannot be accelerated by pp? if so please list the names and a general sentence of what... (2 Replies)
Discussion started by: azar.zorn
2 Replies

8. Shell Programming and Scripting

Calling multiple functions in parallel

Hello, I have multiple functions within a shell script. eg. function_database_backup, unix_tar_creation, etc. I would like to run these functions in parallel, as each is independent of the other. If these were simple commands, I could have probably run each of the commands in background. ... (1 Reply)
Discussion started by: neil.k
1 Replies

9. Shell Programming and Scripting

Pass parameters to a function and running functions in parallel

Hi , I have a script which is using a text file as I/P. I want a code where it reads n lines from this file and pass the parameters to a function and now this script should run in such a way where a function can be called in parallel with different parameters. Please find below my script, it... (1 Reply)
Discussion started by: Ravindra Swan
1 Replies

10. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies
funref(3)							SAORD Documentation							 funref(3)

NAME
FunRef - the Funtools Reference Handle SYNOPSIS
A description of how to use a Funtools reference handle to connect a Funtools input file to an output file. DESCRIPTION
The Funtools reference handle connects a Funtools input file to a Funtools output file so that parameters (or even whole extensions) can be copied from the one to the other. To make the connection, the Funtools handle of the input file is passed to the final argument of the FunOpen() call for the output file: if( !(ifun = FunOpen(argv[1], "r", NULL)) ) gerror(stderr, "could not FunOpen input file: %s ", argv[1]); if( !(ofun = FunOpen(argv[2], "w", ifun)) ) gerror(stderr, "could not FunOpen output file: %s ", argv[2]); It does not matter what type of input or output file (or extension) is opened, or whether they are the same type. When the output image or binary table is written using FunImagePut() or FunTableRowPut() an appropriate header will be written first, with parameters copied from the input extension. Of course, invalid parameters will be removed first, e.g., if the input is a binary table and the output is an image, then binary table parameters such as TFORM, TUNIT, etc. parameters will not be copied to the output. Use of a reference handle also allows default values to be passed to FunImagePut() in order to write out an output image with the same dimensions and data type as the input image. To use the defaults from the input, a value of 0 is entered for dim1, dim2, and bitpix. For example: fun = FunOpen(argv[1], "r", NULL); fun2 = FunOpen(argv[2], "w", fun); buf = FunImageGet(fun, NULL, NULL); ... process image data ... FunImagePut(fun2, buf, 0, 0, 0, NULL); Of course, you often want to get information about the data type and dimensions of the image for processing. The above code is equivalent to the following: fun = FunOpen(argv[1], "r", NULL); fun2 = FunOpen(argv[2], "w", fun); buf = FunImageGet(fun, NULL, NULL); FunInfoGet(fun, FUN_SECT_DIM1, &dim1, FUN_SECT_DIM2, &dim2, FUN_SECT_BITPIX, &bitpix, 0); ... process image data ... FunImagePut(fun2, buf, dim1, dim2, bitpix, NULL); It is possible to change the reference handle for a given output Funtools handle using the FunInfoPut() routine: /* make the new extension the reference handle for the output file */ FunInfoPut(fun2, FUN_IFUN, &fun, 0); When this is done, Funtools specially resets the output file to start a new output extension, which is connected to the new input reference handle. You can use this mechanism to process multiple input extensions into a single output file, by successively opening the former and setting the reference handle for the latter. For example: /* open a new output FITS file */ if( !(fun2 = FunOpen(argv[2], "w", NULL)) ) gerror(stderr, "could not FunOpen output file: %s ", argv[2]); /* process each input extension in turn */ for(ext=0; ;ext++){ /* get new extension name */ sprintf(tbuf, "%s[%d]", argv[1], ext); /* open it -- if we cannot open it, we are done */ if( !(fun=FunOpen(tbuf, "r", NULL)) ) break; /* make the new extension the reference handle for the output file */ FunInfoPut(fun2, FUN_IFUN, &fun, 0); ... process ... /* flush output extension (write padding, etc.) */ FunFlush(fun2, NULL); /* close the input extension */ FunClose(fun); } In this example, the output file is opened first. Then each successive input extension is opened, and the output reference handle is set to the newly opened input handle. After data processing is performed, the output extension is flushed and the input extension is closed, in preparation for the next input extension. Finally, a reference handle can be used to copy other extensions from the input file to the output file. Copy of other extensions is con- trolled by adding a "C" or "c" to the mode string of the FunOpen() call of the input reference file. If "C" is specified, then other extensions are always copied (i.e., copy is forced by the application). If "c" is used, then other extensions are copied if the user requests copying by adding a plus sign "+" to the extension name in the bracket specification. For example, the funtable program utilizes user-specified "c" mode so that the second example below will copy all extensions: # copy only the EVENTS extension csh> funtable "test.ev[EVENTS,circle(512,512,10)]" foo.ev # copy ALL extensions csh> funtable "test.ev[EVENTS+,circle(512,512,10)]" foo.ev When extension copy is specified in the input file, the call to FunOpen() on the input file delays the actual file open until the output file also is opened (or until I/O is performed on the input file, which ever happens first). Then, when the output file is opened, the input file is also opened and input extensions are copied to the output file, up to the specific extension being opened. Processing of input and output extensions then proceed. When extension processing is complete, the remaining extensions need to be copied from input to output. This can be done explicitly, using the FunFlush() call with the "copy=remaining" plist: FunFlush(fun, "copy=remaining"); Alternatively, this will happen automatically, if the output file is closed before the input file: /* we could explicitly flush remaining extensions that need copying */ /* FunFlush(fun2, "copy=remaining"); */ /* but if we close output before input, end flush is done automatically */ FunClose(fun2); FunClose(fun); SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funref(3)
All times are GMT -4. The time now is 01:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy