Sponsored Content
Top Forums Shell Programming and Scripting How to make parallel execution on folder ? Post 302922775 by drl on Tuesday 28th of October 2014 08:48:19 AM
Old 10-28-2014
Hi.

One technique for parallelizing processing on large files is to use GNU parallel. If you are interested in re-arranging the order of fields in a file, say with cut or awk, then parallel can create a sequence of processes to do this in chunks.

Here's an example from the man page:
Code:
EXAMPLE: Processing a big file using more cores
       To process a big file or some output you can use --pipe to split up the
       data into blocks and pipe the blocks into the processing program.

       If the program is gzip -9 you can do:

       cat bigfile | parallel --pipe --recend '' -k gzip -9 >bigfile.gz

       This will split bigfile into blocks of 1 MB and pass that to gzip -9 in
       parallel. One gzip will be run per CPU core. The output of gzip -9 will
       be kept in order and saved to bigfile.gz

       gzip works fine if the output is appended, but some processing does not
       work like that - for example sorting. For this GNU parallel can put the
       output of each command into a file. This will sort a big file in
       parallel:

       cat bigfile | parallel --pipe --files sort | parallel -Xj1 sort -m {}
       ';' rm {} >bigfile.sort

       Here bigfile is split into blocks of around 1MB, each block ending in
       '\n' (which is the default for --recend). Each block is passed to sort
       and the output from sort is saved into files. These files are passed to
       the second parallel that runs sort -m on the files before it removes
       the files. The output is saved to bigfile.sort.

Note that it covers two cases, the second being where the process cannot do simple appending.

Best wishes ... cheers, drl
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel Execution

Hello, I wish to run parallel process forked from one script. Currently I doing is submit them in background. For example: --------------------------------------------------------------- #!/usr/bin/ksh process1 & process2 & process3 & ..... ..... #here I check for completion of... (4 Replies)
Discussion started by: RishiPahuja
4 Replies

2. AIX

Make : parallel execution

Hi, Is there any way to run parallel jobs using make command? I am using non-GNU make utility on AIX 4.3. I want to run 2 jobs symultaneously using make utility. Thanks. Suman (0 Replies)
Discussion started by: suman_jakkula
0 Replies

3. Shell Programming and Scripting

Job Execution in Parallel and Maintain Dependency

All, We need to run multiple jobs in parallel with the dependencies intact. I was trying to use the same with &, but in vain. Is there is any other way to resolve this issue. I need to run Set1 A.sh B.sh C.sh Set2 D.sh E.sh Set3 F.sh (10 Replies)
Discussion started by: anand_msm
10 Replies

4. High Performance Computing

Parallel Execution on Multiple System

Hi All, I am working on a project where I need to execute set of arguments (around 500) on a Simulator. If I execute this on one linux (RedHat 8.0) machine it will approximately takes 2~3 days. Since I am having more linux machines am thinking of executing these on different machines in... (7 Replies)
Discussion started by: 123an
7 Replies

5. Shell Programming and Scripting

Parallel Job Execution

Hi All, At present I am using a UNIX Script which is running a set of JOBS. These JOBS are to be repeated for 20 times, means the same set of JOBS are repeated by just passing different arguments (From 1 to 20). Is there any way by which I can execute them parallel? At present its all... (4 Replies)
Discussion started by: Prashantckc
4 Replies

6. Shell Programming and Scripting

Conditional execution and parallel jobs

how can i process jobs parallel with conditions below. Script1.ksh Script2.ksh Script3.ksh Script4.ksh Script5.ksh Script6.ksh Script7.ksh Script8.ksh Script9.ksh Script10.ksh After successful completion of Script1.ksh I need to run Script7.ksh. After successful... (4 Replies)
Discussion started by: ford2020
4 Replies

7. Shell Programming and Scripting

Parallel execution of script not syncronize

I am haveing 2 scripts, 1st script calls 2ed script for each parameter.(parameters are kept in a different txt file) 1st script for x in `cat Export_Tables_List.txt` do sh Exp_Table.sh $x & done echo -e "1) following tables are successfully exported : \n" > temp cat... (1 Reply)
Discussion started by: sbmk_design
1 Replies

8. Programming

Parallel Execution of Programs

Since there've been a few requests for a method to execute commands on multiple CPUs (logical or physical), with various levels of shell-, make-, or Perl-based solutions, ranging from well-done to well-meant, and mostly specific to a certain problem, I've started to write a C-based solution... (4 Replies)
Discussion started by: pludi
4 Replies

9. UNIX for Advanced & Expert Users

Parallel Execution of Command

Hi All, We have a table that has to store around 80-100 million records. The table is partitioned by a column called Market Code. There are 30 partitions each corresponding to one MRKT_CD. The source of this table is a join between 3-4 other tables. We are loading this table through SQLPLUS... (2 Replies)
Discussion started by: jerome_rajan
2 Replies

10. Homework & Coursework Questions

Parallel execution on multiple servers in UNIX

I have a requirement (in a shell script) to connect to several servers at once and execute a series of commands. I am aware that ssh can be used for sequential execution. But since most of the commands that I need to execute take a long time, I have to go for the parallel option. Is there... (2 Replies)
Discussion started by: sneha1887
2 Replies
PSORT(3)						   BSD Library Functions Manual 						  PSORT(3)

NAME
psort, psort_b, psort_r -- parallel sort functions SYNOPSIS
#include <stdlib.h> void psort(void *base, size_t nel, size_t width, int (*compar)(const void *, const void *)); void psort_b(void *base, size_t nel, size_t width, int (^compar)(const void *, const void *)); void psort_r(void *base, size_t nel, size_t width, void *thunk, int (*compar)(void *, const void *, const void *)); DESCRIPTION
The psort(), psort_b(), and psort_r() functions are parallel sort routines that are drop-in compatible with the corresponding qsort() func- tion (see qsort(3) for a description of the arguments). On multiprocessor machines, multiple threads may be created to simultaneously per- form the sort calculations, resulting in an overall faster sort result. Overhead in managing the threads limits the maximum speed improve- ment to somewhat less that the number of processors available. For example, on a 4-processor machine, a typical sort on a large array might result in 3.2 times faster sorting than a regular qsort(). RESTRICTIONS
Because of the multi-threaded nature of the sort, the comparison function is expected to perform its own synchronization that might be required for data physically outside the two objects passed to the comparison function. However, no synchronization is required for the two object themselves, unless some third party is also accessing those objects. Additional memory is temporary allocated to deal with the parallel nature of the computation. Because of the overhead of maintaining multiple threads, the psort() family of routines may choose to just call qsort(3) when there is no advantage to parallelizing (for example, when the number of objects in the array is too small, or only one processor is available). Like qsort(3), the sort is not stable. RETURN VALUES
The psort(), psort_b() and psort_r() functions return no value. SEE ALSO
qsort(3) Mac OS X Nov 25, 2008 Mac OS X
All times are GMT -4. The time now is 02:26 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy