One technique for parallelizing processing on large files is to use GNU parallel. If you are interested in re-arranging the order of fields in a file, say with cut or awk, then parallel can create a sequence of processes to do this in chunks.
Here's an example from the man page:
Note that it covers two cases, the second being where the process cannot do simple appending.
Hello,
I wish to run parallel process forked from one script.
Currently I doing is submit them in background.
For example:
---------------------------------------------------------------
#!/usr/bin/ksh
process1 &
process2 &
process3 &
.....
.....
#here I check for completion of... (4 Replies)
Hi,
Is there any way to run parallel jobs using make command?
I am using non-GNU make utility on AIX 4.3.
I want to run 2 jobs symultaneously using make utility.
Thanks.
Suman (0 Replies)
All,
We need to run multiple jobs in parallel with the dependencies intact.
I was trying to use the same with &, but in vain. Is there is any other way to resolve this issue.
I need to run
Set1
A.sh
B.sh
C.sh
Set2
D.sh
E.sh
Set3
F.sh (10 Replies)
Hi All,
I am working on a project where I need to execute set of arguments (around 500) on a Simulator. If I execute this on one linux (RedHat 8.0) machine it will approximately takes 2~3 days. Since I am having more linux machines am thinking of executing these on different machines in... (7 Replies)
Hi All,
At present I am using a UNIX Script which is running a set of JOBS. These JOBS are to be repeated for 20 times, means the same set of JOBS are repeated by just passing different arguments (From 1 to 20).
Is there any way by which I can execute them parallel?
At present its all... (4 Replies)
how can i process jobs parallel with conditions below.
Script1.ksh
Script2.ksh
Script3.ksh
Script4.ksh
Script5.ksh
Script6.ksh
Script7.ksh
Script8.ksh
Script9.ksh
Script10.ksh
After successful completion of Script1.ksh I need to run Script7.ksh.
After successful... (4 Replies)
I am haveing 2 scripts, 1st script calls 2ed script for each parameter.(parameters are kept in a different txt file)
1st script
for x in `cat Export_Tables_List.txt`
do
sh Exp_Table.sh $x &
done
echo -e "1) following tables are successfully exported : \n" > temp
cat... (1 Reply)
Since there've been a few requests for a method to execute commands on multiple CPUs (logical or physical), with various levels of shell-, make-, or Perl-based solutions, ranging from well-done to well-meant, and mostly specific to a certain problem, I've started to write a C-based solution... (4 Replies)
Hi All,
We have a table that has to store around 80-100 million records. The table is partitioned by a column called Market Code. There are 30 partitions each corresponding to one MRKT_CD.
The source of this table is a join between 3-4 other tables. We are loading this table through SQLPLUS... (2 Replies)
I have a requirement (in a shell script) to connect to several servers at once and execute a series of commands.
I am aware that ssh can be used for sequential execution. But since most of the commands that I need to execute take a long time, I have to go for the parallel option.
Is there... (2 Replies)
Discussion started by: sneha1887
2 Replies
LEARN ABOUT MOJAVE
psort
PSORT(3) BSD Library Functions Manual PSORT(3)NAME
psort, psort_b, psort_r -- parallel sort functions
SYNOPSIS
#include <stdlib.h>
void
psort(void *base, size_t nel, size_t width, int (*compar)(const void *, const void *));
void
psort_b(void *base, size_t nel, size_t width, int (^compar)(const void *, const void *));
void
psort_r(void *base, size_t nel, size_t width, void *thunk, int (*compar)(void *, const void *, const void *));
DESCRIPTION
The psort(), psort_b(), and psort_r() functions are parallel sort routines that are drop-in compatible with the corresponding qsort() func-
tion (see qsort(3) for a description of the arguments). On multiprocessor machines, multiple threads may be created to simultaneously per-
form the sort calculations, resulting in an overall faster sort result. Overhead in managing the threads limits the maximum speed improve-
ment to somewhat less that the number of processors available. For example, on a 4-processor machine, a typical sort on a large array might
result in 3.2 times faster sorting than a regular qsort().
RESTRICTIONS
Because of the multi-threaded nature of the sort, the comparison function is expected to perform its own synchronization that might be
required for data physically outside the two objects passed to the comparison function. However, no synchronization is required for the two
object themselves, unless some third party is also accessing those objects.
Additional memory is temporary allocated to deal with the parallel nature of the computation.
Because of the overhead of maintaining multiple threads, the psort() family of routines may choose to just call qsort(3) when there is no
advantage to parallelizing (for example, when the number of objects in the array is too small, or only one processor is available).
Like qsort(3), the sort is not stable.
RETURN VALUES
The psort(), psort_b() and psort_r() functions return no value.
SEE ALSO qsort(3)SEE ALSO sort(1), radixsort(3)
Hoare, C.A.R., "Quicksort", The Computer Journal, 5:1, pp. 10-15, 1962.
Williams, J.W.J, "Heapsort", Communications of the ACM, 7:1, pp. 347-348, 1964.
Knuth, D.E., "Sorting and Searching", The Art of Computer Programming, Vol. 3, pp. 114-123, 145-149, 1968.
McIlroy, P.M., "Optimistic Sorting and Information Theoretic Complexity", Fourth Annual ACM-SIAM Symposium on Discrete Algorithms, January
1992.
Bentley, J.L. and McIlroy, M.D., "Engineering a Sort Function", Software--Practice and Experience, Vol. 23(11), pp. 1249-1265,
November 1993.
STANDARDS
The qsort() function conforms to ISO/IEC 9899:1990 (``ISO C90'').
Mac OS X Nov 25, 2008 Mac OS X