Making use of multiple cores for running sed and awk scripts
Hi All,
After reading that the sort command in Linux can be made to use many processor cores just by using a simple script which I found on the internet, I was wondering if I can use similar techniques for programs like the awk and sed?
Code:
#!/bin/bash
# Usage: psort filename <chunksize> <threads>
# In this example a the file largefile is split into chunks of 20 MB.
# The part are sorted in 4 simultaneous threads before getting merged.
#
# psort largefile.txt 20m 4
#
# by h.p.
split -b $2 $1 $1.part
suffix=sorttemp.`date +%s`
nthreads=$3
i=0
for fname in `ls *$1.part*`
do
let i++
sort $fname > $fname.$suffix &
mres=$(($i % $nthreads))
test "$mres" -eq 0 && wait
done
wait
sort -m *.$suffix
rm $1.part*
Previously, I used to use sort without using the above script and it used to take several minutes to sort a very large file. By default sort command only uses one core of the processor.
My school has just purchased a 16 core server with Linux and 96 GB RAM, so I am currently fiddling with it.
Now, a thought comes to my mind: Can sed and awk be used in the same way so that they make use of all the 16 cores of the processor?
I ask this because once I tried to fiddle with a huge Wikipedia file dump which I downloaded from the internet. The XML file is 30 GB in size and contains some 3.5 million articles.
I then ran this script in order to parse the individual articles and store them in separate files:
Code:
awk '/<page>/{c++}{print > c ".dat"}' wikipedia_dump.xml
To my horror, it took about 10-12 days to complete the task. I am wondering, if it is possible to use awk in such a way that it could use all the cores of the processor and run in a multi-threaded fashion? I ran the above awk script on the same new server running Linux.
Hello, I've tried for a while now to run a bash script that continues to the end, while opening new shells as needed.
I've tried
xterm -e "somecommand"; &
xterm -e " somecommand";
I've also tried
screen -S "somecommand"; &
screen -S "somecommand";
All without any luck, they... (5 Replies)
Hi all I have a requirement where I have a flow like
Script1
script2 Script3 Script 4 Script 5 Script 6
script7
where script2 to script6 will... (3 Replies)
Hi
I have a shell script A which calls another 10 shell scripts which run in background. How do i make the parent script wait for the child scripts complete, or in other words, i must be able to do a grep of parent script to find out if the child scripts are still running.
My Code:
... (5 Replies)
Hi,
I have created 3 shell scripts which has to run one by one first two shell scripts will create a .txt files...which are used by the third shell script.Now I want to create a master script and run all these in a single script.
Please give a pseudo code on how to so the same.
... (4 Replies)
I want to make the first character of some words to be uppercase. I have a file like the one below.
uid,givenname,sn,cn,mail,telephonenumber
mattj,matt,johnson,matt johnson,mattj@gmail.com
markv,mark,vennet,matt s vennet,markv@gmail.com
mikea,mike,austi,mike austin,mike@gmail.com
I want... (3 Replies)
Hi, I was hoping that someone could help me. I have a problem that i am trying to work on and it requires me to change text within multiple files using sed. I use the program to change an occurance of a word throughout different files that are being tested. At first i had to Create a new script,... (1 Reply)
hi all,
I have 3 individual scripts to perform the task . 2nd script should run only after the 1st script and 3rd script must run only after first 2 scripts are executed successfully.
i want to have a single script that calls all this 3 scripts .this single script should execute the 2nd script... (1 Reply)
Hi
I probably dont have GNU extended sed in my SUNOS . and its creating lot of problems
ex:
a simple sed command like this is not working
sed '/WORD/ a\
sample text line 1 \
sample text line 1
' filename
sed: command garbled: /WORD/ a
I took precaution to have a new line after... (11 Replies)
Hi Team,
I have the below 4 scripts which I will be running in sequential order.
This run will start for today's business date.
If all the 4 scripts are success for today that means script has ran succesfully.
Howver if any one of these 4 scripts failed then it has to take the next... (1 Reply)
Hello!
I have a scriptA.ksh and in this script I need to call script1.ksh, script2.ksh, script3.ksh, script4.ksh and script5.ksh. But want to run in two batches like
1st script1.ksh, script2.ksh, script3.ksh, once all 3 are completed then
script4.ksh script5.ksh
I have given the syntax... (1 Reply)
Discussion started by: karumudi7
1 Replies
LEARN ABOUT OSX
psort_r
PSORT(3) BSD Library Functions Manual PSORT(3)NAME
psort, psort_b, psort_r -- parallel sort functions
SYNOPSIS
#include <stdlib.h>
void
psort(void *base, size_t nel, size_t width, int (*compar)(const void *, const void *));
void
psort_b(void *base, size_t nel, size_t width, int (^compar)(const void *, const void *));
void
psort_r(void *base, size_t nel, size_t width, void *thunk, int (*compar)(void *, const void *, const void *));
DESCRIPTION
The psort(), psort_b(), and psort_r() functions are parallel sort routines that are drop-in compatible with the corresponding qsort() func-
tion (see qsort(3) for a description of the arguments). On multiprocessor machines, multiple threads may be created to simultaneously per-
form the sort calculations, resulting in an overall faster sort result. Overhead in managing the threads limits the maximum speed improve-
ment to somewhat less that the number of processors available. For example, on a 4-processor machine, a typical sort on a large array might
result in 3.2 times faster sorting than a regular qsort().
RESTRICTIONS
Because of the multi-threaded nature of the sort, the comparison function is expected to perform its own synchronization that might be
required for data physically outside the two objects passed to the comparison function. However, no synchronization is required for the two
object themselves, unless some third party is also accessing those objects.
Additional memory is temporary allocated to deal with the parallel nature of the computation.
Because of the overhead of maintaining multiple threads, the psort() family of routines may choose to just call qsort(3) when there is no
advantage to parallelizing (for example, when the number of objects in the array is too small, or only one processor is available).
Like qsort(3), the sort is not stable.
RETURN VALUES
The psort(), psort_b() and psort_r() functions return no value.
SEE ALSO qsort(3)Mac OS X Nov 25, 2008 Mac OS X
08-20-2011
