Parallelize a task that have for

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parallelize a task that have for
# 1  
Old 12-15-2011
Parallelize a task that have for

Dear all,
I'm a newbie in programming and I would like to know if it is possible to parallelize the script:
for l in {1..1000}
cut -f$l quase2 |tr "\n" "," |sed 's/$/\
/g' |sed '/^$/d' >a_$l.t

I tried:
for l in {1..1000}
cut -f$l quase2 |tr "\n" "," |sed 's/$/\
/g' |sed '/^$/d' >a_$l.t &

But it was showed a message "fork: Resource temporarily unavailable"

Is it possible to do something? I need to do this because the number of columns of the real file is too long. So, the work it will be impossible to do in the current way.

Thanks in advance
Moderator's Comments:
Mod Comment
Please use code tags when posting data and code samples!

Last edited by vgersh99; 12-15-2011 at 07:02 PM.. Reason: code tags, please
# 2  
Old 12-15-2011
You seem to have run out of resource (system...) what kind of box is this yaou are working on? It looks like you will have to do some kernel tuning... Cant say more not knowing the OS more the kernel system parameters.
# 3  
Old 12-15-2011
As you are running 1000 instances, you should consider optimising the code within the process as much as possible.

You could probably use 1 awk script in the place of your 2 x sed + cut + tr subscript. This would reduce the number of running processes working on the task from around 4000 to 1000.

Edit: perhaps something like this:
awk -v F=$l '{ printf $F"," } END { print ""} ' infile > a_$l.t

Last edited by Chubler_XL; 12-15-2011 at 09:09 PM..
# 4  
Old 12-15-2011
Thanks for the information vbe.
I'm using a computer with a quad core processor, 12 GB of RAM and MAC OSX of OS.
The real file has 4 GB and 40,000,000 of columns.

Is it enough to help me?

Thanks for the attention.
# 5  
Old 12-15-2011
Your task is pretty easy to split up into smaller sub tasks so you should try running 10 background jobs and see how it utilises your resources. If your system is still pretty idle increase to 20.

This is much better than running 1000's of tasks when you only have 16 cores as system control overheads (swapping processes in and out, etc) will take most of the resources and not leave much for the actual work.

doit() {
while [ $l -le $2 ]
    awk -v F=$l '{ printf $F"," } END { print ""} ' infile > a_$l.t
    let l=l+1
doit 1 100 &
doit 101 200 &
doit 201 300 &
doit 301 400 &
doit 401 500 &
doit 501 600 &
doit 601 700 &
doit 701 800 &
doit 801 900 &
doit 901 1000 &

Last edited by Chubler_XL; 12-16-2011 at 12:06 AM..
# 6  
Old 12-16-2011
Its not a MacOSX server OS isn't it ?
What version?
# 7  
Old 12-16-2011
I know there is a conf file /etc/sysctl* in macosx 10.3 server but I am completely novice... and trying to find on my powerbook... You will have to use sysctl command

While I am at it I will explain my thoughts: Not being able to fork usually happen when you reach a system parameter limit, I think of two possibles: max number of process for whole system or max process/user, their names differ between OS, on HP-UX we have maxuprc and nproc, for mac you would have to search a bit...
If you wish, we could move this to more suitable forum (MacOSX...)

Last edited by vbe; 12-16-2011 at 07:37 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallelize bash commands/jobs

Hello, I have a bunch of jobs (cp, cat or ln -s) on big files (10xGB in size): # cp file1 path1/newfile1 cp file2 path1/newfile2 cp file3 path1/newfile3 ...... cat file11 path2/file21 path1/newfile11 cat file12 path2/file22 path1/newfile12 cat file13 path2/file23... (5 Replies)
Discussion started by: yifangt
5 Replies

2. Homework & Coursework Questions

[HELP] Easy task

I have a simple task for my school work. I'm new with unix, so i need help. I need to write a scenario. Task is. From created txt file read first 3 words and create a 3 catalogs with those 3 words. 2 of those new catalogs should be transferred to other directory. If someone could help me just... (1 Reply)
Discussion started by: justynykas
1 Replies

3. Shell Programming and Scripting

Can Anyone help do my task in simple way..

Hi... I am trying extract data using 2 files.. 1. Data file - DATA.TXT containing data to be extracted 2. Column file - LIST.TXT header column to be extracted output not coming properly.. if the header field name matches with data file header extract data...this actually I wanted ... ... (7 Replies)
Discussion started by: nex_asp
7 Replies

4. Shell Programming and Scripting

need help with an easy task

Hello everyone my name is Telis and i just registered in this forum.. i just started programming in linux shell and i need help with this easy task.. this is my code #!/bin/sh echo "What is your First name?" read name if ; then echo "Hello Master." else echo "Who are you?" fi when i... (2 Replies)
Discussion started by: Telis
2 Replies

5. Shell Programming and Scripting


Hi experts, I have a problem with the below shell task: I need to modify the file creatin a paired row , per each row which matches filter (e.g. number of nonempty columns = 5) Output should look like this: second row is original one from the input, first row(red) is pairing row, it's... (29 Replies)
Discussion started by: hernand
29 Replies

6. Shell Programming and Scripting


Hi all, I'm newbie and stuck here. Thanks for any help. Input(txt file) a b X c d Y e f Z g h W Requested output: a b X Y c d Y X e f Z W g h W Z Please use code tags when posting data and code samples! (10 Replies)
Discussion started by: hernand
10 Replies

7. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores... (10 Replies)
Discussion started by: vhope07
10 Replies

8. Shell Programming and Scripting

Parse an XML task list to create each task.xml file

I have an task definition listing xml file that contains a list of tasks such as <TASKLIST <TASK definition="Completion date" id="Taskname1" Some other <CODE name="Code12" <Parameter pname="Dog" input="5.6" units="feet" etc /Parameter> <Parameter... (3 Replies)
Discussion started by: MissI
3 Replies

9. Shell Programming and Scripting

comment and Uncomment single task out of multiple task

I have a file contains TASK gsnmpproxy { CommandLine = $SMCHOME/bin/gsnmpProxy.exe } TASK gsnmpdbgui { CommandLine = $SMCHOME/bin/gsnmpdbgui.exe I would like to comment and than uncomment specific task eg TASK gsnmpproxy Pls suggest how to do in shell script (9 Replies)
Discussion started by: madhusmita
9 Replies
Login or Register to Ask a Question

Featured Tech Videos