Sponsored Content
Top Forums Shell Programming and Scripting Parallelize a task that have for Post 302582412 by Chubler_XL on Thursday 15th of December 2011 09:59:07 PM
Old 12-15-2011
Your task is pretty easy to split up into smaller sub tasks so you should try running 10 background jobs and see how it utilises your resources. If your system is still pretty idle increase to 20.

This is much better than running 1000's of tasks when you only have 16 cores as system control overheads (swapping processes in and out, etc) will take most of the resources and not leave much for the actual work.

Code:
doit() {
l=$1
while [ $l -le $2 ]
do
    awk -v F=$l '{ printf $F"," } END { print ""} ' infile > a_$l.t
    let l=l+1
done
}
 
doit 1 100 &
doit 101 200 &
doit 201 300 &
doit 301 400 &
doit 401 500 &
doit 501 600 &
doit 601 700 &
doit 701 800 &
doit 801 900 &
doit 901 1000 &


Last edited by Chubler_XL; 12-15-2011 at 11:06 PM..
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

comment and Uncomment single task out of multiple task

I have a file contains TASK gsnmpproxy { CommandLine = $SMCHOME/bin/gsnmpProxy.exe } TASK gsnmpdbgui { CommandLine = $SMCHOME/bin/gsnmpdbgui.exe I would like to comment and than uncomment specific task eg TASK gsnmpproxy Pls suggest how to do in shell script (9 Replies)
Discussion started by: madhusmita
9 Replies

2. Shell Programming and Scripting

Parse an XML task list to create each task.xml file

I have an task definition listing xml file that contains a list of tasks such as <TASKLIST <TASK definition="Completion date" id="Taskname1" Some other <CODE name="Code12" <Parameter pname="Dog" input="5.6" units="feet" etc /Parameter> <Parameter... (3 Replies)
Discussion started by: MissI
3 Replies

3. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores... (10 Replies)
Discussion started by: vhope07
10 Replies

4. Shell Programming and Scripting

task

Hi all, I'm newbie and stuck here. Thanks for any help. Input(txt file) a b X c d Y e f Z g h W Requested output: a b X Y c d Y X e f Z W g h W Z Please use code tags when posting data and code samples! (10 Replies)
Discussion started by: hernand
10 Replies

5. Shell Programming and Scripting

Task

Hi experts, I have a problem with the below shell task: I need to modify the file creatin a paired row , per each row which matches filter (e.g. number of nonempty columns = 5) Output should look like this: second row is original one from the input, first row(red) is pairing row, it's... (29 Replies)
Discussion started by: hernand
29 Replies

6. Shell Programming and Scripting

need help with an easy task

Hello everyone my name is Telis and i just registered in this forum.. i just started programming in linux shell and i need help with this easy task.. this is my code #!/bin/sh echo "What is your First name?" read name if ; then echo "Hello Master." else echo "Who are you?" fi when i... (2 Replies)
Discussion started by: Telis
2 Replies

7. Shell Programming and Scripting

Can Anyone help me..to do my task in simple way..

Hi... I am trying extract data using 2 files.. 1. Data file - DATA.TXT containing data to be extracted 2. Column file - LIST.TXT header column to be extracted output not coming properly.. if the header field name matches with data file header extract data...this actually I wanted ... ... (7 Replies)
Discussion started by: nex_asp
7 Replies

8. Homework & Coursework Questions

[HELP] Easy task

I have a simple task for my school work. I'm new with unix, so i need help. I need to write a scenario. Task is. From created txt file read first 3 words and create a 3 catalogs with those 3 words. 2 of those new catalogs should be transferred to other directory. If someone could help me just... (1 Reply)
Discussion started by: justynykas
1 Replies

9. Shell Programming and Scripting

Parallelize bash commands/jobs

Hello, I have a bunch of jobs (cp, cat or ln -s) on big files (10xGB in size): # commands_script.sh: cp file1 path1/newfile1 cp file2 path1/newfile2 cp file3 path1/newfile3 ...... cat file11 path2/file21 path1/newfile11 cat file12 path2/file22 path1/newfile12 cat file13 path2/file23... (5 Replies)
Discussion started by: yifangt
5 Replies
CONTROL.CTL(5)							File Formats Manual						    CONTROL.CTL(5)

NAME
control.ctl - specify handling of Usenet control messages DESCRIPTION
The file /etc/news/control.ctl is used to determine what action is taken when a control message is received. It is read by the parsecon- trol script, which is called by all the control scripts. (For an explanation of how the control scripts are invoked, see innd(8).) The file consists of a series of lines; blank lines and lines beginning with a number sign (``#'') are ignored. All other lines consist of four fields separated by a colon: message:from:newsgroups:action The first field is the name of the message for which this line is valid. It should be either the name of the control message, or the word ``all'' to mean that it is valid for all messages. The second field is a shell-style pattern that matches the email address of the person posting the message. (The poster's address is first converted to lowercase.) The matching is done using the shell's case statement; see sh (1) for details. If the control message is ``newgroup'' or ``rmgroup'' then the third field specifies the shell-style pattern that must match the group being created or removed. If the control message is of a different type, then this field is ignored. The fourth field specifies what action to take if this line is selected for the message. The following actions are understood: doit The action requested by the control message should be performed. In most cases the control script will also send mail to usenet. doifarg If the control message has an argument, this is treated as a ``doit'' action. If no argument was given, it is treated as a ``mail'' entry. This is used in ``sendsys'' entries script so that a site can request its own newsfeeds(5) entry by posting a ``sendsys mysite'' article. On the other hand, sendsys ``bombs'' ask that the entire newsfeeds file be sent to a forged reply-to address; by using ``doifarg'' such messages will not be processed automatically. doit=file The action is performed, but a log entry is written to the specified log file, file. If file is the word ``mail'' then the record is mailed. A null string is equivalent to /dev/null. A pathname that starts with a slash is taken as the absolute filename to use as the log. All other pathnames are written to /var/log/news/file.log. The log is written by writelog (see newslog(8)). drop No action is taken; the message is ignored. log A one-line log notice is sent to standard error. Innd normally directs this to the file /var/log/news/errlog. log=file A log entry is written to the specified log file, file, which is interpreted as described above. mail A mail message is sent to the news administrator. Lines are matched in order; the last match found in the file is the one that is used. For example, with the following three lines: newgroup:*:*:drop newgroup:tale@*.uu.net:comp.*|misc.*|news.*|rec.*|sci.*|soc.*|talk.*:doit newgroup:kre@munnari.oz.au:aus.*:mail A newgroup coming from ``tale'' at a UUNET machine will be honored if it is in the mainstream Usenet hierarchy. If ``kre'' posts a new- group message creating ``aus.foo'', then mail will be sent. All other newgroup messages are ignored. HISTORY
Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews. This is revision 1.8, dated 1996/09/06. SEE ALSO
innd(8), newsfeeds(5), scanlogs(8). CONTROL.CTL(5)
All times are GMT -4. The time now is 06:22 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy