Sponsored Content
Top Forums Shell Programming and Scripting Parallelize bash commands/jobs Post 302974551 by stomp on Tuesday 31st of May 2016 03:12:45 PM
Old 05-31-2016
Hi,

the script example looks very basic. Hard to believe this can not be compacted in several ways. Maybe you show us the real script, so there is a better idea of what you are doing.

Thought's so far

If you're on a multi-user-system it may be not such a good idea to parallel i/o-heavy processes because you may completely eat up the available i/o of your machine and cause extreme loads at within your server. This could slow down the whole server very greatly. I would suggest you keep the i/o tasks sequentially.

The "ln"-commands may be executed sequentially too, because they should be so fast, it should be a duration that is hardly noticed to create some thousand symlinks / hardlinks. (1000 Links in one second on my 10 year old desktop machine).

As I can see from your small snippet, cpu-power is not required here at all, but only raw i/o so you will not profit from parallel executing your script.

Last edited by stomp; 05-31-2016 at 05:23 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

background jobs exit status and limit the number of jobs to run

i need to execute 5 jobs at a time in background and need to get the exit status of all the jobs i wrote small script below , i'm not sure this is right way to do it.any ideas please help. $cat run_job.ksh #!/usr/bin/ksh #################################### typeset -u SCHEMA_NAME=$1 ... (1 Reply)
Discussion started by: GrepMe
1 Replies

2. Shell Programming and Scripting

Can BASH execute commands on a remote server when the commands are embedded in shell

I want to log into a remote server transfer over a new config and then backup the existing config, replace with the new config. I am not sure if I can do this with BASH scripting. I have set up password less login by adding my public key to authorized_keys file, it works. I am a little... (1 Reply)
Discussion started by: bash_in_my_head
1 Replies

3. Shell Programming and Scripting

commands to be executed in order for a batch jobs!

Hi All, I am trying to run this script. I have a small problem: each "./goada.sh" command when done produces three files (file1, file2, file3) then they are moved to their respective directory as can be seem from this script snippet here. The script goada.sh sends some commands for some... (1 Reply)
Discussion started by: faizlo
1 Replies

4. Shell Programming and Scripting

General Q: how to run/schedule a php script from cron jobs maybe via bash from shell?

Status quo is, within a web application, which is coded completely in php (not by me, I dont know php), I have to fill out several fields, and execute it manually by clicking the "go" button in my browser, several times a day. Thats because: The script itself pulls data (textfiles) from a... (3 Replies)
Discussion started by: lowmaster
3 Replies

5. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores... (10 Replies)
Discussion started by: vhope07
10 Replies

6. Shell Programming and Scripting

waiting on jobs in bash, allowing limited parallel jobs at one time, and then for all to finish

Hello, I am running GNU bash, version 3.2.39(1)-release (x86_64-pc-linux-gnu). I have a specific question pertaining to waiting on jobs run in sub-shells, based on the max number of parallel processes I want to allow, and then wait... (1 Reply)
Discussion started by: srao
1 Replies

7. Shell Programming and Scripting

Parallelize a task that have for

Dear all, I'm a newbie in programming and I would like to know if it is possible to parallelize the script: for l in {1..1000} do cut -f$l quase2 |tr "\n" "," |sed 's/$/\ /g' |sed '/^$/d' >a_$l.t done I tried: for l in {1..1000} do cut -f$l quase2 |tr "\n" "," |sed 's/$/\ /g' |sed... (7 Replies)
Discussion started by: valente
7 Replies

8. Shell Programming and Scripting

Bash scripts as commands

Hello, the bulk of my work is run by scripts. An example is as such: #!/bin/bash awk '{print first line}' Input.in > Intermediate.ter awk '{print second line}' Input.in > Intermediate_2.ter command Intermediate.ter Intermediate_2.ter > Output.out It works the way I want it to, but it's not... (1 Reply)
Discussion started by: Leo_Boon
1 Replies

9. Shell Programming and Scripting

Shell script to run multiple jobs and it's dependent jobs

I have multiple jobs and each job dependent on other job. Each Job generates a log and If job completed successfully log file end's with JOB ENDED SUCCESSFULLY message and if it failed then it will end with JOB ENDED with FAILURE. I need an help how to start. Attaching the JOB dependency... (3 Replies)
Discussion started by: santoshkumarkal
3 Replies

10. Shell Programming and Scripting

How to run several bash commands put in bash command line?

How to run several bash commands put in bash command line without needing and requiring a script file. Because I'm actually a windows guy and new here so for illustration is sort of : $ bash "echo ${PATH} & echo have a nice day!" will do output, for example:... (4 Replies)
Discussion started by: abdulbadii
4 Replies
INSTALLSIEVE(1) 					      General Commands Manual						   INSTALLSIEVE(1)

 *

NAME
installsieve - user utility for managing sieve scripts SYNOPSIS
installsieve DESCRIPTION
installsieve is a utility that allows users to manage their sieve scripts kept on the server. OPTIONS
-v <name> View script with the given name. The script if retrieved sucessfully is output to standard output. -l List all of the scripts currently on the server. If one of the scripts is active a arrow is printed indicating that it is the active script. -p <port> Port to connect to. If left off this defaults to "sieve" as defined in /etc/services. -i <file> Install a file onto the server. If a script with the same name already exists on the server it is overwritten. Upon sucessfully putting the script on the server the script is set active. If <file> has the extension .script it is chopped when put on the server since sieve names may not contain a '.'. -a <name> Set <name> as the active script. The list of available names can be obtained from the '-l' option.. -d <name> Delete the sieve script on the server with <name>. -m <mechanism> Force installsieve to use <mechanism> for authentication. If not specified the strongest authentication mechanism is chosen. -g <name> Get the sieve script with <name> and save it to disk with a ".script" extension. If a file with that name already exists it is overwritten. -u <user> Userid/Authname to use for authentication; by default, the current user. CMU
Project Cyrus INSTALLSIEVE(1)
All times are GMT -4. The time now is 03:15 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy