Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Parallelize bash commands/jobs
Top Forums Shell Programming and Scripting Parallelize bash commands/jobs Post 302974551 by stomp on Tuesday 31st of May 2016 03:12:45 PM
Old 05-31-2016
Hi,

the script example looks very basic. Hard to believe this can not be compacted in several ways. Maybe you show us the real script, so there is a better idea of what you are doing.

Thought's so far

If you're on a multi-user-system it may be not such a good idea to parallel i/o-heavy processes because you may completely eat up the available i/o of your machine and cause extreme loads at within your server. This could slow down the whole server very greatly. I would suggest you keep the i/o tasks sequentially.

The "ln"-commands may be executed sequentially too, because they should be so fast, it should be a duration that is hardly noticed to create some thousand symlinks / hardlinks. (1000 Links in one second on my 10 year old desktop machine).

As I can see from your small snippet, cpu-power is not required here at all, but only raw i/o so you will not profit from parallel executing your script.
Last edited by stomp; 05-31-2016 at 05:23 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

background jobs exit status and limit the number of jobs to run

i need to execute 5 jobs at a time in background and need to get the exit status of all the jobs i wrote small script below , i'm not sure this is right way to do it.any ideas please help. $cat run_job.ksh #!/usr/bin/ksh #################################### typeset -u SCHEMA_NAME=$1 ... (1 Reply)
Discussion started by: GrepMe
1 Replies

2. Shell Programming and Scripting

Can BASH execute commands on a remote server when the commands are embedded in shell

I want to log into a remote server transfer over a new config and then backup the existing config, replace with the new config. I am not sure if I can do this with BASH scripting. I have set up password less login by adding my public key to authorized_keys file, it works. I am a little... (1 Reply)
Discussion started by: bash_in_my_head
1 Replies

3. Shell Programming and Scripting

commands to be executed in order for a batch jobs!

Hi All, I am trying to run this script. I have a small problem: each "./goada.sh" command when done produces three files (file1, file2, file3) then they are moved to their respective directory as can be seem from this script snippet here. The script goada.sh sends some commands for some... (1 Reply)
Discussion started by: faizlo
1 Replies

4. Shell Programming and Scripting

General Q: how to run/schedule a php script from cron jobs maybe via bash from shell?

Status quo is, within a web application, which is coded completely in php (not by me, I dont know php), I have to fill out several fields, and execute it manually by clicking the "go" button in my browser, several times a day. Thats because: The script itself pulls data (textfiles) from a... (3 Replies)
Discussion started by: lowmaster
3 Replies

5. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores... (10 Replies)
Discussion started by: vhope07
10 Replies

6. Shell Programming and Scripting

waiting on jobs in bash, allowing limited parallel jobs at one time, and then for all to finish

Hello, I am running GNU bash, version 3.2.39(1)-release (x86_64-pc-linux-gnu). I have a specific question pertaining to waiting on jobs run in sub-shells, based on the max number of parallel processes I want to allow, and then wait... (1 Reply)
Discussion started by: srao
1 Replies

7. Shell Programming and Scripting

Parallelize a task that have for

Dear all, I'm a newbie in programming and I would like to know if it is possible to parallelize the script: for l in {1..1000} do cut -f$l quase2 |tr "\n" "," |sed 's/$/\ /g' |sed '/^$/d' >a_$l.t done I tried: for l in {1..1000} do cut -f$l quase2 |tr "\n" "," |sed 's/$/\ /g' |sed... (7 Replies)
Discussion started by: valente
7 Replies

8. Shell Programming and Scripting

Bash scripts as commands

Hello, the bulk of my work is run by scripts. An example is as such: #!/bin/bash awk '{print first line}' Input.in > Intermediate.ter awk '{print second line}' Input.in > Intermediate_2.ter command Intermediate.ter Intermediate_2.ter > Output.out It works the way I want it to, but it's not... (1 Reply)
Discussion started by: Leo_Boon
1 Replies

9. Shell Programming and Scripting

Shell script to run multiple jobs and it's dependent jobs

I have multiple jobs and each job dependent on other job. Each Job generates a log and If job completed successfully log file end's with JOB ENDED SUCCESSFULLY message and if it failed then it will end with JOB ENDED with FAILURE. I need an help how to start. Attaching the JOB dependency... (3 Replies)
Discussion started by: santoshkumarkal
3 Replies

10. Shell Programming and Scripting

How to run several bash commands put in bash command line?

How to run several bash commands put in bash command line without needing and requiring a script file. Because I'm actually a windows guy and new here so for illustration is sort of : $ bash "echo ${PATH} & echo have a nice day!" will do output, for example:... (4 Replies)
Discussion started by: abdulbadii
4 Replies

LEARN ABOUT OPENSOLARIS

vino-server

vino-server(1)							   User Commands						    vino-server(1)

NAME

       vino-server - remote desktop server

SYNOPSIS

       vino-server

DESCRIPTION

       vino-server  is	a remote desktop server based on VNC protocol.	vino-server provides remote desktop takeover and remote desktop view func-
       tions. vino-server allows multiple client to access a GNOME desktop session remotely. It is based on the VNC technology.  vino-server  pro-
       vides  service at port 5800. Each desktop has a vino server which disabled by default and a Java client application which is made available
       to a client via a web browser (via port 5900 of the server machine is the service is enabled by the user).

FILES

       The following files are used by this application:

	    /usr/lib/vino-server

	    The executable file for the vino server application.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWgnome-remote-desktop	   |
       +-----------------------------+-----------------------------+
       |Interface stability	     |Volatile			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       vino-passwd(1), vino-preferences(1), attributes(5)

NOTES

       Written by Steven Zhang, Sun Microsystems Inc., 2006.

       Updated by Halton Huo, Sun Microsystems Inc., 2008.

SunOS 5.11							    15 Aug 2008 						    vino-server(1)
All times are GMT -4. The time now is 02:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy