Sponsored Content
Top Forums Shell Programming and Scripting Help needed on restart-from-point-of-failure in Parallel Processing Post 302995545 by saps19 on Friday 7th of April 2017 06:48:20 AM
Old 04-07-2017
Thank you rbattle1,
Quote:
You might want this:-

Code:
(/path/to/product/script/data_process "${fileLines}" || exit -1) &
make sense and corrected the script(though earlier it was running fine (Smilie), don't know how).

Currently I'm trying to get the command I'm running using below sample test script:
Code:
 $ [] cat bgPIDTest.ksh
# Some function that takes a long time to process
longprocess() {
        # Sleep up to 14 seconds
        #sleep $((RANDOM % 15))
        sleepTime=$((RANDOM % 5))
        # Randomly exit with 0 or 1
        exitCode=$((RANDOM % 2))
        echo "sleeping for: $sleepTime with exit code: $exitCode "
        sleep $sleepTime
        exit $exitCode
}
 pids=""
failCnt=0
# Run five concurrent processes
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
 
# Wait for all processes to finish, will take max 14s
echo "initial failCnt is $failCnt"
for p in $pids; do
        #if wait $p; then
        if ! wait $p; then
            cmdJobNM=`ps -p $p -o command=`
            failCnt=`expr $failCnt + 1`
            echo "failed command is --> $cmdJobNM, PID: $p"
        fi
done
echo "total failCnt is $failCnt"
if [ $failCnt -gt 0 ]; then
    exit -1
fi

Output:
Code:
 $ [] ksh bgPIDTest.ksh
PID 15858
sleeping for: 2 with exit code: 1
PID 15858 15859
sleeping for: 4 with exit code: 0
PID 15858 15859 15860
sleeping for: 0 with exit code: 0
PID 15858 15859 15860 15861
sleeping for: 3 with exit code: 0
PID 15858 15859 15860 15861 15862
sleeping for: 3 with exit code: 1
PID 15858 15859 15860 15861 15862 15863
initial failCnt is 0
sleeping for: 1 with exit code: 0
failed command is --> , PID: 15858
failed command is --> , PID: 15862
total failCnt is 2

Still I'm not getting the command, so that I can awk'ed the passed argument and put into a file. So the logic will be:
Once I'll restart script:
if this new file exist
take this new file
else
use old config file

once successfully done, I'll remove the error file (if any). Please suggest if this feasible.

Cheers,
Saps.
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to do parallel processing??

Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a... (1 Reply)
Discussion started by: zing
1 Replies

2. UNIX for Dummies Questions & Answers

Restart on power failure

How do I configure my workstation (Solaris 9) to restart and perform any check disk automatically if there is a power failure? Thanks. (1 Reply)
Discussion started by: here2learn
1 Replies

3. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

4. Shell Programming and Scripting

Need Help With Parallel Processing

Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or... (6 Replies)
Discussion started by: imnewtothis23
6 Replies

5. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

6. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

7. Shell Programming and Scripting

script parallel processing

How to write script which run multiple scripts parllely, i have script called A,which has to execute B,C,D,E scripts parllely.. (2 Replies)
Discussion started by: machpee
2 Replies

8. Shell Programming and Scripting

Re run the script from the point of failure

Hello I have a shell script with multiple sections in it. Like, verify pre-requisites, ch co version, stop services , install product , post migration steps, start services, send status email. If the script fails at certain step, (like after product installation) it can't be re-run since the... (2 Replies)
Discussion started by: mo12
2 Replies

9. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies
FEBOOTSTRAP-SUPERMIN-HELPER(8)				      Virtualization Support				    FEBOOTSTRAP-SUPERMIN-HELPER(8)

NAME
febootstrap-supermin-helper - Reconstruct initramfs from supermin appliance. SYNOPSIS
febootstrap-supermin-helper supermin.img hostfiles.txt host_cpu kernel initrd febootstrap-supermin-helper input [...] host_cpu kernel initrd febootstrap-supermin-helper -f ext2 input [...] host_cpu kernel initrd appliance febootstrap-supermin-helper -f checksum input [...] host_cpu DESCRIPTION
febootstrap-supermin-helper reconstructs a bootable kernel and initramfs from a supermin appliance. First you should be familiar with febootstrap(8). PARAMETERS
Of the required parameters, the first few are input files, and the last two or three are output files. "supermin.img" and "hostfiles.txt" are the input files which describe the supermin appliance. (You can also use a directory name here which is searched for files). "host_cpu" should be the host CPU, eg. "x86_64" or "i686". "kernel", "initrd" and "appliance" are the temporary output files that this script produces. These output files are meant to be used just for booting the appliance, and should be deleted straight afterwards. The extra "appliance" parameter is only required when the format is "ext2". None of these parameters are needed for the checksum output "-f checksum". OPTIONS
-f fmt --format fmt Select the output format for the appliance. Possible formats are: cpio A Linux initramfs. This is the default. In this case you have to supply names for the "kernel" and "initrd", where the "initrd" is the appliance. ext2 An ext2 filesystem. In this case you have to supply names for the "kernel", a small "initrd" which is used just to locate the appliance, and the "appliance" (the ext2 filesystem). checksum Output a checksum. This prints a checksum which only changes when one of the input files changes. You can use this in order to cache the output of a previous run of this program: computing the checksum is much quicker than building an appliance, and you only need to invalidate the cache (and consequently rebuild the appliance) when the checksum changes. Note that the host_cpu and the UID of the current user are included in the checksum. -k file --kmods file If this option is specified, then "file" should be a list of wildcards matching kernel module names, eg: virtio*.ko scsi*.ko piix.ko In this case, only kernel modules matching those wildcards will be included in the output appliance. Note: You must resolve any dependencies yourself as this does not pull in dependent modules automatically. If this option is not specified, then every kernel module from the host will be included. This is safer, but can produce rather large appliances which need a lot more memory to boot. -u user --user user -g group --group group Run febootstrap-supermin-helper as an alternate user and/or group. "user" and "group" can be specified as either a name, which will be resolved using the system name service, or a uid/gid. Use of these options requires root privileges. Use of these options is required if running febootstrap-supermin-helper as root with the effective uid/gid set to non-root. Bash will reset the effective uid/gid to the real uid/gid when invoked. As febootstrap-supermin-helper uses bash in parts, this will result in the creation of an appliance with a mixture of ownerships. SPEED
In libguestfs, on a mid-range Intel-based PC, we reconstruct the initramfs using this script in around 1/5th of a second (assuming a "hot cache" - it's rather slower when run the first time on a cold cache). Some tips to improve performance: o Use a kernel module whitelist (the "--kmods" option), and only list the kernel modules you really need. o Minimize the appliance, removing as much extraneous junk as possible. ENVIRONMENT VARIABLES
FEBOOTSTRAP_KERNEL If this environment variable is set, then automatic selection of the kernel is bypassed and this kernel is used. The environment variable should point to a kernel file, eg. "/boot/vmlinuz-3.0.x86_64" The corresponding module path is guessed from the kernel name, but you can override that by setting "FEBOOTSTRAP_MODULES". FEBOOTSTRAP_MODULES If "FEBOOTSTRAP_KERNEL" and "FEBOOTSTRAP_MODULES" are both set, then automatic selection of the kernel is bypassed and the kernel and module path are set to these values. The environment variable should point to a module directory, eg. "/lib/modules/3.0.x86_64/" This has no effect if "FEBOOTSTRAP_KERNEL" is not set. SEE ALSO
febootstrap(8). AUTHORS
Richard W.M. Jones <rjones @ redhat . com> COPYRIGHT
(C) Copyright 2009-2011 Red Hat Inc., <http://people.redhat.com/~rjones/febootstrap>. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. febootstrap-3.10 2011-09-14 FEBOOTSTRAP-SUPERMIN-HELPER(8)
All times are GMT -4. The time now is 10:18 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy