Sponsored Content
Top Forums Shell Programming and Scripting Help needed on restart-from-point-of-failure in Parallel Processing Post 302995545 by saps19 on Friday 7th of April 2017 06:48:20 AM
Old 04-07-2017
Thank you rbattle1,
Quote:
You might want this:-

Code:
(/path/to/product/script/data_process "${fileLines}" || exit -1) &
make sense and corrected the script(though earlier it was running fine (Smilie), don't know how).

Currently I'm trying to get the command I'm running using below sample test script:
Code:
 $ [] cat bgPIDTest.ksh
# Some function that takes a long time to process
longprocess() {
        # Sleep up to 14 seconds
        #sleep $((RANDOM % 15))
        sleepTime=$((RANDOM % 5))
        # Randomly exit with 0 or 1
        exitCode=$((RANDOM % 2))
        echo "sleeping for: $sleepTime with exit code: $exitCode "
        sleep $sleepTime
        exit $exitCode
}
 pids=""
failCnt=0
# Run five concurrent processes
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
 
# Wait for all processes to finish, will take max 14s
echo "initial failCnt is $failCnt"
for p in $pids; do
        #if wait $p; then
        if ! wait $p; then
            cmdJobNM=`ps -p $p -o command=`
            failCnt=`expr $failCnt + 1`
            echo "failed command is --> $cmdJobNM, PID: $p"
        fi
done
echo "total failCnt is $failCnt"
if [ $failCnt -gt 0 ]; then
    exit -1
fi

Output:
Code:
 $ [] ksh bgPIDTest.ksh
PID 15858
sleeping for: 2 with exit code: 1
PID 15858 15859
sleeping for: 4 with exit code: 0
PID 15858 15859 15860
sleeping for: 0 with exit code: 0
PID 15858 15859 15860 15861
sleeping for: 3 with exit code: 0
PID 15858 15859 15860 15861 15862
sleeping for: 3 with exit code: 1
PID 15858 15859 15860 15861 15862 15863
initial failCnt is 0
sleeping for: 1 with exit code: 0
failed command is --> , PID: 15858
failed command is --> , PID: 15862
total failCnt is 2

Still I'm not getting the command, so that I can awk'ed the passed argument and put into a file. So the logic will be:
Once I'll restart script:
if this new file exist
take this new file
else
use old config file

once successfully done, I'll remove the error file (if any). Please suggest if this feasible.

Cheers,
Saps.
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to do parallel processing??

Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a... (1 Reply)
Discussion started by: zing
1 Replies

2. UNIX for Dummies Questions & Answers

Restart on power failure

How do I configure my workstation (Solaris 9) to restart and perform any check disk automatically if there is a power failure? Thanks. (1 Reply)
Discussion started by: here2learn
1 Replies

3. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

4. Shell Programming and Scripting

Need Help With Parallel Processing

Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or... (6 Replies)
Discussion started by: imnewtothis23
6 Replies

5. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

6. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

7. Shell Programming and Scripting

script parallel processing

How to write script which run multiple scripts parllely, i have script called A,which has to execute B,C,D,E scripts parllely.. (2 Replies)
Discussion started by: machpee
2 Replies

8. Shell Programming and Scripting

Re run the script from the point of failure

Hello I have a shell script with multiple sections in it. Like, verify pre-requisites, ch co version, stop services , install product , post migration steps, start services, send status email. If the script fails at certain step, (like after product installation) it can't be re-run since the... (2 Replies)
Discussion started by: mo12
2 Replies

9. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies
IP-NETNS(8)							       Linux							       IP-NETNS(8)

NAME
ip-netns - process network namespace management SYNOPSIS
ip [ OPTIONS ] netns { COMMAND | help } ip netns [ list ] ip netns add NETNSNAME ip [-all] netns del [ NETNSNAME ] ip netns set NETNSNAME NETNSID ip netns identify [ PID ] ip netns pids NETNSNAME ip [-all] netns exec [ NETNSNAME ] command... ip netns monitor ip netns list-id DESCRIPTION
A network namespace is logically another copy of the network stack, with its own routes, firewall rules, and network devices. By default a process inherits its network namespace from its parent. Initially all the processes share the same default network namespace from the init process. By convention a named network namespace is an object at /var/run/netns/NAME that can be opened. The file descriptor resulting from opening /var/run/netns/NAME refers to the specified network namespace. Holding that file descriptor open keeps the network namespace alive. The file descriptor can be used with the setns(2) system call to change the network namespace associated with a task. For applications that are aware of network namespaces, the convention is to look for global network configuration files first in /etc/netns/NAME/ then in /etc/. For example, if you want a different version of /etc/resolv.conf for a network namespace used to isolate your vpn you would name it /etc/netns/myvpn/resolv.conf. ip netns exec automates handling of this configuration, file convention for network namespace unaware applications, by creating a mount namespace and bind mounting all of the per network namespace configure files into their traditional location in /etc. ip netns list - show all of the named network namespaces This command displays all of the network namespaces in /var/run/netns ip netns add NAME - create a new named network namespace If NAME is available in /var/run/netns/ this command creates a new network namespace and assigns NAME. ip [-all] netns delete [ NAME ] - delete the name of a network namespace(s) If NAME is present in /var/run/netns it is umounted and the mount point is removed. If this is the last user of the network names- pace the network namespace will be freed and all physical devices will be moved to the default one, otherwise the network namespace persists until it has no more users. ip netns delete may fail if the mount point is in use in another mount namespace. If -all option was specified then all the network namespace names will be removed. It is possible to lose the physical device when it was moved to netns and then this netns was deleted with a running process: $ ip netns add net0 $ ip link set dev eth0 netns net0 $ ip netns exec net0 SOME_PROCESS_IN_BACKGROUND $ ip netns del net0 and eth0 will appear in the default netns only after SOME_PROCESS_IN_BACKGROUND will exit or will be killed. To prevent this the processes running in net0 should be killed before deleting the netns: $ ip netns pids net0 | xargs kill $ ip netns del net0 ip netns set NAME NETNSID - assign an id to a peer network namespace This command assigns a id to a peer network namespace. This id is valid only in the current network namespace. If the keyword "auto" is specified an available nsid will be chosen. This id will be used by the kernel in some netlink messages. If no id is assigned when the kernel needs it, it will be automatically assigned by the kernel. Once it is assigned, it's not possible to change it. ip netns identify [PID] - Report network namespaces names for process This command walks through /var/run/netns and finds all the network namespace names for network namespace of the specified process, if PID is not specified then the current process will be used. ip netns pids NAME - Report processes in the named network namespace This command walks through proc and finds all of the process who have the named network namespace as their primary network names- pace. ip [-all] netns exec [ NAME ] cmd ... - Run cmd in the named network namespace This command allows applications that are network namespace unaware to be run in something other than the default network namespace with all of the configuration for the specified network namespace appearing in the customary global locations. A network namespace and bind mounts are used to move files from their network namespace specific location to their default locations without affecting other processes. If -all option was specified then cmd will be executed synchronously on the each named network namespace even if cmd fails on some of them. Network namespace name is printed on each cmd executing. ip netns monitor - Report as network namespace names are added and deleted This command watches network namespace name addition and deletion events and prints a line for each event it sees. ip netns list-id - list network namespace ids (nsid) Network namespace ids are used to identify a peer network namespace. This command displays nsid of the current network namespace and provides the corresponding iproute2 netns name (from /var/run/netns) if any. EXAMPLES
ip netns list Shows the list of current named network namespaces ip netns add vpn Creates a network namespace and names it vpn ip netns exec vpn ip link set lo up Bring up the loopback interface in the vpn network namespace. SEE ALSO
ip(8) AUTHOR
Original Manpage by Eric W. Biederman iproute2 16 Jan 2013 IP-NETNS(8)
All times are GMT -4. The time now is 11:57 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy