Sponsored Content
Top Forums Shell Programming and Scripting Help needed on restart-from-point-of-failure in Parallel Processing Post 302995545 by saps19 on Friday 7th of April 2017 06:48:20 AM
Old 04-07-2017
Thank you rbattle1,
Quote:
You might want this:-

Code:
(/path/to/product/script/data_process "${fileLines}" || exit -1) &
make sense and corrected the script(though earlier it was running fine (Smilie), don't know how).

Currently I'm trying to get the command I'm running using below sample test script:
Code:
 $ [] cat bgPIDTest.ksh
# Some function that takes a long time to process
longprocess() {
        # Sleep up to 14 seconds
        #sleep $((RANDOM % 15))
        sleepTime=$((RANDOM % 5))
        # Randomly exit with 0 or 1
        exitCode=$((RANDOM % 2))
        echo "sleeping for: $sleepTime with exit code: $exitCode "
        sleep $sleepTime
        exit $exitCode
}
 pids=""
failCnt=0
# Run five concurrent processes
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
        ( longprocess ) &
        # store PID of process
        pids+=" $!"
        echo PID $pids
 
# Wait for all processes to finish, will take max 14s
echo "initial failCnt is $failCnt"
for p in $pids; do
        #if wait $p; then
        if ! wait $p; then
            cmdJobNM=`ps -p $p -o command=`
            failCnt=`expr $failCnt + 1`
            echo "failed command is --> $cmdJobNM, PID: $p"
        fi
done
echo "total failCnt is $failCnt"
if [ $failCnt -gt 0 ]; then
    exit -1
fi

Output:
Code:
 $ [] ksh bgPIDTest.ksh
PID 15858
sleeping for: 2 with exit code: 1
PID 15858 15859
sleeping for: 4 with exit code: 0
PID 15858 15859 15860
sleeping for: 0 with exit code: 0
PID 15858 15859 15860 15861
sleeping for: 3 with exit code: 0
PID 15858 15859 15860 15861 15862
sleeping for: 3 with exit code: 1
PID 15858 15859 15860 15861 15862 15863
initial failCnt is 0
sleeping for: 1 with exit code: 0
failed command is --> , PID: 15858
failed command is --> , PID: 15862
total failCnt is 2

Still I'm not getting the command, so that I can awk'ed the passed argument and put into a file. So the logic will be:
Once I'll restart script:
if this new file exist
take this new file
else
use old config file

once successfully done, I'll remove the error file (if any). Please suggest if this feasible.

Cheers,
Saps.
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to do parallel processing??

Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a... (1 Reply)
Discussion started by: zing
1 Replies

2. UNIX for Dummies Questions & Answers

Restart on power failure

How do I configure my workstation (Solaris 9) to restart and perform any check disk automatically if there is a power failure? Thanks. (1 Reply)
Discussion started by: here2learn
1 Replies

3. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

4. Shell Programming and Scripting

Need Help With Parallel Processing

Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or... (6 Replies)
Discussion started by: imnewtothis23
6 Replies

5. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

6. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

7. Shell Programming and Scripting

script parallel processing

How to write script which run multiple scripts parllely, i have script called A,which has to execute B,C,D,E scripts parllely.. (2 Replies)
Discussion started by: machpee
2 Replies

8. Shell Programming and Scripting

Re run the script from the point of failure

Hello I have a shell script with multiple sections in it. Like, verify pre-requisites, ch co version, stop services , install product , post migration steps, start services, send status email. If the script fails at certain step, (like after product installation) it can't be re-run since the... (2 Replies)
Discussion started by: mo12
2 Replies

9. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies
kdump-tools.conf(5)					Linux System Administrator's Manual				       kdump-tools.conf(5)

NAME
kdump-tools.conf - configuration file for the kdump-tools init script SYNOPSIS
/etc/default/kdump-tools DESCRIPTION
kdump-tools manages the kdump feature of the Linux kernel. The /etc/default/kdump-tools file contains local configuration settings of kdump. This file is sourced into a shell script, so it's format should be consistent with shell scripting. OPTIONS
USE_KDUMP Controls whether or not kdump-tools will take any action. If not set or 0, the kdump-tools init script will not run. KDUMP_SYSCTL Controls when a panic occurs, using the sysctl(8) interface. Each time a kdump kernel is loaded, "sysctl -w $KDUMP_SYSCTL" is also executed, thus allowing the system adiminstrator to maintain customizable conditions for a kdump. The contents of this variable should be the "variable=value ..." portion of the "sysctl -w" command. If not set, "kernel.panic_on_oops=1" will be used. This feature can be disabled by setting KDUMP_SYSCTL=" " See sysctl(8) for more info. KDUMP_KERNEL A full pathname to a kdump kernel (the kernel that is kexec'd at crash time in a reserved memory area, exposing the old kernel's memory for dumping). If not set, kdump-config will use the boot kernel as the kdump kernel if it is relocatable; otherwise you must set KDUMP_KERNEL in order for kdump-tools to work. KDUMP_INITRD A full pathname to the kdump initrd (if used). If KDUMP_KERNEL is set and KDUMP_INITRD is not set, a warning message will be printed, and an initrd will not be used. KDUMP_COREDIR Full path to a directory where the vmcore will be saved. Date stamped subdirectories are created each time a vmcore file is pro- cessed. If not set, /var/crash will be used. KDUMP_FAIL_CMD This variable specifies a command to run if the vmcore save fails. If not set, no special action is taken. DEBUG_KERNEL A debug version of the running kernel. If not set, kdump-config will use /usr/lib/debug/vmlinux-$(uname-r) if it exists. Other- wise, kdump will still work, but the dump will take longer and will be much larger. MAKEDUMP_ARGS Extra arguments passed to makedumpfile(8). If not set, "-c -d 31" will be used. This tells makedumpfile to use compression, and reduce the corefile to in-use kernel pages only. See makedumpfile(8) for details. KDUMP_KEXEC_ARGS Additional arguments to the kexec command used to load the kdump kernel. KDUMP_CMDLINE Normally, the current kernel commandline is obtained from /proc/cmdline. Set this variable to override /proc/cmdline. KDUMP_CMDLINE_APPEND Additional arguments to append to the command line for the kdump kernel. If not set, "irqpoll maxcpus=1 nousb" will be used. USAGE
kdump-tools is as automated as can be at this point but there are some prerequisites to using it. Additionally, some manual configuration is still required. Manual Configuration 1. USE_KDUMP is set to 0 by default. To enable kdump-tools, edit the /etc/default/kdump-tools configuration file and set USE_KDUMP=1. 2. Kernel Command line parameters - the kernel must be booted with a crashkernel= command line parameter. Some example crashkernel parameters: ia64: crashkernel=384M x86: crashkernel=128M x86_64: crashkernel=256M Some users may also want to add nmi_watchdog=1 on certain systems. The nmi watchdog will cause the kernel to panic (and kdump) if a system hang is detected. The kernel command line parameter is generally set in one of these files: /etc/default/grub, /boot/grub/menu.lst, /etc/elilo.conf, or /etc/lilo.conf. If the command line parameter is changed, a reboot is required in order for it to take effect. 3. Architectural considerations A) x86 && PAE && memory > 4 Gigabytes - use KDUMP_KEXEC_ARGS="--elf64-core-headers" B) x86 and x86_64 - Some systems can take advantage of the nmi watchdog. Add nmi_watchdog=1 to the boot commandline to turn on the watchdog. The nmi interrupt will call panic if activated. C) ia64 - Some systems may need KDUMP_KEXEC_ARGS="--noio". Use this if the system hangs after a panic, but before the kdump kernel begins to boot. Prerequisites 1. Boot Kernel Configuration - The boot kernel must be configured with CONFIG_KEXEC=y and, if it is also to be used as the kdump ker- nel, CONFIG_CRASHDUMP=y. For ia64, only makedumpfile level 1 will work if the memory model selected is CONFIG_DISCONTIG. CONFIG_SPARSEMEM is recommended instead. 2. Kdump Kernel Configuration - The kdump kernel must be relocated or relocatable. ia64 is relocatable by default, but x86, x86_64, and powerpc must be built with CONFIG_RELOCATABLE=y. Other architectures may require a predermined start location via CONFIG_PHYSI- CAL_START. If the boot kernel is relocatable, kdump-tools will automatically use that kernel. Otherwise, a relocatable or relo- cated kernel will need to be provided. The kdump kernel can be specified in the /etc/default/kdump-tools file. Set the KDUMP_KER- NEL variable and if necessary the KDUMP_INITRD variable to point to the provided kernel and its initrd. The kdump kernel must be configured with: CONFIG_CRASH_DUMP=y 3. Debug Kernel - An uncompressed debug kernel should be provided in order for makedumpfile to process the vmcore file. Without a debug kernel, the transfer process is reduced to using "makedumpfile -c -d 1". If /usr/lib/debug/vmlinux-$(uname -r) exists, kdump- tools will automatically use that kernel. Otherwise, A) set DEBUG_KERNEL in /etc/default/kdump-tools to point to a debug version of the currently booted kernel or B) do with- out - makedumpfile will still work, but the dumpfile will be larger and take longer to save to disk. EXAMPLES
Also panic and kdump on oom: KDUMP_SYSCTL="kernel.panic_on_oops=1 vm.panic_on_oom=1" Use this option on x86 systems with PAE and more than 4 gig of memory: KDUMP_KEXEC_ARGS="--elf64-core-headers" This option starts a shell if kdump-tools cannot save the vmcore file: KDUMP_FAIL_CMD="/bin/bash; reboot -f" FILES
/etc/init.d/kdump-tools an init script to automatically load a kdump kernel, or save a vmcore and reboot. /etc/default/kdump-tools the kdump-tools configuration file /var/crash/kernel_link a link to the current debug kernel /var/crash/kexec_cmd the last kexec_cmd executed by kdump-config DIAGNOSTICS
See kdump-config(8) for explanations of various error messages. SEE ALSO
/usr/share/doc/kdump-tools/README /usr/share/doc/kdump-tools/README.Debian kdump-config(8), kexec(8), sysctl(8), makedumpfile(8), crash(8), gdb(1), AUTHOR
Terry Loftin <terry.loftin@hp.com> kdump-tools v1.1.3 April 2007 kdump-tools.conf(5)
All times are GMT -4. The time now is 11:12 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy