Sponsored Content
Top Forums Shell Programming and Scripting Help needed on restart-from-point-of-failure in Parallel Processing Post 302995455 by saps19 on Thursday 6th of April 2017 01:55:03 AM
Old 04-06-2017
Help needed on restart-from-point-of-failure in Parallel Processing

Hi Gurus,
Good morning... Smilie
OS Info:
Linux 2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11 17:27:00 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

I have a script which takes multiples parameters from a properties file one by one and run in background (to do parallel processing). As example:
Code:
$ cat properties.file
Account
Customer
Address
Phone

part of customCombinedScripts.sh
Code:
 
  
 while IFS= read -r fileLines; do
    /path/to/product/script/data_process "${fileLines}" || exit -1 &
    dpPIDs+=" $!"
 done < properties.file
 ## logic to check background job status (success or failed)
for chkPIDs in $dpPIDs; do
    if ! wait $chkPIDs; then
        failCnt=`expr $failCnt + 1`
    fi
done
 if [ $failCnt -gt 0 ]; then
    echo "[`date`][FATAL] There are errors in Data Processing, total number of DP jobs failure is $failCnt. Check log."
    exit -1
else
    echo "[`date`][SUCCESS] Data Processing for received files has been completed..."
fi

Now my requirement is: Suppose "Address" failed, how to restart the script which will take the failed parameters (i.e Address) only?

I have implemented restart-from-point-of-failure concept in my another script which has "Sequential processing" using help from this article, posted by Corona688 Resume from last failed command

How to implement the same concept in "Parallel Processing"?

Kindly help / provide ideas.

Cheers,
Saptarshi
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to do parallel processing??

Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a... (1 Reply)
Discussion started by: zing
1 Replies

2. UNIX for Dummies Questions & Answers

Restart on power failure

How do I configure my workstation (Solaris 9) to restart and perform any check disk automatically if there is a power failure? Thanks. (1 Reply)
Discussion started by: here2learn
1 Replies

3. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

4. Shell Programming and Scripting

Need Help With Parallel Processing

Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or... (6 Replies)
Discussion started by: imnewtothis23
6 Replies

5. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

6. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

7. Shell Programming and Scripting

script parallel processing

How to write script which run multiple scripts parllely, i have script called A,which has to execute B,C,D,E scripts parllely.. (2 Replies)
Discussion started by: machpee
2 Replies

8. Shell Programming and Scripting

Re run the script from the point of failure

Hello I have a shell script with multiple sections in it. Like, verify pre-requisites, ch co version, stop services , install product , post migration steps, start services, send status email. If the script fails at certain step, (like after product installation) it can't be re-run since the... (2 Replies)
Discussion started by: mo12
2 Replies

9. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies
GRUB-FILE(1)                                                       User Commands                                                      GRUB-FILE(1)

NAME
grub-file - check file type SYNOPSIS
file OPTIONS FILE DESCRIPTION
Check if FILE is of specified type. --is-i386-xen-pae-domu Check if FILE can be booted as i386 PAE Xen unprivileged guest kernel --is-x86_64-xen-domu Check if FILE can be booted as x86_64 Xen unprivileged guest kernel --is-x86-xen-dom0 Check if FILE can be used as Xen x86 privileged guest kernel --is-x86-multiboot Check if FILE can be used as x86 multiboot kernel --is-x86-multiboot2 Check if FILE can be used as x86 multiboot2 kernel --is-arm-linux Check if FILE is ARM Linux --is-arm64-linux Check if FILE is ARM64 Linux --is-ia64-linux Check if FILE is IA64 Linux --is-mips-linux Check if FILE is MIPS Linux --is-mipsel-linux Check if FILE is MIPSEL Linux --is-sparc64-linux Check if FILE is SPARC64 Linux --is-powerpc-linux Check if FILE is POWERPC Linux --is-x86-linux Check if FILE is x86 Linux --is-x86-linux32 Check if FILE is x86 Linux supporting 32-bit protocol --is-x86-kfreebsd Check if FILE is x86 kFreeBSD --is-i386-kfreebsd Check if FILE is i386 kFreeBSD --is-x86_64-kfreebsd Check if FILE is x86_64 kFreeBSD --is-x86-knetbsd Check if FILE is x86 kNetBSD --is-i386-knetbsd Check if FILE is i386 kNetBSD --is-x86_64-knetbsd Check if FILE is x86_64 kNetBSD --is-i386-efi Check if FILE is i386 EFI file --is-x86_64-efi Check if FILE is x86_64 EFI file --is-ia64-efi Check if FILE is IA64 EFI file --is-arm64-efi Check if FILE is ARM64 EFI file --is-arm-efi Check if FILE is ARM EFI file --is-hibernated-hiberfil Check if FILE is hiberfil.sys in hibernated state --is-x86_64-xnu Check if FILE is x86_64 XNU (Mac OS X kernel) --is-i386-xnu Check if FILE is i386 XNU (Mac OS X kernel) --is-xnu-hibr Check if FILE is XNU (Mac OS X kernel) hibernated image --is-x86-bios-bootsector Check if FILE is BIOS bootsector -h, --help Display this help and exit. -u, --usage Display the usage of this command and exit. SEE ALSO
The full documentation for grub-file is maintained as a Texinfo manual. If the info and grub-file programs are properly installed at your site, the command info grub-file should give you access to the complete manual. grub-file (GRUB) 2.02-2ubuntu8.3 July 2018 GRUB-FILE(1)
All times are GMT -4. The time now is 06:17 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy