Sponsored Content
Top Forums Shell Programming and Scripting Help needed on restart-from-point-of-failure in Parallel Processing Post 302995455 by saps19 on Thursday 6th of April 2017 01:55:03 AM
Old 04-06-2017
Help needed on restart-from-point-of-failure in Parallel Processing

Hi Gurus,
Good morning... Smilie
OS Info:
Linux 2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11 17:27:00 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

I have a script which takes multiples parameters from a properties file one by one and run in background (to do parallel processing). As example:
Code:
$ cat properties.file
Account
Customer
Address
Phone

part of customCombinedScripts.sh
Code:
 
  
 while IFS= read -r fileLines; do
    /path/to/product/script/data_process "${fileLines}" || exit -1 &
    dpPIDs+=" $!"
 done < properties.file
 ## logic to check background job status (success or failed)
for chkPIDs in $dpPIDs; do
    if ! wait $chkPIDs; then
        failCnt=`expr $failCnt + 1`
    fi
done
 if [ $failCnt -gt 0 ]; then
    echo "[`date`][FATAL] There are errors in Data Processing, total number of DP jobs failure is $failCnt. Check log."
    exit -1
else
    echo "[`date`][SUCCESS] Data Processing for received files has been completed..."
fi

Now my requirement is: Suppose "Address" failed, how to restart the script which will take the failed parameters (i.e Address) only?

I have implemented restart-from-point-of-failure concept in my another script which has "Sequential processing" using help from this article, posted by Corona688 Resume from last failed command

How to implement the same concept in "Parallel Processing"?

Kindly help / provide ideas.

Cheers,
Saptarshi
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to do parallel processing??

Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a... (1 Reply)
Discussion started by: zing
1 Replies

2. UNIX for Dummies Questions & Answers

Restart on power failure

How do I configure my workstation (Solaris 9) to restart and perform any check disk automatically if there is a power failure? Thanks. (1 Reply)
Discussion started by: here2learn
1 Replies

3. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

4. Shell Programming and Scripting

Need Help With Parallel Processing

Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or... (6 Replies)
Discussion started by: imnewtothis23
6 Replies

5. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

6. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

7. Shell Programming and Scripting

script parallel processing

How to write script which run multiple scripts parllely, i have script called A,which has to execute B,C,D,E scripts parllely.. (2 Replies)
Discussion started by: machpee
2 Replies

8. Shell Programming and Scripting

Re run the script from the point of failure

Hello I have a shell script with multiple sections in it. Like, verify pre-requisites, ch co version, stop services , install product , post migration steps, start services, send status email. If the script fails at certain step, (like after product installation) it can't be re-run since the... (2 Replies)
Discussion started by: mo12
2 Replies

9. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies
IMAP_MAILBOXMSGINFO(3)							 1						    IMAP_MAILBOXMSGINFO(3)

imap_mailboxmsginfo - Get information about the current mailbox

SYNOPSIS
object imap_mailboxmsginfo (resource $imap_stream) DESCRIPTION
Checks the current mailbox status on the server. It is similar to imap_status(3), but will additionally sum up the size of all messages in the mailbox, which will take some additional time to execute. PARAMETERS
o $ imap_stream -An IMAP stream returned by imap_open(3). RETURN VALUES
Returns the information in an object with following properties: Mailbox properties +--------+----------------------------------------+ | Date | | | | | | | date of last change (current datetime) | | | | |Driver | | | | | | | driver | | | | |Mailbox | | | | | | | name of the mailbox | | | | | Nmsgs | | | | | | | number of messages | | | | |Recent | | | | | | | number of recent messages | | | | |Unread | | | | | | | number of unread messages | | | | |Deleted | | | | | | | number of deleted messages | | | | | Size | | | | | | | mailbox size | | | | +--------+----------------------------------------+ Returns FALSE on failure. EXAMPLES
Example #1 imap_mailboxmsginfo(3) example <?php $mbox = imap_open("{imap.example.org}INBOX", "username", "password") or die("can't connect: " . imap_last_error()); $check = imap_mailboxmsginfo($mbox); if ($check) { echo "Date: " . $check->Date . "<br /> " ; echo "Driver: " . $check->Driver . "<br /> " ; echo "Mailbox: " . $check->Mailbox . "<br /> " ; echo "Messages: " . $check->Nmsgs . "<br /> " ; echo "Recent: " . $check->Recent . "<br /> " ; echo "Unread: " . $check->Unread . "<br /> " ; echo "Deleted: " . $check->Deleted . "<br /> " ; echo "Size: " . $check->Size . "<br /> " ; } else { echo "imap_mailboxmsginfo() failed: " . imap_last_error() . "<br /> "; } imap_close($mbox); ?> PHP Documentation Group IMAP_MAILBOXMSGINFO(3)
All times are GMT -4. The time now is 01:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy