Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Qsub + job array + while read; do; done

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Qsub + job array + while read; do; done
# 1  
Old 03-08-2018
Qsub + job array + while read; do; done
Code:
Linux Server

Hi!

just some help...

I'm using a "Sun Grid Engine" with "qsub"

In one hand I have this code

Code:
while read -r H; do echo ${H}; done < list

The number of the lines in the list file is 100.

On the other hand I want to create a job array with the option -t

Code:
echo "echo $H" | qsub -N loop -t 1-100


How can I merge the two codes? In the way to create a job array (1-100) where each job read a line from the list file...?



Many thanks
# 2  
Old 03-08-2018
It's not entirely clear what you are after here. Can you show us a worked example if list file contained 3 lines like this

Code:
harry
bob
mary

For example you may want -t value incremented for each line something like this:

Code:
   echo "harry" | qsub -N loop -t 1
   echo "bob"   | qsub -N loop -t 2 
   echo "mary"  | qsub -N loop -t 3

Last edited by Chubler_XL; 03-08-2018 at 04:11 PM..
# 3  
Old 03-09-2018
Hi,

with your code are 3 different jobs and not three jobs in one array

Code:
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
372502  1.00000 loop       manolis      r     03/09/2018 11:19:58 all.q@localhost                1     1
372503  1.00000 loop       manolis      r     03/09/2018 11:19:58 all.q@localhost                1     2
372504  1.00000 loop       manolis      r     03/09/2018 11:19:58 all.q@localhost                1     3

I tried with 20 jobs
Code:
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
 372527 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     1
 372528 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     2
 372529 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     3
 372530 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     3
 372531 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     4
 372532 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     5
 372533 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     6
 372534 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     7
 372535 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     8
 372536 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     9
 372537 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     10
 372538 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     11
 372539 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     12
 372540 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     13
 372541 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     14
 372542 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     15
 372543 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     16
 372544 0.00000 loop       manolis      qw    03/09/2018 11:26:01                                1     17
 372545 0.00000 loop       manolis      qw    03/09/2018 11:26:02                                1     19
 372546 0.00000 loop       manolis      qw    03/09/2018 11:26:02                                1     20

What I need is something like this:
Code:
 372546 0.00000 loop       manolis      qw    03/09/2018 11:26:02                                1     1-20

where the script "loop" read the 20 line (names in this case) from a file (each line would be a job of an array job)
Last edited by echo manolis; 03-09-2018 at 06:34 AM..
# 4  
Old 03-09-2018
How about:

Code:
qsub -N loop -t 1-100 < list

This User Gave Thanks to Chubler_XL For This Post:
RudiC
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

C++ how would you read this file into an array.

here is the pesudo file. REREREEEEEERRRREER SOMEStrinG1234 RERRRR EEERRRREER SOMEStrinG1224 REREREEEREERRR REE SOMEStrinG1214 REREREREREREREEEER SOMEStrinG1204 RERE EEEEEERRRRRRR SOMEStrinG1294 REREEREEE ERRRREER SOMEStrinG1284 REREREEEEEERR REER here is my attempted code #include... (3 Replies)
Discussion started by: briandanielz
3 Replies

2. Shell Programming and Scripting

PERL : Read an array and write to another array with intial string pattern checks

I have an array and two variables as below, I need to check if $datevar is present in $filename. If so, i need to replace $filename with the values in the array. I need the output inside an ARRAY How can this be done. Any help will be appreciated. Thanks in advance. (2 Replies)
Discussion started by: irudayaraj
2 Replies

3. UNIX for Advanced & Expert Users

Collecting job id from qsub command

Hello. Hopefully this is the right forum, did a search and found some similar questions here. I'm trying to find out if there is a way to get the job id while submitting a job using qsub. I want to monitor the list of active jobs using qstat and when my jobid is no longer on the active list, I... (3 Replies)
Discussion started by: aeoleon88
3 Replies

4. UNIX for Dummies Questions & Answers

Qsub and printf

Hello, I have some issue with qsub and the standard output : I launch a script that "echoes" the string "abc" and then executes a C program in which I print informations with the printf function ; the .o file contains "abc" but not the information displayed by printf. I also tried... (0 Replies)
Discussion started by: Shaderw
0 Replies

5. UNIX for Advanced & Expert Users

How to set the SGE job execution order for nested qsub?

Dear All I am trying to write a script to set the SGE job execution order. I named each job with 'job1', 'job2' and 'job3'. I want my script to do: When 'job1' execution is complete, 'job2' is executed; when both 'job1' and 'job2' are complete, 'job3' is executed. First, I tried this... (0 Replies)
Discussion started by: cliffyiu
0 Replies

6. Shell Programming and Scripting

qsub

Hey all, I have a binary file, named "exe1" that receives a parameter. (I don't have its source code). Executing "exe1" results in creating 3 output files. I have a bash script that runs "exe1". I want to enter "exe1" to cluster queue (using subq command) But i want to do it more than... (1 Reply)
Discussion started by: elad2109
1 Replies

7. Shell Programming and Scripting

cron job issue..i hav read the basic threads already...

hi friends well m facing a different sort of issue in my cron. i hav set job like this 30 09 * * 1 /bin/backup14M 01 14 * * 1 /bin/backup14N 20 18 * * 1 /bin/backup14E that is for every Monday at three different times. but, first job executes well, later ones do not. I checked my... (13 Replies)
Discussion started by: oracle.test2
13 Replies

8. Shell Programming and Scripting

Read array from a file

Hi I've a config file like: file1 #comment k_array: 1 2 3 4 5 n_array: 7 8 9 0 11 I'd like to write a script that read it and store k_array and n_array in 2 arrays. I mean the script should be able to use both as array. I've tried to use awk as (only for one array): ... (5 Replies)
Discussion started by: Dedalus
5 Replies

9. Shell Programming and Scripting

how to read a var value into array

Hi I need to read a value of the variable into array so each character/digit will become an array element,for example: A=147921231432545436547568678679870 The resulting array should hold each digit as an element. Thanks a lot for any help -A (7 Replies)
Discussion started by: aoussenko
7 Replies

10. Shell Programming and Scripting

read input file for batch job

hi all, I am a newbie in unix shell script. May I know how to write a bacth job to read a list of files afrom the inout file and do the chmod ? Thanks. i.e. inside input.txt ==== a.txt b.txt c.txt and I want to write a batch job to read the input .txt and do the chmod 755 for all 3... (14 Replies)
Discussion started by: kinmak
14 Replies
Login or Register to Ask a Question