Sponsored Content
Special Forums UNIX and Linux Applications High Performance Computing Python code runs on login node but not on cluster Post 302985382 by devinmgibson on Tuesday 8th of November 2016 05:29:10 PM
Old 11-08-2016
Python code runs on login node but not on cluster

I work for one of my professors and we are trying to run SU2 in parallel on a cluster owned by the university that uses slurm for its workload manager. The problem we are running into is that when we ssh into the cluster and run the command:

Code:
parallel_computation.py -f SU2.cfg

on an assigned node by slurm (using sbatch), the code hangs and wont run. The weird thing about this is if we run the same command on the login node, it works just fine. Do any of you know what could possibly be the problem?

Here is some additional information:
- We talked with the IT guy in charge of the cluster and he doesn't have enough background to know what is going on.
- On some of our output files we would get the escape key [!0134h, when we changed the terminal settings to get rid of the escape key the code behavior was consistent as above.
- We can run SU2_CFD "config file", the code in serial, on both the login node and the cluster just fine
- We have tried running an interactive session on a node (using srun), no change in behavior

Any thoughts would be appreciated! We really want to be able to run the code in-house instead of outsource.


Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 11-09-2016 at 04:07 AM.. Reason: Added CODE tags.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

The other node name of a SUN cluster

Hello, Under ksh I have to run a script on one of the nodes of a Solaris 8 cluster which at some time must execute a command on the alternate node: # rsh <name> "command" I have to implement this script on all the clusters of my company (a lot of...). Fortunately, the names of the two nodes... (11 Replies)
Discussion started by: heartwork
11 Replies

2. HP-UX

Node can't join cluster

Need help guys! when running cmrunnode batch i'm getting this error cmrunnode : Waiting for cluster to... (1 Reply)
Discussion started by: Tris
1 Replies

3. HP-UX

MC/SG Fail to join cluster node

Hi, Please advise me whereas I have two node cluster server configured with MC/SG. Application and DB are running on Node 1, while Node 2 is standby. All the volume group devices are part of cluster environment. There is only one package running at node 1. Node 2 is having the problem to... (1 Reply)
Discussion started by: rauphelhunter
1 Replies

4. High Performance Computing

Removed crashed node from Solaris Cluster 3.0

All- I am new to these forums so please excuse me if this post is in the wrong place. I had a node crash in a 4 node cluster and mgmt has determined this node will not be part of the cluster when rebuilt. I am researching how to remove it from the cluster information on the other 3 nodes and... (2 Replies)
Discussion started by: bluescreen
2 Replies

5. High Performance Computing

Setting up 2 node cluster using solaris 10

hi, i am trying to setup a 2 node cluster environment. following is what i have; 1. 2 x sun ultra60 - 450MHz procs, 1GB RAM, 9GB HDD, solaris 10 2. 2 x HBA cards 3. 2 x Connection leads to connect ultra60 with D1000 4. 1 x D1000 storage box. 5. 3 x 9GB HDD + 2 x 36GB HDD first of all,... (1 Reply)
Discussion started by: solman17
1 Replies

6. Solaris

Active Sun cluster node?

I now the logical name and Virtual IP of the cluster. How can I find the active sun cluster node without having root access? (3 Replies)
Discussion started by: sreeniatbp
3 Replies

7. HP-UX

Identify cluster active node

Hello, Is there any way to identify the active node in a HP-UX cluster without root privileges? (3 Replies)
Discussion started by: psimoes79
3 Replies

8. Solaris

How to remove single node cluster

Hi Gurus, I am very new to clustering and for test i have created a single node cluster, now i want to remove the system from cluster. Did some googling however as a newbee in cluster unable to co related the info. Please help Thanks (1 Reply)
Discussion started by: kumarmani
1 Replies

9. Solaris

SVM metaset on 2 node Solaris cluster storage replicated to non-clustered Solaris node

Hi, Is it possible to have a Solaris cluster of 2 nodes at SITE-A using SVM and creating metaset using say 2 LUNs (on SAN). Then replicating these 2 LUNs to remote site SITE-B via storage based replication and then using these LUNs by importing them as a metaset on a server at SITE-B which is... (0 Replies)
Discussion started by: dn2011
0 Replies

10. AIX

Cluster node not starting

Setting up HACMP 6.1 on a two node cluster. The other node works fine and can start properly on STABLE state (VGs varied, FS mounted, Service IP aliased). However, the other node is always stuck on ST_JOINING state. Its taking forever and you can't stop the cluster as well or recover from script... (2 Replies)
Discussion started by: depam
2 Replies
cmhaltnode(1m)															    cmhaltnode(1m)

NAME
cmhaltnode - halt a node in a high availability cluster SYNOPSIS
cmhaltnode [-f] [-v] [-t] [node_name...] DESCRIPTION
cmhaltnode causes a node to halt its cluster daemon and remove itself from the existing cluster. To halt cluster on the node, a user must either be superuser(UID=0), or have an access policy of FULL_ADMIN allowed in the cluster configu- ration file. See access policy in cmquerycl. When cmhaltnode is run on a node, the cluster daemon is halted and, optionally, all packages that were running on that node are moved to other nodes if possible. If node_name is not specified, the cluster daemon running on the local node will be halted and removed from the existing cluster. If you issue this command while a cluster is still in the process of forming, the command will fail with the message "Unable to connect to daemon." If this happens, wait for the cluster to form successfully, then issue the command again. Options cmhaltnode supports the following options: -f Force the node to halt even if there are packages or group members running on it. The group members on the node will be terminated. The halt scripts for all packages running on the node will be run; based on priority or dependency relation- ships, this may affect packages on other nodes. In other words, packages on other nodes may either start or halt based on this package halting. If the package configuration and current cluster membership permit, and if the package halt script succeeds, the packages will be started on other nodes. Without this option, if packages are running on the given node, the command will fail. If a package fails to halt, the node halt will also fail. -v Verbose output will be displayed. -t Test only. Provide an assessment of the package placement without affecting the current state of the nodes or packages. This option validates the node's eligibility with respect to the package dependencies as well as the external dependencies such as EMS resources, package subnets, and storage before predicting any package placement decisions. If there is a pack- age in maintenance mode running on the nodes being halted, the package will always be halted and not failover to another node; the report will not display an assessment for that package. node_name... The name of the node(s) to halt. RETURN VALUE
cmhaltnode returns the following value: 0 Successful completion. 1 Command failed. EXAMPLES
Halt the cluster daemon on two other nodes: cmhaltnode node2 node3 AUTHOR
cmhaltnode was developed by HP. SEE ALSO
cmquerycl(1m), cmhaltcl(1m), cmruncl(1m), cmrunnode(1m), cmviewcl(1m), cmeval(1m). Requires Optional Serviceguard Software cmhaltnode(1m)
All times are GMT -4. The time now is 02:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy