0
  1 [node18:02644] [[37701,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 142
  2 [node18:02644] [[37701,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
  3 [node18:02644] [[37701,0],0] ORTE_ERROR_LOG: File open fail[node18:02645] [[37700,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
  4 [node18:02645] [[37700,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
  5 [node18:02645] [[37700,0],0] ORTE_ERROR_LOG: File open fail[node18:02633] [[37720,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
  6 [node18:02633] [[37720,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
  7 [node18:02633] [[37720,0],0] ORTE_ERROR_LOG: File open fail[node18:02635] [[37722,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
  8 [node18:02635] [[37722,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
  9 [node18:02635] [[37722,0],0] ORTE_ERROR_LOG: File open fail[node18:02646] [[37703,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 10 [node18:02646] [[37703,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 11 [node18:02646] [[37703,0],0] ORTE_ERROR_LOG: File open fail[node18:02643] [[37698,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 12 [node18:02643] [[37698,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 13 [node18:02643] [[37698,0],0] ORTE_ERROR_LOG: File open fail[node18:02647] [[37702,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 14 [node18:02647] [[37702,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 15 [node18:02647] [[37702,0],0] ORTE_ERROR_LOG: File open fail[node18:02637] [[37724,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 16 [node18:02637] [[37724,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 17 [node18:02637] [[37724,0],0] ORTE_ERROR_LOG: File open fail[node18:02641] [[37696,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 18 [node18:02641] [[37696,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 19 [node18:02641] [[37696,0],0] ORTE_ERROR_LOG: File open fail[node18:02636] [[37725,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 20 [node18:02636] [[37725,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 21 [node18:02636] [[37725,0],0] ORTE_ERROR_LOG: File open fail[node18:02634] [[37723,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 22 [node18:02634] [[37723,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 23 [node18:02634] [[37723,0],0] ORTE_ERROR_LOG: File open fail[node18:02640] [[37697,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 24 [node18:02640] [[37697,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 25 [node18:02640] [[37697,0],0] ORTE_ERROR_LOG: File open fail[node18:02638] [[37727,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 26 [node18:02638] [[37727,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 27 [node18:02638] [[37727,0],0] ORTE_ERROR_LOG: File open fail[node18:02648] [[37705,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 28 [node18:02648] [[37705,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 29 [node18:02648] [[37705,0],0] ORTE_ERROR_LOG: File open fail[node18:02632] [[37721,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 30 [node18:02632] [[37721,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 31 [node18:02632] [[37721,0],0] ORTE_ERROR_LOG: File open fail[node18:02642] [[37699,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c     at line 142
 32 [node18:02642] [[37699,0],0] ORTE_ERROR_LOG: File open failure in file ras_tm_module.c at line 82
 33 [node18:02642] [[37699,0],0] ORTE_ERROR_LOG: File open failure in file base/ras_base_allocate.c at line 149
 34 [node18:02644] [[37701,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 35 ure in file base/ras_base_allocate.c at line 149
 36 [node18:02645] [[37700,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 37 ure in file base/ras_base_allocate.c at line 149
 38 [node18:02633] [[37720,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 39 ure in file base/ras_base_allocate.c at line 149
 40 [node18:02635] [[37722,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 41 ure in file base/ras_base_allocate.c at line 149
 42 [node18:02646] [[37703,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 43 ure in file base/ras_base_allocate.c at line 149
 44 [node18:02643] [[37698,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 45 ure in file base/ras_base_allocate.c at line 149
 46 [node18:02647] [[37702,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 47 ure in file base/ras_base_allocate.c at line 149
 48 [node18:02637] [[37724,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 49 ure in file base/ras_base_allocate.c at line 149
 50 [node18:02641] [[37696,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 51 ure in file base/ras_base_allocate.c at line 149
 52 [node18:02636] [[37725,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 53 ure in file base/ras_base_allocate.c at line 149
 54 [node18:02634] [[37723,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 55 ure in file base/ras_base_allocate.c at line 149
 56 [node18:02640] [[37697,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 57 ure in file base/ras_base_allocate.c at line 149
 58 [node18:02638] [[37727,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 59 ure in file base/ras_base_allocate.c at line 149
 60 [node18:02648] [[37705,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 61 ure in file base/ras_base_allocate.c at line 149
 62 [node18:02632] [[37721,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
 63 ure in file base/ras_base_allocate.c at line 149
 64 [node19:02642] [[37699,0],0] ORTE_ERROR_LOG: File open failure in file orted/orted_main.c at line 574
~                                                                                                           

I get 64 such messages in one of my error file after i submit a job and it gets over. I am supposed to get some numbers after the job completion in one of the files it creates. Instead i get a few NaNs and a few numbers.

For the record, node 18 at my cluster works fine. every time, the message has different node number (it was 3 some days ago and i tried to redo all my computation)

I googled the error. It looked like ORTE_ERROR has something to do with the MPI package. MPI version installed in my cluster account is 1.6.5.

Do you think there is wrong with some values i am inputting or if any package is missing or outdated?

  • Is that the complete error message? What is the program and how do you run it? – Kusalananda Nov 9 '18 at 11:14
  • i have posted the complete error. my program is a job script. it calls a python file which has multiple functions defined. Its a very big architecture. – Prasad Mani Nov 9 '18 at 11:18
  • It's hard do debug since you don't mention how your are starting your jobs or what the jobs are doing. I'm assuming you Python code calls a C or Fortran program that is using MPI. – Kusalananda Nov 9 '18 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.