We use cookies to ensure you get the best experience on our website
Página Principal Explorar Ajuda News
Registe-se Iniciar Sessão
felixh
/
ds003720_ReproVBM
1
0
Fork 0
Ficheiros Problemas 0 Pull Requests 0 Wiki
Ramo: master
Ramos Etiquetas
ds003720_Rep...
/
code
/
process.condor_dag.dagman.out

process.condor_dag.dagman.out 33 KB
Link permanente Histórico Download

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418 
  1. 07/12/22 12:21:28 ******************************************************
    2. 

  2. 07/12/22 12:21:28 ** condor_scheduniv_exec.931968.0 (CONDOR_DAGMAN) STARTING UP
    3. 

  3. 07/12/22 12:21:28 ** /usr/bin/condor_dagman
    4. 

  4. 07/12/22 12:21:28 ** SubsystemInfo: name=DAGMAN type=DAGMAN(10) class=DAEMON(1)
    5. 

  5. 07/12/22 12:21:28 ** Configuration: subsystem:DAGMAN local:<NONE> class:DAEMON
    6. 

  6. 07/12/22 12:21:28 ** $CondorVersion: 8.8.6 Nov 19 2019 BuildID: Debian-8.8.6~dfsg.1-1 Debian-8.8.6~dfsg.1-1 $
    7. 

  7. 07/12/22 12:21:28 ** $CondorPlatform: X86_64-Debian_10 $
    8. 

  8. 07/12/22 12:21:28 ** PID = 2264448
    9. 

  9. 07/12/22 12:21:28 ** Log last touched time unavailable (No such file or directory)
    10. 

  10. 07/12/22 12:21:28 ******************************************************
    11. 

  11. 07/12/22 12:21:28 Using config source: /etc/condor/condor_config
    12. 

  12. 07/12/22 12:21:28 Using local config sources: 
    13. 

  13. 07/12/22 12:21:28    /etc/condor/config.d/50_inm7_master
    14. 

  14. 07/12/22 12:21:28    /etc/condor/condor_config.local
    15. 

  15. 07/12/22 12:21:28 config Macros = 81, Sorted = 81, StringBytes = 2405, TablesBytes = 2972
    16. 

  16. 07/12/22 12:21:28 CLASSAD_CACHING is ENABLED
    17. 

  17. 07/12/22 12:21:28 Daemon Log is logging: D_ALWAYS D_ERROR
    18. 

  18. 07/12/22 12:21:28 DaemonCore: No command port requested.
    19. 

  19. 07/12/22 12:21:28 DAGMAN_USE_STRICT setting: 1
    20. 

  20. 07/12/22 12:21:28 DAGMAN_VERBOSITY setting: 3
    21. 

  21. 07/12/22 12:21:28 DAGMAN_DEBUG_CACHE_SIZE setting: 5242880
    22. 

  22. 07/12/22 12:21:28 DAGMAN_DEBUG_CACHE_ENABLE setting: False
    23. 

  23. 07/12/22 12:21:28 DAGMAN_SUBMIT_DELAY setting: 0
    24. 

  24. 07/12/22 12:21:28 DAGMAN_MAX_SUBMIT_ATTEMPTS setting: 6
    25. 

  25. 07/12/22 12:21:28 DAGMAN_STARTUP_CYCLE_DETECT setting: False
    26. 

  26. 07/12/22 12:21:28 DAGMAN_MAX_SUBMITS_PER_INTERVAL setting: 100
    27. 

  27. 07/12/22 12:21:28 DAGMAN_AGGRESSIVE_SUBMIT setting: False
    28. 

  28. 07/12/22 12:21:28 DAGMAN_USER_LOG_SCAN_INTERVAL setting: 5
    29. 

  29. 07/12/22 12:21:28 DAGMAN_QUEUE_UPDATE_INTERVAL setting: 300
    30. 

  30. 07/12/22 12:21:28 DAGMAN_DEFAULT_PRIORITY setting: 0
    31. 

  31. 07/12/22 12:21:28 DAGMAN_SUPPRESS_NOTIFICATION setting: True
    32. 

  32. 07/12/22 12:21:28 allow_events (DAGMAN_ALLOW_EVENTS) setting: 114
    33. 

  33. 07/12/22 12:21:28 DAGMAN_RETRY_SUBMIT_FIRST setting: True
    34. 

  34. 07/12/22 12:21:28 DAGMAN_RETRY_NODE_FIRST setting: False
    35. 

  35. 07/12/22 12:21:28 DAGMAN_MAX_JOBS_IDLE setting: 1000
    36. 

  36. 07/12/22 12:21:28 DAGMAN_MAX_JOBS_SUBMITTED setting: 0
    37. 

  37. 07/12/22 12:21:28 DAGMAN_MAX_PRE_SCRIPTS setting: 20
    38. 

  38. 07/12/22 12:21:28 DAGMAN_MAX_POST_SCRIPTS setting: 20
    39. 

  39. 07/12/22 12:21:28 DAGMAN_MUNGE_NODE_NAMES setting: True
    40. 

  40. 07/12/22 12:21:28 DAGMAN_PROHIBIT_MULTI_JOBS setting: False
    41. 

  41. 07/12/22 12:21:28 DAGMAN_SUBMIT_DEPTH_FIRST setting: False
    42. 

  42. 07/12/22 12:21:28 DAGMAN_ALWAYS_RUN_POST setting: False
    43. 

  43. 07/12/22 12:21:28 DAGMAN_CONDOR_SUBMIT_EXE setting: /usr/bin/condor_submit
    44. 

  44. 07/12/22 12:21:28 DAGMAN_USE_CONDOR_SUBMIT setting: True
    45. 

  45. 07/12/22 12:21:28 DAGMAN_ABORT_DUPLICATES setting: True
    46. 

  46. 07/12/22 12:21:28 DAGMAN_ABORT_ON_SCARY_SUBMIT setting: True
    47. 

  47. 07/12/22 12:21:28 DAGMAN_PENDING_REPORT_INTERVAL setting: 600
    48. 

  48. 07/12/22 12:21:28 DAGMAN_AUTO_RESCUE setting: True
    49. 

  49. 07/12/22 12:21:28 DAGMAN_MAX_RESCUE_NUM setting: 100
    50. 

  50. 07/12/22 12:21:28 DAGMAN_WRITE_PARTIAL_RESCUE setting: True
    51. 

  51. 07/12/22 12:21:28 DAGMAN_DEFAULT_NODE_LOG setting: @(DAG_DIR)/@(DAG_FILE).nodes.log
    52. 

  52. 07/12/22 12:21:28 DAGMAN_GENERATE_SUBDAG_SUBMITS setting: True
    53. 

  53. 07/12/22 12:21:28 DAGMAN_MAX_JOB_HOLDS setting: 100
    54. 

  54. 07/12/22 12:21:28 DAGMAN_HOLD_CLAIM_TIME setting: 20
    55. 

  55. 07/12/22 12:21:28 ALL_DEBUG setting: 
    56. 

  56. 07/12/22 12:21:28 DAGMAN_DEBUG setting: 
    57. 

  57. 07/12/22 12:21:28 DAGMAN_SUPPRESS_JOB_LOGS setting: False
    58. 

  58. 07/12/22 12:21:28 DAGMAN_REMOVE_NODE_JOBS setting: True
    59. 

  59. 07/12/22 12:21:28 argv[0] == "condor_scheduniv_exec.931968.0"
    60. 

  60. 07/12/22 12:21:28 argv[1] == "-Lockfile"
    61. 

  61. 07/12/22 12:21:28 argv[2] == "code/process.condor_dag.lock"
    62. 

  62. 07/12/22 12:21:28 argv[3] == "-AutoRescue"
    63. 

  63. 07/12/22 12:21:28 argv[4] == "1"
    64. 

  64. 07/12/22 12:21:28 argv[5] == "-DoRescueFrom"
    65. 

  65. 07/12/22 12:21:28 argv[6] == "0"
    66. 

  66. 07/12/22 12:21:28 argv[7] == "-Dag"
    67. 

  67. 07/12/22 12:21:28 argv[8] == "code/process.condor_dag"
    68. 

  68. 07/12/22 12:21:28 argv[9] == "-Suppress_notification"
    69. 

  69. 07/12/22 12:21:28 argv[10] == "-CsdVersion"
    70. 

  70. 07/12/22 12:21:28 argv[11] == "$CondorVersion: 8.8.6 Nov 19 2019 BuildID: Debian-8.8.6~dfsg.1-1 Debian-8.8.6~dfsg.1-1 $"
    71. 

  71. 07/12/22 12:21:28 argv[12] == "-Dagman"
    72. 

  72. 07/12/22 12:21:28 argv[13] == "/usr/bin/condor_dagman"
    73. 

  73. 07/12/22 12:21:28 Workflow batch-name: <process.condor_dag+931968>
    74. 

  74. 07/12/22 12:21:28 Workflow accounting_group: <>
    75. 

  75. 07/12/22 12:21:28 Workflow accounting_group_user: <>
    76. 

  76. 07/12/22 12:21:28 Warning: failed to get attribute DAGNodeName
    77. 

  77. 07/12/22 12:21:28 DAGMAN_LOG_ON_NFS_IS_ERROR setting: False
    78. 

  78. 07/12/22 12:21:28 Default node log file is: </data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log>
    79. 

  79. 07/12/22 12:21:28 DAG Lockfile will be written to code/process.condor_dag.lock
    80. 

  80. 07/12/22 12:21:28 DAG Input file is code/process.condor_dag
    81. 

  81. 07/12/22 12:21:28 Parsing 1 dagfiles
    82. 

  82. 07/12/22 12:21:28 Parsing code/process.condor_dag ...
    83. 

  83. 07/12/22 12:21:28 Dag contains 5 total jobs
    84. 

  84. 07/12/22 12:21:28 Sleeping for 3 seconds to ensure ProcessId uniqueness
    85. 

  85. 07/12/22 12:21:31 Bootstrapping...
    86. 

  86. 07/12/22 12:21:31 Number of pre-completed nodes: 0
    87. 

  87. 07/12/22 12:21:31 MultiLogFiles: truncating log file /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
    88. 

  88. 07/12/22 12:21:31 DAG status: 0 (DAG_STATUS_OK)
    89. 

  89. 07/12/22 12:21:31 Of 5 nodes total:
    90. 

  90. 07/12/22 12:21:31  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
    91. 

  91. 07/12/22 12:21:31   ===     ===      ===     ===     ===        ===      ===
    92. 

  92. 07/12/22 12:21:31     0       0        0       0       5          0        0
    93. 

  93. 07/12/22 12:21:31 0 job proc(s) currently held
    94. 

  94. 07/12/22 12:21:31 DAGMan Runtime Statistics: [ EventCycleTimeSum = 0.0; EventCycleTimeCount = 0.0; LogProcessCycleTimeSum = 0.0; SleepCycleTimeSum = 0.0; LogProcessCycleTimeCount = 0.0; SleepCycleTimeCount = 0.0; SubmitCycleTimeCount = 0.0; SubmitCycleTimeSum = 0.0; ]
    95. 

  95. 07/12/22 12:21:31 Registering condor_event_timer...
    96. 

  96. 07/12/22 12:21:32 Submitting HTCondor Node sub-001 job(s)...
    97. 

  97. 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
    98. 

  98. 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
    99. 

  99. 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
    100. 

  100. 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-001 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-001 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-001 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
    101. 

  101. 07/12/22 12:21:32 From submit: Submitting job(s).
    102. 

  102. 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931969.
    103. 

  103. 07/12/22 12:21:32 	assigned HTCondor ID (931969.0.0)
    104. 

  104. 07/12/22 12:21:32 Submitting HTCondor Node sub-002 job(s)...
    105. 

  105. 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
    106. 

  106. 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
    107. 

  107. 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
    108. 

  108. 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-002 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-002 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-002 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
    109. 

  109. 07/12/22 12:21:32 From submit: Submitting job(s).
    110. 

  110. 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931970.
    111. 

  111. 07/12/22 12:21:32 	assigned HTCondor ID (931970.0.0)
    112. 

  112. 07/12/22 12:21:32 Submitting HTCondor Node sub-005 job(s)...
    113. 

  113. 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
    114. 

  114. 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
    115. 

  115. 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
    116. 

  116. 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-005 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-005 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-005 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
    117. 

  117. 07/12/22 12:21:32 From submit: Submitting job(s).
    118. 

  118. 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931971.
    119. 

  119. 07/12/22 12:21:32 	assigned HTCondor ID (931971.0.0)
    120. 

  120. 07/12/22 12:21:32 Submitting HTCondor Node sub-004 job(s)...
    121. 

  121. 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
    122. 

  122. 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
    123. 

  123. 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
    124. 

  124. 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-004 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-004 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-004 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
    125. 

  125. 07/12/22 12:21:32 From submit: Submitting job(s).
    126. 

  126. 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931972.
    127. 

  127. 07/12/22 12:21:32 	assigned HTCondor ID (931972.0.0)
    128. 

  128. 07/12/22 12:21:32 Submitting HTCondor Node sub-003 job(s)...
    129. 

  129. 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
    130. 

  130. 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
    131. 

  131. 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
    132. 

  132. 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-003 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-003 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-003 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
    133. 

  133. 07/12/22 12:21:32 From submit: Submitting job(s).
    134. 

  134. 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931973.
    135. 

  135. 07/12/22 12:21:32 	assigned HTCondor ID (931973.0.0)
    136. 

  136. 07/12/22 12:21:32 Just submitted 5 jobs this cycle...
    137. 

  137. 07/12/22 12:21:32 DAG status: 0 (DAG_STATUS_OK)
    138. 

  138. 07/12/22 12:21:32 Of 5 nodes total:
    139. 

  139. 07/12/22 12:21:32  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
    140. 

  140. 07/12/22 12:21:32   ===     ===      ===     ===     ===        ===      ===
    141. 

  141. 07/12/22 12:21:32     0       0        5       0       0          0        0
    142. 

  142. 07/12/22 12:21:32 0 job proc(s) currently held
    143. 

  143. 07/12/22 12:21:32 DAGMan Runtime Statistics: [ EventCycleTimeSum = 0.0; SleepCycleTimeCount = 0.0; EventCycleTimeCount = 0.0; SubmitCycleTimeStd = 0.2237379550933838; SubmitCycleTimeMax = 0.2237379550933838; LogProcessCycleTimeSum = 0.0; LogProcessCycleTimeCount = 0.0; SubmitCycleTimeMin = 0.2237379550933838; SubmitCycleTimeCount = 1.0; SubmitCycleTimeSum = 0.2237379550933838; SleepCycleTimeSum = 0.0; SubmitCycleTimeAvg = 0.2237379550933838; ]
    144. 

  144. 07/12/22 12:21:37 Currently monitoring 1 HTCondor log file(s)
    145. 

  145. 07/12/22 12:21:37 Reassigning the id of job sub-001 from (931969.0.0) to (931969.0.0)
    146. 

  146. 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-001 (931969.0.0) {07/12/22 12:21:32}
    147. 

  147. 07/12/22 12:21:37 Number of idle job procs: 1
    148. 

  148. 07/12/22 12:21:37 Reassigning the id of job sub-002 from (931970.0.0) to (931970.0.0)
    149. 

  149. 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-002 (931970.0.0) {07/12/22 12:21:32}
    150. 

  150. 07/12/22 12:21:37 Number of idle job procs: 2
    151. 

  151. 07/12/22 12:21:37 Reassigning the id of job sub-005 from (931971.0.0) to (931971.0.0)
    152. 

  152. 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-005 (931971.0.0) {07/12/22 12:21:32}
    153. 

  153. 07/12/22 12:21:37 Number of idle job procs: 3
    154. 

  154. 07/12/22 12:21:37 Reassigning the id of job sub-004 from (931972.0.0) to (931972.0.0)
    155. 

  155. 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-004 (931972.0.0) {07/12/22 12:21:32}
    156. 

  156. 07/12/22 12:21:37 Number of idle job procs: 4
    157. 

  157. 07/12/22 12:21:37 Reassigning the id of job sub-003 from (931973.0.0) to (931973.0.0)
    158. 

  158. 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-003 (931973.0.0) {07/12/22 12:21:32}
    159. 

  159. 07/12/22 12:21:37 Number of idle job procs: 5
    160. 

  160. 07/12/22 12:31:38 601 seconds since last log event
    161. 

  161. 07/12/22 12:31:38 Pending DAG nodes:
    162. 

  162. 07/12/22 12:31:38   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    163. 

  163. 07/12/22 12:31:38   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    164. 

  164. 07/12/22 12:31:38   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    165. 

  165. 07/12/22 12:31:38   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    166. 

  166. 07/12/22 12:31:38   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    167. 

  167. 07/12/22 12:41:38 1201 seconds since last log event
    168. 

  168. 07/12/22 12:41:38 Pending DAG nodes:
    169. 

  169. 07/12/22 12:41:38   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    170. 

  170. 07/12/22 12:41:38   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    171. 

  171. 07/12/22 12:41:38   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    172. 

  172. 07/12/22 12:41:38   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    173. 

  173. 07/12/22 12:41:38   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    174. 

  174. 07/12/22 12:51:39 1802 seconds since last log event
    175. 

  175. 07/12/22 12:51:39 Pending DAG nodes:
    176. 

  176. 07/12/22 12:51:39   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    177. 

  177. 07/12/22 12:51:39   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    178. 

  178. 07/12/22 12:51:39   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    179. 

  179. 07/12/22 12:51:39   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    180. 

  180. 07/12/22 12:51:39   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    181. 

  181. 07/12/22 13:01:40 2403 seconds since last log event
    182. 

  182. 07/12/22 13:01:40 Pending DAG nodes:
    183. 

  183. 07/12/22 13:01:40   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    184. 

  184. 07/12/22 13:01:40   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    185. 

  185. 07/12/22 13:01:40   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    186. 

  186. 07/12/22 13:01:40   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    187. 

  187. 07/12/22 13:01:40   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    188. 

  188. 07/12/22 13:11:40 3003 seconds since last log event
    189. 

  189. 07/12/22 13:11:40 Pending DAG nodes:
    190. 

  190. 07/12/22 13:11:40   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    191. 

  191. 07/12/22 13:11:40   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    192. 

  192. 07/12/22 13:11:40   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    193. 

  193. 07/12/22 13:11:40   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    194. 

  194. 07/12/22 13:11:40   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    195. 

  195. 07/12/22 13:21:41 3604 seconds since last log event
    196. 

  196. 07/12/22 13:21:41 Pending DAG nodes:
    197. 

  197. 07/12/22 13:21:41   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    198. 

  198. 07/12/22 13:21:41   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    199. 

  199. 07/12/22 13:21:41   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    200. 

  200. 07/12/22 13:21:41   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    201. 

  201. 07/12/22 13:21:41   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    202. 

  202. 07/12/22 13:31:41 4204 seconds since last log event
    203. 

  203. 07/12/22 13:31:41 Pending DAG nodes:
    204. 

  204. 07/12/22 13:31:41   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    205. 

  205. 07/12/22 13:31:41   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    206. 

  206. 07/12/22 13:31:41   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    207. 

  207. 07/12/22 13:31:41   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    208. 

  208. 07/12/22 13:31:41   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    209. 

  209. 07/12/22 13:41:42 4805 seconds since last log event
    210. 

  210. 07/12/22 13:41:42 Pending DAG nodes:
    211. 

  211. 07/12/22 13:41:42   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    212. 

  212. 07/12/22 13:41:42   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    213. 

  213. 07/12/22 13:41:42   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    214. 

  214. 07/12/22 13:41:42   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    215. 

  215. 07/12/22 13:41:42   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    216. 

  216. 07/12/22 13:51:42 5405 seconds since last log event
    217. 

  217. 07/12/22 13:51:42 Pending DAG nodes:
    218. 

  218. 07/12/22 13:51:42   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    219. 

  219. 07/12/22 13:51:42   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    220. 

  220. 07/12/22 13:51:42   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    221. 

  221. 07/12/22 13:51:42   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    222. 

  222. 07/12/22 13:51:42   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    223. 

  223. 07/12/22 14:01:42 6005 seconds since last log event
    224. 

  224. 07/12/22 14:01:42 Pending DAG nodes:
    225. 

  225. 07/12/22 14:01:42   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    226. 

  226. 07/12/22 14:01:42   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    227. 

  227. 07/12/22 14:01:42   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    228. 

  228. 07/12/22 14:01:42   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    229. 

  229. 07/12/22 14:01:42   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    230. 

  230. 07/12/22 14:11:43 6606 seconds since last log event
    231. 

  231. 07/12/22 14:11:43 Pending DAG nodes:
    232. 

  232. 07/12/22 14:11:43   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    233. 

  233. 07/12/22 14:11:43   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    234. 

  234. 07/12/22 14:11:43   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    235. 

  235. 07/12/22 14:11:43   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    236. 

  236. 07/12/22 14:11:43   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    237. 

  237. 07/12/22 14:21:43 7206 seconds since last log event
    238. 

  238. 07/12/22 14:21:43 Pending DAG nodes:
    239. 

  239. 07/12/22 14:21:43   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    240. 

  240. 07/12/22 14:21:43   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    241. 

  241. 07/12/22 14:21:43   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    242. 

  242. 07/12/22 14:21:43   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    243. 

  243. 07/12/22 14:21:43   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    244. 

  244. 07/12/22 14:31:44 7807 seconds since last log event
    245. 

  245. 07/12/22 14:31:44 Pending DAG nodes:
    246. 

  246. 07/12/22 14:31:44   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    247. 

  247. 07/12/22 14:31:44   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    248. 

  248. 07/12/22 14:31:44   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    249. 

  249. 07/12/22 14:31:44   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    250. 

  250. 07/12/22 14:31:44   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    251. 

  251. 07/12/22 14:41:44 8407 seconds since last log event
    252. 

  252. 07/12/22 14:41:44 Pending DAG nodes:
    253. 

  253. 07/12/22 14:41:44   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    254. 

  254. 07/12/22 14:41:44   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    255. 

  255. 07/12/22 14:41:44   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    256. 

  256. 07/12/22 14:41:44   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    257. 

  257. 07/12/22 14:41:44   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    258. 

  258. 07/12/22 14:51:45 9008 seconds since last log event
    259. 

  259. 07/12/22 14:51:45 Pending DAG nodes:
    260. 

  260. 07/12/22 14:51:45   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    261. 

  261. 07/12/22 14:51:45   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    262. 

  262. 07/12/22 14:51:45   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    263. 

  263. 07/12/22 14:51:45   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    264. 

  264. 07/12/22 14:51:45   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    265. 

  265. 07/12/22 15:01:45 9608 seconds since last log event
    266. 

  266. 07/12/22 15:01:45 Pending DAG nodes:
    267. 

  267. 07/12/22 15:01:45   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    268. 

  268. 07/12/22 15:01:45   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    269. 

  269. 07/12/22 15:01:45   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    270. 

  270. 07/12/22 15:01:45   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    271. 

  271. 07/12/22 15:01:45   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    272. 

  272. 07/12/22 15:11:46 10209 seconds since last log event
    273. 

  273. 07/12/22 15:11:46 Pending DAG nodes:
    274. 

  274. 07/12/22 15:11:46   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    275. 

  275. 07/12/22 15:11:46   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    276. 

  276. 07/12/22 15:11:46   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    277. 

  277. 07/12/22 15:11:46   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    278. 

  278. 07/12/22 15:11:46   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    279. 

  279. 07/12/22 15:21:46 10809 seconds since last log event
    280. 

  280. 07/12/22 15:21:46 Pending DAG nodes:
    281. 

  281. 07/12/22 15:21:46   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    282. 

  282. 07/12/22 15:21:46   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    283. 

  283. 07/12/22 15:21:46   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    284. 

  284. 07/12/22 15:21:46   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    285. 

  285. 07/12/22 15:21:46   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    286. 

  286. 07/12/22 15:31:47 11410 seconds since last log event
    287. 

  287. 07/12/22 15:31:47 Pending DAG nodes:
    288. 

  288. 07/12/22 15:31:47   Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
    289. 

  289. 07/12/22 15:31:47   Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
    290. 

  290. 07/12/22 15:31:47   Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
    291. 

  291. 07/12/22 15:31:47   Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
    292. 

  292. 07/12/22 15:31:47   Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
    293. 

  293. 07/12/22 15:36:07 Currently monitoring 1 HTCondor log file(s)
    294. 

  294. 07/12/22 15:36:07 Event: ULOG_EXECUTE for HTCondor Node sub-003 (931973.0.0) {07/12/22 15:36:06}
    295. 

  295. 07/12/22 15:36:07 Number of idle job procs: 4
    296. 

  296. 07/12/22 15:36:07 Event: ULOG_EXECUTE for HTCondor Node sub-004 (931972.0.0) {07/12/22 15:36:06}
    297. 

  297. 07/12/22 15:36:07 Number of idle job procs: 3
    298. 

  298. 07/12/22 15:36:22 Currently monitoring 1 HTCondor log file(s)
    299. 

  299. 07/12/22 15:36:22 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-004 (931972.0.0) {07/12/22 15:36:17}
    300. 

  300. 07/12/22 15:36:22 Number of idle job procs: 3
    301. 

  301. 07/12/22 15:36:22 Node sub-004 job proc (931972.0.0) failed with status 1.
    302. 

  302. 07/12/22 15:36:22 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-003 (931973.0.0) {07/12/22 15:36:21}
    303. 

  303. 07/12/22 15:36:22 Number of idle job procs: 3
    304. 

  304. 07/12/22 15:36:22 Node sub-003 job proc (931973.0.0) failed with status 1.
    305. 

  305. 07/12/22 15:36:22 DAG status: 2 (DAG_STATUS_NODE_FAILED)
    306. 

  306. 07/12/22 15:36:22 Of 5 nodes total:
    307. 

  307. 07/12/22 15:36:22  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
    308. 

  308. 07/12/22 15:36:22   ===     ===      ===     ===     ===        ===      ===
    309. 

  309. 07/12/22 15:36:22     0       0        3       0       0          0        2
    310. 

  310. 07/12/22 15:36:22 0 job proc(s) currently held
    311. 

  311. 07/12/22 15:36:22 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.005493951815109337; EventCycleTimeMax = 0.2239258289337158; EventCycleTimeMin = 2.09808349609375E-05; EventCycleTimeAvg = 0.0003880991102897958; EventCycleTimeSum = 0.9065995216369629; EventCycleTimeCount = 2336.0; SleepCycleTimeStd = 0.02079933325079659; LogProcessCycleTimeMin = 0.0005269050598144531; LogProcessCycleTimeMax = 0.0007569789886474609; SubmitCycleTimeCount = 2337.0; LogProcessCycleTimeAvg = 0.0006422996520996094; LogProcessCycleTimeCount = 3.0; LogProcessCycleTimeStd = 0.0001150386320991401; LogProcessCycleTimeSum = 0.001926898956298828; SubmitCycleTimeSum = 0.5614738464355469; SubmitCycleTimeAvg = 0.000240254106305326; SleepCycleTimeSum = 11689.4080324173; SubmitCycleTimeMin = 7.867813110351562E-06; SubmitCycleTimeMax = 0.2237379550933838; SleepCycleTimeAvg = 5.004027411137542; SleepCycleTimeMax = 5.068968057632446; SleepCycleTimeCount = 2336.0; SubmitCycleTimeStd = 0.00498719534509626; SleepCycleTimeMin = 4.004912853240967; ]
    312. 

  312. 07/12/22 15:40:42 Currently monitoring 1 HTCondor log file(s)
    313. 

  313. 07/12/22 15:40:42 Event: ULOG_EXECUTE for HTCondor Node sub-001 (931969.0.0) {07/12/22 15:40:38}
    314. 

  314. 07/12/22 15:40:42 Number of idle job procs: 2
    315. 

  315. 07/12/22 15:40:42 Event: ULOG_EXECUTE for HTCondor Node sub-002 (931970.0.0) {07/12/22 15:40:38}
    316. 

  316. 07/12/22 15:40:42 Number of idle job procs: 1
    317. 

  317. 07/12/22 15:40:42 Event: ULOG_EXECUTE for HTCondor Node sub-005 (931971.0.0) {07/12/22 15:40:38}
    318. 

  318. 07/12/22 15:40:42 Number of idle job procs: 0
    319. 

  319. 07/12/22 15:40:52 Currently monitoring 1 HTCondor log file(s)
    320. 

  320. 07/12/22 15:40:52 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-005 (931971.0.0) {07/12/22 15:40:50}
    321. 

  321. 07/12/22 15:40:52 Number of idle job procs: 0
    322. 

  322. 07/12/22 15:40:52 Node sub-005 job proc (931971.0.0) failed with status 1.
    323. 

  323. 07/12/22 15:40:52 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-002 (931970.0.0) {07/12/22 15:40:50}
    324. 

  324. 07/12/22 15:40:52 Number of idle job procs: 0
    325. 

  325. 07/12/22 15:40:52 Node sub-002 job proc (931970.0.0) failed with status 1.
    326. 

  326. 07/12/22 15:40:52 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-001 (931969.0.0) {07/12/22 15:40:50}
    327. 

  327. 07/12/22 15:40:52 Number of idle job procs: 0
    328. 

  328. 07/12/22 15:40:52 Node sub-001 job proc (931969.0.0) failed with status 1.
    329. 

  329. 07/12/22 15:40:52 DAG status: 2 (DAG_STATUS_NODE_FAILED)
    330. 

  330. 07/12/22 15:40:52 Of 5 nodes total:
    331. 

  331. 07/12/22 15:40:52  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
    332. 

  332. 07/12/22 15:40:52   ===     ===      ===     ===     ===        ===      ===
    333. 

  333. 07/12/22 15:40:52     0       0        0       0       0          0        5
    334. 

  334. 07/12/22 15:40:52 0 job proc(s) currently held
    335. 

  335. 07/12/22 15:40:52 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.005433997287527276; EventCycleTimeMax = 0.2239258289337158; EventCycleTimeMin = 2.09808349609375E-05; EventCycleTimeAvg = 0.0003853633313997021; EventCycleTimeSum = 0.9210183620452881; EventCycleTimeCount = 2390.0; SleepCycleTimeStd = 0.02056334142362197; LogProcessCycleTimeMin = 0.0005269050598144531; LogProcessCycleTimeMax = 0.0009350776672363281; SubmitCycleTimeCount = 2391.0; LogProcessCycleTimeAvg = 0.0006816387176513672; LogProcessCycleTimeCount = 5.0; LogProcessCycleTimeStd = 0.0001685829264184372; LogProcessCycleTimeSum = 0.003408193588256836; SubmitCycleTimeSum = 0.5647740364074707; SubmitCycleTimeAvg = 0.0002362082962808326; SleepCycleTimeSum = 11959.61773061752; SubmitCycleTimeMin = 7.867813110351562E-06; SubmitCycleTimeMax = 0.2237379550933838; SleepCycleTimeAvg = 5.004024155070093; SleepCycleTimeMax = 5.068968057632446; SleepCycleTimeCount = 2390.0; SubmitCycleTimeStd = 0.004930626397422213; SleepCycleTimeMin = 4.004912853240967; ]
    336. 

  336. 07/12/22 15:40:52 ERROR: the following job(s) failed:
    337. 

  337. 07/12/22 15:40:52 ---------------------- Job ----------------------
    338. 

  338. 07/12/22 15:40:52       Node Name: sub-001
    339. 

  339. 07/12/22 15:40:52            Noop: false
    340. 

  340. 07/12/22 15:40:52          NodeID: 0
    341. 

  341. 07/12/22 15:40:52     Node Status: STATUS_ERROR    
    342. 

  342. 07/12/22 15:40:52 Node return val: 1
    343. 

  343. 07/12/22 15:40:52           Error: Job proc (931969.0.0) failed with status 1
    344. 

  344. 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
    345. 

  345. 07/12/22 15:40:52  HTCondor Job ID: (931969.0.0)
    346. 

  346. 07/12/22 15:40:52       Q_PARENTS: <END>
    347. 

  347. 07/12/22 15:40:52       Q_WAITING: <END>
    348. 

  348. 07/12/22 15:40:52      Q_CHILDREN: <END>
    349. 

  349. 07/12/22 15:40:52 ---------------------- Job ----------------------
    350. 

  350. 07/12/22 15:40:52       Node Name: sub-002
    351. 

  351. 07/12/22 15:40:52            Noop: false
    352. 

  352. 07/12/22 15:40:52          NodeID: 1
    353. 

  353. 07/12/22 15:40:52     Node Status: STATUS_ERROR    
    354. 

  354. 07/12/22 15:40:52 Node return val: 1
    355. 

  355. 07/12/22 15:40:52           Error: Job proc (931970.0.0) failed with status 1
    356. 

  356. 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
    357. 

  357. 07/12/22 15:40:52  HTCondor Job ID: (931970.0.0)
    358. 

  358. 07/12/22 15:40:52       Q_PARENTS: <END>
    359. 

  359. 07/12/22 15:40:52       Q_WAITING: <END>
    360. 

  360. 07/12/22 15:40:52      Q_CHILDREN: <END>
    361. 

  361. 07/12/22 15:40:52 ---------------------- Job ----------------------
    362. 

  362. 07/12/22 15:40:52       Node Name: sub-005
    363. 

  363. 07/12/22 15:40:52            Noop: false
    364. 

  364. 07/12/22 15:40:52          NodeID: 2
    365. 

  365. 07/12/22 15:40:52     Node Status: STATUS_ERROR    
    366. 

  366. 07/12/22 15:40:52 Node return val: 1
    367. 

  367. 07/12/22 15:40:52           Error: Job proc (931971.0.0) failed with status 1
    368. 

  368. 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
    369. 

  369. 07/12/22 15:40:52  HTCondor Job ID: (931971.0.0)
    370. 

  370. 07/12/22 15:40:52       Q_PARENTS: <END>
    371. 

  371. 07/12/22 15:40:52       Q_WAITING: <END>
    372. 

  372. 07/12/22 15:40:52      Q_CHILDREN: <END>
    373. 

  373. 07/12/22 15:40:52 ---------------------- Job ----------------------
    374. 

  374. 07/12/22 15:40:52       Node Name: sub-004
    375. 

  375. 07/12/22 15:40:52            Noop: false
    376. 

  376. 07/12/22 15:40:52          NodeID: 3
    377. 

  377. 07/12/22 15:40:52     Node Status: STATUS_ERROR    
    378. 

  378. 07/12/22 15:40:52 Node return val: 1
    379. 

  379. 07/12/22 15:40:52           Error: Job proc (931972.0.0) failed with status 1
    380. 

  380. 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
    381. 

  381. 07/12/22 15:40:52  HTCondor Job ID: (931972.0.0)
    382. 

  382. 07/12/22 15:40:52       Q_PARENTS: <END>
    383. 

  383. 07/12/22 15:40:52       Q_WAITING: <END>
    384. 

  384. 07/12/22 15:40:52      Q_CHILDREN: <END>
    385. 

  385. 07/12/22 15:40:52 ---------------------- Job ----------------------
    386. 

  386. 07/12/22 15:40:52       Node Name: sub-003
    387. 

  387. 07/12/22 15:40:52            Noop: false
    388. 

  388. 07/12/22 15:40:52          NodeID: 4
    389. 

  389. 07/12/22 15:40:52     Node Status: STATUS_ERROR    
    390. 

  390. 07/12/22 15:40:52 Node return val: 1
    391. 

  391. 07/12/22 15:40:52           Error: Job proc (931973.0.0) failed with status 1
    392. 

  392. 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
    393. 

  393. 07/12/22 15:40:52  HTCondor Job ID: (931973.0.0)
    394. 

  394. 07/12/22 15:40:52       Q_PARENTS: <END>
    395. 

  395. 07/12/22 15:40:52       Q_WAITING: <END>
    396. 

  396. 07/12/22 15:40:52      Q_CHILDREN: <END>
    397. 

  397. 07/12/22 15:40:52 ---------------------------------------	<END>
    398. 

  398. 07/12/22 15:40:52 Aborting DAG...
    399. 

  399. 07/12/22 15:40:52 Writing Rescue DAG to code/process.condor_dag.rescue001...
    400. 

  400. 07/12/22 15:40:52 Removing submitted jobs...
    401. 

  401. 07/12/22 15:40:52 Removing any/all submitted HTCondor jobs...
    402. 

  402. 07/12/22 15:40:52 Running: /usr/bin/condor_rm -const DAGManJobId' '=?=' '931968
    403. 

  403. 07/12/22 15:40:52 Note: 0 total job deferrals because of -MaxJobs limit (0)
    404. 

  404. 07/12/22 15:40:52 Note: 0 total job deferrals because of -MaxIdle limit (1000)
    405. 

  405. 07/12/22 15:40:52 Note: 0 total job deferrals because of node category throttles
    406. 

  406. 07/12/22 15:40:52 Note: 0 total PRE script deferrals because of -MaxPre limit (20) or DEFER
    407. 

  407. 07/12/22 15:40:52 Note: 0 total POST script deferrals because of -MaxPost limit (20) or DEFER
    408. 

  408. 07/12/22 15:40:52 DAG status: 2 (DAG_STATUS_NODE_FAILED)
    409. 

  409. 07/12/22 15:40:52 Of 5 nodes total:
    410. 

  410. 07/12/22 15:40:52  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
    411. 

  411. 07/12/22 15:40:52   ===     ===      ===     ===     ===        ===      ===
    412. 

  412. 07/12/22 15:40:52     0       0        0       0       0          0        5
    413. 

  413. 07/12/22 15:40:52 0 job proc(s) currently held
    414. 

  414. 07/12/22 15:40:52 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.005433997287527276; EventCycleTimeMax = 0.2239258289337158; EventCycleTimeMin = 2.09808349609375E-05; EventCycleTimeAvg = 0.0003853633313997021; EventCycleTimeSum = 0.9210183620452881; EventCycleTimeCount = 2390.0; SleepCycleTimeStd = 0.02056334142362197; LogProcessCycleTimeMin = 0.0005269050598144531; LogProcessCycleTimeMax = 0.0009350776672363281; SubmitCycleTimeCount = 2391.0; LogProcessCycleTimeAvg = 0.0006816387176513672; LogProcessCycleTimeCount = 5.0; LogProcessCycleTimeStd = 0.0001685829264184372; LogProcessCycleTimeSum = 0.003408193588256836; SubmitCycleTimeSum = 0.5647740364074707; SubmitCycleTimeAvg = 0.0002362082962808326; SleepCycleTimeSum = 11959.61773061752; SubmitCycleTimeMin = 7.867813110351562E-06; SubmitCycleTimeMax = 0.2237379550933838; SleepCycleTimeAvg = 5.004024155070093; SleepCycleTimeMax = 5.068968057632446; SleepCycleTimeCount = 2390.0; SubmitCycleTimeStd = 0.004930626397422213; SleepCycleTimeMin = 4.004912853240967; ]
    415. 

  415. 07/12/22 15:40:52 Wrote metrics file code/process.condor_dag.metrics.
    416. 

  416. 07/12/22 15:40:52 Metrics not sent because of PEGASUS_METRICS or CONDOR_DEVELOPERS setting.
    417. 

  417. 07/12/22 15:40:52 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.005433997287527276; EventCycleTimeMax = 0.2239258289337158; EventCycleTimeMin = 2.09808349609375E-05; EventCycleTimeAvg = 0.0003853633313997021; EventCycleTimeSum = 0.9210183620452881; EventCycleTimeCount = 2390.0; SleepCycleTimeStd = 0.02056334142362197; LogProcessCycleTimeMin = 0.0005269050598144531; LogProcessCycleTimeMax = 0.0009350776672363281; SubmitCycleTimeCount = 2391.0; LogProcessCycleTimeAvg = 0.0006816387176513672; LogProcessCycleTimeCount = 5.0; LogProcessCycleTimeStd = 0.0001685829264184372; LogProcessCycleTimeSum = 0.003408193588256836; SubmitCycleTimeSum = 0.5647740364074707; SubmitCycleTimeAvg = 0.0002362082962808326; SleepCycleTimeSum = 11959.61773061752; SubmitCycleTimeMin = 7.867813110351562E-06; SubmitCycleTimeMax = 0.2237379550933838; SleepCycleTimeAvg = 5.004024155070093; SleepCycleTimeMax = 5.068968057632446; SleepCycleTimeCount = 2390.0; SubmitCycleTimeStd = 0.004930626397422213; SleepCycleTimeMin = 4.004912853240967; ]
    418. 

  418. 07/12/22 15:40:52 **** condor_scheduniv_exec.931968.0 (condor_DAGMAN) pid 2264448 EXITING WITH STATUS 1
    419.