ds003639_ReproVBM/code/process.condor_dag.dagman.out

07/12/22 12:10:11 ******************************************************
07/12/22 12:10:11 ** condor_scheduniv_exec.930575.0 (CONDOR_DAGMAN) STARTING UP
07/12/22 12:10:11 ** /usr/bin/condor_dagman
07/12/22 12:10:11 ** SubsystemInfo: name=DAGMAN type=DAGMAN(10) class=DAEMON(1)
07/12/22 12:10:11 ** Configuration: subsystem:DAGMAN local:<NONE> class:DAEMON
07/12/22 12:10:11 ** $CondorVersion: 8.8.6 Nov 19 2019 BuildID: Debian-8.8.6~dfsg.1-1 Debian-8.8.6~dfsg.1-1 $
07/12/22 12:10:11 ** $CondorPlatform: X86_64-Debian_10 $
07/12/22 12:10:11 ** PID = 2158359
07/12/22 12:10:11 ** Log last touched time unavailable (No such file or directory)
07/12/22 12:10:11 ******************************************************
07/12/22 12:10:11 Using config source: /etc/condor/condor_config
07/12/22 12:10:11 Using local config sources: 
07/12/22 12:10:11    /etc/condor/config.d/50_inm7_master
07/12/22 12:10:11    /etc/condor/condor_config.local
07/12/22 12:10:11 config Macros = 81, Sorted = 81, StringBytes = 2405, TablesBytes = 2972
07/12/22 12:10:11 CLASSAD_CACHING is ENABLED
07/12/22 12:10:11 Daemon Log is logging: D_ALWAYS D_ERROR
07/12/22 12:10:11 DaemonCore: No command port requested.
07/12/22 12:10:11 DAGMAN_USE_STRICT setting: 1
07/12/22 12:10:11 DAGMAN_VERBOSITY setting: 3
07/12/22 12:10:11 DAGMAN_DEBUG_CACHE_SIZE setting: 5242880
07/12/22 12:10:11 DAGMAN_DEBUG_CACHE_ENABLE setting: False
07/12/22 12:10:11 DAGMAN_SUBMIT_DELAY setting: 0
07/12/22 12:10:11 DAGMAN_MAX_SUBMIT_ATTEMPTS setting: 6
07/12/22 12:10:11 DAGMAN_STARTUP_CYCLE_DETECT setting: False
07/12/22 12:10:11 DAGMAN_MAX_SUBMITS_PER_INTERVAL setting: 100
07/12/22 12:10:11 DAGMAN_AGGRESSIVE_SUBMIT setting: False
07/12/22 12:10:11 DAGMAN_USER_LOG_SCAN_INTERVAL setting: 5
07/12/22 12:10:11 DAGMAN_QUEUE_UPDATE_INTERVAL setting: 300
07/12/22 12:10:11 DAGMAN_DEFAULT_PRIORITY setting: 0
07/12/22 12:10:11 DAGMAN_SUPPRESS_NOTIFICATION setting: True
07/12/22 12:10:11 allow_events (DAGMAN_ALLOW_EVENTS) setting: 114
07/12/22 12:10:11 DAGMAN_RETRY_SUBMIT_FIRST setting: True
07/12/22 12:10:11 DAGMAN_RETRY_NODE_FIRST setting: False
07/12/22 12:10:11 DAGMAN_MAX_JOBS_IDLE setting: 1000
07/12/22 12:10:11 DAGMAN_MAX_JOBS_SUBMITTED setting: 0
07/12/22 12:10:11 DAGMAN_MAX_PRE_SCRIPTS setting: 20
07/12/22 12:10:11 DAGMAN_MAX_POST_SCRIPTS setting: 20
07/12/22 12:10:11 DAGMAN_MUNGE_NODE_NAMES setting: True
07/12/22 12:10:11 DAGMAN_PROHIBIT_MULTI_JOBS setting: False
07/12/22 12:10:11 DAGMAN_SUBMIT_DEPTH_FIRST setting: False
07/12/22 12:10:11 DAGMAN_ALWAYS_RUN_POST setting: False
07/12/22 12:10:11 DAGMAN_CONDOR_SUBMIT_EXE setting: /usr/bin/condor_submit
07/12/22 12:10:11 DAGMAN_USE_CONDOR_SUBMIT setting: True
07/12/22 12:10:11 DAGMAN_ABORT_DUPLICATES setting: True
07/12/22 12:10:11 DAGMAN_ABORT_ON_SCARY_SUBMIT setting: True
07/12/22 12:10:11 DAGMAN_PENDING_REPORT_INTERVAL setting: 600
07/12/22 12:10:11 DAGMAN_AUTO_RESCUE setting: True
07/12/22 12:10:11 DAGMAN_MAX_RESCUE_NUM setting: 100
07/12/22 12:10:11 DAGMAN_WRITE_PARTIAL_RESCUE setting: True
07/12/22 12:10:11 DAGMAN_DEFAULT_NODE_LOG setting: @(DAG_DIR)/@(DAG_FILE).nodes.log
07/12/22 12:10:11 DAGMAN_GENERATE_SUBDAG_SUBMITS setting: True
07/12/22 12:10:11 DAGMAN_MAX_JOB_HOLDS setting: 100
07/12/22 12:10:11 DAGMAN_HOLD_CLAIM_TIME setting: 20
07/12/22 12:10:11 ALL_DEBUG setting: 
07/12/22 12:10:11 DAGMAN_DEBUG setting: 
07/12/22 12:10:11 DAGMAN_SUPPRESS_JOB_LOGS setting: False
07/12/22 12:10:11 DAGMAN_REMOVE_NODE_JOBS setting: True
07/12/22 12:10:11 argv[0] == "condor_scheduniv_exec.930575.0"
07/12/22 12:10:11 argv[1] == "-Lockfile"
07/12/22 12:10:11 argv[2] == "code/process.condor_dag.lock"
07/12/22 12:10:11 argv[3] == "-AutoRescue"
07/12/22 12:10:11 argv[4] == "1"
07/12/22 12:10:11 argv[5] == "-DoRescueFrom"
07/12/22 12:10:11 argv[6] == "0"
07/12/22 12:10:11 argv[7] == "-Dag"
07/12/22 12:10:11 argv[8] == "code/process.condor_dag"
07/12/22 12:10:11 argv[9] == "-Suppress_notification"
07/12/22 12:10:11 argv[10] == "-CsdVersion"
07/12/22 12:10:11 argv[11] == "$CondorVersion: 8.8.6 Nov 19 2019 BuildID: Debian-8.8.6~dfsg.1-1 Debian-8.8.6~dfsg.1-1 $"
07/12/22 12:10:11 argv[12] == "-Dagman"
07/12/22 12:10:11 argv[13] == "/usr/bin/condor_dagman"
07/12/22 12:10:11 Workflow batch-name: <process.condor_dag+930575>
07/12/22 12:10:11 Workflow accounting_group: <>
07/12/22 12:10:11 Workflow accounting_group_user: <>
07/12/22 12:10:11 Warning: failed to get attribute DAGNodeName
07/12/22 12:10:11 DAGMAN_LOG_ON_NFS_IS_ERROR setting: False
07/12/22 12:10:11 Default node log file is: </data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log>
07/12/22 12:10:11 DAG Lockfile will be written to code/process.condor_dag.lock
07/12/22 12:10:11 DAG Input file is code/process.condor_dag
07/12/22 12:10:11 Parsing 1 dagfiles
07/12/22 12:10:11 Parsing code/process.condor_dag ...
07/12/22 12:10:11 Dag contains 29 total jobs
07/12/22 12:10:11 Sleeping for 3 seconds to ensure ProcessId uniqueness
07/12/22 12:10:14 Bootstrapping...
07/12/22 12:10:14 Number of pre-completed nodes: 0
07/12/22 12:10:14 MultiLogFiles: truncating log file /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:14 DAG status: 0 (DAG_STATUS_OK)
07/12/22 12:10:14 Of 29 nodes total:
07/12/22 12:10:14  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 12:10:14   ===     ===      ===     ===     ===        ===      ===
07/12/22 12:10:14     0       0        0       0      29          0        0
07/12/22 12:10:14 0 job proc(s) currently held
07/12/22 12:10:14 DAGMan Runtime Statistics: [ EventCycleTimeSum = 0.0; EventCycleTimeCount = 0.0; LogProcessCycleTimeSum = 0.0; SleepCycleTimeSum = 0.0; LogProcessCycleTimeCount = 0.0; SleepCycleTimeCount = 0.0; SubmitCycleTimeCount = 0.0; SubmitCycleTimeSum = 0.0; ]
07/12/22 12:10:14 Registering condor_event_timer...
07/12/22 12:10:15 Submitting HTCondor Node sub-NC251 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC251 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC251 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC251 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930603.
07/12/22 12:10:15 	assigned HTCondor ID (930603.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC242 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC242 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC242 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC242 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930606.
07/12/22 12:10:15 	assigned HTCondor ID (930606.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC235 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC235 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC235 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC235 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930609.
07/12/22 12:10:15 	assigned HTCondor ID (930609.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC248 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC248 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC248 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC248 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930612.
07/12/22 12:10:15 	assigned HTCondor ID (930612.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC232 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC232 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC232 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC232 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930614.
07/12/22 12:10:15 	assigned HTCondor ID (930614.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC245 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC245 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC245 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC245 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930616.
07/12/22 12:10:15 	assigned HTCondor ID (930616.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC189 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC189 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC189 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC189 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930618.
07/12/22 12:10:15 	assigned HTCondor ID (930618.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC183 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC183 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC183 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC183 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930620.
07/12/22 12:10:15 	assigned HTCondor ID (930620.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC225 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC225 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC225 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC225 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930622.
07/12/22 12:10:15 	assigned HTCondor ID (930622.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC252 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC252 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC252 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC252 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930624.
07/12/22 12:10:15 	assigned HTCondor ID (930624.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC236 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC236 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC236 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC236 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:15 From submit: Submitting job(s).
07/12/22 12:10:15 From submit: 1 job(s) submitted to cluster 930626.
07/12/22 12:10:15 	assigned HTCondor ID (930626.0.0)
07/12/22 12:10:15 Submitting HTCondor Node sub-NC246 job(s)...
07/12/22 12:10:15 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:15 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:15 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:15 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC246 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC246 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC246 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930628.
07/12/22 12:10:16 	assigned HTCondor ID (930628.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC231 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC231 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC231 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC231 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930630.
07/12/22 12:10:16 	assigned HTCondor ID (930630.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC255 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC255 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC255 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC255 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930632.
07/12/22 12:10:16 	assigned HTCondor ID (930632.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC188 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC188 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC188 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC188 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930634.
07/12/22 12:10:16 	assigned HTCondor ID (930634.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC239 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC239 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC239 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC239 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930636.
07/12/22 12:10:16 	assigned HTCondor ID (930636.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC244 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC244 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC244 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC244 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930638.
07/12/22 12:10:16 	assigned HTCondor ID (930638.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC233 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC233 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC233 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC233 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930640.
07/12/22 12:10:16 	assigned HTCondor ID (930640.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC249 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC249 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC249 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC249 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930642.
07/12/22 12:10:16 	assigned HTCondor ID (930642.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC234 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC234 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC234 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC234 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930644.
07/12/22 12:10:16 	assigned HTCondor ID (930644.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC243 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC243 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC243 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC243 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930646.
07/12/22 12:10:16 	assigned HTCondor ID (930646.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC227 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC227 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC227 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC227 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930648.
07/12/22 12:10:16 	assigned HTCondor ID (930648.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC250 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC250 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC250 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC250 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930650.
07/12/22 12:10:16 	assigned HTCondor ID (930650.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC254 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC254 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC254 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC254 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930652.
07/12/22 12:10:16 	assigned HTCondor ID (930652.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC247 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC247 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC247 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC247 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930654.
07/12/22 12:10:16 	assigned HTCondor ID (930654.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC240 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC240 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC240 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC240 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930656.
07/12/22 12:10:16 	assigned HTCondor ID (930656.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC237 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC237 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC237 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC237 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930658.
07/12/22 12:10:16 	assigned HTCondor ID (930658.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC150 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC150 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC150 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC150 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930660.
07/12/22 12:10:16 	assigned HTCondor ID (930660.0.0)
07/12/22 12:10:16 Submitting HTCondor Node sub-NC216 job(s)...
07/12/22 12:10:16 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log
07/12/22 12:10:16 Masking the events recorded in the DAGMAN workflow log
07/12/22 12:10:16 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
07/12/22 12:10:16 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-NC216 -a +DAGManJobId' '=' '930575 -a DAGManJobId' '=' '930575 -batch-name process.condor_dag+930575 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-NC216 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003639_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-NC216 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
07/12/22 12:10:16 From submit: Submitting job(s).
07/12/22 12:10:16 From submit: 1 job(s) submitted to cluster 930662.
07/12/22 12:10:16 	assigned HTCondor ID (930662.0.0)
07/12/22 12:10:16 Just submitted 29 jobs this cycle...
07/12/22 12:10:16 DAG status: 0 (DAG_STATUS_OK)
07/12/22 12:10:16 Of 29 nodes total:
07/12/22 12:10:16  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 12:10:16   ===     ===      ===     ===     ===        ===      ===
07/12/22 12:10:16     0       0       29       0       0          0        0
07/12/22 12:10:16 0 job proc(s) currently held
07/12/22 12:10:16 DAGMan Runtime Statistics: [ EventCycleTimeSum = 0.0; SleepCycleTimeCount = 0.0; EventCycleTimeCount = 0.0; SubmitCycleTimeStd = 1.098250865936279; SubmitCycleTimeMax = 1.098250865936279; LogProcessCycleTimeSum = 0.0; LogProcessCycleTimeCount = 0.0; SubmitCycleTimeMin = 1.098250865936279; SubmitCycleTimeCount = 1.0; SubmitCycleTimeSum = 1.098250865936279; SleepCycleTimeSum = 0.0; SubmitCycleTimeAvg = 1.098250865936279; ]
07/12/22 12:10:21 Currently monitoring 1 HTCondor log file(s)
07/12/22 12:10:21 Reassigning the id of job sub-NC251 from (930603.0.0) to (930603.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC251 (930603.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 1
07/12/22 12:10:21 Reassigning the id of job sub-NC242 from (930606.0.0) to (930606.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC242 (930606.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 2
07/12/22 12:10:21 Reassigning the id of job sub-NC235 from (930609.0.0) to (930609.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC235 (930609.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 3
07/12/22 12:10:21 Reassigning the id of job sub-NC248 from (930612.0.0) to (930612.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC248 (930612.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 4
07/12/22 12:10:21 Reassigning the id of job sub-NC232 from (930614.0.0) to (930614.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC232 (930614.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 5
07/12/22 12:10:21 Reassigning the id of job sub-NC245 from (930616.0.0) to (930616.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC245 (930616.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 6
07/12/22 12:10:21 Reassigning the id of job sub-NC189 from (930618.0.0) to (930618.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC189 (930618.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 7
07/12/22 12:10:21 Reassigning the id of job sub-NC183 from (930620.0.0) to (930620.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC183 (930620.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 8
07/12/22 12:10:21 Reassigning the id of job sub-NC225 from (930622.0.0) to (930622.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC225 (930622.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 9
07/12/22 12:10:21 Reassigning the id of job sub-NC252 from (930624.0.0) to (930624.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC252 (930624.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 10
07/12/22 12:10:21 Reassigning the id of job sub-NC236 from (930626.0.0) to (930626.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC236 (930626.0.0) {07/12/22 12:10:15}
07/12/22 12:10:21 Number of idle job procs: 11
07/12/22 12:10:21 Reassigning the id of job sub-NC246 from (930628.0.0) to (930628.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC246 (930628.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 12
07/12/22 12:10:21 Reassigning the id of job sub-NC231 from (930630.0.0) to (930630.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC231 (930630.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 13
07/12/22 12:10:21 Reassigning the id of job sub-NC255 from (930632.0.0) to (930632.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC255 (930632.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 14
07/12/22 12:10:21 Reassigning the id of job sub-NC188 from (930634.0.0) to (930634.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC188 (930634.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 15
07/12/22 12:10:21 Reassigning the id of job sub-NC239 from (930636.0.0) to (930636.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC239 (930636.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 16
07/12/22 12:10:21 Reassigning the id of job sub-NC244 from (930638.0.0) to (930638.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC244 (930638.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 17
07/12/22 12:10:21 Reassigning the id of job sub-NC233 from (930640.0.0) to (930640.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC233 (930640.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 18
07/12/22 12:10:21 Reassigning the id of job sub-NC249 from (930642.0.0) to (930642.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC249 (930642.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 19
07/12/22 12:10:21 Reassigning the id of job sub-NC234 from (930644.0.0) to (930644.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC234 (930644.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 20
07/12/22 12:10:21 Reassigning the id of job sub-NC243 from (930646.0.0) to (930646.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC243 (930646.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 21
07/12/22 12:10:21 Reassigning the id of job sub-NC227 from (930648.0.0) to (930648.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC227 (930648.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 22
07/12/22 12:10:21 Reassigning the id of job sub-NC250 from (930650.0.0) to (930650.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC250 (930650.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 23
07/12/22 12:10:21 Reassigning the id of job sub-NC254 from (930652.0.0) to (930652.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC254 (930652.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 24
07/12/22 12:10:21 Reassigning the id of job sub-NC247 from (930654.0.0) to (930654.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC247 (930654.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 25
07/12/22 12:10:21 Reassigning the id of job sub-NC240 from (930656.0.0) to (930656.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC240 (930656.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 26
07/12/22 12:10:21 Reassigning the id of job sub-NC237 from (930658.0.0) to (930658.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC237 (930658.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 27
07/12/22 12:10:21 Reassigning the id of job sub-NC150 from (930660.0.0) to (930660.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC150 (930660.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 28
07/12/22 12:10:21 Reassigning the id of job sub-NC216 from (930662.0.0) to (930662.0.0)
07/12/22 12:10:21 Event: ULOG_SUBMIT for HTCondor Node sub-NC216 (930662.0.0) {07/12/22 12:10:16}
07/12/22 12:10:21 Number of idle job procs: 29
07/12/22 12:20:22 601 seconds since last log event
07/12/22 12:20:22 Pending DAG nodes:
07/12/22 12:20:22   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC189, HTCondor ID 930618, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC183, HTCondor ID 930620, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC188, HTCondor ID 930634, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC227, HTCondor ID 930648, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 12:20:22   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 12:30:23 1202 seconds since last log event
07/12/22 12:30:23 Pending DAG nodes:
07/12/22 12:30:23   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC189, HTCondor ID 930618, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC183, HTCondor ID 930620, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC188, HTCondor ID 930634, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC227, HTCondor ID 930648, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 12:30:23   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 12:40:23 1802 seconds since last log event
07/12/22 12:40:23 Pending DAG nodes:
07/12/22 12:40:23   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC189, HTCondor ID 930618, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC183, HTCondor ID 930620, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC188, HTCondor ID 930634, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC227, HTCondor ID 930648, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 12:40:23   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 12:50:24 2403 seconds since last log event
07/12/22 12:50:24 Pending DAG nodes:
07/12/22 12:50:24   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC189, HTCondor ID 930618, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC183, HTCondor ID 930620, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC188, HTCondor ID 930634, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC227, HTCondor ID 930648, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 12:50:24   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 13:00:24 3003 seconds since last log event
07/12/22 13:00:24 Pending DAG nodes:
07/12/22 13:00:24   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC189, HTCondor ID 930618, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC183, HTCondor ID 930620, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC188, HTCondor ID 930634, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC227, HTCondor ID 930648, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 13:00:24   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 13:10:25 3604 seconds since last log event
07/12/22 13:10:25 Pending DAG nodes:
07/12/22 13:10:25   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC189, HTCondor ID 930618, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC183, HTCondor ID 930620, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC188, HTCondor ID 930634, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC227, HTCondor ID 930648, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 13:10:25   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 13:20:25 4204 seconds since last log event
07/12/22 13:20:25 Pending DAG nodes:
07/12/22 13:20:25   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC189, HTCondor ID 930618, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC183, HTCondor ID 930620, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC188, HTCondor ID 930634, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC227, HTCondor ID 930648, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 13:20:25   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 13:30:25 4804 seconds since last log event
07/12/22 13:30:25 Pending DAG nodes:
07/12/22 13:30:25   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC189, HTCondor ID 930618, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC183, HTCondor ID 930620, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC188, HTCondor ID 930634, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC227, HTCondor ID 930648, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 13:30:25   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 13:40:25 5404 seconds since last log event
07/12/22 13:40:25 Pending DAG nodes:
07/12/22 13:40:25   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC189, HTCondor ID 930618, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC183, HTCondor ID 930620, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC188, HTCondor ID 930634, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC227, HTCondor ID 930648, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 13:40:25   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 13:46:31 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:46:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC251 (930603.0.0) {07/12/22 13:46:26}
07/12/22 13:46:31 Number of idle job procs: 28
07/12/22 13:46:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC235 (930609.0.0) {07/12/22 13:46:26}
07/12/22 13:46:31 Number of idle job procs: 27
07/12/22 13:46:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC248 (930612.0.0) {07/12/22 13:46:26}
07/12/22 13:46:31 Number of idle job procs: 26
07/12/22 13:46:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC242 (930606.0.0) {07/12/22 13:46:26}
07/12/22 13:46:31 Number of idle job procs: 25
07/12/22 13:46:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC232 (930614.0.0) {07/12/22 13:46:26}
07/12/22 13:46:31 Number of idle job procs: 24
07/12/22 13:46:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC245 (930616.0.0) {07/12/22 13:46:26}
07/12/22 13:46:31 Number of idle job procs: 23
07/12/22 13:46:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC189 (930618.0.0) {07/12/22 13:46:26}
07/12/22 13:46:31 Number of idle job procs: 22
07/12/22 13:46:36 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:46:36 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC189 (930618.0.0) {07/12/22 13:46:35}
07/12/22 13:46:36 Number of idle job procs: 22
07/12/22 13:46:36 Node sub-NC189 job proc (930618.0.0) failed with status 1.
07/12/22 13:46:36 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 13:46:36 Of 29 nodes total:
07/12/22 13:46:36  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 13:46:36   ===     ===      ===     ===     ===        ===      ===
07/12/22 13:46:36     0       0       28       0       0          0        1
07/12/22 13:46:36 0 job proc(s) currently held
07/12/22 13:46:36 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.03268407171566606; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.001442952795978232; EventCycleTimeSum = 1.666610479354858; EventCycleTimeCount = 1155.0; SleepCycleTimeStd = 0.02954030335453569; LogProcessCycleTimeMin = 0.0003359317779541016; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 1156.0; LogProcessCycleTimeAvg = 0.001290241877237956; LogProcessCycleTimeCount = 3.0; LogProcessCycleTimeStd = 0.001315540364430074; LogProcessCycleTimeSum = 0.003870725631713867; SubmitCycleTimeSum = 1.377048015594482; SubmitCycleTimeAvg = 0.001191218006569622; SleepCycleTimeSum = 5778.953231573105; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003422711318706; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 1155.0; SubmitCycleTimeStd = 0.03248898562773909; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 13:46:46 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:46:46 Event: ULOG_EXECUTE for HTCondor Node sub-NC183 (930620.0.0) {07/12/22 13:46:45}
07/12/22 13:46:46 Number of idle job procs: 21
07/12/22 13:46:51 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:46:51 Event: ULOG_EXECUTE for HTCondor Node sub-NC225 (930622.0.0) {07/12/22 13:46:47}
07/12/22 13:46:51 Number of idle job procs: 20
07/12/22 13:46:56 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:46:56 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC183 (930620.0.0) {07/12/22 13:46:53}
07/12/22 13:46:56 Number of idle job procs: 20
07/12/22 13:46:56 Node sub-NC183 job proc (930620.0.0) failed with status 1.
07/12/22 13:46:56 Event: ULOG_EXECUTE for HTCondor Node sub-NC252 (930624.0.0) {07/12/22 13:46:55}
07/12/22 13:46:56 Number of idle job procs: 19
07/12/22 13:46:56 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 13:46:56 Of 29 nodes total:
07/12/22 13:46:56  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 13:46:56   ===     ===      ===     ===     ===        ===      ===
07/12/22 13:46:56     0       0       27       0       0          0        2
07/12/22 13:46:56 0 job proc(s) currently held
07/12/22 13:46:56 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.03262789902441825; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.00144369340950914; EventCycleTimeSum = 1.673240661621094; EventCycleTimeCount = 1159.0; SleepCycleTimeStd = 0.02948939027885912; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 1160.0; LogProcessCycleTimeAvg = 0.0008008082707722982; LogProcessCycleTimeCount = 6.0; LogProcessCycleTimeStd = 0.0009928205094057109; LogProcessCycleTimeSum = 0.004804849624633789; SubmitCycleTimeSum = 1.377624750137329; SubmitCycleTimeAvg = 0.001187607543221835; SleepCycleTimeSum = 5798.97333741188; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003428246257015; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 1159.0; SubmitCycleTimeStd = 0.032432933849096; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 13:48:31 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:48:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC246 (930628.0.0) {07/12/22 13:48:26}
07/12/22 13:48:31 Number of idle job procs: 18
07/12/22 13:48:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC231 (930630.0.0) {07/12/22 13:48:26}
07/12/22 13:48:31 Number of idle job procs: 17
07/12/22 13:50:31 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:50:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC255 (930632.0.0) {07/12/22 13:50:26}
07/12/22 13:50:31 Number of idle job procs: 16
07/12/22 13:50:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC188 (930634.0.0) {07/12/22 13:50:26}
07/12/22 13:50:31 Number of idle job procs: 15
07/12/22 13:50:36 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:50:36 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC188 (930634.0.0) {07/12/22 13:50:34}
07/12/22 13:50:36 Number of idle job procs: 15
07/12/22 13:50:36 Node sub-NC188 job proc (930634.0.0) failed with status 1.
07/12/22 13:50:36 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 13:50:36 Of 29 nodes total:
07/12/22 13:50:36  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 13:50:36   ===     ===      ===     ===     ===        ===      ===
07/12/22 13:50:36     0       0       26       0       0          0        3
07/12/22 13:50:36 0 job proc(s) currently held
07/12/22 13:50:36 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.03202612463587928; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.001395686507125943; EventCycleTimeSum = 1.67901086807251; EventCycleTimeCount = 1203.0; SleepCycleTimeStd = 0.02894631195548085; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 1204.0; LogProcessCycleTimeAvg = 0.0006630685594346789; LogProcessCycleTimeCount = 9.0; LogProcessCycleTimeStd = 0.0008137830026383995; LogProcessCycleTimeSum = 0.005967617034912109; SubmitCycleTimeSum = 1.379941940307617; SubmitCycleTimeAvg = 0.001146131179657489; SleepCycleTimeSum = 6019.186182022095; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003479785554526; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 1203.0; SubmitCycleTimeStd = 0.03183500804528302; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 13:53:26 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:53:26 Event: ULOG_EXECUTE for HTCondor Node sub-NC236 (930626.0.0) {07/12/22 13:53:26}
07/12/22 13:53:26 Number of idle job procs: 14
07/12/22 13:53:26 Event: ULOG_EXECUTE for HTCondor Node sub-NC239 (930636.0.0) {07/12/22 13:53:26}
07/12/22 13:53:26 Number of idle job procs: 13
07/12/22 13:53:31 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:53:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC244 (930638.0.0) {07/12/22 13:53:26}
07/12/22 13:53:31 Number of idle job procs: 12
07/12/22 13:53:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC233 (930640.0.0) {07/12/22 13:53:26}
07/12/22 13:53:31 Number of idle job procs: 11
07/12/22 13:53:36 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:53:36 Event: ULOG_EXECUTE for HTCondor Node sub-NC249 (930642.0.0) {07/12/22 13:53:36}
07/12/22 13:53:36 Number of idle job procs: 10
07/12/22 13:55:26 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:55:26 Event: ULOG_EXECUTE for HTCondor Node sub-NC234 (930644.0.0) {07/12/22 13:55:26}
07/12/22 13:55:26 Number of idle job procs: 9
07/12/22 13:55:31 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:55:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC243 (930646.0.0) {07/12/22 13:55:26}
07/12/22 13:55:31 Number of idle job procs: 8
07/12/22 13:56:26 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:56:26 Event: ULOG_EXECUTE for HTCondor Node sub-NC227 (930648.0.0) {07/12/22 13:56:26}
07/12/22 13:56:26 Number of idle job procs: 7
07/12/22 13:56:26 Event: ULOG_EXECUTE for HTCondor Node sub-NC250 (930650.0.0) {07/12/22 13:56:26}
07/12/22 13:56:26 Number of idle job procs: 6
07/12/22 13:56:26 Event: ULOG_EXECUTE for HTCondor Node sub-NC254 (930652.0.0) {07/12/22 13:56:26}
07/12/22 13:56:26 Number of idle job procs: 5
07/12/22 13:56:26 Event: ULOG_EXECUTE for HTCondor Node sub-NC247 (930654.0.0) {07/12/22 13:56:26}
07/12/22 13:56:26 Number of idle job procs: 4
07/12/22 13:56:31 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:56:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC240 (930656.0.0) {07/12/22 13:56:26}
07/12/22 13:56:31 Number of idle job procs: 3
07/12/22 13:56:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC237 (930658.0.0) {07/12/22 13:56:26}
07/12/22 13:56:31 Number of idle job procs: 2
07/12/22 13:56:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC150 (930660.0.0) {07/12/22 13:56:27}
07/12/22 13:56:31 Number of idle job procs: 1
07/12/22 13:56:36 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:56:36 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC227 (930648.0.0) {07/12/22 13:56:34}
07/12/22 13:56:36 Number of idle job procs: 1
07/12/22 13:56:36 Node sub-NC227 job proc (930648.0.0) failed with status 1.
07/12/22 13:56:36 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 13:56:36 Of 29 nodes total:
07/12/22 13:56:36  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 13:56:36   ===     ===      ===     ===     ===        ===      ===
07/12/22 13:56:36     0       0       25       0       0          0        4
07/12/22 13:56:36 0 job proc(s) currently held
07/12/22 13:56:36 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.03110935133897414; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.001325392816580978; EventCycleTimeSum = 1.689875841140747; EventCycleTimeCount = 1275.0; SleepCycleTimeStd = 0.02811895216618614; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 1276.0; LogProcessCycleTimeAvg = 0.000532991745892693; LogProcessCycleTimeCount = 17.0; LogProcessCycleTimeStd = 0.0006030132133239038; LogProcessCycleTimeSum = 0.009060859680175781; SubmitCycleTimeSum = 1.384816408157349; SubmitCycleTimeAvg = 0.001085279316737734; SleepCycleTimeSum = 6379.521847248077; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003546546861237; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 1275.0; SubmitCycleTimeStd = 0.0309240980759125; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 13:57:31 Currently monitoring 1 HTCondor log file(s)
07/12/22 13:57:31 Event: ULOG_EXECUTE for HTCondor Node sub-NC216 (930662.0.0) {07/12/22 13:57:26}
07/12/22 13:57:31 Number of idle job procs: 0
07/12/22 14:07:32 601 seconds since last log event
07/12/22 14:07:32 Pending DAG nodes:
07/12/22 14:07:32   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 14:07:32   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 14:17:32 1201 seconds since last log event
07/12/22 14:17:32 Pending DAG nodes:
07/12/22 14:17:32   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 14:17:32   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 14:27:33 1802 seconds since last log event
07/12/22 14:27:33 Pending DAG nodes:
07/12/22 14:27:33   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 14:27:33   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 14:37:34 2403 seconds since last log event
07/12/22 14:37:34 Pending DAG nodes:
07/12/22 14:37:34   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 14:37:34   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 14:47:34 3003 seconds since last log event
07/12/22 14:47:34 Pending DAG nodes:
07/12/22 14:47:34   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 14:47:34   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 14:57:34 3603 seconds since last log event
07/12/22 14:57:34 Pending DAG nodes:
07/12/22 14:57:34   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 14:57:34   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 15:07:35 4204 seconds since last log event
07/12/22 15:07:35 Pending DAG nodes:
07/12/22 15:07:35   Node sub-NC251, HTCondor ID 930603, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC242, HTCondor ID 930606, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC235, HTCondor ID 930609, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC248, HTCondor ID 930612, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC232, HTCondor ID 930614, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC245, HTCondor ID 930616, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC225, HTCondor ID 930622, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC252, HTCondor ID 930624, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC236, HTCondor ID 930626, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC246, HTCondor ID 930628, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC231, HTCondor ID 930630, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC255, HTCondor ID 930632, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC247, HTCondor ID 930654, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 15:07:35   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 15:16:55 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:16:55 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC235 (930609.0.0) {07/12/22 15:16:53}
07/12/22 15:16:55 Number of idle job procs: 0
07/12/22 15:16:55 Node sub-NC235 job proc (930609.0.0) completed successfully.
07/12/22 15:16:55 Node sub-NC235 job completed
07/12/22 15:16:55 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:16:55 Of 29 nodes total:
07/12/22 15:16:55  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:16:55   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:16:55     1       0       24       0       0          0        4
07/12/22 15:16:55 0 job proc(s) currently held
07/12/22 15:16:55 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.0234850639673462; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.000807421669776787; EventCycleTimeSum = 1.807009696960449; EventCycleTimeCount = 2238.0; SleepCycleTimeStd = 0.02123541443353945; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2239.0; LogProcessCycleTimeAvg = 0.0005369939302143298; LogProcessCycleTimeCount = 19.0; LogProcessCycleTimeStd = 0.0005708456185621395; LogProcessCycleTimeSum = 0.01020288467407227; SubmitCycleTimeSum = 1.443668127059937; SubmitCycleTimeAvg = 0.0006447825489325308; SleepCycleTimeSum = 11198.72505426407; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003898594398601; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2238.0; SubmitCycleTimeStd = 0.02334682062757023; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:19:06 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:19:06 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC251 (930603.0.0) {07/12/22 15:19:04}
07/12/22 15:19:06 Number of idle job procs: 0
07/12/22 15:19:06 Node sub-NC251 job proc (930603.0.0) completed successfully.
07/12/22 15:19:06 Node sub-NC251 job completed
07/12/22 15:19:06 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:19:06 Of 29 nodes total:
07/12/22 15:19:06  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:19:06   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:19:06     2       0       23       0       0          0        4
07/12/22 15:19:06 0 job proc(s) currently held
07/12/22 15:19:06 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.0233504209365598; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0008032211355943984; EventCycleTimeSum = 1.818492650985718; EventCycleTimeCount = 2264.0; SleepCycleTimeStd = 0.02111339811395652; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2265.0; LogProcessCycleTimeAvg = 0.0005495429039001464; LogProcessCycleTimeCount = 20.0; LogProcessCycleTimeStd = 0.0005584473836395932; LogProcessCycleTimeSum = 0.01099085807800293; SubmitCycleTimeSum = 1.445442676544189; SubmitCycleTimeAvg = 0.0006381645371056024; SleepCycleTimeSum = 11328.83724141121; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003903375181629; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2264.0; SubmitCycleTimeStd = 0.02321245872157906; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:22:41 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:22:41 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC252 (930624.0.0) {07/12/22 15:22:36}
07/12/22 15:22:41 Number of idle job procs: 0
07/12/22 15:22:41 Node sub-NC252 job proc (930624.0.0) completed successfully.
07/12/22 15:22:41 Node sub-NC252 job completed
07/12/22 15:22:41 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:22:41 Of 29 nodes total:
07/12/22 15:22:41  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:22:41   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:22:41     3       0       22       0       0          0        4
07/12/22 15:22:41 0 job proc(s) currently held
07/12/22 15:22:41 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.0231318846257654; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007906757911976457; EventCycleTimeSum = 1.824089050292969; EventCycleTimeCount = 2307.0; SleepCycleTimeStd = 0.02091610695425774; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2308.0; LogProcessCycleTimeAvg = 0.0005566619691394624; LogProcessCycleTimeCount = 21.0; LogProcessCycleTimeStd = 0.0005452839711258034; LogProcessCycleTimeSum = 0.01168990135192871; SubmitCycleTimeSum = 1.447896957397461; SubmitCycleTimeAvg = 0.0006273383697562656; SleepCycleTimeSum = 11544.02438282967; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.00391173941468; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2307.0; SubmitCycleTimeStd = 0.02299525090137036; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:23:11 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:23:11 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC232 (930614.0.0) {07/12/22 15:23:10}
07/12/22 15:23:11 Number of idle job procs: 0
07/12/22 15:23:11 Node sub-NC232 job proc (930614.0.0) completed successfully.
07/12/22 15:23:11 Node sub-NC232 job completed
07/12/22 15:23:11 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:23:11 Of 29 nodes total:
07/12/22 15:23:11  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:23:11   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:23:11     4       0       21       0       0          0        4
07/12/22 15:23:11 0 job proc(s) currently held
07/12/22 15:23:11 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02310263464731745; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007931366659478609; EventCycleTimeSum = 1.834525108337402; EventCycleTimeCount = 2313.0; SleepCycleTimeStd = 0.02088897304840855; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2314.0; LogProcessCycleTimeAvg = 0.0005628087303855202; LogProcessCycleTimeCount = 22.0; LogProcessCycleTimeStd = 0.0005329231061588319; LogProcessCycleTimeSum = 0.01238179206848145; SubmitCycleTimeSum = 1.44839882850647; SubmitCycleTimeAvg = 0.0006259286207893127; SleepCycleTimeSum = 11574.04921579361; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003912328488374; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2313.0; SubmitCycleTimeStd = 0.02296542327959653; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:24:01 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:24:01 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC231 (930630.0.0) {07/12/22 15:23:57}
07/12/22 15:24:01 Number of idle job procs: 0
07/12/22 15:24:01 Node sub-NC231 job proc (930630.0.0) completed successfully.
07/12/22 15:24:01 Node sub-NC231 job completed
07/12/22 15:24:01 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:24:01 Of 29 nodes total:
07/12/22 15:24:01  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:24:01   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:24:01     5       0       20       0       0          0        4
07/12/22 15:24:01 0 job proc(s) currently held
07/12/22 15:24:01 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02305287756909302; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007904605315360744; EventCycleTimeSum = 1.836239814758301; EventCycleTimeCount = 2323.0; SleepCycleTimeStd = 0.02084413281385595; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2324.0; LogProcessCycleTimeAvg = 0.000566814256751019; LogProcessCycleTimeCount = 23.0; LogProcessCycleTimeStd = 0.0005210246096446829; LogProcessCycleTimeSum = 0.01303672790527344; SubmitCycleTimeSum = 1.448861598968506; SubmitCycleTimeAvg = 0.0006234344229640731; SleepCycleTimeSum = 11624.09291505814; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003914298346163; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2323.0; SubmitCycleTimeStd = 0.0229159712397632; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:24:46 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:24:46 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC246 (930628.0.0) {07/12/22 15:24:42}
07/12/22 15:24:46 Number of idle job procs: 0
07/12/22 15:24:46 Node sub-NC246 job proc (930628.0.0) completed successfully.
07/12/22 15:24:46 Node sub-NC246 job completed
07/12/22 15:24:46 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:24:46 Of 29 nodes total:
07/12/22 15:24:46  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:24:46   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:24:46     6       0       19       0       0          0        4
07/12/22 15:24:46 0 job proc(s) currently held
07/12/22 15:24:46 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02300836836459964; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007881214442833824; EventCycleTimeSum = 1.837899208068848; EventCycleTimeCount = 2332.0; SleepCycleTimeStd = 0.02080396919926844; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2333.0; LogProcessCycleTimeAvg = 0.0005726118882497152; LogProcessCycleTimeCount = 24.0; LogProcessCycleTimeStd = 0.0005103630541490401; LogProcessCycleTimeSum = 0.01374268531799316; SubmitCycleTimeSum = 1.449421882629395; SubmitCycleTimeAvg = 0.0006212695596354027; SleepCycleTimeSum = 11669.13650083542; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003917882004896; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2332.0; SubmitCycleTimeStd = 0.02287173565645759; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:27:41 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:27:41 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC242 (930606.0.0) {07/12/22 15:27:40}
07/12/22 15:27:41 Number of idle job procs: 0
07/12/22 15:27:41 Node sub-NC242 job proc (930606.0.0) completed successfully.
07/12/22 15:27:41 Node sub-NC242 job completed
07/12/22 15:27:41 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:27:41 Of 29 nodes total:
07/12/22 15:27:41  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:27:41   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:27:41     7       0       18       0       0          0        4
07/12/22 15:27:41 0 job proc(s) currently held
07/12/22 15:27:41 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02283769707527778; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007787175857421463; EventCycleTimeSum = 1.84322452545166; EventCycleTimeCount = 2367.0; SleepCycleTimeStd = 0.02065015467936619; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2368.0; LogProcessCycleTimeAvg = 0.0005790328979492187; LogProcessCycleTimeCount = 25.0; LogProcessCycleTimeStd = 0.000500647826207222; LogProcessCycleTimeSum = 0.01447582244873047; SubmitCycleTimeSum = 1.452025651931763; SubmitCycleTimeAvg = 0.0006131865084171295; SleepCycleTimeSum = 11844.27802348137; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003919739535855; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2367.0; SubmitCycleTimeStd = 0.02270210792475625; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:34:26 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:34:26 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC245 (930616.0.0) {07/12/22 15:34:24}
07/12/22 15:34:26 Number of idle job procs: 0
07/12/22 15:34:26 Node sub-NC245 job proc (930616.0.0) completed successfully.
07/12/22 15:34:26 Node sub-NC245 job completed
07/12/22 15:34:26 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:34:26 Of 29 nodes total:
07/12/22 15:34:26  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:34:26   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:34:26     8       0       17       0       0          0        4
07/12/22 15:34:26 0 job proc(s) currently held
07/12/22 15:34:26 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02245711063704705; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007597921915303648; EventCycleTimeSum = 1.859971284866333; EventCycleTimeCount = 2448.0; SleepCycleTimeStd = 0.02030647243318049; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2449.0; LogProcessCycleTimeAvg = 0.0005803016515878531; LogProcessCycleTimeCount = 26.0; LogProcessCycleTimeStd = 0.0004905753450605323; LogProcessCycleTimeSum = 0.01508784294128418; SubmitCycleTimeSum = 1.458717346191406; SubmitCycleTimeAvg = 0.000595637952711885; SleepCycleTimeSum = 12249.60916161537; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.00392531111739; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2448.0; SubmitCycleTimeStd = 0.02232357670613508; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:35:01 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:35:01 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC255 (930632.0.0) {07/12/22 15:34:57}
07/12/22 15:35:01 Number of idle job procs: 0
07/12/22 15:35:01 Node sub-NC255 job proc (930632.0.0) completed successfully.
07/12/22 15:35:01 Node sub-NC255 job completed
07/12/22 15:35:01 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:35:01 Of 29 nodes total:
07/12/22 15:35:01  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:35:01   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:35:01     9       0       16       0       0          0        4
07/12/22 15:35:01 0 job proc(s) currently held
07/12/22 15:35:01 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02242534371423057; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007603500622595886; EventCycleTimeSum = 1.86665940284729; EventCycleTimeCount = 2455.0; SleepCycleTimeStd = 0.02027754723321812; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2456.0; LogProcessCycleTimeAvg = 0.000576372499819155; LogProcessCycleTimeCount = 27.0; LogProcessCycleTimeStd = 0.0004814817621569487; LogProcessCycleTimeSum = 0.01556205749511719; SubmitCycleTimeSum = 1.459134340286255; SubmitCycleTimeAvg = 0.0005941100734064555; SleepCycleTimeSum = 12284.63899159431; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003926269488519; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2455.0; SubmitCycleTimeStd = 0.02229174676160399; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:37:32 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:37:32 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC248 (930612.0.0) {07/12/22 15:37:28}
07/12/22 15:37:32 Number of idle job procs: 0
07/12/22 15:37:32 Node sub-NC248 job proc (930612.0.0) completed successfully.
07/12/22 15:37:32 Node sub-NC248 job completed
07/12/22 15:37:32 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:37:32 Of 29 nodes total:
07/12/22 15:37:32  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:37:32   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:37:32    10       0       15       0       0          0        4
07/12/22 15:37:32 0 job proc(s) currently held
07/12/22 15:37:32 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02228962115835265; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007530478164703554; EventCycleTimeSum = 1.871323823928833; EventCycleTimeCount = 2485.0; SleepCycleTimeStd = 0.02015495423031842; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2486.0; LogProcessCycleTimeAvg = 0.0005838189806256976; LogProcessCycleTimeCount = 28.0; LogProcessCycleTimeStd = 0.000474121492762954; LogProcessCycleTimeSum = 0.01634693145751953; SubmitCycleTimeSum = 1.461542367935181; SubmitCycleTimeAvg = 0.0005879092389119794; SleepCycleTimeSum = 12434.76190114021; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003928330438717; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2485.0; SubmitCycleTimeStd = 0.02215685572463739; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:38:12 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:38:12 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC225 (930622.0.0) {07/12/22 15:38:10}
07/12/22 15:38:12 Number of idle job procs: 0
07/12/22 15:38:12 Node sub-NC225 job proc (930622.0.0) completed successfully.
07/12/22 15:38:12 Node sub-NC225 job completed
07/12/22 15:38:12 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC236 (930626.0.0) {07/12/22 15:38:10}
07/12/22 15:38:12 Number of idle job procs: 0
07/12/22 15:38:12 Node sub-NC236 job proc (930626.0.0) completed successfully.
07/12/22 15:38:12 Node sub-NC236 job completed
07/12/22 15:38:12 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:38:12 Of 29 nodes total:
07/12/22 15:38:12  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:38:12   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:38:12    12       0       13       0       0          0        4
07/12/22 15:38:12 0 job proc(s) currently held
07/12/22 15:38:12 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.0222538412176792; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007514780513550735; EventCycleTimeSum = 1.873434782028198; EventCycleTimeCount = 2493.0; SleepCycleTimeStd = 0.02012265726444573; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2494.0; LogProcessCycleTimeAvg = 0.0005945501656367861; LogProcessCycleTimeCount = 29.0; LogProcessCycleTimeStd = 0.0004691508549903339; LogProcessCycleTimeSum = 0.0172419548034668; SubmitCycleTimeSum = 1.462277173995972; SubmitCycleTimeAvg = 0.0005863180328772942; SleepCycleTimeSum = 12474.7919151783; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003927763809988; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2493.0; SubmitCycleTimeStd = 0.022121295329864; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:41:37 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:41:37 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC247 (930654.0.0) {07/12/22 15:41:35}
07/12/22 15:41:37 Number of idle job procs: 0
07/12/22 15:41:37 Node sub-NC247 job proc (930654.0.0) completed successfully.
07/12/22 15:41:37 Node sub-NC247 job completed
07/12/22 15:41:37 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:41:37 Of 29 nodes total:
07/12/22 15:41:37  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:41:37   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:41:37    13       0       12       0       0          0        4
07/12/22 15:41:37 0 job proc(s) currently held
07/12/22 15:41:37 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02207314673197089; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007418161691787587; EventCycleTimeSum = 1.879762172698975; EventCycleTimeCount = 2534.0; SleepCycleTimeStd = 0.01995946013595138; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2535.0; LogProcessCycleTimeAvg = 0.0006031672159830729; LogProcessCycleTimeCount = 30.0; LogProcessCycleTimeStd = 0.0004634008946969397; LogProcessCycleTimeSum = 0.01809501647949219; SubmitCycleTimeSum = 1.465208053588867; SubmitCycleTimeAvg = 0.0005779913426386064; SleepCycleTimeSum = 12679.95641875267; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003929131315181; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2534.0; SubmitCycleTimeStd = 0.021941705049106; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 15:51:37 600 seconds since last log event
07/12/22 15:51:37 Pending DAG nodes:
07/12/22 15:51:37   Node sub-NC239, HTCondor ID 930636, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC244, HTCondor ID 930638, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC233, HTCondor ID 930640, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC249, HTCondor ID 930642, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC234, HTCondor ID 930644, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC243, HTCondor ID 930646, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC250, HTCondor ID 930650, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC254, HTCondor ID 930652, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC240, HTCondor ID 930656, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC237, HTCondor ID 930658, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC150, HTCondor ID 930660, status STATUS_SUBMITTED
07/12/22 15:51:37   Node sub-NC216, HTCondor ID 930662, status STATUS_SUBMITTED
07/12/22 15:54:22 Currently monitoring 1 HTCondor log file(s)
07/12/22 15:54:22 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC244 (930638.0.0) {07/12/22 15:54:21}
07/12/22 15:54:22 Number of idle job procs: 0
07/12/22 15:54:22 Node sub-NC244 job proc (930638.0.0) completed successfully.
07/12/22 15:54:22 Node sub-NC244 job completed
07/12/22 15:54:22 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 15:54:22 Of 29 nodes total:
07/12/22 15:54:22  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 15:54:22   ===     ===      ===     ===     ===        ===      ===
07/12/22 15:54:22    14       0       11       0       0          0        4
07/12/22 15:54:22 0 job proc(s) currently held
07/12/22 15:54:22 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02143606769008347; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.883506774902344E-05; EventCycleTimeAvg = 0.0007100548050329949; EventCycleTimeSum = 1.907917261123657; EventCycleTimeCount = 2687.0; SleepCycleTimeStd = 0.01938367533174665; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2688.0; LogProcessCycleTimeAvg = 0.0005955542287518901; LogProcessCycleTimeCount = 31.0; LogProcessCycleTimeStd = 0.0004575795679927592; LogProcessCycleTimeSum = 0.01846218109130859; SubmitCycleTimeSum = 1.477378368377686; SubmitCycleTimeAvg = 0.0005496199287119366; SleepCycleTimeSum = 13445.56185269356; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003930722997231; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2687.0; SubmitCycleTimeStd = 0.02130819141918409; SleepCycleTimeMin = 4.004153966903687; ]
07/12/22 16:02:42 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:02:42 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC239 (930636.0.0) {07/12/22 16:02:38}
07/12/22 16:02:42 Number of idle job procs: 0
07/12/22 16:02:42 Node sub-NC239 job proc (930636.0.0) completed successfully.
07/12/22 16:02:42 Node sub-NC239 job completed
07/12/22 16:02:42 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:02:42 Of 29 nodes total:
07/12/22 16:02:42  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:02:42   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:02:42    15       0       10       0       0          0        4
07/12/22 16:02:42 0 job proc(s) currently held
07/12/22 16:02:42 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02104827897540271; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006907739047299245; EventCycleTimeSum = 1.9251868724823; EventCycleTimeCount = 2787.0; SleepCycleTimeStd = 0.02685194776728333; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2788.0; LogProcessCycleTimeAvg = 0.0005978196859359741; LogProcessCycleTimeCount = 32.0; LogProcessCycleTimeStd = 0.0004503211423695709; LogProcessCycleTimeSum = 0.01913022994995117; SubmitCycleTimeSum = 1.48489785194397; SubmitCycleTimeAvg = 0.0005326032467517825; SleepCycleTimeSum = 13945.00879716873; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003591244050495; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2787.0; SubmitCycleTimeStd = 0.02092262044423706; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:03:22 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:03:22 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC243 (930646.0.0) {07/12/22 16:03:18}
07/12/22 16:03:22 Number of idle job procs: 0
07/12/22 16:03:22 Node sub-NC243 job proc (930646.0.0) completed successfully.
07/12/22 16:03:22 Node sub-NC243 job completed
07/12/22 16:03:22 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:03:22 Of 29 nodes total:
07/12/22 16:03:22  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:03:22   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:03:22    16       0        9       0       0          0        4
07/12/22 16:03:22 0 job proc(s) currently held
07/12/22 16:03:22 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.0210183412685139; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006910807757983268; EventCycleTimeSum = 1.931570768356323; EventCycleTimeCount = 2795.0; SleepCycleTimeStd = 0.0268136370005388; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2796.0; LogProcessCycleTimeAvg = 0.0005907651149865353; LogProcessCycleTimeCount = 33.0; LogProcessCycleTimeStd = 0.0004450778369737871; LogProcessCycleTimeSum = 0.01949524879455566; SubmitCycleTimeSum = 1.485641717910767; SubmitCycleTimeAvg = 0.000531345392671948; SleepCycleTimeSum = 13985.04700613022; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003594635466984; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2795.0; SubmitCycleTimeStd = 0.02089267038671967; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:04:02 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:04:02 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC234 (930644.0.0) {07/12/22 16:03:59}
07/12/22 16:04:02 Number of idle job procs: 0
07/12/22 16:04:02 Node sub-NC234 job proc (930644.0.0) completed successfully.
07/12/22 16:04:02 Node sub-NC234 job completed
07/12/22 16:04:02 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:04:02 Of 29 nodes total:
07/12/22 16:04:02  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:04:02   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:04:02    17       0        8       0       0          0        4
07/12/22 16:04:02 0 job proc(s) currently held
07/12/22 16:04:02 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02098834009415785; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006894368679660412; EventCycleTimeSum = 1.932491540908813; EventCycleTimeCount = 2803.0; SleepCycleTimeStd = 0.02677540354692731; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2804.0; LogProcessCycleTimeAvg = 0.0005882417454439052; LogProcessCycleTimeCount = 34.0; LogProcessCycleTimeStd = 0.0004385292639689768; LogProcessCycleTimeSum = 0.02000021934509277; SubmitCycleTimeSum = 1.486221551895142; SubmitCycleTimeAvg = 0.0005300362167956996; SleepCycleTimeSum = 14025.08032536507; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003596263062813; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2803.0; SubmitCycleTimeStd = 0.02086284996627704; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:04:22 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:04:22 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC249 (930642.0.0) {07/12/22 16:04:19}
07/12/22 16:04:22 Number of idle job procs: 0
07/12/22 16:04:22 Node sub-NC249 job proc (930642.0.0) completed successfully.
07/12/22 16:04:22 Node sub-NC249 job completed
07/12/22 16:04:22 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:04:22 Of 29 nodes total:
07/12/22 16:04:22  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:04:22   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:04:22    18       0        7       0       0          0        4
07/12/22 16:04:22 0 job proc(s) currently held
07/12/22 16:04:22 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02097338456205347; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006890975583860957; EventCycleTimeSum = 1.934296846389771; EventCycleTimeCount = 2807.0; SleepCycleTimeStd = 0.02675641058563077; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2808.0; LogProcessCycleTimeAvg = 0.0005819456917898995; LogProcessCycleTimeCount = 35.0; LogProcessCycleTimeStd = 0.0004336348814453269; LogProcessCycleTimeSum = 0.02036809921264648; SubmitCycleTimeSum = 1.486782789230347; SubmitCycleTimeAvg = 0.0005294810502957075; SleepCycleTimeSum = 14045.09830117226; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003597542277255; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2807.0; SubmitCycleTimeStd = 0.02084798630879364; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:04:47 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:04:47 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC233 (930640.0.0) {07/12/22 16:04:46}
07/12/22 16:04:47 Number of idle job procs: 0
07/12/22 16:04:47 Node sub-NC233 job proc (930640.0.0) completed successfully.
07/12/22 16:04:47 Node sub-NC233 job completed
07/12/22 16:04:47 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:04:47 Of 29 nodes total:
07/12/22 16:04:47  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:04:47   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:04:47    19       0        6       0       0          0        4
07/12/22 16:04:47 0 job proc(s) currently held
07/12/22 16:04:47 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02095473739301369; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006881681818032841; EventCycleTimeSum = 1.935128927230835; EventCycleTimeCount = 2812.0; SleepCycleTimeStd = 0.02673266571370244; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2813.0; LogProcessCycleTimeAvg = 0.0005808075269063314; LogProcessCycleTimeCount = 36.0; LogProcessCycleTimeStd = 0.0004274497591907564; LogProcessCycleTimeSum = 0.02090907096862793; SubmitCycleTimeSum = 1.486894130706787; SubmitCycleTimeAvg = 0.0005285794990070342; SleepCycleTimeSum = 14070.11890602112; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003598472980483; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2812.0; SubmitCycleTimeStd = 0.02082945418554102; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:05:02 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:05:02 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC254 (930652.0.0) {07/12/22 16:05:01}
07/12/22 16:05:02 Number of idle job procs: 0
07/12/22 16:05:02 Node sub-NC254 job proc (930652.0.0) completed successfully.
07/12/22 16:05:02 Node sub-NC254 job completed
07/12/22 16:05:02 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:05:02 Of 29 nodes total:
07/12/22 16:05:02  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:05:02   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:05:02    20       0        5       0       0          0        4
07/12/22 16:05:02 0 job proc(s) currently held
07/12/22 16:05:02 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02094357156172772; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006877516345063069; EventCycleTimeSum = 1.936020851135254; EventCycleTimeCount = 2815.0; SleepCycleTimeStd = 0.02671843137816933; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2816.0; LogProcessCycleTimeAvg = 0.0005778686420337574; LogProcessCycleTimeCount = 37.0; LogProcessCycleTimeStd = 0.0004218500893922239; LogProcessCycleTimeSum = 0.02138113975524902; SubmitCycleTimeSum = 1.487356185913086; SubmitCycleTimeAvg = 0.0005281804637475448; SleepCycleTimeSum = 14085.13225722313; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003599380896316; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2815.0; SubmitCycleTimeStd = 0.02081835650832857; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:06:52 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:06:52 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC237 (930658.0.0) {07/12/22 16:06:48}
07/12/22 16:06:52 Number of idle job procs: 0
07/12/22 16:06:52 Node sub-NC237 job proc (930658.0.0) completed successfully.
07/12/22 16:06:52 Node sub-NC237 job completed
07/12/22 16:06:52 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:06:52 Of 29 nodes total:
07/12/22 16:06:52  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:06:52   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:06:52    21       0        4       0       0          0        4
07/12/22 16:06:52 0 job proc(s) currently held
07/12/22 16:06:52 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02086224540192354; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006836507788501158; EventCycleTimeSum = 1.939517259597778; EventCycleTimeCount = 2837.0; SleepCycleTimeStd = 0.02661482972669981; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2838.0; LogProcessCycleTimeAvg = 0.0005809005938078228; LogProcessCycleTimeCount = 38.0; LogProcessCycleTimeStd = 0.0004165299024242617; LogProcessCycleTimeSum = 0.02207422256469727; SubmitCycleTimeSum = 1.488794803619385; SubmitCycleTimeAvg = 0.0005245929540589799; SleepCycleTimeSum = 14195.22535467148; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003604284339612; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2837.0; SubmitCycleTimeStd = 0.02073752215844846; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:10:17 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:10:17 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC216 (930662.0.0) {07/12/22 16:10:17}
07/12/22 16:10:17 Number of idle job procs: 0
07/12/22 16:10:17 Node sub-NC216 job proc (930662.0.0) completed successfully.
07/12/22 16:10:17 Node sub-NC216 job completed
07/12/22 16:10:17 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:10:17 Of 29 nodes total:
07/12/22 16:10:17  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:10:17   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:10:17    22       0        3       0       0          0        4
07/12/22 16:10:17 0 job proc(s) currently held
07/12/22 16:10:17 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02071316178690071; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006764180300714574; EventCycleTimeSum = 1.946731090545654; EventCycleTimeCount = 2878.0; SleepCycleTimeStd = 0.02642499182712548; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2879.0; LogProcessCycleTimeAvg = 0.0005880074623303536; LogProcessCycleTimeCount = 39.0; LogProcessCycleTimeStd = 0.000413402032547685; LogProcessCycleTimeSum = 0.02293229103088379; SubmitCycleTimeSum = 1.492846250534058; SubmitCycleTimeAvg = 0.0005185294374901207; SleepCycleTimeSum = 14400.40100193024; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003613968704044; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2878.0; SubmitCycleTimeStd = 0.02058934920523793; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:14:53 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:14:53 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC150 (930660.0.0) {07/12/22 16:14:52}
07/12/22 16:14:53 Number of idle job procs: 0
07/12/22 16:14:53 Node sub-NC150 job proc (930660.0.0) completed successfully.
07/12/22 16:14:53 Node sub-NC150 job completed
07/12/22 16:14:53 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:14:53 Of 29 nodes total:
07/12/22 16:14:53  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:14:53   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:14:53    23       0        2       0       0          0        4
07/12/22 16:14:53 0 job proc(s) currently held
07/12/22 16:14:53 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02051842279444979; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006688302017664528; EventCycleTimeSum = 1.961678981781006; EventCycleTimeCount = 2933.0; SleepCycleTimeStd = 0.02617647758468073; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002790927886962891; SubmitCycleTimeCount = 2934.0; LogProcessCycleTimeAvg = 0.0005906045436859131; LogProcessCycleTimeCount = 40.0; LogProcessCycleTimeStd = 0.0004083980297297431; LogProcessCycleTimeSum = 0.02362418174743652; SubmitCycleTimeSum = 1.498642206192017; SubmitCycleTimeAvg = 0.0005107846646871222; SleepCycleTimeSum = 14675.63482189178; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003625919499415; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2933.0; SubmitCycleTimeStd = 0.02039549539532848; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:18:03 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:18:03 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC240 (930656.0.0) {07/12/22 16:18:00}
07/12/22 16:18:03 Number of idle job procs: 0
07/12/22 16:18:03 Node sub-NC240 job proc (930656.0.0) completed successfully.
07/12/22 16:18:03 Node sub-NC240 job completed
07/12/22 16:18:03 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:18:03 Of 29 nodes total:
07/12/22 16:18:03  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:18:03   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:18:03    24       0        1       0       0          0        4
07/12/22 16:18:03 0 job proc(s) currently held
07/12/22 16:18:03 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.020386962316123; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006640111505283085; EventCycleTimeSum = 1.972777128219604; EventCycleTimeCount = 2971.0; SleepCycleTimeStd = 0.02600870302116088; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002995014190673828; SubmitCycleTimeCount = 2972.0; LogProcessCycleTimeAvg = 0.0006492486814173256; LogProcessCycleTimeCount = 41.0; LogProcessCycleTimeStd = 0.0005510206506299507; LogProcessCycleTimeSum = 0.02661919593811035; SubmitCycleTimeSum = 1.505520343780518; SubmitCycleTimeAvg = 0.0005065680833716412; SleepCycleTimeSum = 14865.77980589867; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003628342611466; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2971.0; SubmitCycleTimeStd = 0.02026483592487191; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:18:48 Currently monitoring 1 HTCondor log file(s)
07/12/22 16:18:48 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-NC250 (930650.0.0) {07/12/22 16:18:45}
07/12/22 16:18:48 Number of idle job procs: 0
07/12/22 16:18:48 Node sub-NC250 job proc (930650.0.0) completed successfully.
07/12/22 16:18:48 Node sub-NC250 job completed
07/12/22 16:18:48 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:18:48 Of 29 nodes total:
07/12/22 16:18:48  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:18:48   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:18:48    25       0        0       0       0          0        4
07/12/22 16:18:48 0 job proc(s) currently held
07/12/22 16:18:48 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02035672529709174; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006651098296146264; EventCycleTimeSum = 1.982027292251587; EventCycleTimeCount = 2980.0; SleepCycleTimeStd = 0.02596952898047455; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002995014190673828; SubmitCycleTimeCount = 2981.0; LogProcessCycleTimeAvg = 0.0006439345223563058; LogProcessCycleTimeCount = 42.0; LogProcessCycleTimeStd = 0.0005453479549394807; LogProcessCycleTimeSum = 0.02704524993896484; SubmitCycleTimeSum = 1.505898475646973; SubmitCycleTimeAvg = 0.0005051655403042511; SleepCycleTimeSum = 14910.81469774246; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003629093202169; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2980.0; SubmitCycleTimeStd = 0.02023422783895091; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:18:48 ERROR: the following job(s) failed:
07/12/22 16:18:48 ---------------------- Job ----------------------
07/12/22 16:18:48       Node Name: sub-NC189
07/12/22 16:18:48            Noop: false
07/12/22 16:18:48          NodeID: 6
07/12/22 16:18:48     Node Status: STATUS_ERROR    
07/12/22 16:18:48 Node return val: 1
07/12/22 16:18:48           Error: Job proc (930618.0.0) failed with status 1
07/12/22 16:18:48 Job Submit File: code/process.condor_submit
07/12/22 16:18:48  HTCondor Job ID: (930618.0.0)
07/12/22 16:18:48       Q_PARENTS: <END>
07/12/22 16:18:48       Q_WAITING: <END>
07/12/22 16:18:48      Q_CHILDREN: <END>
07/12/22 16:18:48 ---------------------- Job ----------------------
07/12/22 16:18:48       Node Name: sub-NC183
07/12/22 16:18:48            Noop: false
07/12/22 16:18:48          NodeID: 7
07/12/22 16:18:48     Node Status: STATUS_ERROR    
07/12/22 16:18:48 Node return val: 1
07/12/22 16:18:48           Error: Job proc (930620.0.0) failed with status 1
07/12/22 16:18:48 Job Submit File: code/process.condor_submit
07/12/22 16:18:48  HTCondor Job ID: (930620.0.0)
07/12/22 16:18:48       Q_PARENTS: <END>
07/12/22 16:18:48       Q_WAITING: <END>
07/12/22 16:18:48      Q_CHILDREN: <END>
07/12/22 16:18:48 ---------------------- Job ----------------------
07/12/22 16:18:48       Node Name: sub-NC188
07/12/22 16:18:48            Noop: false
07/12/22 16:18:48          NodeID: 14
07/12/22 16:18:48     Node Status: STATUS_ERROR    
07/12/22 16:18:48 Node return val: 1
07/12/22 16:18:48           Error: Job proc (930634.0.0) failed with status 1
07/12/22 16:18:48 Job Submit File: code/process.condor_submit
07/12/22 16:18:48  HTCondor Job ID: (930634.0.0)
07/12/22 16:18:48       Q_PARENTS: <END>
07/12/22 16:18:48       Q_WAITING: <END>
07/12/22 16:18:48      Q_CHILDREN: <END>
07/12/22 16:18:48 ---------------------- Job ----------------------
07/12/22 16:18:48       Node Name: sub-NC227
07/12/22 16:18:48            Noop: false
07/12/22 16:18:48          NodeID: 21
07/12/22 16:18:48     Node Status: STATUS_ERROR    
07/12/22 16:18:48 Node return val: 1
07/12/22 16:18:48           Error: Job proc (930648.0.0) failed with status 1
07/12/22 16:18:48 Job Submit File: code/process.condor_submit
07/12/22 16:18:48  HTCondor Job ID: (930648.0.0)
07/12/22 16:18:48       Q_PARENTS: <END>
07/12/22 16:18:48       Q_WAITING: <END>
07/12/22 16:18:48      Q_CHILDREN: <END>
07/12/22 16:18:48 ---------------------------------------	<END>
07/12/22 16:18:48 Aborting DAG...
07/12/22 16:18:48 Writing Rescue DAG to code/process.condor_dag.rescue001...
07/12/22 16:18:48 Removing submitted jobs...
07/12/22 16:18:48 Removing any/all submitted HTCondor jobs...
07/12/22 16:18:48 Running: /usr/bin/condor_rm -const DAGManJobId' '=?=' '930575
07/12/22 16:18:48 Note: 0 total job deferrals because of -MaxJobs limit (0)
07/12/22 16:18:48 Note: 0 total job deferrals because of -MaxIdle limit (1000)
07/12/22 16:18:48 Note: 0 total job deferrals because of node category throttles
07/12/22 16:18:48 Note: 0 total PRE script deferrals because of -MaxPre limit (20) or DEFER
07/12/22 16:18:48 Note: 0 total POST script deferrals because of -MaxPost limit (20) or DEFER
07/12/22 16:18:48 DAG status: 2 (DAG_STATUS_NODE_FAILED)
07/12/22 16:18:48 Of 29 nodes total:
07/12/22 16:18:48  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
07/12/22 16:18:48   ===     ===      ===     ===     ===        ===      ===
07/12/22 16:18:48    25       0        0       0       0          0        4
07/12/22 16:18:48 0 job proc(s) currently held
07/12/22 16:18:48 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02035672529709174; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006651098296146264; EventCycleTimeSum = 1.982027292251587; EventCycleTimeCount = 2980.0; SleepCycleTimeStd = 0.02596952898047455; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002995014190673828; SubmitCycleTimeCount = 2981.0; LogProcessCycleTimeAvg = 0.0006439345223563058; LogProcessCycleTimeCount = 42.0; LogProcessCycleTimeStd = 0.0005453479549394807; LogProcessCycleTimeSum = 0.02704524993896484; SubmitCycleTimeSum = 1.505898475646973; SubmitCycleTimeAvg = 0.0005051655403042511; SleepCycleTimeSum = 14910.81469774246; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003629093202169; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2980.0; SubmitCycleTimeStd = 0.02023422783895091; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:18:48 Wrote metrics file code/process.condor_dag.metrics.
07/12/22 16:18:48 Metrics not sent because of PEGASUS_METRICS or CONDOR_DEVELOPERS setting.
07/12/22 16:18:48 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.02035672529709174; EventCycleTimeMax = 1.098453044891357; EventCycleTimeMin = 1.406669616699219E-05; EventCycleTimeAvg = 0.0006651098296146264; EventCycleTimeSum = 1.982027292251587; EventCycleTimeCount = 2980.0; SleepCycleTimeStd = 0.02596952898047455; LogProcessCycleTimeMin = 0.0001718997955322266; LogProcessCycleTimeMax = 0.002995014190673828; SubmitCycleTimeCount = 2981.0; LogProcessCycleTimeAvg = 0.0006439345223563058; LogProcessCycleTimeCount = 42.0; LogProcessCycleTimeStd = 0.0005453479549394807; LogProcessCycleTimeSum = 0.02704524993896484; SubmitCycleTimeSum = 1.505898475646973; SubmitCycleTimeAvg = 0.0005051655403042511; SleepCycleTimeSum = 14910.81469774246; SubmitCycleTimeMin = 5.006790161132812E-06; SubmitCycleTimeMax = 1.098250865936279; SleepCycleTimeAvg = 5.003629093202169; SleepCycleTimeMax = 5.059043884277344; SleepCycleTimeCount = 2980.0; SubmitCycleTimeStd = 0.02023422783895091; SleepCycleTimeMin = 4.004049062728882; ]
07/12/22 16:18:48 **** condor_scheduniv_exec.930575.0 (condor_DAGMAN) pid 2158359 EXITING WITH STATUS 1