123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418 |
- 07/12/22 12:21:28 ******************************************************
- 07/12/22 12:21:28 ** condor_scheduniv_exec.931968.0 (CONDOR_DAGMAN) STARTING UP
- 07/12/22 12:21:28 ** /usr/bin/condor_dagman
- 07/12/22 12:21:28 ** SubsystemInfo: name=DAGMAN type=DAGMAN(10) class=DAEMON(1)
- 07/12/22 12:21:28 ** Configuration: subsystem:DAGMAN local:<NONE> class:DAEMON
- 07/12/22 12:21:28 ** $CondorVersion: 8.8.6 Nov 19 2019 BuildID: Debian-8.8.6~dfsg.1-1 Debian-8.8.6~dfsg.1-1 $
- 07/12/22 12:21:28 ** $CondorPlatform: X86_64-Debian_10 $
- 07/12/22 12:21:28 ** PID = 2264448
- 07/12/22 12:21:28 ** Log last touched time unavailable (No such file or directory)
- 07/12/22 12:21:28 ******************************************************
- 07/12/22 12:21:28 Using config source: /etc/condor/condor_config
- 07/12/22 12:21:28 Using local config sources:
- 07/12/22 12:21:28 /etc/condor/config.d/50_inm7_master
- 07/12/22 12:21:28 /etc/condor/condor_config.local
- 07/12/22 12:21:28 config Macros = 81, Sorted = 81, StringBytes = 2405, TablesBytes = 2972
- 07/12/22 12:21:28 CLASSAD_CACHING is ENABLED
- 07/12/22 12:21:28 Daemon Log is logging: D_ALWAYS D_ERROR
- 07/12/22 12:21:28 DaemonCore: No command port requested.
- 07/12/22 12:21:28 DAGMAN_USE_STRICT setting: 1
- 07/12/22 12:21:28 DAGMAN_VERBOSITY setting: 3
- 07/12/22 12:21:28 DAGMAN_DEBUG_CACHE_SIZE setting: 5242880
- 07/12/22 12:21:28 DAGMAN_DEBUG_CACHE_ENABLE setting: False
- 07/12/22 12:21:28 DAGMAN_SUBMIT_DELAY setting: 0
- 07/12/22 12:21:28 DAGMAN_MAX_SUBMIT_ATTEMPTS setting: 6
- 07/12/22 12:21:28 DAGMAN_STARTUP_CYCLE_DETECT setting: False
- 07/12/22 12:21:28 DAGMAN_MAX_SUBMITS_PER_INTERVAL setting: 100
- 07/12/22 12:21:28 DAGMAN_AGGRESSIVE_SUBMIT setting: False
- 07/12/22 12:21:28 DAGMAN_USER_LOG_SCAN_INTERVAL setting: 5
- 07/12/22 12:21:28 DAGMAN_QUEUE_UPDATE_INTERVAL setting: 300
- 07/12/22 12:21:28 DAGMAN_DEFAULT_PRIORITY setting: 0
- 07/12/22 12:21:28 DAGMAN_SUPPRESS_NOTIFICATION setting: True
- 07/12/22 12:21:28 allow_events (DAGMAN_ALLOW_EVENTS) setting: 114
- 07/12/22 12:21:28 DAGMAN_RETRY_SUBMIT_FIRST setting: True
- 07/12/22 12:21:28 DAGMAN_RETRY_NODE_FIRST setting: False
- 07/12/22 12:21:28 DAGMAN_MAX_JOBS_IDLE setting: 1000
- 07/12/22 12:21:28 DAGMAN_MAX_JOBS_SUBMITTED setting: 0
- 07/12/22 12:21:28 DAGMAN_MAX_PRE_SCRIPTS setting: 20
- 07/12/22 12:21:28 DAGMAN_MAX_POST_SCRIPTS setting: 20
- 07/12/22 12:21:28 DAGMAN_MUNGE_NODE_NAMES setting: True
- 07/12/22 12:21:28 DAGMAN_PROHIBIT_MULTI_JOBS setting: False
- 07/12/22 12:21:28 DAGMAN_SUBMIT_DEPTH_FIRST setting: False
- 07/12/22 12:21:28 DAGMAN_ALWAYS_RUN_POST setting: False
- 07/12/22 12:21:28 DAGMAN_CONDOR_SUBMIT_EXE setting: /usr/bin/condor_submit
- 07/12/22 12:21:28 DAGMAN_USE_CONDOR_SUBMIT setting: True
- 07/12/22 12:21:28 DAGMAN_ABORT_DUPLICATES setting: True
- 07/12/22 12:21:28 DAGMAN_ABORT_ON_SCARY_SUBMIT setting: True
- 07/12/22 12:21:28 DAGMAN_PENDING_REPORT_INTERVAL setting: 600
- 07/12/22 12:21:28 DAGMAN_AUTO_RESCUE setting: True
- 07/12/22 12:21:28 DAGMAN_MAX_RESCUE_NUM setting: 100
- 07/12/22 12:21:28 DAGMAN_WRITE_PARTIAL_RESCUE setting: True
- 07/12/22 12:21:28 DAGMAN_DEFAULT_NODE_LOG setting: @(DAG_DIR)/@(DAG_FILE).nodes.log
- 07/12/22 12:21:28 DAGMAN_GENERATE_SUBDAG_SUBMITS setting: True
- 07/12/22 12:21:28 DAGMAN_MAX_JOB_HOLDS setting: 100
- 07/12/22 12:21:28 DAGMAN_HOLD_CLAIM_TIME setting: 20
- 07/12/22 12:21:28 ALL_DEBUG setting:
- 07/12/22 12:21:28 DAGMAN_DEBUG setting:
- 07/12/22 12:21:28 DAGMAN_SUPPRESS_JOB_LOGS setting: False
- 07/12/22 12:21:28 DAGMAN_REMOVE_NODE_JOBS setting: True
- 07/12/22 12:21:28 argv[0] == "condor_scheduniv_exec.931968.0"
- 07/12/22 12:21:28 argv[1] == "-Lockfile"
- 07/12/22 12:21:28 argv[2] == "code/process.condor_dag.lock"
- 07/12/22 12:21:28 argv[3] == "-AutoRescue"
- 07/12/22 12:21:28 argv[4] == "1"
- 07/12/22 12:21:28 argv[5] == "-DoRescueFrom"
- 07/12/22 12:21:28 argv[6] == "0"
- 07/12/22 12:21:28 argv[7] == "-Dag"
- 07/12/22 12:21:28 argv[8] == "code/process.condor_dag"
- 07/12/22 12:21:28 argv[9] == "-Suppress_notification"
- 07/12/22 12:21:28 argv[10] == "-CsdVersion"
- 07/12/22 12:21:28 argv[11] == "$CondorVersion: 8.8.6 Nov 19 2019 BuildID: Debian-8.8.6~dfsg.1-1 Debian-8.8.6~dfsg.1-1 $"
- 07/12/22 12:21:28 argv[12] == "-Dagman"
- 07/12/22 12:21:28 argv[13] == "/usr/bin/condor_dagman"
- 07/12/22 12:21:28 Workflow batch-name: <process.condor_dag+931968>
- 07/12/22 12:21:28 Workflow accounting_group: <>
- 07/12/22 12:21:28 Workflow accounting_group_user: <>
- 07/12/22 12:21:28 Warning: failed to get attribute DAGNodeName
- 07/12/22 12:21:28 DAGMAN_LOG_ON_NFS_IS_ERROR setting: False
- 07/12/22 12:21:28 Default node log file is: </data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log>
- 07/12/22 12:21:28 DAG Lockfile will be written to code/process.condor_dag.lock
- 07/12/22 12:21:28 DAG Input file is code/process.condor_dag
- 07/12/22 12:21:28 Parsing 1 dagfiles
- 07/12/22 12:21:28 Parsing code/process.condor_dag ...
- 07/12/22 12:21:28 Dag contains 5 total jobs
- 07/12/22 12:21:28 Sleeping for 3 seconds to ensure ProcessId uniqueness
- 07/12/22 12:21:31 Bootstrapping...
- 07/12/22 12:21:31 Number of pre-completed nodes: 0
- 07/12/22 12:21:31 MultiLogFiles: truncating log file /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
- 07/12/22 12:21:31 DAG status: 0 (DAG_STATUS_OK)
- 07/12/22 12:21:31 Of 5 nodes total:
- 07/12/22 12:21:31 Done Pre Queued Post Ready Un-Ready Failed
- 07/12/22 12:21:31 === === === === === === ===
- 07/12/22 12:21:31 0 0 0 0 5 0 0
- 07/12/22 12:21:31 0 job proc(s) currently held
- 07/12/22 12:21:31 DAGMan Runtime Statistics: [ EventCycleTimeSum = 0.0; EventCycleTimeCount = 0.0; LogProcessCycleTimeSum = 0.0; SleepCycleTimeSum = 0.0; LogProcessCycleTimeCount = 0.0; SleepCycleTimeCount = 0.0; SubmitCycleTimeCount = 0.0; SubmitCycleTimeSum = 0.0; ]
- 07/12/22 12:21:31 Registering condor_event_timer...
- 07/12/22 12:21:32 Submitting HTCondor Node sub-001 job(s)...
- 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
- 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
- 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
- 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-001 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-001 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-001 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
- 07/12/22 12:21:32 From submit: Submitting job(s).
- 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931969.
- 07/12/22 12:21:32 assigned HTCondor ID (931969.0.0)
- 07/12/22 12:21:32 Submitting HTCondor Node sub-002 job(s)...
- 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
- 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
- 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
- 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-002 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-002 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-002 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
- 07/12/22 12:21:32 From submit: Submitting job(s).
- 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931970.
- 07/12/22 12:21:32 assigned HTCondor ID (931970.0.0)
- 07/12/22 12:21:32 Submitting HTCondor Node sub-005 job(s)...
- 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
- 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
- 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
- 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-005 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-005 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-005 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
- 07/12/22 12:21:32 From submit: Submitting job(s).
- 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931971.
- 07/12/22 12:21:32 assigned HTCondor ID (931971.0.0)
- 07/12/22 12:21:32 Submitting HTCondor Node sub-004 job(s)...
- 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
- 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
- 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
- 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-004 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-004 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-004 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
- 07/12/22 12:21:32 From submit: Submitting job(s).
- 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931972.
- 07/12/22 12:21:32 assigned HTCondor ID (931972.0.0)
- 07/12/22 12:21:32 Submitting HTCondor Node sub-003 job(s)...
- 07/12/22 12:21:32 Adding a DAGMan workflow log /data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log
- 07/12/22 12:21:32 Masking the events recorded in the DAGMAN workflow log
- 07/12/22 12:21:32 Mask for workflow log is 0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36
- 07/12/22 12:21:32 submitting: /usr/bin/condor_submit -a dag_node_name' '=' 'sub-003 -a +DAGManJobId' '=' '931968 -a DAGManJobId' '=' '931968 -batch-name process.condor_dag+931968 -a submit_event_notes' '=' 'DAG' 'Node:' 'sub-003 -a dagman_log' '=' '/data/project/infrasound/ReproVBM/ds003720_ReproVBM/code/process.condor_dag.nodes.log -a +DAGManNodesMask' '=' '"0,1,2,4,5,7,9,10,11,12,13,16,17,24,27,35,36" -a subject' '=' 'sub-003 -a DAG_STATUS' '=' '0 -a FAILED_COUNT' '=' '0 -a notification' '=' 'never -a +DAGParentNodeNames' '=' '"" code/process.condor_submit
- 07/12/22 12:21:32 From submit: Submitting job(s).
- 07/12/22 12:21:32 From submit: 1 job(s) submitted to cluster 931973.
- 07/12/22 12:21:32 assigned HTCondor ID (931973.0.0)
- 07/12/22 12:21:32 Just submitted 5 jobs this cycle...
- 07/12/22 12:21:32 DAG status: 0 (DAG_STATUS_OK)
- 07/12/22 12:21:32 Of 5 nodes total:
- 07/12/22 12:21:32 Done Pre Queued Post Ready Un-Ready Failed
- 07/12/22 12:21:32 === === === === === === ===
- 07/12/22 12:21:32 0 0 5 0 0 0 0
- 07/12/22 12:21:32 0 job proc(s) currently held
- 07/12/22 12:21:32 DAGMan Runtime Statistics: [ EventCycleTimeSum = 0.0; SleepCycleTimeCount = 0.0; EventCycleTimeCount = 0.0; SubmitCycleTimeStd = 0.2237379550933838; SubmitCycleTimeMax = 0.2237379550933838; LogProcessCycleTimeSum = 0.0; LogProcessCycleTimeCount = 0.0; SubmitCycleTimeMin = 0.2237379550933838; SubmitCycleTimeCount = 1.0; SubmitCycleTimeSum = 0.2237379550933838; SleepCycleTimeSum = 0.0; SubmitCycleTimeAvg = 0.2237379550933838; ]
- 07/12/22 12:21:37 Currently monitoring 1 HTCondor log file(s)
- 07/12/22 12:21:37 Reassigning the id of job sub-001 from (931969.0.0) to (931969.0.0)
- 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-001 (931969.0.0) {07/12/22 12:21:32}
- 07/12/22 12:21:37 Number of idle job procs: 1
- 07/12/22 12:21:37 Reassigning the id of job sub-002 from (931970.0.0) to (931970.0.0)
- 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-002 (931970.0.0) {07/12/22 12:21:32}
- 07/12/22 12:21:37 Number of idle job procs: 2
- 07/12/22 12:21:37 Reassigning the id of job sub-005 from (931971.0.0) to (931971.0.0)
- 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-005 (931971.0.0) {07/12/22 12:21:32}
- 07/12/22 12:21:37 Number of idle job procs: 3
- 07/12/22 12:21:37 Reassigning the id of job sub-004 from (931972.0.0) to (931972.0.0)
- 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-004 (931972.0.0) {07/12/22 12:21:32}
- 07/12/22 12:21:37 Number of idle job procs: 4
- 07/12/22 12:21:37 Reassigning the id of job sub-003 from (931973.0.0) to (931973.0.0)
- 07/12/22 12:21:37 Event: ULOG_SUBMIT for HTCondor Node sub-003 (931973.0.0) {07/12/22 12:21:32}
- 07/12/22 12:21:37 Number of idle job procs: 5
- 07/12/22 12:31:38 601 seconds since last log event
- 07/12/22 12:31:38 Pending DAG nodes:
- 07/12/22 12:31:38 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 12:31:38 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 12:31:38 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 12:31:38 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 12:31:38 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 12:41:38 1201 seconds since last log event
- 07/12/22 12:41:38 Pending DAG nodes:
- 07/12/22 12:41:38 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 12:41:38 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 12:41:38 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 12:41:38 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 12:41:38 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 12:51:39 1802 seconds since last log event
- 07/12/22 12:51:39 Pending DAG nodes:
- 07/12/22 12:51:39 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 12:51:39 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 12:51:39 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 12:51:39 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 12:51:39 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 13:01:40 2403 seconds since last log event
- 07/12/22 13:01:40 Pending DAG nodes:
- 07/12/22 13:01:40 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 13:01:40 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 13:01:40 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 13:01:40 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 13:01:40 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 13:11:40 3003 seconds since last log event
- 07/12/22 13:11:40 Pending DAG nodes:
- 07/12/22 13:11:40 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 13:11:40 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 13:11:40 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 13:11:40 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 13:11:40 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 13:21:41 3604 seconds since last log event
- 07/12/22 13:21:41 Pending DAG nodes:
- 07/12/22 13:21:41 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 13:21:41 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 13:21:41 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 13:21:41 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 13:21:41 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 13:31:41 4204 seconds since last log event
- 07/12/22 13:31:41 Pending DAG nodes:
- 07/12/22 13:31:41 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 13:31:41 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 13:31:41 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 13:31:41 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 13:31:41 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 13:41:42 4805 seconds since last log event
- 07/12/22 13:41:42 Pending DAG nodes:
- 07/12/22 13:41:42 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 13:41:42 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 13:41:42 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 13:41:42 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 13:41:42 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 13:51:42 5405 seconds since last log event
- 07/12/22 13:51:42 Pending DAG nodes:
- 07/12/22 13:51:42 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 13:51:42 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 13:51:42 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 13:51:42 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 13:51:42 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 14:01:42 6005 seconds since last log event
- 07/12/22 14:01:42 Pending DAG nodes:
- 07/12/22 14:01:42 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 14:01:42 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 14:01:42 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 14:01:42 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 14:01:42 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 14:11:43 6606 seconds since last log event
- 07/12/22 14:11:43 Pending DAG nodes:
- 07/12/22 14:11:43 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 14:11:43 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 14:11:43 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 14:11:43 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 14:11:43 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 14:21:43 7206 seconds since last log event
- 07/12/22 14:21:43 Pending DAG nodes:
- 07/12/22 14:21:43 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 14:21:43 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 14:21:43 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 14:21:43 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 14:21:43 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 14:31:44 7807 seconds since last log event
- 07/12/22 14:31:44 Pending DAG nodes:
- 07/12/22 14:31:44 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 14:31:44 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 14:31:44 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 14:31:44 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 14:31:44 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 14:41:44 8407 seconds since last log event
- 07/12/22 14:41:44 Pending DAG nodes:
- 07/12/22 14:41:44 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 14:41:44 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 14:41:44 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 14:41:44 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 14:41:44 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 14:51:45 9008 seconds since last log event
- 07/12/22 14:51:45 Pending DAG nodes:
- 07/12/22 14:51:45 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 14:51:45 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 14:51:45 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 14:51:45 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 14:51:45 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 15:01:45 9608 seconds since last log event
- 07/12/22 15:01:45 Pending DAG nodes:
- 07/12/22 15:01:45 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 15:01:45 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 15:01:45 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 15:01:45 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 15:01:45 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 15:11:46 10209 seconds since last log event
- 07/12/22 15:11:46 Pending DAG nodes:
- 07/12/22 15:11:46 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 15:11:46 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 15:11:46 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 15:11:46 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 15:11:46 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 15:21:46 10809 seconds since last log event
- 07/12/22 15:21:46 Pending DAG nodes:
- 07/12/22 15:21:46 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 15:21:46 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 15:21:46 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 15:21:46 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 15:21:46 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 15:31:47 11410 seconds since last log event
- 07/12/22 15:31:47 Pending DAG nodes:
- 07/12/22 15:31:47 Node sub-001, HTCondor ID 931969, status STATUS_SUBMITTED
- 07/12/22 15:31:47 Node sub-002, HTCondor ID 931970, status STATUS_SUBMITTED
- 07/12/22 15:31:47 Node sub-005, HTCondor ID 931971, status STATUS_SUBMITTED
- 07/12/22 15:31:47 Node sub-004, HTCondor ID 931972, status STATUS_SUBMITTED
- 07/12/22 15:31:47 Node sub-003, HTCondor ID 931973, status STATUS_SUBMITTED
- 07/12/22 15:36:07 Currently monitoring 1 HTCondor log file(s)
- 07/12/22 15:36:07 Event: ULOG_EXECUTE for HTCondor Node sub-003 (931973.0.0) {07/12/22 15:36:06}
- 07/12/22 15:36:07 Number of idle job procs: 4
- 07/12/22 15:36:07 Event: ULOG_EXECUTE for HTCondor Node sub-004 (931972.0.0) {07/12/22 15:36:06}
- 07/12/22 15:36:07 Number of idle job procs: 3
- 07/12/22 15:36:22 Currently monitoring 1 HTCondor log file(s)
- 07/12/22 15:36:22 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-004 (931972.0.0) {07/12/22 15:36:17}
- 07/12/22 15:36:22 Number of idle job procs: 3
- 07/12/22 15:36:22 Node sub-004 job proc (931972.0.0) failed with status 1.
- 07/12/22 15:36:22 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-003 (931973.0.0) {07/12/22 15:36:21}
- 07/12/22 15:36:22 Number of idle job procs: 3
- 07/12/22 15:36:22 Node sub-003 job proc (931973.0.0) failed with status 1.
- 07/12/22 15:36:22 DAG status: 2 (DAG_STATUS_NODE_FAILED)
- 07/12/22 15:36:22 Of 5 nodes total:
- 07/12/22 15:36:22 Done Pre Queued Post Ready Un-Ready Failed
- 07/12/22 15:36:22 === === === === === === ===
- 07/12/22 15:36:22 0 0 3 0 0 0 2
- 07/12/22 15:36:22 0 job proc(s) currently held
- 07/12/22 15:36:22 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.005493951815109337; EventCycleTimeMax = 0.2239258289337158; EventCycleTimeMin = 2.09808349609375E-05; EventCycleTimeAvg = 0.0003880991102897958; EventCycleTimeSum = 0.9065995216369629; EventCycleTimeCount = 2336.0; SleepCycleTimeStd = 0.02079933325079659; LogProcessCycleTimeMin = 0.0005269050598144531; LogProcessCycleTimeMax = 0.0007569789886474609; SubmitCycleTimeCount = 2337.0; LogProcessCycleTimeAvg = 0.0006422996520996094; LogProcessCycleTimeCount = 3.0; LogProcessCycleTimeStd = 0.0001150386320991401; LogProcessCycleTimeSum = 0.001926898956298828; SubmitCycleTimeSum = 0.5614738464355469; SubmitCycleTimeAvg = 0.000240254106305326; SleepCycleTimeSum = 11689.4080324173; SubmitCycleTimeMin = 7.867813110351562E-06; SubmitCycleTimeMax = 0.2237379550933838; SleepCycleTimeAvg = 5.004027411137542; SleepCycleTimeMax = 5.068968057632446; SleepCycleTimeCount = 2336.0; SubmitCycleTimeStd = 0.00498719534509626; SleepCycleTimeMin = 4.004912853240967; ]
- 07/12/22 15:40:42 Currently monitoring 1 HTCondor log file(s)
- 07/12/22 15:40:42 Event: ULOG_EXECUTE for HTCondor Node sub-001 (931969.0.0) {07/12/22 15:40:38}
- 07/12/22 15:40:42 Number of idle job procs: 2
- 07/12/22 15:40:42 Event: ULOG_EXECUTE for HTCondor Node sub-002 (931970.0.0) {07/12/22 15:40:38}
- 07/12/22 15:40:42 Number of idle job procs: 1
- 07/12/22 15:40:42 Event: ULOG_EXECUTE for HTCondor Node sub-005 (931971.0.0) {07/12/22 15:40:38}
- 07/12/22 15:40:42 Number of idle job procs: 0
- 07/12/22 15:40:52 Currently monitoring 1 HTCondor log file(s)
- 07/12/22 15:40:52 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-005 (931971.0.0) {07/12/22 15:40:50}
- 07/12/22 15:40:52 Number of idle job procs: 0
- 07/12/22 15:40:52 Node sub-005 job proc (931971.0.0) failed with status 1.
- 07/12/22 15:40:52 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-002 (931970.0.0) {07/12/22 15:40:50}
- 07/12/22 15:40:52 Number of idle job procs: 0
- 07/12/22 15:40:52 Node sub-002 job proc (931970.0.0) failed with status 1.
- 07/12/22 15:40:52 Event: ULOG_JOB_TERMINATED for HTCondor Node sub-001 (931969.0.0) {07/12/22 15:40:50}
- 07/12/22 15:40:52 Number of idle job procs: 0
- 07/12/22 15:40:52 Node sub-001 job proc (931969.0.0) failed with status 1.
- 07/12/22 15:40:52 DAG status: 2 (DAG_STATUS_NODE_FAILED)
- 07/12/22 15:40:52 Of 5 nodes total:
- 07/12/22 15:40:52 Done Pre Queued Post Ready Un-Ready Failed
- 07/12/22 15:40:52 === === === === === === ===
- 07/12/22 15:40:52 0 0 0 0 0 0 5
- 07/12/22 15:40:52 0 job proc(s) currently held
- 07/12/22 15:40:52 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.005433997287527276; EventCycleTimeMax = 0.2239258289337158; EventCycleTimeMin = 2.09808349609375E-05; EventCycleTimeAvg = 0.0003853633313997021; EventCycleTimeSum = 0.9210183620452881; EventCycleTimeCount = 2390.0; SleepCycleTimeStd = 0.02056334142362197; LogProcessCycleTimeMin = 0.0005269050598144531; LogProcessCycleTimeMax = 0.0009350776672363281; SubmitCycleTimeCount = 2391.0; LogProcessCycleTimeAvg = 0.0006816387176513672; LogProcessCycleTimeCount = 5.0; LogProcessCycleTimeStd = 0.0001685829264184372; LogProcessCycleTimeSum = 0.003408193588256836; SubmitCycleTimeSum = 0.5647740364074707; SubmitCycleTimeAvg = 0.0002362082962808326; SleepCycleTimeSum = 11959.61773061752; SubmitCycleTimeMin = 7.867813110351562E-06; SubmitCycleTimeMax = 0.2237379550933838; SleepCycleTimeAvg = 5.004024155070093; SleepCycleTimeMax = 5.068968057632446; SleepCycleTimeCount = 2390.0; SubmitCycleTimeStd = 0.004930626397422213; SleepCycleTimeMin = 4.004912853240967; ]
- 07/12/22 15:40:52 ERROR: the following job(s) failed:
- 07/12/22 15:40:52 ---------------------- Job ----------------------
- 07/12/22 15:40:52 Node Name: sub-001
- 07/12/22 15:40:52 Noop: false
- 07/12/22 15:40:52 NodeID: 0
- 07/12/22 15:40:52 Node Status: STATUS_ERROR
- 07/12/22 15:40:52 Node return val: 1
- 07/12/22 15:40:52 Error: Job proc (931969.0.0) failed with status 1
- 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
- 07/12/22 15:40:52 HTCondor Job ID: (931969.0.0)
- 07/12/22 15:40:52 Q_PARENTS: <END>
- 07/12/22 15:40:52 Q_WAITING: <END>
- 07/12/22 15:40:52 Q_CHILDREN: <END>
- 07/12/22 15:40:52 ---------------------- Job ----------------------
- 07/12/22 15:40:52 Node Name: sub-002
- 07/12/22 15:40:52 Noop: false
- 07/12/22 15:40:52 NodeID: 1
- 07/12/22 15:40:52 Node Status: STATUS_ERROR
- 07/12/22 15:40:52 Node return val: 1
- 07/12/22 15:40:52 Error: Job proc (931970.0.0) failed with status 1
- 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
- 07/12/22 15:40:52 HTCondor Job ID: (931970.0.0)
- 07/12/22 15:40:52 Q_PARENTS: <END>
- 07/12/22 15:40:52 Q_WAITING: <END>
- 07/12/22 15:40:52 Q_CHILDREN: <END>
- 07/12/22 15:40:52 ---------------------- Job ----------------------
- 07/12/22 15:40:52 Node Name: sub-005
- 07/12/22 15:40:52 Noop: false
- 07/12/22 15:40:52 NodeID: 2
- 07/12/22 15:40:52 Node Status: STATUS_ERROR
- 07/12/22 15:40:52 Node return val: 1
- 07/12/22 15:40:52 Error: Job proc (931971.0.0) failed with status 1
- 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
- 07/12/22 15:40:52 HTCondor Job ID: (931971.0.0)
- 07/12/22 15:40:52 Q_PARENTS: <END>
- 07/12/22 15:40:52 Q_WAITING: <END>
- 07/12/22 15:40:52 Q_CHILDREN: <END>
- 07/12/22 15:40:52 ---------------------- Job ----------------------
- 07/12/22 15:40:52 Node Name: sub-004
- 07/12/22 15:40:52 Noop: false
- 07/12/22 15:40:52 NodeID: 3
- 07/12/22 15:40:52 Node Status: STATUS_ERROR
- 07/12/22 15:40:52 Node return val: 1
- 07/12/22 15:40:52 Error: Job proc (931972.0.0) failed with status 1
- 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
- 07/12/22 15:40:52 HTCondor Job ID: (931972.0.0)
- 07/12/22 15:40:52 Q_PARENTS: <END>
- 07/12/22 15:40:52 Q_WAITING: <END>
- 07/12/22 15:40:52 Q_CHILDREN: <END>
- 07/12/22 15:40:52 ---------------------- Job ----------------------
- 07/12/22 15:40:52 Node Name: sub-003
- 07/12/22 15:40:52 Noop: false
- 07/12/22 15:40:52 NodeID: 4
- 07/12/22 15:40:52 Node Status: STATUS_ERROR
- 07/12/22 15:40:52 Node return val: 1
- 07/12/22 15:40:52 Error: Job proc (931973.0.0) failed with status 1
- 07/12/22 15:40:52 Job Submit File: code/process.condor_submit
- 07/12/22 15:40:52 HTCondor Job ID: (931973.0.0)
- 07/12/22 15:40:52 Q_PARENTS: <END>
- 07/12/22 15:40:52 Q_WAITING: <END>
- 07/12/22 15:40:52 Q_CHILDREN: <END>
- 07/12/22 15:40:52 --------------------------------------- <END>
- 07/12/22 15:40:52 Aborting DAG...
- 07/12/22 15:40:52 Writing Rescue DAG to code/process.condor_dag.rescue001...
- 07/12/22 15:40:52 Removing submitted jobs...
- 07/12/22 15:40:52 Removing any/all submitted HTCondor jobs...
- 07/12/22 15:40:52 Running: /usr/bin/condor_rm -const DAGManJobId' '=?=' '931968
- 07/12/22 15:40:52 Note: 0 total job deferrals because of -MaxJobs limit (0)
- 07/12/22 15:40:52 Note: 0 total job deferrals because of -MaxIdle limit (1000)
- 07/12/22 15:40:52 Note: 0 total job deferrals because of node category throttles
- 07/12/22 15:40:52 Note: 0 total PRE script deferrals because of -MaxPre limit (20) or DEFER
- 07/12/22 15:40:52 Note: 0 total POST script deferrals because of -MaxPost limit (20) or DEFER
- 07/12/22 15:40:52 DAG status: 2 (DAG_STATUS_NODE_FAILED)
- 07/12/22 15:40:52 Of 5 nodes total:
- 07/12/22 15:40:52 Done Pre Queued Post Ready Un-Ready Failed
- 07/12/22 15:40:52 === === === === === === ===
- 07/12/22 15:40:52 0 0 0 0 0 0 5
- 07/12/22 15:40:52 0 job proc(s) currently held
- 07/12/22 15:40:52 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.005433997287527276; EventCycleTimeMax = 0.2239258289337158; EventCycleTimeMin = 2.09808349609375E-05; EventCycleTimeAvg = 0.0003853633313997021; EventCycleTimeSum = 0.9210183620452881; EventCycleTimeCount = 2390.0; SleepCycleTimeStd = 0.02056334142362197; LogProcessCycleTimeMin = 0.0005269050598144531; LogProcessCycleTimeMax = 0.0009350776672363281; SubmitCycleTimeCount = 2391.0; LogProcessCycleTimeAvg = 0.0006816387176513672; LogProcessCycleTimeCount = 5.0; LogProcessCycleTimeStd = 0.0001685829264184372; LogProcessCycleTimeSum = 0.003408193588256836; SubmitCycleTimeSum = 0.5647740364074707; SubmitCycleTimeAvg = 0.0002362082962808326; SleepCycleTimeSum = 11959.61773061752; SubmitCycleTimeMin = 7.867813110351562E-06; SubmitCycleTimeMax = 0.2237379550933838; SleepCycleTimeAvg = 5.004024155070093; SleepCycleTimeMax = 5.068968057632446; SleepCycleTimeCount = 2390.0; SubmitCycleTimeStd = 0.004930626397422213; SleepCycleTimeMin = 4.004912853240967; ]
- 07/12/22 15:40:52 Wrote metrics file code/process.condor_dag.metrics.
- 07/12/22 15:40:52 Metrics not sent because of PEGASUS_METRICS or CONDOR_DEVELOPERS setting.
- 07/12/22 15:40:52 DAGMan Runtime Statistics: [ EventCycleTimeStd = 0.005433997287527276; EventCycleTimeMax = 0.2239258289337158; EventCycleTimeMin = 2.09808349609375E-05; EventCycleTimeAvg = 0.0003853633313997021; EventCycleTimeSum = 0.9210183620452881; EventCycleTimeCount = 2390.0; SleepCycleTimeStd = 0.02056334142362197; LogProcessCycleTimeMin = 0.0005269050598144531; LogProcessCycleTimeMax = 0.0009350776672363281; SubmitCycleTimeCount = 2391.0; LogProcessCycleTimeAvg = 0.0006816387176513672; LogProcessCycleTimeCount = 5.0; LogProcessCycleTimeStd = 0.0001685829264184372; LogProcessCycleTimeSum = 0.003408193588256836; SubmitCycleTimeSum = 0.5647740364074707; SubmitCycleTimeAvg = 0.0002362082962808326; SleepCycleTimeSum = 11959.61773061752; SubmitCycleTimeMin = 7.867813110351562E-06; SubmitCycleTimeMax = 0.2237379550933838; SleepCycleTimeAvg = 5.004024155070093; SleepCycleTimeMax = 5.068968057632446; SleepCycleTimeCount = 2390.0; SubmitCycleTimeStd = 0.004930626397422213; SleepCycleTimeMin = 4.004912853240967; ]
- 07/12/22 15:40:52 **** condor_scheduniv_exec.931968.0 (condor_DAGMAN) pid 2264448 EXITING WITH STATUS 1
|