makai Path

trajectory

20220513T204805 [CommandExec]:got command set NAL9602.time_fix 1652474871 second

20220513T204845 [CommandExec]:got command strobe off

20220513T204845 [CommandExec]:Deactivating strobe

20220513T204853 [MissionManager]:Started mission Default

20220513T204910 [CommandExec]:got command failComponent

20220513T204910 [CommandExec]:Failed components:

20220513T204910 [CommandExec]:No failed Components.

20220513T205110 [DVL_micro]:only read 3 of 4 data items

20220513T205154 [NAL9602]:MT Queue status failed to be acquired within timeout. Will not retry this session.

20220513T205856 [DVL_micro]:only read 3 of 4 data items

20220513T210022 [DVL_micro]:only read 0 of 1 data item for BIT error

20220513T210026 [Default:CheckIn:E]:Default mission has been running for 11.548638 min

20220513T210028 [NAL9602]:GPS failed to acquire within timeout.

20220513T210028 [CBIT]:Data Fault in component: NAL9602

20220513T210613 [NAL9602]:MT Queue status failed to be acquired within timeout. Will not retry this session.

20220513T211054 [Default:CheckIn:E]:Default mission has been running for 22.019820 min

20220513T211127 [NAL9602]:Failed to receive proper response when querying signal strength for MT queue check.

20220513T211246 [DVL_micro]:only read 0 of 1 data item for BIT error

20220513T211253 [DVL_micro]:only read 3 of 4 data items

20220513T211558 [NAL9602]:MT Queue status failed to be acquired within timeout. Will not retry this session.

20220513T212123 [Default:CheckIn:E]:Default mission has been running for 32.497502 min

20220513T212125 [NAL9602]:GPS failed to acquire within timeout.

20220513T212125 [CBIT]:Data Fault in component: NAL9602

20220513T212753 [DVL_micro]:only read 0 of 1 data item for BIT error

20220513T213225 [NAL9602]:Failed to receive proper response when querying signal strength for MT queue check.

20220513T214222 [NAL9602]:GPS failed to acquire within timeout.

20220513T214222 [CBIT]:Data Fault in component: NAL9602

20220513T221419 [NAL9602]:Failed to receive proper response when querying signal strength for MT queue check.

20220513T222100 [DVL_micro]:DVL uart error: serial timeout

20220513T222100 [DVL_micro]:Communications Fault

20220513T222100 [DVL_micro]:Failed to parse:

20220513T222100 [CBIT]:Communications Fault in component: DVL_micro

20220513T222635 [CommandExec]:got command ! onESPclient methane --Cartridge.release 60

20220513T222746 [CommandExec]:got command ! onESPclient methane --Cartridge.release 59

20220513T222925 [CommandExec]:got command ! onESPclient methane --Cartridge.release 58, 57

20220513T223041 [CommandExec]:got command ! check

20220513T223041 [CommandExec]:Waiting for ESP to boot... Connected.# Fri May 13 22:30:22 UTC 2022 -- ssh esp@10.89.5.3 espclient methane --Cartridge.release 58, 57# /home/esp/bin/espclient: unrecognized option `--Cartridge.release'# /opt/mbari/lib/ruby/1.8/getoptlong.rb:403:in `set_error': unrecognized option `--Cartridge.release' (GetoptLong::InvalidOption)# #from /opt/mbari/lib/ruby/1.8/getoptlong.rb:510:in `get_option'# #from /opt/mbari/lib/ruby/1.8/getoptlong.rb:611:in `each'# #from /opt/mbari/lib/ruby/1.8/getoptlong.rb:610:in `loop'# #from /opt/mbari/lib/ruby/1.8/getoptlong.rb:610:in `each'# #from /home/esp/bin/espclient:43# shutting ESP off#

20220513T223158 [CommandExec]:got command set NAL9602.time_fix 1652481085 second

20220513T223849 [CommandExec]:got command ! onESPclient methane -- Cartridge.release 60

20220513T223959 [CommandExec]:got command ! check

20220513T223959 [CommandExec]:Waiting for ESP to boot... Connected.# Fri May 13 22:39:49 UTC 2022 -- ssh esp@10.89.5.3 espclient methane# 1# shutting ESP off#

20220513T224454 [CommandExec]:got command ! onESPclient methane -- Cartridge.release 59, 58

20220513T224634 [CommandExec]:got command ! onESPclient methane -- spare PhytoSampler: 100

20220513T225441 [CommandExec]:got command load ./Missions/Maintenance/sample_lab.xml

20220513T225441 [CommandExec]:Loaded ./Missions/Maintenance/sample_lab.xml

20220513T225445 [CommandExec]:got command set sample_lab.MissionTimeout 180 minute

20220513T225445 [CommandExec]:got command set sample_lab.CartridgeType -10 count

20220513T225445 [CommandExec]:got command set sample_lab.NumberOfSamples 2 count

20220513T225445 [CommandExec]:got command run

20220513T225445 [CommandExec]:Running

20220513T225445 [MissionManager]:Started mission sample_lab

20220513T225445 [sample_lab:SampleRepeater:A]:Sampling ESP cartridge type -10.000000 count

20220513T225447 [ESPComponent]:Setting regex for ESP Log summary: 'Selecting

20220513T225457 [ESPComponent]:Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8

20220513T225457 [ESPComponent]:Waiting for ESP to connect (timeout=150.0000)

20220513T225555 [ESPComponent]:ESP has connected as client

20220513T225556 [ESPComponent]:[sample #1] Setting cartridge argument to -10

20220513T225556 [ESPComponent]:[sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT

20220513T225557 [ESPComponent]:[sample #1] ESP sampling state: S_PREPARING_SHOW_LOG

20220513T225558 [ESPComponent]:[sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS

20220513T225559 [ESPComponent]:[sample #1] ESP sampling state: S_LOADING_CARTRIDGE

20220513T225640 [ESPComponent]:[sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@60,daSPR_phyto)

20220513T225643 [ESPComponent]:[sample #1] ESP sampling state: S_PRIMING (@100.0)

20220513T225643 [ESPComponent]:Filtering timeoutMinutes=17.070000 issuedCmd_=Cmd.startFiltering

20220513T225643 [ESPComponent]:[sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)

20220513T225852 [ESPComponent]:[sample #1] ESP sampling state: S_FILTERING (@0.0)

20220513T225853 [ESPComponent]:[sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)

20220513T230318 [ESPComponent]:[sample #1] ESP sampling state: S_PAUSED (@88.8)

20220513T230318 [ESPComponent]:[sample #1] ESP sampling state: S_PROCESSING (@88.8)

20220513T230320 [ESPComponent]:Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing

20220513T230321 [ESPComponent]:[sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@60,daSPR_phyto)

20220514T000004 [ESPComponent]:[sample #1] ESP sampling state: S_STOPPING (@60,daSPR_phyto)

20220514T000015 [ESPComponent]:Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &

20220514T000015 [ESPComponent]:[sample #1] ESP sampling state: S_STOPPED

20220514T000515 [ESPComponent]:[sample #1] ESP sampling sequence completed normally.

20220514T000515 [ESPComponent]:[sample #1] ESP log summary report (4 messages):#@15:56:08.27 Selecting Cartridge 60#@16:03:12.89 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged#@16:03:13.09 Sampled 88.8ml#@16:53:16.56 SPRsummary:-141.799,-58.4227,-52.952,-66.2467 after 3143s#

20220514T000516 [sample_lab:SampleRepeater:A]:Sampling ESP cartridge type -10.000000 count

20220514T000527 [ESPComponent]:Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8

20220514T000527 [ESPComponent]:Waiting for ESP to connect (timeout=150.0000)

20220514T000626 [ESPComponent]:ESP has connected as client

20220514T000627 [ESPComponent]:[sample #2] Setting cartridge argument to -10

20220514T000627 [ESPComponent]:[sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT

20220514T000628 [ESPComponent]:[sample #2] ESP sampling state: S_PREPARING_SHOW_LOG

20220514T000628 [ESPComponent]:[sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS

20220514T000630 [ESPComponent]:[sample #2] ESP sampling state: S_LOADING_CARTRIDGE

20220514T000711 [ESPComponent]:[sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@59,daSPR_phyto)

20220514T000713 [ESPComponent]:[sample #2] ESP sampling state: S_PRIMING (@100.0)

20220514T000713 [ESPComponent]:Filtering timeoutMinutes=17.070000 issuedCmd_=Cmd.startFiltering

20220514T000713 [ESPComponent]:[sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)

20220514T000924 [ESPComponent]:[sample #2] ESP sampling state: S_FILTERING (@0.0)

20220514T000924 [ESPComponent]:[sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)

20220514T001354 [ESPComponent]:[sample #2] ESP sampling state: S_PAUSED (@90.3)

20220514T001354 [ESPComponent]:[sample #2] ESP sampling state: S_PROCESSING (@90.3)

20220514T001356 [ESPComponent]:Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing

20220514T001357 [ESPComponent]:[sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@59,daSPR_phyto)

20220514T002800 [ESPComponent]:In state S_WAITING_FOR_PROCESSED got exception while awaiting completion of cmd='Cmd.startProcessing' exception=<>. Elapsed time since command issued: 845.9962

20220514T002800 [ESPComponent]:[sample #2] ESP sampling state: S_STOPPING (@59,daSPR_phyto)

20220514T002811 [ESPComponent]:Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &

20220514T002812 [ESPComponent]:[sample #2] ESP sampling state: S_STOPPED

20220514T003312 [ESPComponent]:[sample #2] ESP sampling sequence terminated with error: In state S_WAITING_FOR_PROCESSED got exception while awaiting completion of cmd='Cmd.startProcessing' exception=<>. Elapsed time since command issued: 845.9962

20220514T003312 [ESPComponent]:Communications Fault

20220514T003312 [CBIT]:Communications Fault in component: ESPComponent

20220514T003312 [ESPComponent]:[sample #2] ESP log summary report (5 messages):#@17:06:39.05 Selecting Cartridge 59#@17:13:49.06 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged#@17:13:49.26 Sampled 90.3ml#@17:27:53.39 Underpressure in PROCESSING -- Reached only 28.8 of 50.0psi after pumping 13840uL#@17:27:53.52 Thread::ParentDied in stopSPRafterError -- :PROCESSING#

20220514T003312 [MissionManager]:Started mission Default

20220514T004108 [CommandExec]:got command ! start onESP upsync

20220514T005658 [DVL_micro]:DVL uart error: serial timeout

20220514T005658 [DVL_micro]:Communications Fault

20220514T005658 [DVL_micro]:Failed to parse:

20220514T005658 [CBIT]:Communications Fault in component: DVL_micro

20220514T010601 [CommandExec]:got command ! onESPclient methane -- Cartridge.release 58

20220514T010811 [CommandExec]:got command ! onESPclient methane -- spare BACsampler: 100

20220514T011100 [CommandExec]:got command load ./Missions/Maintenance/sample_lab.xml

20220514T011100 [CommandExec]:Loaded ./Missions/Maintenance/sample_lab.xml

20220514T011102 [CommandExec]:got command set sample_lab.CartridgeType -6 count

20220514T011102 [CommandExec]:got command run

20220514T011102 [CommandExec]:Running

20220514T011102 [MissionManager]:Started mission sample_lab

20220514T011102 [sample_lab:SampleRepeater:A]:Sampling ESP cartridge type -6.000000 count

20220514T011108 [CommandExec]:got command ! check

20220514T011109 [CommandExec]:Waiting for ESP to boot... Connected.# Sat May 14 01:09:07 UTC 2022 -- ssh esp@10.89.5.3 espclient methane# {BACsampler: 100,# PhytoSampler: 100}# shutting ESP off#

20220514T011114 [ESPComponent]:Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8

20220514T011114 [ESPComponent]:Waiting for ESP to connect (timeout=150.0000)

20220514T011213 [ESPComponent]:ESP has connected as client

20220514T011213 [ESPComponent]:[sample #3] Setting cartridge argument to -6

20220514T011213 [ESPComponent]:[sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT

20220514T011214 [ESPComponent]:[sample #3] ESP sampling state: S_PREPARING_SHOW_LOG

20220514T011215 [ESPComponent]:[sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS

20220514T011217 [ESPComponent]:[sample #3] ESP sampling state: S_LOADING_CARTRIDGE

20220514T011256 [ESPComponent]:[sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@58,archive_bac)

20220514T011259 [ESPComponent]:[sample #3] ESP sampling state: S_PRIMING (@100.0)

20220514T011259 [ESPComponent]:Filtering timeoutMinutes=17.070000 issuedCmd_=Cmd.startFiltering

20220514T011259 [ESPComponent]:[sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)

20220514T011503 [ESPComponent]:[sample #3] ESP sampling state: S_FILTERING (@0.0)

20220514T011503 [ESPComponent]:[sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)

20220514T011937 [ESPComponent]:[sample #3] ESP sampling state: S_PAUSED (@100.0)

20220514T011937 [ESPComponent]:[sample #3] ESP sampling state: S_PROCESSING (@100.0)

20220514T011939 [ESPComponent]:Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing

20220514T011940 [ESPComponent]:[sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@58,archive_bac)

20220514T013144 [ESPComponent]:[sample #3] ESP sampling state: S_STOPPING (@58,archive_bac)

20220514T013153 [ESPComponent]:Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &

20220514T013154 [ESPComponent]:[sample #3] ESP sampling state: S_STOPPED

20220514T013654 [ESPComponent]:[sample #3] ESP sampling sequence completed normally.

20220514T013654 [ESPComponent]:[sample #3] ESP log summary report (2 messages):#@18:12:25.52 Selecting Cartridge 58#@18:19:33.27 Sampled 100.0ml#

20220514T013654 [MissionManager]:Started mission Default

20220514T014547 [CommandExec]:got command ! onESPclient methane -- spare BACsampler: 1000

20220514T014627 [CommandExec]:got command ! onESPclient methane -- spare PhytoSampler: 1000

20220514T015009 [CommandExec]:Sat May 14 01:46:27 UTC 2022 -- ssh esp@10.89.5.3 espclient methane# ssh: connect to host 10.89.5.3 port 22: Connection refused##

20220514T015430 [DVL_micro]:DVL uart error: serial timeout

20220514T015430 [DVL_micro]:Communications Fault

20220514T015430 [DVL_micro]:Failed to parse:

20220514T015430 [CBIT]:Communications Fault in component: DVL_micro

20220514T024642 [CBIT]:Beginning ground fault scan

20220514T031528 [DVL_micro]:No DVL communication! Re-initializing

20220514T032951 [CommandExec]:got command ! onESPclient methane -- Cartridge.release 55

20220514T033000 [CommandExec]:Waiting for ESP to boot...

20220514T033849 [CommandExec]:got command ! onESPclient methane -- spare PhytoSampler: 1000

20220514T034113 [CommandExec]:got command load ./Missions/Maintenance/sample_lab.xml

20220514T034113 [CommandExec]:Loaded ./Missions/Maintenance/sample_lab.xml

20220514T034115 [CommandExec]:got command set sample_lab.CartridgeType -6 count

20220514T034115 [CommandExec]:got command run

20220514T034115 [CommandExec]:Running

20220514T034115 [MissionManager]:Started mission sample_lab

20220514T034310 [ESPComponent]:[sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@55,archive_bac)

20220514T041940 [ESPComponent]:[sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@55,archive_bac)

20220514T043144 [ESPComponent]:[sample #4] ESP sampling state: S_STOPPING (@55,archive_bac)

20220514T043654 [ESPComponent]:[sample #4] ESP sampling sequence completed normally.

20220514T043654 [ESPComponent]:[sample #4] ESP log summary report (2 messages):#@20:42:39.42 Selecting Cartridge 55#@21:19:33.06 Sampled 1000.0ml#

20220514T060448 [CommandExec]:got command set sample_lab.MissionTimeout 120 minute

20220514T060643 [ESPComponent]:[sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@56,daSPR_phyto)

20220514T063623 [DVL_micro]:No DVL communication! Re-initializing

20220514T064128 [DVL_micro]:No DVL communication! Re-initializing

20220514T064528 [ESPComponent]:[sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@56,daSPR_phyto)

20220514T074323 [ESPComponent]:[sample #5] ESP sampling state: S_STOPPING (@56,daSPR_phyto)

20220514T084653 [CBIT]:Beginning ground fault scan

20220514T100538 [DVL_micro]:DVL BIT error. See manual. Result code: 54

20220514T144704 [CBIT]:Beginning ground fault scan

20220514T153756 [CommandExec]:got command ! start onESP upsync

20220514T172711 [DVL_micro]:DVL BIT error. See manual. Result code: 74

20220514T213022 [CommandExec]:got command set sample_lab.MissionTimeout 180 minute

20220514T215057 [DVL_micro]:DVL BIT error. See manual. Result code: 304

20220514T232026 [CommandExec]:got command ! start onESP upsync

20220514T235306 [ESPComponent]:[sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@57,archive_bac)

20220515T002919 [ESPComponent]:[sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@57,archive_bac)

20220515T004123 [ESPComponent]:[sample #7] ESP sampling state: S_STOPPING (@57,archive_bac)

20220515T004633 [ESPComponent]:[sample #7] ESP log summary report (2 messages):#@16:52:35.79 Selecting Cartridge 57#@17:29:12.21 Sampled 1000.0ml#

20220515T010211 [CommandExec]:got command ! onESPclient methane -- Cmd.runStandard

20220515T052621 [CommandExec]:got command quit