2022-05-13T20:46:07.181Z,1652474767.181 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg 2022-05-13T20:46:10.656Z,1652474770.656 [SBIT](IMPORTANT): git: 2022-04-25-37-g2a944135f 2022-05-13T20:46:10.657Z,1652474770.657 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8 2022-05-13T20:46:31.371Z,1652474791.371 [SBIT](IMPORTANT): Beginning Startup BIT 2022-05-13T20:46:31.407Z,1652474791.407 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-13T20:46:42.424Z,1652474802.424 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.015891 CHAN A1 (24V): 0.000928 CHAN A2 (12V): -0.005865 CHAN A3 (5V): -0.002452 CHAN B0 (3.3V): -0.000904 CHAN B1 (3.15aV): -0.001255 CHAN B2 (3.15bV): -0.000953 CHAN B3 (GND): 0.000046 OPEN: 0.007007 Full Scale: +/- 1 mA 2022-05-13T20:46:44.458Z,1652474804.458 [SBIT](FAULT): Elevator: EXPECTED:15.000000 ACTUAL:14.799875 2022-05-13T20:47:25.115Z,1652474845.115 [SBIT](CRITICAL): SBIT FAILED 2022-05-13T20:47:25.116Z,1652474845.116 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2022-05-13T20:47:25.117Z,1652474845.117 [SBIT](IMPORTANT): BPC1.loadAtStartup=0 bool; 2022-05-13T20:47:25.117Z,1652474845.117 [SBIT](IMPORTANT): Express linearApproximation CBIT.ampHoursUsed 1.000000 ampere_hour; 2022-05-13T20:47:25.117Z,1652474845.117 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.latVertex1=34.4368 degree; 2022-05-13T20:47:25.117Z,1652474845.117 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.latVertex2=34.267 degree; 2022-05-13T20:47:25.117Z,1652474845.117 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.latVertex3=34.107 degree; 2022-05-13T20:47:25.117Z,1652474845.117 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.latVertex4=34.37 degree; 2022-05-13T20:47:25.117Z,1652474845.117 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.loadAtStartup=1 bool; 2022-05-13T20:47:25.117Z,1652474845.117 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.lonVertex1=-120.476 degree; 2022-05-13T20:47:25.118Z,1652474845.118 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.lonVertex2=-119.333 degree; 2022-05-13T20:47:25.118Z,1652474845.118 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.lonVertex3=-119.33 degree; 2022-05-13T20:47:25.118Z,1652474845.118 [SBIT](IMPORTANT): ShortestDistanceToPolygonSidesCalculator.lonVertex4=-120.5 degree; 2022-05-13T20:47:25.118Z,1652474845.118 [SBIT](IMPORTANT): VerticalControl.massDefault=0 centimeter; 2022-05-13T20:47:25.118Z,1652474845.118 [SBIT](IMPORTANT): VerticalControl.massPositionLimitAft=-28 millimeter; 2022-05-13T20:47:25.119Z,1652474845.119 [CommandExec](FAULT): Scheduling is paused 2022-05-13T20:47:25.523Z,1652474845.523 [MissionManager](IMPORTANT): Started mission Startup 2022-05-13T20:47:51.992Z,1652474871.992 [CommandExec](IMPORTANT): got command maintain sensor NAL9602.latitude_fix 36.8 degree 2022-05-13T20:47:51.996Z,1652474871.996 [CommandExec](IMPORTANT): got command maintain sensor NAL9602.longitude_fix -121.9 degree 2022-05-13T20:48:05.190Z,1652474885.190 [CommandExec](IMPORTANT): got command set NAL9602.time_fix 1652474871 second 2022-05-13T20:48:45.001Z,1652474925.001 [CommandExec](IMPORTANT): got command strobe off 2022-05-13T20:48:45.001Z,1652474925.001 [CommandExec](IMPORTANT): Deactivating strobe 2022-05-13T20:48:53.172Z,1652474933.172 [MissionManager](IMPORTANT): Started mission Default 2022-05-13T20:49:10.788Z,1652474950.788 [CommandExec](IMPORTANT): got command failComponent 2022-05-13T20:49:10.788Z,1652474950.788 [CommandExec](IMPORTANT): Failed components: 2022-05-13T20:49:10.790Z,1652474950.790 [CommandExec](IMPORTANT): No failed Components. 2022-05-13T20:51:54.562Z,1652475114.562 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T21:00:26.494Z,1652475626.494 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.548638 min 2022-05-13T21:00:28.092Z,1652475628.092 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-13T21:06:13.588Z,1652475973.588 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T21:10:54.759Z,1652476254.759 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 22.019820 min 2022-05-13T21:15:58.594Z,1652476558.594 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T21:21:23.429Z,1652476883.429 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 32.497502 min 2022-05-13T21:21:25.024Z,1652476885.024 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-13T21:27:10.472Z,1652477230.472 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T21:31:52.544Z,1652477512.544 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 42.982731 min 2022-05-13T21:36:56.334Z,1652477816.334 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T21:42:20.792Z,1652478140.792 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 53.453544 min 2022-05-13T21:42:22.396Z,1652478142.396 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-13T21:48:07.821Z,1652478487.821 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T21:52:49.463Z,1652478769.463 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 63.931291 min 2022-05-13T21:57:53.657Z,1652479073.657 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T22:03:17.694Z,1652479397.694 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 74.402067 min 2022-05-13T22:03:19.360Z,1652479399.360 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-13T22:09:04.730Z,1652479744.730 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T22:13:46.346Z,1652480026.346 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 84.879281 min 2022-05-13T22:18:50.525Z,1652480330.525 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T22:24:15.027Z,1652480655.027 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 95.357202 min 2022-05-13T22:24:16.599Z,1652480656.599 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-13T22:24:16.624Z,1652480656.624 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-13T22:26:28.057Z,1652480788.057 [DataOverHttps](IMPORTANT): SBD MTMSN=20220513T222624 2022-05-13T22:26:35.514Z,1652480795.514 [CommandExec](IMPORTANT): got command ! onESPclient methane --Cartridge.release 60 2022-05-13T22:26:35.865Z,1652480795.865 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-13UTC222635_onESPclient 2022-05-13T22:27:38.525Z,1652480858.525 [DataOverHttps](IMPORTANT): SBD MTMSN=20220513T222734 2022-05-13T22:27:46.017Z,1652480866.017 [CommandExec](IMPORTANT): got command ! onESPclient methane --Cartridge.release 59 2022-05-13T22:27:46.349Z,1652480866.349 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-13UTC222746_onESPclient 2022-05-13T22:29:18.046Z,1652480958.046 [DataOverHttps](IMPORTANT): SBD MOMSN=16780046, MTMSN=20220513T222914 2022-05-13T22:29:25.486Z,1652480965.486 [CommandExec](IMPORTANT): got command ! onESPclient methane --Cartridge.release 58, 57 2022-05-13T22:29:25.838Z,1652480965.838 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-13UTC222925_onESPclient 2022-05-13T22:30:32.330Z,1652481032.330 [DataOverHttps](IMPORTANT): SBD MOMSN=16780073, MTMSN=20220513T223028 2022-05-13T22:30:41.721Z,1652481041.721 [CommandExec](IMPORTANT): got command ! check 2022-05-13T22:30:41.990Z,1652481041.990 [CommandExec](IMPORTANT): Waiting for ESP to boot... Connected. Fri May 13 22:30:22 UTC 2022 -- ssh esp@10.89.5.3 espclient methane --Cartridge.release 58, 57 /home/esp/bin/espclient: unrecognized option `--Cartridge.release' /opt/mbari/lib/ruby/1.8/getoptlong.rb:403:in `set_error': unrecognized option `--Cartridge.release' (GetoptLong::InvalidOption) from /opt/mbari/lib/ruby/1.8/getoptlong.rb:510:in `get_option' from /opt/mbari/lib/ruby/1.8/getoptlong.rb:611:in `each' from /opt/mbari/lib/ruby/1.8/getoptlong.rb:610:in `loop' from /opt/mbari/lib/ruby/1.8/getoptlong.rb:610:in `each' from /home/esp/bin/espclient:43 shutting ESP off 2022-05-13T22:31:31.889Z,1652481091.889 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T22:31:58.191Z,1652481118.191 [CommandExec](IMPORTANT): got command set NAL9602.time_fix 1652481085 second 2022-05-13T22:36:28.422Z,1652481388.422 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 107.580688 min 2022-05-13T22:36:30.025Z,1652481390.025 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-13T22:38:42.217Z,1652481522.217 [DataOverHttps](IMPORTANT): SBD MTMSN=20220513T223838 2022-05-13T22:38:49.759Z,1652481529.759 [CommandExec](IMPORTANT): got command ! onESPclient methane -- Cartridge.release 60 2022-05-13T22:38:50.114Z,1652481530.114 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-13UTC223850_onESPclient 2022-05-13T22:39:51.969Z,1652481591.969 [DataOverHttps](IMPORTANT): SBD MTMSN=20220513T223948 2022-05-13T22:39:59.794Z,1652481599.794 [CommandExec](IMPORTANT): got command ! check 2022-05-13T22:39:59.985Z,1652481599.985 [CommandExec](IMPORTANT): Waiting for ESP to boot... Connected. Fri May 13 22:39:49 UTC 2022 -- ssh esp@10.89.5.3 espclient methane 1 shutting ESP off 2022-05-13T22:42:15.616Z,1652481735.616 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T22:44:46.853Z,1652481886.853 [DataOverHttps](IMPORTANT): SBD MTMSN=20220513T224442 2022-05-13T22:44:54.358Z,1652481894.358 [CommandExec](IMPORTANT): got command ! onESPclient methane -- Cartridge.release 59, 58 2022-05-13T22:44:54.710Z,1652481894.710 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-13UTC224454_onESPclient 2022-05-13T22:46:26.737Z,1652481986.737 [DataOverHttps](IMPORTANT): SBD MTMSN=20220513T224622 2022-05-13T22:46:34.199Z,1652481994.199 [CommandExec](IMPORTANT): got command ! onESPclient methane -- spare PhytoSampler: 100 2022-05-13T22:46:34.528Z,1652481994.528 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-13UTC224634_onESPclient 2022-05-13T22:47:08.537Z,1652482028.537 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 118.249406 min 2022-05-13T22:52:12.383Z,1652482332.383 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-13T22:54:33.622Z,1652482473.622 [DataOverHttps](IMPORTANT): SBD MTMSN=20220513T225429 2022-05-13T22:54:41.079Z,1652482481.079 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.xml 2022-05-13T22:54:41.173Z,1652482481.173 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.xml 2022-05-13T22:54:45.462Z,1652482485.462 [CommandExec](IMPORTANT): got command set sample_lab.MissionTimeout 180 minute 2022-05-13T22:54:45.463Z,1652482485.463 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -10 count 2022-05-13T22:54:45.504Z,1652482485.504 [CommandExec](IMPORTANT): got command set sample_lab.NumberOfSamples 2 count 2022-05-13T22:54:45.505Z,1652482485.505 [CommandExec](IMPORTANT): got command run 2022-05-13T22:54:45.507Z,1652482485.507 [CommandExec](IMPORTANT): Running 2022-05-13T22:54:45.591Z,1652482485.591 [MissionManager](IMPORTANT): Started mission sample_lab 2022-05-13T22:54:45.592Z,1652482485.592 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -10.000000 count 2022-05-13T22:54:47.085Z,1652482487.085 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:' 2022-05-13T22:54:57.224Z,1652482497.224 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-13T22:54:57.483Z,1652482497.483 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-13T22:55:55.772Z,1652482555.772 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-13T22:55:56.586Z,1652482556.586 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -10 2022-05-13T22:55:56.587Z,1652482556.587 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-13T22:55:57.793Z,1652482557.793 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-13T22:55:58.201Z,1652482558.201 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-13T22:55:59.813Z,1652482559.813 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-13T22:56:40.213Z,1652482600.213 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@60,daSPR_phyto) 2022-05-13T22:56:43.041Z,1652482603.041 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2022-05-13T22:56:43.042Z,1652482603.042 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=17.070000 issuedCmd_=Cmd.startFiltering 2022-05-13T22:56:43.445Z,1652482603.445 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-13T22:58:52.725Z,1652482732.725 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2022-05-13T22:58:53.129Z,1652482733.129 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-13T23:03:18.197Z,1652482998.197 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@88.8) 2022-05-13T23:03:18.601Z,1652482998.601 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@88.8) 2022-05-13T23:03:20.623Z,1652483000.623 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2022-05-13T23:03:21.026Z,1652483001.026 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@60,daSPR_phyto) 2022-05-14T00:00:04.425Z,1652486404.425 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@60,daSPR_phyto) 2022-05-14T00:00:15.334Z,1652486415.334 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-14T00:00:15.738Z,1652486415.738 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2022-05-14T00:05:15.505Z,1652486715.505 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2022-05-14T00:05:16.075Z,1652486716.075 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -10.000000 count 2022-05-14T00:05:27.732Z,1652486727.732 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-14T00:05:27.864Z,1652486727.864 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-14T00:06:26.609Z,1652486786.609 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-14T00:06:27.426Z,1652486787.426 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -10 2022-05-14T00:06:27.427Z,1652486787.427 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-14T00:06:28.225Z,1652486788.225 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-14T00:06:28.629Z,1652486788.629 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-14T00:06:30.649Z,1652486790.649 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-14T00:07:11.049Z,1652486831.049 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@59,daSPR_phyto) 2022-05-14T00:07:13.473Z,1652486833.473 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2022-05-14T00:07:13.474Z,1652486833.474 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=17.070000 issuedCmd_=Cmd.startFiltering 2022-05-14T00:07:13.877Z,1652486833.877 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-14T00:09:24.369Z,1652486964.369 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2022-05-14T00:09:24.773Z,1652486964.773 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-14T00:13:54.242Z,1652487234.242 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@90.3) 2022-05-14T00:13:54.645Z,1652487234.645 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@90.3) 2022-05-14T00:13:56.677Z,1652487236.677 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2022-05-14T00:13:57.070Z,1652487237.070 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@59,daSPR_phyto) 2022-05-14T00:28:00.641Z,1652488080.641 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@59,daSPR_phyto) 2022-05-14T00:28:11.961Z,1652488091.961 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-14T00:28:12.361Z,1652488092.361 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2022-05-14T00:33:12.126Z,1652488392.126 [ESPComponent](CRITICAL): [sample #2] ESP sampling sequence terminated with error: In state S_WAITING_FOR_PROCESSED got exception while awaiting completion of cmd='Cmd.startProcessing' exception=<>. Elapsed time since command issued: 845.9962 2022-05-14T00:33:12.980Z,1652488392.980 [MissionManager](IMPORTANT): Started mission Default 2022-05-14T00:33:28.307Z,1652488408.307 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T00:39:13.593Z,1652488753.593 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T00:41:00.785Z,1652488860.785 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T004056 2022-05-14T00:41:08.249Z,1652488868.249 [CommandExec](IMPORTANT): got command ! start onESP upsync 2022-05-14T00:41:08.507Z,1652488868.507 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC004108_onESP 2022-05-14T00:44:23.475Z,1652489063.475 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.168745 min 2022-05-14T00:44:25.084Z,1652489065.084 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T00:50:10.535Z,1652489410.535 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T00:55:09.138Z,1652489709.138 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 21.929140 min 2022-05-14T00:55:10.739Z,1652489710.739 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T01:00:56.226Z,1652490056.226 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T01:05:43.112Z,1652490343.112 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 32.495465 min 2022-05-14T01:05:53.662Z,1652490353.662 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T010549 2022-05-14T01:06:01.170Z,1652490361.170 [CommandExec](IMPORTANT): got command ! onESPclient methane -- Cartridge.release 58 2022-05-14T01:06:01.509Z,1652490361.509 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC010601_onESPclient 2022-05-14T01:08:04.433Z,1652490484.433 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T010800 2022-05-14T01:08:11.902Z,1652490491.902 [CommandExec](IMPORTANT): got command ! onESPclient methane -- spare BACsampler: 100 2022-05-14T01:08:12.293Z,1652490492.293 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC010812_onESPclient 2022-05-14T01:10:47.045Z,1652490647.045 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T01:10:48.295Z,1652490648.295 [DataOverHttps](IMPORTANT): SBD MOMSN=16780655, MTMSN=20220514T011045 2022-05-14T01:11:00.289Z,1652490660.289 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.xml 2022-05-14T01:11:00.326Z,1652490660.326 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.xml 2022-05-14T01:11:01.687Z,1652490661.687 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T011057 2022-05-14T01:11:02.106Z,1652490662.106 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -6 count 2022-05-14T01:11:02.106Z,1652490662.106 [CommandExec](IMPORTANT): got command run 2022-05-14T01:11:02.126Z,1652490662.126 [CommandExec](IMPORTANT): Running 2022-05-14T01:11:02.389Z,1652490662.389 [MissionManager](IMPORTANT): Started mission sample_lab 2022-05-14T01:11:02.391Z,1652490662.391 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -6.000000 count 2022-05-14T01:11:08.885Z,1652490668.885 [CommandExec](IMPORTANT): got command ! check 2022-05-14T01:11:09.081Z,1652490669.081 [CommandExec](IMPORTANT): Waiting for ESP to boot... Connected. Sat May 14 01:09:07 UTC 2022 -- ssh esp@10.89.5.3 espclient methane {BACsampler: 100, PhytoSampler: 100} shutting ESP off 2022-05-14T01:11:14.180Z,1652490674.180 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-14T01:11:14.335Z,1652490674.335 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-14T01:12:13.069Z,1652490733.069 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-14T01:12:13.883Z,1652490733.883 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6 2022-05-14T01:12:13.883Z,1652490733.883 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-14T01:12:14.685Z,1652490734.685 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-14T01:12:15.089Z,1652490735.089 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-14T01:12:17.110Z,1652490737.110 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-14T01:12:56.700Z,1652490776.700 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@58,archive_bac) 2022-05-14T01:12:59.125Z,1652490779.125 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2022-05-14T01:12:59.126Z,1652490779.126 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=17.070000 issuedCmd_=Cmd.startFiltering 2022-05-14T01:12:59.529Z,1652490779.529 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-14T01:15:03.557Z,1652490903.557 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2022-05-14T01:15:03.962Z,1652490903.962 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-14T01:19:37.493Z,1652491177.493 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@100.0) 2022-05-14T01:19:37.897Z,1652491177.897 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@100.0) 2022-05-14T01:19:39.918Z,1652491179.918 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-05-14T01:19:40.321Z,1652491180.321 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@58,archive_bac) 2022-05-14T01:31:44.693Z,1652491904.693 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@58,archive_bac) 2022-05-14T01:31:53.986Z,1652491913.986 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-14T01:31:54.389Z,1652491914.389 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2022-05-14T01:36:54.157Z,1652492214.157 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2022-05-14T01:36:54.723Z,1652492214.723 [MissionManager](IMPORTANT): Started mission Default 2022-05-14T01:37:09.932Z,1652492229.932 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T01:37:09.980Z,1652492229.980 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T01:44:25.080Z,1652492665.080 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T01:45:40.417Z,1652492740.417 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T014536 2022-05-14T01:45:47.927Z,1652492747.927 [CommandExec](IMPORTANT): got command ! onESPclient methane -- spare BACsampler: 1000 2022-05-14T01:45:48.269Z,1652492748.269 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC014548_onESPclient 2022-05-14T01:46:19.641Z,1652492779.641 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T014615 2022-05-14T01:46:27.163Z,1652492787.163 [CommandExec](IMPORTANT): got command ! onESPclient methane -- spare PhytoSampler: 1000 2022-05-14T01:46:27.478Z,1652492787.478 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC014627_onESPclient 2022-05-14T01:49:03.093Z,1652492943.093 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.133021 min 2022-05-14T01:50:01.985Z,1652493001.985 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T014958 2022-05-14T01:50:09.484Z,1652493009.484 [CommandExec](IMPORTANT): got command ! check 2022-05-14T01:50:09.666Z,1652493009.666 [CommandExec](IMPORTANT): Sat May 14 01:46:27 UTC 2022 -- ssh esp@10.89.5.3 espclient methane ssh: connect to host 10.89.5.3 port 22: Connection refused 2022-05-14T01:54:06.897Z,1652493246.897 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T01:59:49.122Z,1652493589.122 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 22.899896 min 2022-05-14T01:59:50.723Z,1652493590.723 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T02:05:36.188Z,1652493936.188 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T02:10:17.803Z,1652494217.803 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 33.377775 min 2022-05-14T02:15:21.989Z,1652494521.989 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T02:20:46.416Z,1652494846.416 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 43.855107 min 2022-05-14T02:20:48.024Z,1652494848.024 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T02:26:33.462Z,1652495193.462 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T02:31:15.060Z,1652495475.060 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 54.332467 min 2022-05-14T02:36:19.290Z,1652495779.290 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T02:41:43.344Z,1652496103.344 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 64.803483 min 2022-05-14T02:41:44.961Z,1652496104.961 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T02:46:42.745Z,1652496402.745 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-14T02:46:53.641Z,1652496413.641 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.015666 CHAN A1 (24V): 0.000397 CHAN A2 (12V): -0.005367 CHAN A3 (5V): -0.002873 CHAN B0 (3.3V): -0.001537 CHAN B1 (3.15aV): -0.001155 CHAN B2 (3.15bV): -0.004318 CHAN B3 (GND): -0.000079 OPEN: 0.007457 Full Scale: +/- 1 mA 2022-05-14T02:47:30.376Z,1652496450.376 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T02:52:11.577Z,1652496731.577 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 75.273958 min 2022-05-14T02:57:15.364Z,1652497035.364 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T03:02:41.835Z,1652497361.835 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 85.778320 min 2022-05-14T03:02:43.437Z,1652497363.437 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T03:08:28.880Z,1652497708.880 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T03:13:10.478Z,1652497990.478 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 96.256234 min 2022-05-14T03:18:14.682Z,1652498294.682 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T03:23:39.185Z,1652498619.185 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 106.734334 min 2022-05-14T03:23:40.807Z,1652498620.807 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T03:23:40.867Z,1652498620.867 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T03:29:43.909Z,1652498983.909 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T032939 2022-05-14T03:29:51.354Z,1652498991.354 [CommandExec](IMPORTANT): got command ! onESPclient methane -- Cartridge.release 55 2022-05-14T03:29:51.890Z,1652498991.890 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC032951_onESPclient 2022-05-14T03:29:52.306Z,1652498992.306 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T032948 2022-05-14T03:30:00.229Z,1652499000.229 [CommandExec](IMPORTANT): got command ! check 2022-05-14T03:30:00.496Z,1652499000.496 [CommandExec](IMPORTANT): Waiting for ESP to boot... 2022-05-14T03:30:55.918Z,1652499055.918 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T03:34:07.834Z,1652499247.834 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 117.211800 min 2022-05-14T03:38:42.349Z,1652499522.349 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T033838 2022-05-14T03:38:49.883Z,1652499529.883 [CommandExec](IMPORTANT): got command ! onESPclient methane -- spare PhytoSampler: 1000 2022-05-14T03:38:50.195Z,1652499530.195 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC033850_onESPclient 2022-05-14T03:39:11.649Z,1652499551.649 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T03:41:03.878Z,1652499663.878 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T034059 2022-05-14T03:41:13.791Z,1652499673.791 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.xml 2022-05-14T03:41:13.864Z,1652499673.864 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.xml 2022-05-14T03:41:15.279Z,1652499675.279 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -6 count 2022-05-14T03:41:15.280Z,1652499675.280 [CommandExec](IMPORTANT): got command run 2022-05-14T03:41:15.284Z,1652499675.284 [CommandExec](IMPORTANT): Running 2022-05-14T03:41:15.374Z,1652499675.374 [MissionManager](IMPORTANT): Started mission sample_lab 2022-05-14T03:41:15.375Z,1652499675.375 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -6.000000 count 2022-05-14T03:41:27.160Z,1652499687.160 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-14T03:41:27.296Z,1652499687.296 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-14T03:42:26.845Z,1652499746.845 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-14T03:42:27.656Z,1652499747.656 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6 2022-05-14T03:42:27.656Z,1652499747.656 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-14T03:42:28.461Z,1652499748.461 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-14T03:42:28.865Z,1652499748.865 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-14T03:42:30.885Z,1652499750.885 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-14T03:43:10.477Z,1652499790.477 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@55,archive_bac) 2022-05-14T03:43:13.305Z,1652499793.305 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2022-05-14T03:43:13.306Z,1652499793.306 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-05-14T03:43:13.710Z,1652499793.710 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-14T03:45:19.758Z,1652499919.758 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2022-05-14T03:45:20.161Z,1652499920.161 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-14T04:19:37.366Z,1652501977.366 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@1000.0) 2022-05-14T04:19:37.769Z,1652501977.769 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@1000.0) 2022-05-14T04:19:39.790Z,1652501979.790 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-05-14T04:19:40.194Z,1652501980.194 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@55,archive_bac) 2022-05-14T04:31:44.595Z,1652502704.595 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@55,archive_bac) 2022-05-14T04:31:53.886Z,1652502713.886 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-14T04:31:54.289Z,1652502714.289 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2022-05-14T04:36:54.089Z,1652503014.089 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2022-05-14T04:36:54.671Z,1652503014.671 [MissionManager](IMPORTANT): Started mission Default 2022-05-14T04:37:09.872Z,1652503029.872 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T04:42:55.357Z,1652503375.357 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T04:47:53.520Z,1652503673.520 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 10.974006 min 2022-05-14T04:47:55.116Z,1652503675.116 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T04:53:40.565Z,1652504020.565 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T04:58:22.173Z,1652504302.173 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 21.451396 min 2022-05-14T05:03:26.370Z,1652504606.370 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T05:08:50.782Z,1652504930.782 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.928902 min 2022-05-14T05:08:52.391Z,1652504932.391 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T05:14:37.815Z,1652505277.815 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T05:19:20.218Z,1652505560.218 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 42.419051 min 2022-05-14T05:24:24.842Z,1652505864.842 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T05:29:48.861Z,1652506188.861 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 52.896615 min 2022-05-14T05:29:50.464Z,1652506190.464 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T05:35:35.911Z,1652506535.911 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T05:40:17.913Z,1652506817.913 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 63.380680 min 2022-05-14T05:45:21.778Z,1652507121.778 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T05:50:46.269Z,1652507446.269 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 73.852799 min 2022-05-14T05:50:47.823Z,1652507447.823 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T05:50:47.843Z,1652507447.843 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T05:58:02.957Z,1652507882.957 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T05:59:23.821Z,1652507963.821 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T055919 2022-05-14T05:59:30.329Z,1652507970.329 [CommandExec](IMPORTANT): got command ! onESPclient methane -- Cartridge.release 56 2022-05-14T05:59:30.629Z,1652507970.629 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC055930_onESPclient 2022-05-14T06:01:14.926Z,1652508074.926 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 84.331022 min 2022-05-14T06:04:38.038Z,1652508278.038 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T060434 2022-05-14T06:04:45.482Z,1652508285.482 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.xml 2022-05-14T06:04:45.577Z,1652508285.577 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.xml 2022-05-14T06:04:48.385Z,1652508288.385 [CommandExec](IMPORTANT): got command set sample_lab.MissionTimeout 120 minute 2022-05-14T06:04:48.386Z,1652508288.386 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -10 count 2022-05-14T06:04:48.386Z,1652508288.386 [CommandExec](IMPORTANT): got command run 2022-05-14T06:04:48.388Z,1652508288.388 [CommandExec](IMPORTANT): Running 2022-05-14T06:04:48.687Z,1652508288.687 [MissionManager](IMPORTANT): Started mission sample_lab 2022-05-14T06:04:48.689Z,1652508288.689 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -10.000000 count 2022-05-14T06:05:00.436Z,1652508300.436 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-14T06:05:00.578Z,1652508300.578 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-14T06:05:59.313Z,1652508359.313 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-14T06:06:00.125Z,1652508360.125 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -10 2022-05-14T06:06:00.125Z,1652508360.125 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-14T06:06:00.929Z,1652508360.929 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-14T06:06:01.333Z,1652508361.333 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-14T06:06:03.353Z,1652508363.353 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-14T06:06:18.737Z,1652508378.737 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T06:06:43.753Z,1652508403.753 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@56,daSPR_phyto) 2022-05-14T06:06:46.582Z,1652508406.582 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2022-05-14T06:06:46.583Z,1652508406.583 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-05-14T06:06:46.985Z,1652508406.985 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-14T06:09:01.114Z,1652508541.114 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2022-05-14T06:09:01.517Z,1652508541.517 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-14T06:45:25.239Z,1652510725.239 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0) 2022-05-14T06:45:25.642Z,1652510725.642 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0) 2022-05-14T06:45:27.663Z,1652510727.663 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2022-05-14T06:45:28.065Z,1652510728.065 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@56,daSPR_phyto) 2022-05-14T07:43:23.854Z,1652514203.854 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@56,daSPR_phyto) 2022-05-14T07:43:34.763Z,1652514214.763 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-14T07:43:35.165Z,1652514215.165 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2022-05-14T07:48:34.941Z,1652514514.941 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2022-05-14T07:48:35.535Z,1652514515.535 [MissionManager](IMPORTANT): Started mission Default 2022-05-14T07:48:50.712Z,1652514530.712 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T07:54:36.135Z,1652514876.135 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T07:59:17.334Z,1652515157.334 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 10.690434 min 2022-05-14T08:04:21.179Z,1652515461.179 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T08:09:45.990Z,1652515785.990 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 21.167920 min 2022-05-14T08:09:47.595Z,1652515787.595 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T08:15:33.043Z,1652516133.043 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T08:20:14.229Z,1652516414.229 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.638184 min 2022-05-14T08:25:18.038Z,1652516718.038 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T08:30:42.062Z,1652517042.062 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 42.101880 min 2022-05-14T08:30:43.663Z,1652517043.663 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T08:36:29.072Z,1652517389.072 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T08:41:11.514Z,1652517671.514 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 52.592712 min 2022-05-14T08:46:16.162Z,1652517976.162 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T08:46:53.744Z,1652518013.744 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-14T08:47:04.654Z,1652518024.654 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.017589 CHAN A1 (24V): 0.000107 CHAN A2 (12V): -0.005066 CHAN A3 (5V): -0.002630 CHAN B0 (3.3V): -0.001382 CHAN B1 (3.15aV): -0.000834 CHAN B2 (3.15bV): -0.001005 CHAN B3 (GND): 0.000118 OPEN: 0.007045 Full Scale: +/- 1 mA 2022-05-14T08:51:40.187Z,1652518300.187 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 63.070805 min 2022-05-14T08:51:41.796Z,1652518301.796 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T08:57:27.237Z,1652518647.237 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T09:02:08.478Z,1652518928.478 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 73.542700 min 2022-05-14T09:07:12.302Z,1652519232.302 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T09:12:36.735Z,1652519556.735 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 84.013184 min 2022-05-14T09:12:38.335Z,1652519558.335 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T09:12:38.354Z,1652519558.354 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T09:19:53.444Z,1652519993.444 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T09:23:06.658Z,1652520186.658 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 94.512142 min 2022-05-14T09:28:10.050Z,1652520490.050 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T09:33:34.908Z,1652520814.908 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 104.982975 min 2022-05-14T09:33:36.508Z,1652520816.508 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T09:39:21.935Z,1652521161.935 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T09:44:03.130Z,1652521443.130 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 115.453369 min 2022-05-14T09:49:06.957Z,1652521746.957 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T09:54:30.998Z,1652522070.998 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 125.917480 min 2022-05-14T09:54:32.616Z,1652522072.616 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T10:00:18.028Z,1652522418.028 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T10:05:00.042Z,1652522700.042 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 136.401912 min 2022-05-14T10:10:03.922Z,1652523003.922 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T10:15:27.953Z,1652523327.953 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 146.866781 min 2022-05-14T10:15:29.555Z,1652523329.555 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T10:21:15.087Z,1652523675.087 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T10:25:56.275Z,1652523956.275 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 157.339079 min 2022-05-14T10:30:59.798Z,1652524259.798 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T10:36:24.241Z,1652524584.241 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 167.805355 min 2022-05-14T10:36:25.837Z,1652524585.837 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T10:42:11.360Z,1652524931.360 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T10:46:52.555Z,1652525212.555 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 178.276839 min 2022-05-14T10:51:55.949Z,1652525515.949 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T10:57:20.775Z,1652525840.775 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 188.747233 min 2022-05-14T10:57:22.381Z,1652525842.381 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T10:57:22.400Z,1652525842.400 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T11:04:37.559Z,1652526277.559 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T11:07:50.283Z,1652526470.283 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 199.238997 min 2022-05-14T11:12:54.493Z,1652526774.493 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T11:18:18.969Z,1652527098.969 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 209.716781 min 2022-05-14T11:18:20.551Z,1652527100.551 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T11:24:05.986Z,1652527445.986 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T11:28:46.770Z,1652527726.770 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 220.180534 min 2022-05-14T11:33:50.970Z,1652528030.970 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T11:39:16.215Z,1652528356.215 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 230.671419 min 2022-05-14T11:39:17.822Z,1652528357.822 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T11:45:03.287Z,1652528703.287 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T11:49:43.275Z,1652528983.275 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 241.122184 min 2022-05-14T11:54:47.089Z,1652529287.089 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T12:00:10.301Z,1652529610.301 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 251.572673 min 2022-05-14T12:00:11.921Z,1652529611.921 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T12:05:57.344Z,1652529957.344 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T12:10:38.985Z,1652530238.985 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 262.050700 min 2022-05-14T12:15:43.230Z,1652530543.230 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T12:21:07.301Z,1652530867.301 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 272.522933 min 2022-05-14T12:21:08.912Z,1652530868.912 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T12:26:54.395Z,1652531214.395 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T12:31:36.454Z,1652531496.454 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 283.008724 min 2022-05-14T12:36:40.260Z,1652531800.260 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T12:42:04.698Z,1652532124.698 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 293.479134 min 2022-05-14T12:42:06.298Z,1652532126.298 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T12:42:06.377Z,1652532126.377 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T12:49:21.415Z,1652532561.415 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T12:52:32.519Z,1652532752.519 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 303.942936 min 2022-05-14T12:57:36.313Z,1652533056.313 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T13:03:01.165Z,1652533381.165 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 314.420345 min 2022-05-14T13:03:02.770Z,1652533382.770 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T13:08:48.211Z,1652533728.211 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T13:13:29.056Z,1652534009.056 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 324.885189 min 2022-05-14T13:18:33.250Z,1652534313.250 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T13:23:56.876Z,1652534636.876 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 335.348828 min 2022-05-14T13:23:58.468Z,1652534638.468 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T13:29:43.920Z,1652534983.920 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T13:34:25.939Z,1652535265.939 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 345.833105 min 2022-05-14T13:39:29.740Z,1652535569.740 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T13:44:54.580Z,1652535894.580 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 356.310547 min 2022-05-14T13:44:56.181Z,1652535896.181 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T13:50:41.599Z,1652536241.599 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T13:55:23.212Z,1652536523.212 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 366.788021 min 2022-05-14T14:00:27.412Z,1652536827.412 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T14:05:54.317Z,1652537154.317 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 377.306576 min 2022-05-14T14:05:55.922Z,1652537155.922 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T14:11:41.447Z,1652537501.447 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T14:16:24.669Z,1652537784.669 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 387.812044 min 2022-05-14T14:21:28.072Z,1652538088.072 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T14:26:52.886Z,1652538412.886 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 398.282585 min 2022-05-14T14:26:54.498Z,1652538414.498 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T14:26:54.558Z,1652538414.558 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T14:34:10.015Z,1652538850.015 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T14:37:21.128Z,1652539041.128 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 408.753060 min 2022-05-14T14:42:24.930Z,1652539344.930 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T14:47:04.929Z,1652539624.929 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-14T14:47:15.840Z,1652539635.840 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.017530 CHAN A1 (24V): 0.000486 CHAN A2 (12V): -0.005473 CHAN A3 (5V): -0.002440 CHAN B0 (3.3V): -0.000790 CHAN B1 (3.15aV): -0.001450 CHAN B2 (3.15bV): -0.001109 CHAN B3 (GND): -0.001369 OPEN: 0.007145 Full Scale: +/- 1 mA 2022-05-14T14:47:48.949Z,1652539668.949 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 419.217155 min 2022-05-14T14:47:50.555Z,1652539670.555 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T14:53:35.972Z,1652540015.972 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T14:58:17.579Z,1652540297.579 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 429.693945 min 2022-05-14T15:03:21.796Z,1652540601.796 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T15:08:46.245Z,1652540926.245 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 440.171322 min 2022-05-14T15:08:47.837Z,1652540927.837 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T15:14:33.263Z,1652541273.263 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T15:19:14.465Z,1652541554.465 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 450.642090 min 2022-05-14T15:24:18.284Z,1652541858.284 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T15:29:42.293Z,1652542182.293 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 461.105794 min 2022-05-14T15:29:43.903Z,1652542183.903 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T15:35:29.312Z,1652542529.312 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T15:37:48.725Z,1652542668.725 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T153744 2022-05-14T15:37:56.185Z,1652542676.185 [CommandExec](IMPORTANT): got command ! start onESP upsync 2022-05-14T15:37:56.467Z,1652542676.467 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC153756_onESP 2022-05-14T15:40:10.529Z,1652542810.529 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 471.576628 min 2022-05-14T15:45:14.326Z,1652543114.326 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T15:50:55.737Z,1652543455.737 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 482.330046 min 2022-05-14T15:50:57.324Z,1652543457.324 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T15:56:42.759Z,1652543802.759 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T16:01:23.945Z,1652544083.945 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 492.800163 min 2022-05-14T16:06:27.817Z,1652544387.817 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T16:11:52.238Z,1652544712.238 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 503.271647 min 2022-05-14T16:11:53.844Z,1652544713.844 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T16:11:53.864Z,1652544713.864 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T16:19:08.964Z,1652545148.964 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T16:22:20.873Z,1652545340.873 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 513.748763 min 2022-05-14T16:27:24.696Z,1652545644.696 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T16:32:49.503Z,1652545969.503 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 524.225781 min 2022-05-14T16:32:51.094Z,1652545971.094 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T16:38:36.519Z,1652546316.519 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T16:43:17.718Z,1652546597.718 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 534.696257 min 2022-05-14T16:48:21.538Z,1652546901.538 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T16:53:45.969Z,1652547225.969 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 545.167122 min 2022-05-14T16:53:47.575Z,1652547227.575 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T16:59:33.011Z,1652547573.011 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T17:04:15.020Z,1652547855.020 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 555.651497 min 2022-05-14T17:09:18.819Z,1652548158.819 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T17:14:43.299Z,1652548483.299 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 566.122266 min 2022-05-14T17:14:44.876Z,1652548484.876 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T17:20:30.298Z,1652548830.298 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T17:25:12.344Z,1652549112.344 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 576.606706 min 2022-05-14T17:30:16.159Z,1652549416.159 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T17:35:41.854Z,1652549741.854 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 587.098828 min 2022-05-14T17:35:43.457Z,1652549743.457 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T17:41:28.913Z,1652550088.913 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T17:46:10.924Z,1652550370.924 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 597.583073 min 2022-05-14T17:51:14.760Z,1652550674.760 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T17:56:38.770Z,1652550998.770 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 608.047266 min 2022-05-14T17:56:40.376Z,1652551000.376 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T17:56:40.396Z,1652551000.396 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T18:03:55.502Z,1652551435.502 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T18:07:07.402Z,1652551627.402 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 618.524284 min 2022-05-14T18:12:11.203Z,1652551931.203 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T18:17:36.437Z,1652552256.437 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 629.008268 min 2022-05-14T18:17:38.051Z,1652552258.051 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T18:23:23.565Z,1652552603.565 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T18:28:05.176Z,1652552885.176 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 639.487044 min 2022-05-14T18:33:09.402Z,1652553189.402 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T18:38:33.823Z,1652553513.823 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 649.964583 min 2022-05-14T18:38:35.432Z,1652553515.432 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T18:44:20.852Z,1652553860.852 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T18:49:02.864Z,1652554142.864 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 660.448698 min 2022-05-14T18:54:06.668Z,1652554446.668 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T18:59:31.882Z,1652554771.882 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 670.932617 min 2022-05-14T18:59:33.492Z,1652554773.492 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T19:05:18.968Z,1652555118.968 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T19:10:00.998Z,1652555400.998 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 681.417969 min 2022-05-14T19:15:04.838Z,1652555704.838 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T19:20:29.289Z,1652556029.289 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 691.889388 min 2022-05-14T19:20:30.905Z,1652556030.905 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T19:26:16.322Z,1652556376.322 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T19:30:57.913Z,1652556657.913 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 702.366602 min 2022-05-14T19:36:02.248Z,1652556962.248 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T19:41:26.686Z,1652557286.686 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 712.845898 min 2022-05-14T19:41:28.288Z,1652557288.288 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T19:41:28.308Z,1652557288.308 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T19:48:43.399Z,1652557723.399 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T19:51:55.718Z,1652557915.718 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 723.329492 min 2022-05-14T19:56:59.146Z,1652558219.146 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T20:02:23.176Z,1652558543.176 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 733.787174 min 2022-05-14T20:02:24.771Z,1652558544.771 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T20:08:10.208Z,1652558890.208 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T20:12:52.229Z,1652559172.229 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 744.271615 min 2022-05-14T20:17:56.059Z,1652559476.059 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T20:23:20.854Z,1652559800.854 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 754.748828 min 2022-05-14T20:23:22.459Z,1652559802.459 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T20:29:07.883Z,1652560147.883 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T20:33:50.693Z,1652560430.693 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 765.245964 min 2022-05-14T20:38:54.902Z,1652560734.902 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T20:44:19.371Z,1652561059.371 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 775.723893 min 2022-05-14T20:44:20.980Z,1652561060.980 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T20:47:15.938Z,1652561235.938 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-14T20:47:26.842Z,1652561246.842 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.017574 CHAN A1 (24V): 0.000044 CHAN A2 (12V): -0.005098 CHAN A3 (5V): -0.002451 CHAN B0 (3.3V): -0.001302 CHAN B1 (3.15aV): -0.001291 CHAN B2 (3.15bV): -0.001102 CHAN B3 (GND): 0.000474 OPEN: 0.007243 Full Scale: +/- 1 mA 2022-05-14T20:50:06.401Z,1652561406.401 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T20:54:48.012Z,1652561688.012 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 786.201172 min 2022-05-14T20:59:52.305Z,1652561992.305 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T21:05:15.150Z,1652562315.150 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 796.653516 min 2022-05-14T21:05:16.760Z,1652562316.760 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T21:11:02.247Z,1652562662.247 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T21:15:46.668Z,1652562946.668 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 807.178776 min 2022-05-14T21:20:51.416Z,1652563251.416 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T21:26:18.271Z,1652563578.271 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 817.705664 min 2022-05-14T21:26:19.876Z,1652563579.876 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T21:26:19.932Z,1652563579.932 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-14T21:27:30.286Z,1652563650.286 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T212726 2022-05-14T21:27:37.859Z,1652563657.859 [CommandExec](IMPORTANT): got command ! onESPclient methane -- Cartridge.release 50 2022-05-14T21:27:38.210Z,1652563658.210 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC212738_onESPclient 2022-05-14T21:30:11.550Z,1652563811.550 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T213007 2022-05-14T21:30:19.115Z,1652563819.115 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.xml 2022-05-14T21:30:19.214Z,1652563819.214 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.xml 2022-05-14T21:30:22.055Z,1652563822.055 [CommandExec](IMPORTANT): got command set sample_lab.MissionTimeout 180 minute 2022-05-14T21:30:22.056Z,1652563822.056 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -10 count 2022-05-14T21:30:22.057Z,1652563822.057 [CommandExec](IMPORTANT): got command run 2022-05-14T21:30:22.059Z,1652563822.059 [CommandExec](IMPORTANT): Running 2022-05-14T21:30:22.296Z,1652563822.296 [MissionManager](IMPORTANT): Started mission sample_lab 2022-05-14T21:30:22.298Z,1652563822.298 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -10.000000 count 2022-05-14T21:30:34.084Z,1652563834.084 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-14T21:30:34.251Z,1652563834.251 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-14T21:31:34.216Z,1652563894.216 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-14T21:31:35.028Z,1652563895.028 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -10 2022-05-14T21:31:35.029Z,1652563895.029 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-14T21:31:35.834Z,1652563895.834 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-14T21:31:36.237Z,1652563896.237 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-14T21:31:38.257Z,1652563898.257 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-14T21:32:18.657Z,1652563938.657 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@50,daSPR_phyto) 2022-05-14T21:32:21.081Z,1652563941.081 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2022-05-14T21:32:21.082Z,1652563941.082 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-05-14T21:32:21.485Z,1652563941.485 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-14T21:33:35.032Z,1652564015.032 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T21:34:32.385Z,1652564072.385 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2022-05-14T21:34:32.790Z,1652564072.790 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-14T22:07:38.873Z,1652566058.873 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1000.0) 2022-05-14T22:07:39.278Z,1652566059.278 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1000.0) 2022-05-14T22:07:41.298Z,1652566061.298 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2022-05-14T22:07:41.702Z,1652566061.702 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@50,daSPR_phyto) 2022-05-14T23:08:30.645Z,1652569710.645 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@50,daSPR_phyto) 2022-05-14T23:08:41.554Z,1652569721.554 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-14T23:08:41.957Z,1652569721.957 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2022-05-14T23:13:41.756Z,1652570021.756 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2022-05-14T23:13:42.750Z,1652570022.750 [MissionManager](IMPORTANT): Started mission Default 2022-05-14T23:13:57.947Z,1652570037.947 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T23:19:43.387Z,1652570383.387 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T23:20:19.161Z,1652570419.161 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T232015 2022-05-14T23:20:26.645Z,1652570426.645 [CommandExec](IMPORTANT): got command ! start onESP upsync 2022-05-14T23:20:26.863Z,1652570426.863 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC232026_onESP 2022-05-14T23:24:43.159Z,1652570683.159 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.000330 min 2022-05-14T23:24:44.773Z,1652570684.773 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T23:30:30.219Z,1652571030.219 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T23:35:29.166Z,1652571329.166 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 21.766901 min 2022-05-14T23:35:30.772Z,1652571330.772 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-14T23:41:16.241Z,1652571676.241 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T23:46:03.913Z,1652571963.913 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 32.345780 min 2022-05-14T23:48:18.125Z,1652572098.125 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T234814 2022-05-14T23:48:25.719Z,1652572105.719 [CommandExec](IMPORTANT): got command ! onESPclient methane -- Cartridge.release 57 2022-05-14T23:48:26.020Z,1652572106.020 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-14UTC234825_onESPclient 2022-05-14T23:51:04.114Z,1652572264.114 [DataOverHttps](IMPORTANT): SBD MTMSN=20220514T235100 2022-05-14T23:51:08.146Z,1652572268.146 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-14T23:51:10.703Z,1652572270.703 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/sample_lab.xml 2022-05-14T23:51:10.776Z,1652572270.776 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/sample_lab.xml 2022-05-14T23:51:12.454Z,1652572272.454 [CommandExec](IMPORTANT): got command set sample_lab.CartridgeType -6 count 2022-05-14T23:51:12.454Z,1652572272.454 [CommandExec](IMPORTANT): got command run 2022-05-14T23:51:12.456Z,1652572272.456 [CommandExec](IMPORTANT): Running 2022-05-14T23:51:12.640Z,1652572272.640 [MissionManager](IMPORTANT): Started mission sample_lab 2022-05-14T23:51:12.642Z,1652572272.642 [sample_lab:SampleRepeater:A](IMPORTANT): Sampling ESP cartridge type -6.000000 count 2022-05-14T23:51:24.392Z,1652572284.392 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-05-14T23:51:24.568Z,1652572284.568 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-05-14T23:52:23.312Z,1652572343.312 [ESPComponent](IMPORTANT): ESP has connected as client 2022-05-14T23:52:24.124Z,1652572344.124 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -6 2022-05-14T23:52:24.125Z,1652572344.125 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-05-14T23:52:24.930Z,1652572344.930 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2022-05-14T23:52:25.333Z,1652572345.333 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-05-14T23:52:27.353Z,1652572347.353 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2022-05-14T23:53:06.945Z,1652572386.945 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@57,archive_bac) 2022-05-14T23:53:09.773Z,1652572389.773 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2022-05-14T23:53:09.774Z,1652572389.774 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-05-14T23:53:10.177Z,1652572390.177 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-05-14T23:55:13.802Z,1652572513.802 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2022-05-14T23:55:14.205Z,1652572514.205 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-05-15T00:29:16.553Z,1652574556.553 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1000.0) 2022-05-15T00:29:16.957Z,1652574556.957 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1000.0) 2022-05-15T00:29:18.978Z,1652574558.978 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-05-15T00:29:19.381Z,1652574559.381 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@57,archive_bac) 2022-05-15T00:41:23.788Z,1652575283.788 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@57,archive_bac) 2022-05-15T00:41:33.082Z,1652575293.082 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-05-15T00:41:33.485Z,1652575293.485 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2022-05-15T00:46:33.253Z,1652575593.253 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2022-05-15T00:46:33.848Z,1652575593.848 [MissionManager](IMPORTANT): Started mission Default 2022-05-15T00:46:49.035Z,1652575609.035 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T00:52:34.504Z,1652575954.504 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T00:57:50.897Z,1652576270.897 [DataOverHttps](IMPORTANT): SBD MTMSN=20220515T005747 2022-05-15T00:57:57.323Z,1652576277.323 [CommandExec](IMPORTANT): got command ! onESPclient methane -- Cartridge.release 53 2022-05-15T00:57:57.658Z,1652576277.658 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-15UTC005757_onESPclient 2022-05-15T00:58:10.255Z,1652576290.255 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.600258 min 2022-05-15T00:58:11.865Z,1652576291.865 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T00:58:11.886Z,1652576291.886 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-15T01:02:03.601Z,1652576523.601 [DataOverHttps](IMPORTANT): SBD MTMSN=20220515T010159 2022-05-15T01:02:11.239Z,1652576531.239 [CommandExec](IMPORTANT): got command ! onESPclient methane -- Cmd.runStandard 2022-05-15T01:02:11.551Z,1652576531.551 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-15UTC010211_onESPclient 2022-05-15T01:05:26.904Z,1652576726.904 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T01:08:50.936Z,1652576930.936 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 22.277968 min 2022-05-15T01:13:54.743Z,1652577234.743 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T01:19:19.981Z,1652577559.981 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 32.763953 min 2022-05-15T01:19:21.587Z,1652577561.587 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T01:25:07.046Z,1652577907.046 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T01:29:51.084Z,1652578191.084 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 43.280363 min 2022-05-15T01:34:54.883Z,1652578494.883 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T01:40:20.108Z,1652578820.108 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 53.764160 min 2022-05-15T01:40:21.723Z,1652578821.723 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T01:46:07.132Z,1652579167.132 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T01:50:48.759Z,1652579448.759 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 64.242891 min 2022-05-15T01:55:52.934Z,1652579752.934 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T02:01:18.162Z,1652580078.162 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 74.731934 min 2022-05-15T02:01:19.791Z,1652580079.791 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T02:07:05.189Z,1652580425.189 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T02:11:47.251Z,1652580707.251 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 85.216634 min 2022-05-15T02:16:51.059Z,1652581011.059 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T02:22:16.344Z,1652581336.344 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 95.701400 min 2022-05-15T02:22:17.941Z,1652581337.941 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T02:28:03.417Z,1652581683.417 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T02:32:44.782Z,1652581964.782 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 106.173592 min 2022-05-15T02:37:48.044Z,1652582268.044 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T02:43:14.090Z,1652582594.090 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 116.664282 min 2022-05-15T02:43:15.691Z,1652582595.691 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T02:43:15.744Z,1652582595.744 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-15T02:47:27.052Z,1652582847.052 [CBIT](IMPORTANT): Beginning ground fault scan 2022-05-15T02:47:37.941Z,1652582857.941 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.016974 CHAN A1 (24V): 0.000116 CHAN A2 (12V): -0.005647 CHAN A3 (5V): -0.002555 CHAN B0 (3.3V): -0.001485 CHAN B1 (3.15aV): 0.000858 CHAN B2 (3.15bV): -0.001752 CHAN B3 (GND): 0.002987 OPEN: 0.007231 Full Scale: +/- 1 mA 2022-05-15T02:50:30.821Z,1652583030.821 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T02:53:42.734Z,1652583222.734 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 127.141520 min 2022-05-15T02:58:46.556Z,1652583526.556 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T03:04:12.228Z,1652583852.228 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 137.633008 min 2022-05-15T03:04:13.827Z,1652583853.827 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T03:09:59.261Z,1652584199.261 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T03:14:41.794Z,1652584481.794 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 148.125570 min 2022-05-15T03:19:45.989Z,1652584785.989 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T03:25:10.423Z,1652585110.423 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 158.602832 min 2022-05-15T03:25:12.029Z,1652585112.029 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T03:30:57.494Z,1652585457.494 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T03:35:39.503Z,1652585739.503 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 169.087695 min 2022-05-15T03:40:43.528Z,1652586043.528 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T03:46:10.376Z,1652586370.376 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 179.602116 min 2022-05-15T03:46:11.995Z,1652586371.995 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T03:51:57.427Z,1652586717.427 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T03:56:40.229Z,1652587000.229 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 190.099854 min 2022-05-15T04:00:58.589Z,1652587258.589 [DataOverHttps](IMPORTANT): SBD MTMSN=20220515T040054 2022-05-15T04:01:06.246Z,1652587266.246 [CommandExec](IMPORTANT): got command ! start onESP upsync 2022-05-15T04:01:06.457Z,1652587266.457 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-05-15UTC040106_onESP 2022-05-15T04:01:44.445Z,1652587304.445 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T04:07:27.052Z,1652587647.052 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 200.880127 min 2022-05-15T04:07:28.652Z,1652587648.652 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T04:13:14.116Z,1652587994.116 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T04:17:56.537Z,1652588276.537 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 211.371354 min 2022-05-15T04:23:01.206Z,1652588581.206 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T04:28:24.794Z,1652588904.794 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 221.842236 min 2022-05-15T04:28:26.402Z,1652588906.402 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T04:28:26.423Z,1652588906.423 [CBIT](CRITICAL): Data Fault in component: NAL9602 2022-05-15T04:35:41.500Z,1652589341.500 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T04:38:54.234Z,1652589534.234 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 232.332780 min 2022-05-15T04:43:58.441Z,1652589838.441 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T04:49:22.851Z,1652590162.851 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 242.809993 min 2022-05-15T04:49:24.456Z,1652590164.456 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T04:55:09.883Z,1652590509.883 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T04:59:52.306Z,1652590792.306 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 253.301025 min 2022-05-15T05:04:56.909Z,1652591096.909 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T05:10:20.927Z,1652591420.927 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 263.777946 min 2022-05-15T05:10:22.531Z,1652591422.531 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-05-15T05:16:07.991Z,1652591767.991 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T05:20:50.004Z,1652592050.004 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 274.262500 min 2022-05-15T05:25:53.804Z,1652592353.804 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-05-15T05:26:21.125Z,1652592381.125 [CommandExec](IMPORTANT): got command quit