2021-08-11T18:26:27.726Z,1628706387.726 [CommandLine](IMPORTANT): got command restart logs 2021-08-11T18:33:58.198Z,1628706838.198 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 123.743677 min 2021-08-11T18:34:38.395Z,1628706878.395 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44084, MT Status=1, MTMSN=3082 2021-08-11T18:34:39.327Z,1628706879.327 [CommandLine](IMPORTANT): got command schedule asap "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2m0p7 1 2.000000 2021-08-11T18:34:39.327Z,1628706879.327 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2m0p7 2021-08-11T18:34:39.328Z,1628706879.328 [CommandLine](IMPORTANT): Scheduled #15 (#1 of 2 with id='2m0p7'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" ASAP 2021-08-11T18:36:49.120Z,1628707009.120 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44085, MT Status=1, MTMSN=3083 2021-08-11T18:36:50.007Z,1628707010.007 [CommandLine](IMPORTANT): got command schedule asap "set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" 2m0p7 2 2.000000 2021-08-11T18:36:50.007Z,1628707010.007 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2m0p7 2021-08-11T18:36:50.008Z,1628707010.008 [CommandLine](IMPORTANT): Scheduled #16 (#2 of 2 with id='2m0p7'): "set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" ASAP 2021-08-11T18:36:50.202Z,1628707010.202 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-11T18:36:53.407Z,1628707013.407 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-11T18:37:06.798Z,1628707026.798 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-11T18:37:06.799Z,1628707026.799 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-11T18:37:06.800Z,1628707026.800 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-11T18:37:06.801Z,1628707026.801 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-11T18:37:06.801Z,1628707026.801 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-11T18:37:22.978Z,1628707042.978 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 5.000000 meter 2021-08-11T18:37:22.979Z,1628707042.979 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 15.000000 meter 2021-08-11T18:37:22.980Z,1628707042.980 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.631634 degree 2021-08-11T18:37:22.981Z,1628707042.981 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.196220 degree 2021-08-11T18:37:22.997Z,1628707042.997 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-11T18:37:22.998Z,1628707042.998 [CommandLine](IMPORTANT): got command run 2021-08-11T18:37:23.036Z,1628707043.036 [CommandLine](IMPORTANT): Running 2021-08-11T18:37:23.312Z,1628707043.312 [MissionManager](IMPORTANT): Started mission Smear 2021-08-11T18:39:45.589Z,1628707185.589 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-11T18:39:46.399Z,1628707186.399 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-11T18:54:48.958Z,1628708088.958 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 12.953700 m , 21.962610 degC , 0.249600 ug/l . 2021-08-11T18:54:50.531Z,1628708090.531 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::' 2021-08-11T18:55:00.679Z,1628708100.679 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-11T18:55:00.981Z,1628708100.981 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-11T18:55:47.312Z,1628708147.312 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-11T18:55:48.238Z,1628708148.238 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -1 2021-08-11T18:55:48.239Z,1628708148.239 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-11T18:55:48.598Z,1628708148.598 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-11T18:55:49.331Z,1628708149.331 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-11T18:55:49.735Z,1628708149.735 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-11T18:55:51.351Z,1628708151.351 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-11T18:56:25.695Z,1628708185.695 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@56,aRNAl_bac) 2021-08-11T18:56:27.311Z,1628708187.311 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2021-08-11T18:56:27.717Z,1628708187.717 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-11T18:56:28.126Z,1628708188.126 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-11T18:58:41.454Z,1628708321.454 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2021-08-11T18:58:41.855Z,1628708321.855 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-11T20:11:37.311Z,1628712697.311 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@1350.6) 2021-08-11T20:11:37.715Z,1628712697.715 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@1350.6) 2021-08-11T20:11:39.736Z,1628712699.736 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-11T20:11:40.139Z,1628712700.139 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@56,aRNAl_bac) 2021-08-11T20:13:15.079Z,1628712795.079 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@56,aRNAl_bac) 2021-08-11T20:13:22.358Z,1628712802.358 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-11T20:13:22.847Z,1628712802.847 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2021-08-11T20:18:22.611Z,1628713102.611 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2021-08-11T20:18:23.015Z,1628713103.015 [ESPComponent](IMPORTANT): [sample #1] ESP log summary report (3 messages): @14:56:01.83 Selecting Cartridge 56 @16:11:34.38 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @16:11:34.58 Sampled 1350.6ml 2021-08-11T20:18:23.854Z,1628713103.854 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-11T20:25:04.812Z,1628713504.812 [MissionManager](IMPORTANT): Started mission Default 2021-08-11T20:25:05.177Z,1628713505.177 [WetLabsBB2FL](FAULT): LCB fault: Software Overcurrent. 2021-08-11T20:31:51.919Z,1628713911.919 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.778654 min 2021-08-11T20:32:09.063Z,1628713929.063 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44110, MT Status=1, MTMSN=3084 2021-08-11T20:32:09.781Z,1628713929.781 [CommandLine](IMPORTANT): got command schedule clear 2021-08-11T20:32:09.782Z,1628713929.782 [CommandLine](IMPORTANT): #Cleared scheduled commands:0 2021-08-11T20:32:09.782Z,1628713929.782 [CommandLine](IMPORTANT): got command schedule resume 2021-08-11T20:32:09.782Z,1628713929.782 [CommandLine](IMPORTANT): Scheduling is resumed 2021-08-11T20:32:30.388Z,1628713950.388 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44111, MT Status=1, MTMSN=3085 2021-08-11T20:32:31.202Z,1628713951.202 [CommandLine](IMPORTANT): got command schedule asap "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2m6b6 1 2.000000 2021-08-11T20:32:31.203Z,1628713951.203 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2m6b6 2021-08-11T20:32:31.204Z,1628713951.204 [CommandLine](IMPORTANT): Scheduled #17 (#1 of 2 with id='2m6b6'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" ASAP 2021-08-11T20:32:40.492Z,1628713960.492 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44112, MT Status=1, MTMSN=3086 2021-08-11T20:32:41.318Z,1628713961.318 [CommandLine](IMPORTANT): got command schedule asap "set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 25 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" 2m6b6 2 2.000000 2021-08-11T20:32:41.319Z,1628713961.319 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2m6b6 2021-08-11T20:32:41.321Z,1628713961.321 [CommandLine](IMPORTANT): Scheduled #18 (#2 of 2 with id='2m6b6'): "set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 25 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" ASAP 2021-08-11T20:32:41.507Z,1628713961.507 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-11T20:32:43.686Z,1628713963.686 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-11T20:32:56.879Z,1628713976.879 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-11T20:32:56.880Z,1628713976.880 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-11T20:32:56.881Z,1628713976.881 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-11T20:32:56.882Z,1628713976.882 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-11T20:32:56.883Z,1628713976.883 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-11T20:33:12.014Z,1628713992.014 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 15.000000 meter 2021-08-11T20:33:12.015Z,1628713992.015 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 25.000000 meter 2021-08-11T20:33:12.015Z,1628713992.015 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.631634 degree 2021-08-11T20:33:12.016Z,1628713992.016 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.196220 degree 2021-08-11T20:33:12.017Z,1628713992.017 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-11T20:33:12.017Z,1628713992.017 [CommandLine](IMPORTANT): got command run 2021-08-11T20:33:12.019Z,1628713992.019 [CommandLine](IMPORTANT): Running 2021-08-11T20:33:12.317Z,1628713992.317 [MissionManager](IMPORTANT): Started mission Smear 2021-08-11T20:35:44.357Z,1628714144.357 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-11T20:35:45.189Z,1628714145.189 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-11T20:50:47.749Z,1628715047.749 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 14.850389 m , 21.963708 degC , 0.426400 ug/l . 2021-08-11T20:50:59.506Z,1628715059.506 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-11T20:50:59.609Z,1628715059.609 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-11T20:51:46.318Z,1628715106.318 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-11T20:51:47.141Z,1628715107.141 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -1 2021-08-11T20:51:47.142Z,1628715107.142 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-11T20:51:47.566Z,1628715107.566 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-11T20:51:47.936Z,1628715107.936 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-11T20:51:48.339Z,1628715108.339 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-11T20:51:50.359Z,1628715110.359 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-11T20:52:24.699Z,1628715144.699 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@55,aRNAl_bac) 2021-08-11T20:52:26.319Z,1628715146.319 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2021-08-11T20:52:26.721Z,1628715146.721 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-11T20:52:27.123Z,1628715147.123 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-11T20:54:49.339Z,1628715289.339 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2021-08-11T20:54:49.743Z,1628715289.743 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-11T21:12:16.942Z,1628716336.942 [DVL_micro](FAULT): Failed to initialize within timeout. 2021-08-11T21:27:54.225Z,1628717274.225 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@619.4) 2021-08-11T21:27:54.620Z,1628717274.620 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@619.4) 2021-08-11T21:27:56.640Z,1628717276.640 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-11T21:27:57.043Z,1628717277.043 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@55,aRNAl_bac) 2021-08-11T21:29:31.984Z,1628717371.984 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@55,aRNAl_bac) 2021-08-11T21:29:39.257Z,1628717379.257 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-11T21:29:39.659Z,1628717379.659 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2021-08-11T21:34:39.455Z,1628717679.455 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2021-08-11T21:34:39.859Z,1628717679.859 [ESPComponent](IMPORTANT): [sample #2] ESP log summary report (3 messages): @16:52:01.05 Selecting Cartridge 55 @17:27:51.90 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @17:27:52.09 Sampled 619.4ml 2021-08-11T21:34:40.705Z,1628717680.705 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-11T21:42:40.148Z,1628718160.148 [MissionManager](IMPORTANT): Started mission Default 2021-08-11T21:48:49.234Z,1628718529.234 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.144376 min 2021-08-11T21:49:35.623Z,1628718575.623 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44139, MT Status=1, MTMSN=3087 2021-08-11T21:49:36.411Z,1628718576.411 [CommandLine](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2m9pa 1 2.000000 2021-08-11T21:49:36.411Z,1628718576.411 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2m9pa 2021-08-11T21:49:36.412Z,1628718576.412 [CommandLine](IMPORTANT): Scheduled #19 (#1 of 2 with id='2m9pa'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2021-08-11T21:50:54.878Z,1628718654.878 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44140, MT Status=1, MTMSN=3088 2021-08-11T21:50:55.701Z,1628718655.701 [CommandLine](IMPORTANT): got command schedule "set Smear.SamplingYoYoMinDepth 30 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" 2m9pa 2 2.000000 2021-08-11T21:50:55.702Z,1628718655.702 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2m9pa 2021-08-11T21:50:55.703Z,1628718655.703 [CommandLine](IMPORTANT): Scheduled #20 (#2 of 2 with id='2m9pa'): "set Smear.SamplingYoYoMinDepth 30 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" 2021-08-11T21:50:55.800Z,1628718655.800 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-11T21:50:58.055Z,1628718658.055 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-11T21:51:10.939Z,1628718670.939 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-11T21:51:10.939Z,1628718670.939 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-11T21:51:10.940Z,1628718670.940 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-11T21:51:10.941Z,1628718670.941 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-11T21:51:10.942Z,1628718670.942 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-11T21:51:25.596Z,1628718685.596 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 30.000000 meter 2021-08-11T21:51:25.596Z,1628718685.596 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 70.000000 meter 2021-08-11T21:51:25.597Z,1628718685.597 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.631634 degree 2021-08-11T21:51:25.598Z,1628718685.598 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.196220 degree 2021-08-11T21:51:25.599Z,1628718685.599 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-11T21:51:25.599Z,1628718685.599 [CommandLine](IMPORTANT): got command run 2021-08-11T21:51:25.601Z,1628718685.601 [CommandLine](IMPORTANT): Running 2021-08-11T21:51:25.813Z,1628718685.813 [MissionManager](IMPORTANT): Started mission Smear 2021-08-11T21:55:23.168Z,1628718923.168 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-11T21:55:23.993Z,1628718923.993 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-11T22:10:26.508Z,1628719826.508 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 42.191662 m , 5.563409 degC , 0.603200 ug/l . 2021-08-11T22:10:38.298Z,1628719838.298 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-11T22:10:38.410Z,1628719838.410 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-11T22:11:25.467Z,1628719885.467 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-11T22:11:26.278Z,1628719886.278 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -1 2021-08-11T22:11:26.278Z,1628719886.278 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-11T22:11:26.720Z,1628719886.720 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-11T22:11:27.083Z,1628719887.083 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-11T22:11:27.487Z,1628719887.487 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-11T22:11:29.507Z,1628719889.507 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-11T22:12:03.851Z,1628719923.851 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@54,aRNAl_bac) 2021-08-11T22:12:05.468Z,1628719925.468 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2021-08-11T22:12:05.872Z,1628719925.872 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-11T22:12:06.286Z,1628719926.286 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-11T22:14:28.083Z,1628720068.083 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2021-08-11T22:14:28.487Z,1628720068.487 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-11T22:51:50.739Z,1628722310.739 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@673.4) 2021-08-11T22:51:51.143Z,1628722311.143 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@673.4) 2021-08-11T22:51:53.165Z,1628722313.165 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-11T22:51:53.570Z,1628722313.570 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@54,aRNAl_bac) 2021-08-11T22:53:28.513Z,1628722408.513 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@54,aRNAl_bac) 2021-08-11T22:53:35.780Z,1628722415.780 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-11T22:53:36.184Z,1628722416.184 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2021-08-11T22:57:48.101Z,1628722668.101 [ThrusterServo](FAULT): getVelocity uart error serial timeout 2021-08-11T22:57:48.101Z,1628722668.101 [ThrusterServo](FAULT): Thruster uart error: serial timeout 2021-08-11T22:58:35.959Z,1628722715.959 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2021-08-11T22:58:36.364Z,1628722716.364 [ESPComponent](IMPORTANT): [sample #3] ESP log summary report (3 messages): @18:11:40.39 Selecting Cartridge 54 @18:51:48.63 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @18:51:48.82 Sampled 673.4ml 2021-08-11T22:58:37.227Z,1628722717.227 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-11T23:02:46.556Z,1628722966.556 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44148, MT Status=1, MTMSN=3089 2021-08-11T23:02:47.334Z,1628722967.334 [CommandLine](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2mcgk 1 3.000000 2021-08-11T23:02:47.335Z,1628722967.335 [CommandLine](IMPORTANT): Scheduling command #1 of 3 with id=2mcgk 2021-08-11T23:02:47.336Z,1628722967.336 [CommandLine](IMPORTANT): Scheduled #21 (#1 of 3 with id='2mcgk'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2021-08-11T23:03:08.229Z,1628722988.229 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44149, MT Status=1, MTMSN=3090 2021-08-11T23:03:09.062Z,1628722989.062 [CommandLine](IMPORTANT): got command schedule "set Smear.SamplingYoYoMinDepth 140 m;set Smear.SamplingYoYoMaxDepth 200 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.YoYoMinAltitude 4 m;set Smear.MinAltitude 3 m" 2mcgk 2 3.000000 2021-08-11T23:03:09.062Z,1628722989.062 [CommandLine](IMPORTANT): Scheduling command #2 of 3 with id=2mcgk 2021-08-11T23:03:09.063Z,1628722989.063 [CommandLine](IMPORTANT): Scheduled #22 (#2 of 3 with id='2mcgk'): "set Smear.SamplingYoYoMinDepth 140 m;set Smear.SamplingYoYoMaxDepth 200 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.YoYoMinAltitude 4 m;set Smear.MinAltitude 3 m" 2021-08-11T23:03:27.237Z,1628723007.237 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44150, MT Status=1, MTMSN=3091 2021-08-11T23:03:27.873Z,1628723007.873 [CommandLine](IMPORTANT): got command schedule "set Smear.NeedCommsTimeVeryLong 8 h;run" 2mcgk 3 3.000000 2021-08-11T23:03:27.874Z,1628723007.874 [CommandLine](IMPORTANT): Scheduling command #3 of 3 with id=2mcgk 2021-08-11T23:03:27.876Z,1628723007.876 [CommandLine](IMPORTANT): Scheduled #23 (#3 of 3 with id='2mcgk'): "set Smear.NeedCommsTimeVeryLong 8 h;run" 2021-08-11T23:03:50.007Z,1628723030.007 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44151, MT Status=1, MTMSN=3092 2021-08-11T23:03:50.706Z,1628723030.706 [CommandLine](IMPORTANT): got command stop 2021-08-11T23:03:50.707Z,1628723030.707 [CommandLine](IMPORTANT): Scheduling is paused for 3 commands 2021-08-11T23:03:50.980Z,1628723030.980 [MissionManager](IMPORTANT): Started mission Default 2021-08-11T23:12:12.608Z,1628723532.608 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.353703 min 2021-08-11T23:12:39.865Z,1628723559.865 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44169, MT Status=1, MTMSN=3093 2021-08-11T23:12:40.558Z,1628723560.558 [CommandLine](IMPORTANT): got command schedule clear 2021-08-11T23:12:40.558Z,1628723560.558 [CommandLine](IMPORTANT): #Cleared scheduled commands:3 2021-08-11T23:12:40.559Z,1628723560.559 [CommandLine](IMPORTANT): got command schedule resume 2021-08-11T23:12:40.559Z,1628723560.559 [CommandLine](IMPORTANT): Scheduling is resumed 2021-08-11T23:18:29.252Z,1628723909.252 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 14.630918 min 2021-08-11T23:18:53.639Z,1628723933.639 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44175, MT Status=1, MTMSN=3094 2021-08-11T23:18:54.518Z,1628723934.518 [CommandLine](IMPORTANT): got command schedule asap "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 210 m" 2mdtz 1 3.000000 2021-08-11T23:18:54.519Z,1628723934.519 [CommandLine](IMPORTANT): Scheduling command #1 of 3 with id=2mdtz 2021-08-11T23:18:54.520Z,1628723934.520 [CommandLine](IMPORTANT): Scheduled #24 (#1 of 3 with id='2mdtz'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 210 m" ASAP 2021-08-11T23:19:23.970Z,1628723963.970 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44176, MT Status=1, MTMSN=3095 2021-08-11T23:19:24.746Z,1628723964.746 [CommandLine](IMPORTANT): got command schedule asap "set Smear.SamplingYoYoMinDepth 140 m;set Smear.SamplingYoYoMaxDepth 200 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.YoYoMinAltitude 4 m;set Smear.MinAltitude 3 m" 2mdtz 2 3.000000 2021-08-11T23:19:24.746Z,1628723964.746 [CommandLine](IMPORTANT): Scheduling command #2 of 3 with id=2mdtz 2021-08-11T23:19:24.748Z,1628723964.748 [CommandLine](IMPORTANT): Scheduled #25 (#2 of 3 with id='2mdtz'): "set Smear.SamplingYoYoMinDepth 140 m;set Smear.SamplingYoYoMaxDepth 200 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.YoYoMinAltitude 4 m;set Smear.MinAltitude 3 m" ASAP 2021-08-11T23:20:19.034Z,1628724019.034 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44177, MT Status=1, MTMSN=3096 2021-08-11T23:20:19.667Z,1628724019.667 [CommandLine](IMPORTANT): got command schedule asap "set Smear.NeedCommsTimeVeryLong 8 h;run" 2mdtz 3 3.000000 2021-08-11T23:20:19.667Z,1628724019.667 [CommandLine](IMPORTANT): Scheduling command #3 of 3 with id=2mdtz 2021-08-11T23:20:19.668Z,1628724019.668 [CommandLine](IMPORTANT): Scheduled #26 (#3 of 3 with id='2mdtz'): "set Smear.NeedCommsTimeVeryLong 8 h;run" ASAP 2021-08-11T23:20:19.777Z,1628724019.777 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-11T23:20:21.981Z,1628724021.981 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-11T23:20:36.662Z,1628724036.662 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-11T23:20:36.663Z,1628724036.663 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-11T23:20:36.664Z,1628724036.664 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-11T23:20:36.664Z,1628724036.664 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-11T23:20:36.665Z,1628724036.665 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-11T23:20:36.666Z,1628724036.666 [CommandLine](IMPORTANT): got command set Smear.MaxDepth 210.000000 meter 2021-08-11T23:20:55.090Z,1628724055.090 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 140.000000 meter 2021-08-11T23:20:55.091Z,1628724055.091 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 200.000000 meter 2021-08-11T23:20:55.091Z,1628724055.091 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.631634 degree 2021-08-11T23:20:55.092Z,1628724055.092 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.196220 degree 2021-08-11T23:20:55.093Z,1628724055.093 [CommandLine](IMPORTANT): got command set Smear.YoYoMinAltitude 4.000000 meter 2021-08-11T23:20:55.094Z,1628724055.094 [CommandLine](IMPORTANT): got command set Smear.MinAltitude 3.000000 meter 2021-08-11T23:20:59.237Z,1628724059.237 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-11T23:20:59.238Z,1628724059.238 [CommandLine](IMPORTANT): got command run 2021-08-11T23:20:59.240Z,1628724059.240 [CommandLine](IMPORTANT): Running 2021-08-11T23:21:00.334Z,1628724060.334 [MissionManager](IMPORTANT): Started mission Smear 2021-08-11T23:29:23.155Z,1628724563.155 [CBIT](CRITICAL): STOP DEPTH REACHED. Terminating Mission. 2021-08-11T23:29:23.746Z,1628724563.746 [CommandLine](FAULT): Scheduling is paused 2021-08-11T23:29:23.904Z,1628724563.904 [MissionManager](IMPORTANT): Started mission Default 2021-08-11T23:31:09.005Z,1628724669.005 [CBIT](IMPORTANT): Beginning ground fault scan 2021-08-11T23:31:19.925Z,1628724679.925 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.004190 CHAN A1 (24V): 0.002122 CHAN A2 (12V): -0.006268 CHAN A3 (5V): -0.005933 CHAN B0 (3.3V): -0.017529 CHAN B1 (3.15aV): -0.001219 CHAN B2 (3.15bV): -0.001347 CHAN B3 (GND): -0.002434 OPEN: 0.012180 Full Scale Calc: 4.765 mA, -1.589 mA 2021-08-11T23:33:28.308Z,1628724808.308 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44186, MT Status=1, MTMSN=3097 2021-08-11T23:33:28.858Z,1628724808.858 [CommandLine](IMPORTANT): got command stop 2021-08-11T23:33:28.858Z,1628724808.858 [CommandLine](IMPORTANT): Scheduling is paused 2021-08-11T23:40:35.612Z,1628725235.612 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.188541 min 2021-08-11T23:46:31.239Z,1628725591.239 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 17.115385 min 2021-08-11T23:47:43.161Z,1628725663.161 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44198, MT Status=1, MTMSN=3098 2021-08-11T23:47:44.083Z,1628725664.083 [CommandLine](IMPORTANT): got command configSet CBIT.abortDepth 210.000000 meter persist 2021-08-11T23:47:44.086Z,1628725664.086 [CommandLine](IMPORTANT): got command configSet CBIT.stopDepth 200.000000 meter persist 2021-08-11T23:55:07.942Z,1628726107.942 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.727246 min 2021-08-11T23:56:02.558Z,1628726162.558 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44204, MT Status=1, MTMSN=3099 2021-08-11T23:56:03.394Z,1628726163.394 [CommandLine](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 195 m" 2mfl7 1 3.000000 2021-08-11T23:56:03.394Z,1628726163.394 [CommandLine](IMPORTANT): Scheduling command #1 of 3 with id=2mfl7 2021-08-11T23:56:03.395Z,1628726163.395 [CommandLine](IMPORTANT): Scheduled #27 (#1 of 3 with id='2mfl7'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 195 m" 2021-08-11T23:56:03.395Z,1628726163.395 [CommandLine](FAULT): Scheduling was paused by a command at 2021-08-11T23:33:28Z 2021-08-11T23:56:52.172Z,1628726212.172 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44205, MT Status=1, MTMSN=3100 2021-08-11T23:56:53.145Z,1628726213.145 [CommandLine](IMPORTANT): got command schedule "set Smear.SamplingYoYoMinDepth 140 m;set Smear.SamplingYoYoMaxDepth 190 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.YoYoMinAltitude 4 m;set Smear.MinAltitude 3 m" 2mfl7 2 3.000000 2021-08-11T23:56:53.147Z,1628726213.147 [CommandLine](IMPORTANT): Scheduling command #2 of 3 with id=2mfl7 2021-08-11T23:56:53.165Z,1628726213.165 [CommandLine](IMPORTANT): Scheduled #28 (#2 of 3 with id='2mfl7'): "set Smear.SamplingYoYoMinDepth 140 m;set Smear.SamplingYoYoMaxDepth 190 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.YoYoMinAltitude 4 m;set Smear.MinAltitude 3 m" 2021-08-11T23:56:53.166Z,1628726213.166 [CommandLine](FAULT): Scheduling was paused by a command at 2021-08-11T23:33:28Z 2021-08-11T23:57:02.469Z,1628726222.469 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44206, MT Status=1, MTMSN=3101 2021-08-11T23:57:03.290Z,1628726223.290 [CommandLine](IMPORTANT): got command schedule "set Smear.NeedCommsTimeVeryLong 8 h;run" 2mfl7 3 3.000000 2021-08-11T23:57:03.290Z,1628726223.290 [CommandLine](IMPORTANT): Scheduling command #3 of 3 with id=2mfl7 2021-08-11T23:57:03.291Z,1628726223.291 [CommandLine](IMPORTANT): Scheduled #29 (#3 of 3 with id='2mfl7'): "set Smear.NeedCommsTimeVeryLong 8 h;run" 2021-08-11T23:57:03.292Z,1628726223.292 [CommandLine](FAULT): Scheduling was paused by a command at 2021-08-11T23:33:28Z 2021-08-12T00:04:07.408Z,1628726647.408 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 34.718408 min 2021-08-12T00:04:38.714Z,1628726678.714 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44212, MT Status=1, MTMSN=3102 2021-08-12T00:04:39.630Z,1628726679.630 [CommandLine](IMPORTANT): got command schedule resume 2021-08-12T00:04:39.630Z,1628726679.630 [CommandLine](IMPORTANT): Scheduling is resumed 2021-08-12T00:04:39.829Z,1628726679.829 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T00:04:41.979Z,1628726681.979 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T00:04:55.750Z,1628726695.750 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-12T00:04:55.751Z,1628726695.751 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-12T00:04:55.752Z,1628726695.752 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-12T00:04:55.753Z,1628726695.753 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-12T00:04:55.753Z,1628726695.753 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-12T00:04:55.754Z,1628726695.754 [CommandLine](IMPORTANT): got command set Smear.MaxDepth 195.000000 meter 2021-08-12T00:05:11.971Z,1628726711.971 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 140.000000 meter 2021-08-12T00:05:11.972Z,1628726711.972 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 190.000000 meter 2021-08-12T00:05:11.973Z,1628726711.973 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.631634 degree 2021-08-12T00:05:11.974Z,1628726711.974 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.196220 degree 2021-08-12T00:05:11.975Z,1628726711.975 [CommandLine](IMPORTANT): got command set Smear.YoYoMinAltitude 4.000000 meter 2021-08-12T00:05:11.975Z,1628726711.975 [CommandLine](IMPORTANT): got command set Smear.MinAltitude 3.000000 meter 2021-08-12T00:05:15.968Z,1628726715.968 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-12T00:05:15.969Z,1628726715.969 [CommandLine](IMPORTANT): got command run 2021-08-12T00:05:15.971Z,1628726715.971 [CommandLine](IMPORTANT): Running 2021-08-12T00:05:16.187Z,1628726716.187 [MissionManager](IMPORTANT): Started mission Smear 2021-08-12T00:16:34.254Z,1628727394.254 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-12T00:16:35.055Z,1628727395.055 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-12T00:31:37.649Z,1628728297.649 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 139.965805 m , 4.115686 degC , 0.083200 ug/l . 2021-08-12T00:31:49.401Z,1628728309.401 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T00:31:49.514Z,1628728309.514 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T00:32:36.559Z,1628728356.559 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T00:32:37.372Z,1628728357.372 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -1 2021-08-12T00:32:37.373Z,1628728357.373 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T00:32:37.806Z,1628728357.806 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-12T00:32:38.175Z,1628728358.175 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T00:32:38.579Z,1628728358.579 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T00:32:40.599Z,1628728360.599 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T00:33:14.947Z,1628728394.947 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@53,aRNAl_bac) 2021-08-12T00:33:16.564Z,1628728396.564 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T00:33:16.971Z,1628728396.971 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-12T00:33:17.372Z,1628728397.372 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T00:35:39.585Z,1628728539.585 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T00:35:39.988Z,1628728539.988 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T01:27:44.682Z,1628731664.682 [VerticalControl](CRITICAL): Failure to ascend, stopping mission after 300.133026 seconds, depthRate=-0.001018 m/s, pitch=-0.726817 deg. 2021-08-12T01:27:44.719Z,1628731664.719 [CommandLine](FAULT): Scheduling is paused 2021-08-12T01:27:45.428Z,1628731665.428 [ESPComponent](IMPORTANT): [sample #4] Stop sampling requested. 2021-08-12T01:27:45.495Z,1628731665.495 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T01:27:45.830Z,1628731665.830 [ESPComponent](IMPORTANT): Stop sampling requested. Issuing Cmd.pauseFiltering 2021-08-12T01:27:51.487Z,1628731671.487 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@1161.4) 2021-08-12T01:27:51.892Z,1628731671.892 [ESPComponent](IMPORTANT): In S_PAUSED received result: 1161.4 2021-08-12T01:27:52.295Z,1628731672.295 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@1161.4) 2021-08-12T01:27:54.316Z,1628731674.316 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T01:27:54.719Z,1628731674.719 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@53,aRNAl_bac) 2021-08-12T01:29:29.659Z,1628731769.659 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@53,aRNAl_bac) 2021-08-12T01:29:36.933Z,1628731776.933 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T01:29:37.335Z,1628731777.335 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2021-08-12T01:32:44.823Z,1628731964.823 [VerticalControl](CRITICAL): Failure to ascend, dropping drop weight after 300.140594 seconds, depthRate=0.002046 m/s, pitch=-0.923591 deg. 2021-08-12T01:34:37.102Z,1628732077.102 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2021-08-12T01:34:37.507Z,1628732077.507 [ESPComponent](IMPORTANT): [sample #4] ESP log summary report (2 messages): @20:32:51.52 Selecting Cartridge 53 @21:27:49.79 Cmd::Paused in FILTERING -- during Sample Pump (SP) move after sampling 1161.4ml 2021-08-12T01:44:25.766Z,1628732665.766 [Default:B.GoToSurface](CRITICAL): Surfacing timeout. Returning calc satisfied. 2021-08-12T01:49:29.584Z,1628732969.584 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2021-08-12T02:01:13.368Z,1628733673.368 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44232, MT Status=1, MTMSN=3103 2021-08-12T02:01:13.953Z,1628733673.953 [CommandLine](IMPORTANT): got command burn off 2021-08-12T02:01:13.953Z,1628733673.953 [CommandLine](IMPORTANT): Deactivating dropweight wire 2021-08-12T02:07:06.088Z,1628734026.088 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 39.336861 min 2021-08-12T02:07:07.701Z,1628734027.701 [NAL9602](FAULT): GPS failed to acquire within timeout. 2021-08-12T02:08:27.702Z,1628734107.702 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44237, MT Status=1, MTMSN=3104 2021-08-12T02:08:28.222Z,1628734108.222 [CommandLine](IMPORTANT): got command failComponent 2021-08-12T02:08:28.222Z,1628734108.222 [CommandLine](IMPORTANT): Failed components: 2021-08-12T02:08:28.222Z,1628734108.222 [CommandLine](IMPORTANT): No failed Components. 2021-08-12T02:15:17.563Z,1628734517.563 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44242, MT Status=1, MTMSN=3105 2021-08-12T02:15:18.141Z,1628734518.141 [CommandLine](IMPORTANT): got command ibit 2021-08-12T02:15:18.379Z,1628734518.379 [IBIT](IMPORTANT): Beginning Initiated BIT 2021-08-12T02:15:18.379Z,1628734518.379 [IBIT](IMPORTANT): Beginning control surface checks. 2021-08-12T02:15:18.383Z,1628734518.383 [CBIT](IMPORTANT): Beginning ground fault scan 2021-08-12T02:15:29.153Z,1628734529.153 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.006448 CHAN A1 (24V): 0.001098 CHAN A2 (12V): -0.005146 CHAN A3 (5V): -0.004415 CHAN B0 (3.3V): 0.004063 CHAN B1 (3.15aV): 0.000340 CHAN B2 (3.15bV): -0.001375 CHAN B3 (GND): -0.001508 OPEN: 0.012206 Full Scale Calc: 4.765 mA, -1.589 mA 2021-08-12T02:17:09.808Z,1628734629.808 [IBIT](FAULT): Error acquiring IBIT communications status. Timeout expired. 2021-08-12T02:17:10.217Z,1628734630.217 [IBIT](IMPORTANT): Battery Status: Battery Charge (AH): 0.000000 Voltage: 16.056000 2021-08-12T02:17:10.218Z,1628734630.218 [IBIT](IMPORTANT): batteryCapacityThreshold: 50.000000 Ah 2021-08-12T02:17:10.218Z,1628734630.218 [IBIT](IMPORTANT): batteryVoltageThreshold: 13.700000 V 2021-08-12T02:17:10.622Z,1628734630.622 [IBIT](IMPORTANT): bitHumidityThreshold: 55.000000 % 2021-08-12T02:17:10.623Z,1628734630.623 [IBIT](IMPORTANT): bitPressureThreshold: 0.750000 psi 2021-08-12T02:17:10.623Z,1628734630.623 [IBIT](IMPORTANT): Pressure:7.730259 PSI 2021-08-12T02:17:10.623Z,1628734630.623 [IBIT](IMPORTANT): Humidity:15.291172 % 2021-08-12T02:17:11.032Z,1628734631.032 [IBIT](IMPORTANT): Vehicle Pitch:-3.448088 degrees 2021-08-12T02:17:11.032Z,1628734631.032 [IBIT](IMPORTANT): Vehicle Roll:2.373942 degrees 2021-08-12T02:17:11.033Z,1628734631.033 [IBIT](IMPORTANT): Vehicle Heading:185.539658 degrees 2021-08-12T02:17:11.462Z,1628734631.462 [IBIT](IMPORTANT): surfaceThreshold: 0.500000 m 2021-08-12T02:17:11.462Z,1628734631.462 [IBIT](IMPORTANT): buoyancyNeutral: 292.263672 cc 2021-08-12T02:17:11.463Z,1628734631.463 [IBIT](IMPORTANT): massDefault: 0.588687 cm 2021-08-12T02:17:11.463Z,1628734631.463 [IBIT](IMPORTANT): stopDepth: 200.000000 m 2021-08-12T02:17:11.463Z,1628734631.463 [IBIT](IMPORTANT): abortDepth: 210.000000 m 2021-08-12T02:17:11.463Z,1628734631.463 [IBIT](IMPORTANT): IBIT FAILED 2021-08-12T02:22:13.247Z,1628734933.247 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 54.455436 min 2021-08-12T02:22:14.820Z,1628734934.820 [NAL9602](FAULT): GPS failed to acquire within timeout. 2021-08-12T02:23:25.110Z,1628735005.110 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44250, MT Status=1, MTMSN=3106 2021-08-12T02:23:25.701Z,1628735005.701 [CommandLine](IMPORTANT): got command schedule clear 2021-08-12T02:23:25.701Z,1628735005.701 [CommandLine](IMPORTANT): #Cleared scheduled commands:0 2021-08-12T02:23:25.702Z,1628735005.702 [CommandLine](IMPORTANT): got command schedule resume 2021-08-12T02:23:25.702Z,1628735005.702 [CommandLine](IMPORTANT): Scheduling is resumed 2021-08-12T02:34:04.663Z,1628735644.663 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 66.312866 min 2021-08-12T02:39:51.809Z,1628735991.809 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 72.098421 min 2021-08-12T02:41:06.930Z,1628736066.930 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44262, MT Status=1, MTMSN=3107 2021-08-12T02:41:07.682Z,1628736067.682 [CommandLine](IMPORTANT): got command schedule asap "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2mnb2 1 2.000000 2021-08-12T02:41:07.683Z,1628736067.683 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2mnb2 2021-08-12T02:41:07.684Z,1628736067.684 [CommandLine](IMPORTANT): Scheduled #30 (#1 of 2 with id='2mnb2'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" ASAP 2021-08-12T02:41:39.575Z,1628736099.575 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44263, MT Status=1, MTMSN=3108 2021-08-12T02:41:40.362Z,1628736100.362 [CommandLine](IMPORTANT): got command schedule asap "set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 25 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" 2mnb2 2 2.000000 2021-08-12T02:41:40.362Z,1628736100.362 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2mnb2 2021-08-12T02:41:40.364Z,1628736100.364 [CommandLine](IMPORTANT): Scheduled #31 (#2 of 2 with id='2mnb2'): "set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 25 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" ASAP 2021-08-12T02:41:40.544Z,1628736100.544 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T02:41:42.729Z,1628736102.729 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T02:41:55.495Z,1628736115.495 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-12T02:41:55.496Z,1628736115.496 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-12T02:41:55.497Z,1628736115.497 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-12T02:41:55.497Z,1628736115.497 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-12T02:41:55.498Z,1628736115.498 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-12T02:42:09.893Z,1628736129.893 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 15.000000 meter 2021-08-12T02:42:09.894Z,1628736129.894 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 25.000000 meter 2021-08-12T02:42:09.895Z,1628736129.895 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.631634 degree 2021-08-12T02:42:09.895Z,1628736129.895 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.196220 degree 2021-08-12T02:42:09.896Z,1628736129.896 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-12T02:42:09.896Z,1628736129.896 [CommandLine](IMPORTANT): got command run 2021-08-12T02:42:09.899Z,1628736129.899 [CommandLine](IMPORTANT): Running 2021-08-12T02:42:10.222Z,1628736130.222 [MissionManager](IMPORTANT): Started mission Smear 2021-08-12T02:45:11.114Z,1628736311.114 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-12T02:45:11.931Z,1628736311.931 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-12T03:00:14.473Z,1628737214.473 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 17.193375 m , 12.728387 degC , 0.468000 ug/l . 2021-08-12T03:00:26.250Z,1628737226.250 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T03:00:26.390Z,1628737226.390 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T03:01:13.412Z,1628737273.412 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T03:01:14.220Z,1628737274.220 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -1 2021-08-12T03:01:14.221Z,1628737274.221 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T03:01:14.710Z,1628737274.710 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-12T03:01:15.024Z,1628737275.024 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T03:01:15.429Z,1628737275.429 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T03:01:17.446Z,1628737277.446 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T03:01:51.787Z,1628737311.787 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@52,aRNAl_bac) 2021-08-12T03:01:53.403Z,1628737313.403 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T03:01:53.812Z,1628737313.812 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-12T03:01:54.213Z,1628737314.213 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T03:04:16.037Z,1628737456.037 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T03:04:16.423Z,1628737456.423 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T03:28:14.687Z,1628738894.687 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@453.7) 2021-08-12T03:28:15.092Z,1628738895.092 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@453.7) 2021-08-12T03:28:17.112Z,1628738897.112 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T03:28:17.515Z,1628738897.515 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@52,aRNAl_bac) 2021-08-12T03:29:52.460Z,1628738992.460 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@52,aRNAl_bac) 2021-08-12T03:29:59.732Z,1628738999.732 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T03:30:00.137Z,1628739000.137 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2021-08-12T03:34:59.911Z,1628739299.911 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2021-08-12T03:35:00.315Z,1628739300.315 [ESPComponent](IMPORTANT): [sample #5] ESP log summary report (3 messages): @23:01:27.39 Selecting Cartridge 52 @23:28:11.77 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @23:28:11.97 Sampled 453.7ml 2021-08-12T03:35:01.155Z,1628739301.155 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-12T03:37:10.436Z,1628739430.436 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44272, MT Status=1, MTMSN=3109 2021-08-12T03:37:11.160Z,1628739431.160 [CommandLine](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2moh5 1 2.000000 2021-08-12T03:37:11.161Z,1628739431.161 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2moh5 2021-08-12T03:37:11.162Z,1628739431.162 [CommandLine](IMPORTANT): Scheduled #32 (#1 of 2 with id='2moh5'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2021-08-12T03:38:12.975Z,1628739492.975 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44273, MT Status=1, MTMSN=3110 2021-08-12T03:38:13.744Z,1628739493.744 [CommandLine](IMPORTANT): got command schedule "set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" 2moh5 2 2.000000 2021-08-12T03:38:13.744Z,1628739493.744 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2moh5 2021-08-12T03:38:13.745Z,1628739493.745 [CommandLine](IMPORTANT): Scheduled #33 (#2 of 2 with id='2moh5'): "set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.NeedCommsTimeVeryLong 8 h;run" 2021-08-12T03:38:38.076Z,1628739518.076 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44274, MT Status=1, MTMSN=3111 2021-08-12T03:38:38.859Z,1628739518.859 [CommandLine](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 80 m" 2mokw 1 3.000000 2021-08-12T03:38:38.859Z,1628739518.859 [CommandLine](IMPORTANT): Scheduling command #1 of 3 with id=2mokw 2021-08-12T03:38:38.860Z,1628739518.860 [CommandLine](IMPORTANT): Scheduled #34 (#1 of 3 with id='2mokw'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 80 m" 2021-08-12T03:38:53.753Z,1628739533.753 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44275, MT Status=1, MTMSN=3112 2021-08-12T03:38:54.535Z,1628739534.535 [CommandLine](IMPORTANT): got command schedule "set Smear.SamplingYoYoMinDepth 30 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2mokw 2 3.000000 2021-08-12T03:38:54.536Z,1628739534.536 [CommandLine](IMPORTANT): Scheduling command #2 of 3 with id=2mokw 2021-08-12T03:38:54.537Z,1628739534.537 [CommandLine](IMPORTANT): Scheduled #35 (#2 of 3 with id='2mokw'): "set Smear.SamplingYoYoMinDepth 30 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 42.631634 degree;set Smear.SamplingEndLongitude -87.196221 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2021-08-12T03:39:11.860Z,1628739551.860 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44276, MT Status=1, MTMSN=3113 2021-08-12T03:39:12.451Z,1628739552.451 [CommandLine](IMPORTANT): got command schedule "set Smear.NeedCommsTimeVeryLong 8 h;run" 2mokw 3 3.000000 2021-08-12T03:39:12.452Z,1628739552.452 [CommandLine](IMPORTANT): Scheduling command #3 of 3 with id=2mokw 2021-08-12T03:39:12.453Z,1628739552.453 [CommandLine](IMPORTANT): Scheduled #36 (#3 of 3 with id='2mokw'): "set Smear.NeedCommsTimeVeryLong 8 h;run" 2021-08-12T03:41:46.459Z,1628739706.459 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T03:41:46.926Z,1628739706.926 [WetLabsBB2FL](FAULT): LCB fault: Software Overcurrent. 2021-08-12T03:41:49.134Z,1628739709.134 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T03:41:49.375Z,1628739709.375 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T03:42:01.309Z,1628739721.309 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-12T03:42:01.310Z,1628739721.310 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-12T03:42:01.310Z,1628739721.310 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-12T03:42:01.311Z,1628739721.311 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-12T03:42:01.312Z,1628739721.312 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-12T03:42:16.735Z,1628739736.735 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 5.000000 meter 2021-08-12T03:42:16.736Z,1628739736.736 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 15.000000 meter 2021-08-12T03:42:16.737Z,1628739736.737 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.631634 degree 2021-08-12T03:42:16.738Z,1628739736.738 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.196220 degree 2021-08-12T03:42:16.739Z,1628739736.739 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-12T03:42:16.739Z,1628739736.739 [CommandLine](IMPORTANT): got command run 2021-08-12T03:42:16.741Z,1628739736.741 [CommandLine](IMPORTANT): Running 2021-08-12T03:42:18.377Z,1628739738.377 [MissionManager](IMPORTANT): Started mission Smear 2021-08-12T03:47:05.455Z,1628740025.455 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-12T03:47:06.285Z,1628740026.285 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-12T04:02:08.804Z,1628740928.804 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 4.423229 m , 22.287408 degC , 0.395200 ug/l . 2021-08-12T04:02:20.590Z,1628740940.590 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T04:02:20.726Z,1628740940.726 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T04:03:07.343Z,1628740987.343 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T04:03:08.154Z,1628740988.154 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -1 2021-08-12T04:03:08.154Z,1628740988.154 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T04:03:08.598Z,1628740988.598 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-12T04:03:08.959Z,1628740988.959 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T04:03:09.362Z,1628740989.362 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T04:03:11.383Z,1628740991.383 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T04:03:45.723Z,1628741025.723 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@51,aRNAl_bac) 2021-08-12T04:03:47.340Z,1628741027.340 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T04:03:47.744Z,1628741027.744 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-12T04:03:48.147Z,1628741028.147 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T04:06:01.871Z,1628741161.871 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T04:06:02.275Z,1628741162.275 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T05:13:24.775Z,1628745204.775 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1205.8) 2021-08-12T05:13:25.180Z,1628745205.180 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1205.8) 2021-08-12T05:13:27.200Z,1628745207.200 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T05:13:27.604Z,1628745207.604 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@51,aRNAl_bac) 2021-08-12T05:15:02.947Z,1628745302.947 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@51,aRNAl_bac) 2021-08-12T05:15:10.224Z,1628745310.224 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T05:15:10.630Z,1628745310.630 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2021-08-12T05:20:10.415Z,1628745610.415 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2021-08-12T05:20:10.822Z,1628745610.822 [ESPComponent](IMPORTANT): [sample #6] ESP log summary report (3 messages): @00:03:22.16 Selecting Cartridge 51 @01:13:22.52 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @01:13:22.72 Sampled 1205.8ml 2021-08-12T05:20:11.658Z,1628745611.658 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-12T05:21:28.415Z,1628745688.415 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44297, MT Status=1, MTMSN=3114 2021-08-12T05:21:29.270Z,1628745689.270 [CommandLine](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 195 m" 2msub 1 3.000000 2021-08-12T05:21:29.270Z,1628745689.270 [CommandLine](IMPORTANT): Scheduling command #1 of 3 with id=2msub 2021-08-12T05:21:29.271Z,1628745689.271 [CommandLine](IMPORTANT): Scheduled #37 (#1 of 3 with id='2msub'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 195 m" 2021-08-12T05:21:51.613Z,1628745711.613 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44298, MT Status=1, MTMSN=3115 2021-08-12T05:21:52.409Z,1628745712.409 [CommandLine](IMPORTANT): got command schedule "set Smear.SamplingYoYoMinDepth 130 m;set Smear.SamplingYoYoMaxDepth 190 m;set Smear.SamplingEndLatitude 42.6 degree;set Smear.SamplingEndLongitude -87.243 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2msub 2 3.000000 2021-08-12T05:21:52.410Z,1628745712.410 [CommandLine](IMPORTANT): Scheduling command #2 of 3 with id=2msub 2021-08-12T05:21:52.410Z,1628745712.410 [CommandLine](IMPORTANT): Scheduled #38 (#2 of 3 with id='2msub'): "set Smear.SamplingYoYoMinDepth 130 m;set Smear.SamplingYoYoMaxDepth 190 m;set Smear.SamplingEndLatitude 42.6 degree;set Smear.SamplingEndLongitude -87.243 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2021-08-12T05:22:11.027Z,1628745731.027 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44299, MT Status=1, MTMSN=3116 2021-08-12T05:22:11.638Z,1628745731.638 [CommandLine](IMPORTANT): got command schedule "set Smear.NeedCommsTimeVeryLong 8 h;run" 2msub 3 3.000000 2021-08-12T05:22:11.638Z,1628745731.638 [CommandLine](IMPORTANT): Scheduling command #3 of 3 with id=2msub 2021-08-12T05:22:11.639Z,1628745731.639 [CommandLine](IMPORTANT): Scheduled #39 (#3 of 3 with id='2msub'): "set Smear.NeedCommsTimeVeryLong 8 h;run" 2021-08-12T05:28:08.877Z,1628746088.877 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T05:28:09.429Z,1628746089.429 [WetLabsBB2FL](FAULT): LCB fault: Software Overcurrent. 2021-08-12T05:28:11.450Z,1628746091.450 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T05:28:11.456Z,1628746091.456 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T05:28:24.906Z,1628746104.906 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-12T05:28:24.906Z,1628746104.906 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-12T05:28:24.907Z,1628746104.907 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-12T05:28:24.908Z,1628746104.908 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-12T05:28:24.932Z,1628746104.932 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-12T05:28:24.933Z,1628746104.933 [CommandLine](IMPORTANT): got command set Smear.MaxDepth 80.000000 meter 2021-08-12T05:28:44.052Z,1628746124.052 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 30.000000 meter 2021-08-12T05:28:44.052Z,1628746124.052 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 70.000000 meter 2021-08-12T05:28:44.057Z,1628746124.057 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.631634 degree 2021-08-12T05:28:44.058Z,1628746124.058 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.196220 degree 2021-08-12T05:28:44.059Z,1628746124.059 [CommandLine](IMPORTANT): got command set Smear.YoYoUpPitch 12.000000 degree 2021-08-12T05:28:44.060Z,1628746124.060 [CommandLine](IMPORTANT): got command set Smear.YoYoDownPitch -12.000000 degree 2021-08-12T05:28:49.900Z,1628746129.900 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-12T05:28:49.901Z,1628746129.901 [CommandLine](IMPORTANT): got command run 2021-08-12T05:28:49.903Z,1628746129.903 [CommandLine](IMPORTANT): Running 2021-08-12T05:28:51.248Z,1628746131.248 [MissionManager](IMPORTANT): Started mission Smear 2021-08-12T05:33:43.626Z,1628746423.626 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-12T05:33:44.434Z,1628746424.434 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-12T05:48:47.023Z,1628747327.023 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 38.430500 m , 5.331506 degC , 0.863200 ug/l . 2021-08-12T05:48:58.767Z,1628747338.767 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T05:48:58.910Z,1628747338.910 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T05:49:46.388Z,1628747386.388 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T05:49:47.199Z,1628747387.199 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -1 2021-08-12T05:49:47.200Z,1628747387.200 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T05:49:47.645Z,1628747387.645 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-12T05:49:48.003Z,1628747388.003 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T05:49:48.408Z,1628747388.408 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T05:49:50.427Z,1628747390.427 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T05:50:24.775Z,1628747424.775 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@50,aRNAl_bac) 2021-08-12T05:50:26.391Z,1628747426.391 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T05:50:26.796Z,1628747426.796 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-12T05:50:27.200Z,1628747427.200 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T05:52:49.004Z,1628747569.004 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T05:52:49.408Z,1628747569.408 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T06:54:10.120Z,1628751250.120 [ThrusterServo](FAULT): Invalid EZ Servo response:"/0@5242" 2021-08-12T06:54:39.415Z,1628751279.415 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1131.7) 2021-08-12T06:54:39.819Z,1628751279.819 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1131.7) 2021-08-12T06:54:41.840Z,1628751281.840 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T06:54:42.245Z,1628751282.245 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@50,aRNAl_bac) 2021-08-12T06:56:17.183Z,1628751377.183 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@50,aRNAl_bac) 2021-08-12T06:56:24.457Z,1628751384.457 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T06:56:24.859Z,1628751384.859 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2021-08-12T07:01:24.627Z,1628751684.627 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2021-08-12T07:01:25.032Z,1628751685.032 [ESPComponent](IMPORTANT): [sample #7] ESP log summary report (3 messages): @01:50:01.18 Selecting Cartridge 50 @02:54:36.95 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @02:54:37.14 Sampled 1131.7ml 2021-08-12T07:01:25.885Z,1628751685.885 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-12T07:04:22.050Z,1628751862.050 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44322, MT Status=1, MTMSN=3117 2021-08-12T07:04:22.803Z,1628751862.803 [CommandLine](IMPORTANT): got command schedule "load Science/sci2.xml;set sci2.MissionTimeout 6 h;set sci2.NeedCommsTime 45 min;set sci2.Lat1 42.631634 degree;set sci2.Lon1 -87.196221 degree;set sci2.YoYoUpPitch 12 degree;set sci2.YoYoDownPitch -12 degree;run" 2021-08-12T07:04:22.803Z,1628751862.803 [CommandLine](IMPORTANT): Scheduled #40: "load Science/sci2.xml;set sci2.MissionTimeout 6 h;set sci2.NeedCommsTime 45 min;set sci2.Lat1 42.631634 degree;set sci2.Lon1 -87.196221 degree;set sci2.YoYoUpPitch 12 degree;set sci2.YoYoDownPitch -12 degree;run" 2021-08-12T07:10:32.287Z,1628752232.287 [Smear:NeedComms:C:A_Timeout:A](IMPORTANT): first GPS update timeout 2021-08-12T07:11:24.168Z,1628752284.168 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44323, MT Status=1, MTMSN=3118 2021-08-12T07:11:24.958Z,1628752284.958 [CommandLine](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2mx3b 1 3.000000 2021-08-12T07:11:24.958Z,1628752284.958 [CommandLine](IMPORTANT): Scheduling command #1 of 3 with id=2mx3b 2021-08-12T07:11:24.959Z,1628752284.959 [CommandLine](IMPORTANT): Scheduled #41 (#1 of 3 with id='2mx3b'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2021-08-12T07:12:00.575Z,1628752320.575 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44324, MT Status=1, MTMSN=3119 2021-08-12T07:12:01.382Z,1628752321.382 [CommandLine](IMPORTANT): got command schedule "set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 42.752012 degree;set Smear.SamplingEndLongitude -87.563339 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2mx3b 2 3.000000 2021-08-12T07:12:01.383Z,1628752321.383 [CommandLine](IMPORTANT): Scheduling command #2 of 3 with id=2mx3b 2021-08-12T07:12:01.384Z,1628752321.384 [CommandLine](IMPORTANT): Scheduled #42 (#2 of 3 with id='2mx3b'): "set Smear.SamplingYoYoMinDepth 5 m;set Smear.SamplingYoYoMaxDepth 15 m;set Smear.SamplingEndLatitude 42.752012 degree;set Smear.SamplingEndLongitude -87.563339 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2021-08-12T07:12:16.361Z,1628752336.361 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44325, MT Status=1, MTMSN=3120 2021-08-12T07:12:16.926Z,1628752336.926 [CommandLine](IMPORTANT): got command schedule "set Smear.NeedCommsTimeVeryLong 8 h;run" 2mx3b 3 3.000000 2021-08-12T07:12:16.927Z,1628752336.927 [CommandLine](IMPORTANT): Scheduling command #3 of 3 with id=2mx3b 2021-08-12T07:12:16.928Z,1628752336.928 [CommandLine](IMPORTANT): Scheduled #43 (#3 of 3 with id='2mx3b'): "set Smear.NeedCommsTimeVeryLong 8 h;run" 2021-08-12T07:18:11.159Z,1628752691.159 [NAL9602](FAULT): GPS failed to acquire within timeout. 2021-08-12T07:25:09.343Z,1628753109.343 [Smear:NeedComms:E:A_Timeout:A](IMPORTANT): second GPS update timeout 2021-08-12T07:25:11.183Z,1628753111.183 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T07:25:11.578Z,1628753111.578 [WetLabsBB2FL](FAULT): LCB fault: Software Overcurrent. 2021-08-12T07:25:13.804Z,1628753113.804 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T07:25:13.879Z,1628753113.879 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T07:25:27.242Z,1628753127.242 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-12T07:25:27.243Z,1628753127.243 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-12T07:25:27.244Z,1628753127.244 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-12T07:25:27.245Z,1628753127.245 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-12T07:25:27.245Z,1628753127.245 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-12T07:25:27.246Z,1628753127.246 [CommandLine](IMPORTANT): got command set Smear.MaxDepth 195.000000 meter 2021-08-12T07:25:42.695Z,1628753142.695 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 130.000000 meter 2021-08-12T07:25:42.696Z,1628753142.696 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 190.000000 meter 2021-08-12T07:25:42.697Z,1628753142.697 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.599998 degree 2021-08-12T07:25:42.697Z,1628753142.697 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.242996 degree 2021-08-12T07:25:42.698Z,1628753142.698 [CommandLine](IMPORTANT): got command set Smear.YoYoUpPitch 12.000000 degree 2021-08-12T07:25:42.699Z,1628753142.699 [CommandLine](IMPORTANT): got command set Smear.YoYoDownPitch -12.000000 degree 2021-08-12T07:25:45.973Z,1628753145.973 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-12T07:25:45.974Z,1628753145.974 [CommandLine](IMPORTANT): got command run 2021-08-12T07:25:45.976Z,1628753145.976 [CommandLine](IMPORTANT): Running 2021-08-12T07:25:46.179Z,1628753146.179 [MissionManager](IMPORTANT): Started mission Smear 2021-08-12T07:28:54.818Z,1628753334.818 [NAL9602](FAULT): GPS failed to acquire within timeout. 2021-08-12T07:32:48.382Z,1628753568.382 [Smear:NeedComms:C:A_Timeout:A](IMPORTANT): first GPS update timeout 2021-08-12T07:50:35.163Z,1628754635.163 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-12T07:50:35.967Z,1628754635.967 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-12T08:05:38.509Z,1628755538.509 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 135.160614 m , 4.105798 degC , 0.104000 ug/l . 2021-08-12T08:05:50.294Z,1628755550.294 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T08:05:50.433Z,1628755550.433 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T08:06:37.458Z,1628755597.458 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T08:06:38.266Z,1628755598.266 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -1 2021-08-12T08:06:38.266Z,1628755598.266 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T08:06:38.718Z,1628755598.718 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-12T08:06:39.075Z,1628755599.075 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T08:06:39.478Z,1628755599.478 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T08:06:41.499Z,1628755601.499 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T08:07:15.839Z,1628755635.839 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@49,aRNAl_bac) 2021-08-12T08:07:17.457Z,1628755637.457 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T08:07:17.873Z,1628755637.873 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-12T08:07:18.263Z,1628755638.263 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T08:09:39.671Z,1628755779.671 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T08:09:40.078Z,1628755780.078 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T08:15:29.305Z,1628756129.305 [CBIT](IMPORTANT): Beginning ground fault scan 2021-08-12T08:15:40.554Z,1628756140.554 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.004174 CHAN A1 (24V): 0.001704 CHAN A2 (12V): -0.007338 CHAN A3 (5V): -0.008252 CHAN B0 (3.3V): -0.004394 CHAN B1 (3.15aV): -0.002246 CHAN B2 (3.15bV): -0.001096 CHAN B3 (GND): -0.002395 OPEN: 0.012362 Full Scale Calc: 4.765 mA, -1.589 mA 2021-08-12T09:56:00.646Z,1628762160.646 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@2000.0) 2021-08-12T09:56:01.047Z,1628762161.047 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@2000.0) 2021-08-12T09:56:03.069Z,1628762163.069 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T09:56:03.471Z,1628762163.471 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,aRNAl_bac) 2021-08-12T09:57:38.413Z,1628762258.413 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@49,aRNAl_bac) 2021-08-12T09:57:45.684Z,1628762265.684 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T09:57:46.087Z,1628762266.087 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2021-08-12T10:02:45.863Z,1628762565.863 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2021-08-12T10:02:46.267Z,1628762566.267 [ESPComponent](IMPORTANT): [sample #8] ESP log summary report (2 messages): @04:06:52.38 Selecting Cartridge 49 @05:55:59.23 Sampled 2000.0ml 2021-08-12T10:02:47.110Z,1628762567.110 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-12T10:15:59.791Z,1628763359.791 [VerticalControl](CRITICAL): Failure to ascend, stopping mission after 300.177917 seconds, depthRate=0.001949 m/s, pitch=13.751524 deg. 2021-08-12T10:15:59.895Z,1628763359.895 [CommandLine](FAULT): Scheduling is paused for 4 commands 2021-08-12T10:16:00.577Z,1628763360.577 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T10:32:09.176Z,1628764329.176 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 16.136874 min 2021-08-12T10:38:39.502Z,1628764719.502 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 22.641829 min 2021-08-12T10:44:27.829Z,1628765067.829 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 28.446759 min 2021-08-12T10:50:01.117Z,1628765401.117 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 34.002391 min 2021-08-12T10:56:08.226Z,1628765768.226 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 40.121000 min 2021-08-12T10:56:45.549Z,1628765805.549 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44382, MT Status=1, MTMSN=3121 2021-08-12T10:56:46.126Z,1628765806.126 [CommandLine](IMPORTANT): got command schedule resume 2021-08-12T10:56:46.126Z,1628765806.126 [CommandLine](IMPORTANT): Scheduling is resumed 2021-08-12T10:56:46.171Z,1628765806.171 [CommandLine](IMPORTANT): got command load ./Missions/Science/sci2.xml 2021-08-12T10:56:47.920Z,1628765807.920 [CommandLine](IMPORTANT): Loaded ./Missions/Science/sci2.xml 2021-08-12T10:56:59.278Z,1628765819.278 [CommandLine](IMPORTANT): got command set sci2.MissionTimeout 6.000000 hour 2021-08-12T10:56:59.279Z,1628765819.279 [CommandLine](IMPORTANT): got command set sci2.NeedCommsTime 45.000000 minute 2021-08-12T10:56:59.280Z,1628765819.280 [CommandLine](IMPORTANT): got command set sci2.Lat1 42.631634 degree 2021-08-12T10:56:59.280Z,1628765819.280 [CommandLine](IMPORTANT): got command set sci2.Lon1 -87.196220 degree 2021-08-12T10:56:59.281Z,1628765819.281 [CommandLine](IMPORTANT): got command set sci2.YoYoUpPitch 12.000000 degree 2021-08-12T10:56:59.282Z,1628765819.282 [CommandLine](IMPORTANT): got command set sci2.YoYoDownPitch -12.000000 degree 2021-08-12T10:56:59.282Z,1628765819.282 [CommandLine](IMPORTANT): got command run 2021-08-12T10:56:59.284Z,1628765819.284 [CommandLine](IMPORTANT): Running 2021-08-12T10:56:59.566Z,1628765819.566 [MissionManager](IMPORTANT): Started mission sci2 2021-08-12T10:57:05.223Z,1628765825.223 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44383, MT Status=1, MTMSN=3122 2021-08-12T10:57:05.794Z,1628765825.794 [CommandLine](IMPORTANT): got command schedule resume 2021-08-12T11:32:56.765Z,1628767976.765 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached Waypoint: 42.631633,-87.196224 2021-08-12T11:32:59.435Z,1628767979.435 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T11:33:01.717Z,1628767981.717 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T11:33:02.066Z,1628767982.066 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T11:33:15.280Z,1628767995.280 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-12T11:33:15.281Z,1628767995.281 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-12T11:33:15.282Z,1628767995.282 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-12T11:33:15.282Z,1628767995.282 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-12T11:33:15.283Z,1628767995.283 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-12T11:33:32.624Z,1628768012.624 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 5.000000 meter 2021-08-12T11:33:32.625Z,1628768012.625 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 15.000000 meter 2021-08-12T11:33:32.626Z,1628768012.626 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.752010 degree 2021-08-12T11:33:32.626Z,1628768012.626 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.563339 degree 2021-08-12T11:33:32.627Z,1628768012.627 [CommandLine](IMPORTANT): got command set Smear.YoYoUpPitch 12.000000 degree 2021-08-12T11:33:32.628Z,1628768012.628 [CommandLine](IMPORTANT): got command set Smear.YoYoDownPitch -12.000000 degree 2021-08-12T11:33:35.881Z,1628768015.881 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-12T11:33:35.881Z,1628768015.881 [CommandLine](IMPORTANT): got command run 2021-08-12T11:33:35.902Z,1628768015.902 [CommandLine](IMPORTANT): Running 2021-08-12T11:33:36.053Z,1628768016.053 [MissionManager](IMPORTANT): Started mission Smear 2021-08-12T11:41:09.058Z,1628768469.058 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-12T11:41:09.851Z,1628768469.851 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-12T11:56:12.416Z,1628769372.416 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 9.910359 m , 22.184900 degC , 0.291200 ug/l . 2021-08-12T11:56:24.181Z,1628769384.181 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T11:56:24.301Z,1628769384.301 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T11:57:11.746Z,1628769431.746 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T11:57:12.562Z,1628769432.562 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -1 2021-08-12T11:57:12.563Z,1628769432.563 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T11:57:12.999Z,1628769432.999 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-12T11:57:13.364Z,1628769433.364 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T11:57:13.767Z,1628769433.767 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T11:57:15.786Z,1628769435.786 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T11:57:50.132Z,1628769470.132 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@48,aRNAl_bac) 2021-08-12T11:57:51.748Z,1628769471.748 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T11:57:52.153Z,1628769472.153 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-12T11:57:52.560Z,1628769472.560 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T12:00:14.769Z,1628769614.769 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T12:00:15.171Z,1628769615.171 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T13:23:17.803Z,1628774597.803 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@1520.9) 2021-08-12T13:23:18.209Z,1628774598.209 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@1520.9) 2021-08-12T13:23:20.230Z,1628774600.230 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T13:23:20.638Z,1628774600.638 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@48,aRNAl_bac) 2021-08-12T13:24:55.977Z,1628774695.977 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@48,aRNAl_bac) 2021-08-12T13:25:03.250Z,1628774703.250 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T13:25:03.652Z,1628774703.652 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2021-08-12T13:30:03.421Z,1628775003.421 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2021-08-12T13:30:03.823Z,1628775003.823 [ESPComponent](IMPORTANT): [sample #9] ESP log summary report (3 messages): @07:57:25.96 Selecting Cartridge 48 @09:23:14.80 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @09:23:15.00 Sampled 1520.9ml 2021-08-12T13:30:04.779Z,1628775004.779 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-12T13:30:50.475Z,1628775050.475 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44401, MT Status=1, MTMSN=3123 2021-08-12T13:30:51.247Z,1628775051.247 [CommandLine](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2nd2u 1 3.000000 2021-08-12T13:30:51.248Z,1628775051.248 [CommandLine](IMPORTANT): Scheduling command #1 of 3 with id=2nd2u 2021-08-12T13:30:51.250Z,1628775051.250 [CommandLine](IMPORTANT): Scheduled #44 (#1 of 3 with id='2nd2u'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool" 2021-08-12T13:31:08.399Z,1628775068.399 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44402, MT Status=1, MTMSN=3124 2021-08-12T13:31:09.207Z,1628775069.207 [CommandLine](IMPORTANT): got command schedule "set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 25 m;set Smear.SamplingEndLatitude 42.752012 degree;set Smear.SamplingEndLongitude -87.563339 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2nd2u 2 3.000000 2021-08-12T13:31:09.208Z,1628775069.208 [CommandLine](IMPORTANT): Scheduling command #2 of 3 with id=2nd2u 2021-08-12T13:31:09.210Z,1628775069.210 [CommandLine](IMPORTANT): Scheduled #45 (#2 of 3 with id='2nd2u'): "set Smear.SamplingYoYoMinDepth 15 m;set Smear.SamplingYoYoMaxDepth 25 m;set Smear.SamplingEndLatitude 42.752012 degree;set Smear.SamplingEndLongitude -87.563339 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2021-08-12T13:31:31.929Z,1628775091.929 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44403, MT Status=1, MTMSN=3125 2021-08-12T13:31:32.558Z,1628775092.558 [CommandLine](IMPORTANT): got command schedule "set Smear.NeedCommsTimeVeryLong 8 h;run" 2nd2u 3 3.000000 2021-08-12T13:31:32.559Z,1628775092.559 [CommandLine](IMPORTANT): Scheduling command #3 of 3 with id=2nd2u 2021-08-12T13:31:32.560Z,1628775092.560 [CommandLine](IMPORTANT): Scheduled #46 (#3 of 3 with id='2nd2u'): "set Smear.NeedCommsTimeVeryLong 8 h;run" 2021-08-12T13:31:59.103Z,1628775119.103 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44404, MT Status=1, MTMSN=3126 2021-08-12T13:31:59.858Z,1628775119.858 [CommandLine](IMPORTANT): got command schedule "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 80 m" 2nh0o 1 3.000000 2021-08-12T13:31:59.858Z,1628775119.858 [CommandLine](IMPORTANT): Scheduling command #1 of 3 with id=2nh0o 2021-08-12T13:31:59.859Z,1628775119.859 [CommandLine](IMPORTANT): Scheduled #47 (#1 of 3 with id='2nh0o'): "load Science/smear_waypoint_sampling.xml;set Smear.CartridgeType -1 count;set Smear.MissionTimeout 8 h;set Smear.WaitDuration 15 min;set Smear.NeedCommsTimeInTransit 45 min;set Smear.ReachEndWaypoint 0 bool;set Smear.MaxDepth 80 m" 2021-08-12T13:33:00.468Z,1628775180.468 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44405, MT Status=1, MTMSN=3127 2021-08-12T13:33:01.291Z,1628775181.291 [CommandLine](IMPORTANT): got command schedule "set Smear.SamplingYoYoMinDepth 30 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 42.752012 degree;set Smear.SamplingEndLongitude -87.563339 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2nh0o 2 3.000000 2021-08-12T13:33:01.291Z,1628775181.291 [CommandLine](IMPORTANT): Scheduling command #2 of 3 with id=2nh0o 2021-08-12T13:33:01.292Z,1628775181.292 [CommandLine](IMPORTANT): Scheduled #48 (#2 of 3 with id='2nh0o'): "set Smear.SamplingYoYoMinDepth 30 m;set Smear.SamplingYoYoMaxDepth 70 m;set Smear.SamplingEndLatitude 42.752012 degree;set Smear.SamplingEndLongitude -87.563339 degree;set Smear.YoYoUpPitch 12 degree;set Smear.YoYoDownPitch -12 degree" 2021-08-12T13:33:24.193Z,1628775204.193 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44406, MT Status=1, MTMSN=3128 2021-08-12T13:33:24.777Z,1628775204.777 [CommandLine](IMPORTANT): got command schedule "set Smear.NeedCommsTimeVeryLong 8 h;run" 2nh0o 3 3.000000 2021-08-12T13:33:24.778Z,1628775204.778 [CommandLine](IMPORTANT): Scheduling command #3 of 3 with id=2nh0o 2021-08-12T13:33:24.779Z,1628775204.779 [CommandLine](IMPORTANT): Scheduled #49 (#3 of 3 with id='2nh0o'): "set Smear.NeedCommsTimeVeryLong 8 h;run" 2021-08-12T13:36:59.505Z,1628775419.505 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T13:37:02.014Z,1628775422.014 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T13:37:02.140Z,1628775422.140 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T13:37:13.834Z,1628775433.834 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-12T13:37:13.835Z,1628775433.835 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-12T13:37:13.836Z,1628775433.836 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-12T13:37:13.837Z,1628775433.837 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-12T13:37:13.861Z,1628775433.861 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-12T13:37:29.520Z,1628775449.520 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 15.000000 meter 2021-08-12T13:37:29.521Z,1628775449.521 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 25.000000 meter 2021-08-12T13:37:29.522Z,1628775449.522 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.752010 degree 2021-08-12T13:37:29.523Z,1628775449.523 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.563339 degree 2021-08-12T13:37:29.523Z,1628775449.523 [CommandLine](IMPORTANT): got command set Smear.YoYoUpPitch 12.000000 degree 2021-08-12T13:37:29.524Z,1628775449.524 [CommandLine](IMPORTANT): got command set Smear.YoYoDownPitch -12.000000 degree 2021-08-12T13:37:33.269Z,1628775453.269 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-12T13:37:33.270Z,1628775453.270 [CommandLine](IMPORTANT): got command run 2021-08-12T13:37:33.272Z,1628775453.272 [CommandLine](IMPORTANT): Running 2021-08-12T13:37:33.816Z,1628775453.816 [MissionManager](IMPORTANT): Started mission Smear 2021-08-12T13:40:53.830Z,1628775653.830 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-12T13:40:54.621Z,1628775654.621 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-12T13:55:57.211Z,1628776557.211 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 18.547474 m , 13.115106 degC , 0.364000 ug/l . 2021-08-12T13:56:08.983Z,1628776568.983 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T13:56:09.092Z,1628776569.092 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T13:56:56.547Z,1628776616.547 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T13:56:57.358Z,1628776617.358 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -1 2021-08-12T13:56:57.359Z,1628776617.359 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T13:56:57.810Z,1628776617.810 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-12T13:56:58.163Z,1628776618.163 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T13:56:58.567Z,1628776618.567 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T13:57:00.589Z,1628776620.589 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T13:57:34.980Z,1628776654.980 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@47,aRNAl_bac) 2021-08-12T13:57:36.595Z,1628776656.595 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T13:57:36.999Z,1628776656.999 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-12T13:57:37.403Z,1628776657.403 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T13:59:58.807Z,1628776798.807 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T13:59:59.214Z,1628776799.214 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T14:15:40.666Z,1628777740.666 [CBIT](IMPORTANT): Beginning ground fault scan 2021-08-12T14:15:51.641Z,1628777751.641 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.004851 CHAN A1 (24V): 0.001408 CHAN A2 (12V): 0.000980 CHAN A3 (5V): -0.005141 CHAN B0 (3.3V): 0.002091 CHAN B1 (3.15aV): -0.001093 CHAN B2 (3.15bV): -0.001031 CHAN B3 (GND): -0.002134 OPEN: 0.011683 Full Scale Calc: 4.765 mA, -1.589 mA 2021-08-12T14:45:14.540Z,1628779514.540 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@828.7) 2021-08-12T14:45:14.943Z,1628779514.943 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@828.7) 2021-08-12T14:45:16.964Z,1628779516.964 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T14:45:17.370Z,1628779517.370 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@47,aRNAl_bac) 2021-08-12T14:46:52.312Z,1628779612.312 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@47,aRNAl_bac) 2021-08-12T14:46:59.585Z,1628779619.585 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T14:46:59.987Z,1628779619.987 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2021-08-12T14:51:59.769Z,1628779919.769 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2021-08-12T14:52:00.167Z,1628779920.167 [ESPComponent](IMPORTANT): [sample #10] ESP log summary report (3 messages): @09:57:11.08 Selecting Cartridge 47 @10:45:11.65 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @10:45:11.84 Sampled 828.7ml 2021-08-12T14:52:01.018Z,1628779921.018 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-12T15:04:08.773Z,1628780648.773 [CommandLine](IMPORTANT): got command load ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T15:04:11.237Z,1628780651.237 [CommandLine](IMPORTANT): Loaded ./Missions/Science/smear_waypoint_sampling.xml 2021-08-12T15:04:11.396Z,1628780651.396 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T15:04:24.479Z,1628780664.479 [CommandLine](IMPORTANT): got command set Smear.CartridgeType -1.000000 count 2021-08-12T15:04:24.480Z,1628780664.480 [CommandLine](IMPORTANT): got command set Smear.MissionTimeout 8.000000 hour 2021-08-12T15:04:24.481Z,1628780664.481 [CommandLine](IMPORTANT): got command set Smear.WaitDuration 15.000000 minute 2021-08-12T15:04:24.482Z,1628780664.482 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeInTransit 45.000000 minute 2021-08-12T15:04:24.483Z,1628780664.483 [CommandLine](IMPORTANT): got command set Smear.ReachEndWaypoint 0.000000 bool 2021-08-12T15:04:24.483Z,1628780664.483 [CommandLine](IMPORTANT): got command set Smear.MaxDepth 80.000000 meter 2021-08-12T15:04:40.632Z,1628780680.632 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMinDepth 30.000000 meter 2021-08-12T15:04:40.633Z,1628780680.633 [CommandLine](IMPORTANT): got command set Smear.SamplingYoYoMaxDepth 70.000000 meter 2021-08-12T15:04:40.634Z,1628780680.634 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLatitude 42.752010 degree 2021-08-12T15:04:40.635Z,1628780680.635 [CommandLine](IMPORTANT): got command set Smear.SamplingEndLongitude -87.563339 degree 2021-08-12T15:04:40.636Z,1628780680.636 [CommandLine](IMPORTANT): got command set Smear.YoYoUpPitch 12.000000 degree 2021-08-12T15:04:40.636Z,1628780680.636 [CommandLine](IMPORTANT): got command set Smear.YoYoDownPitch -12.000000 degree 2021-08-12T15:04:43.692Z,1628780683.692 [CommandLine](IMPORTANT): got command set Smear.NeedCommsTimeVeryLong 8.000000 hour 2021-08-12T15:04:43.693Z,1628780683.693 [CommandLine](IMPORTANT): got command run 2021-08-12T15:04:43.695Z,1628780683.695 [CommandLine](IMPORTANT): Running 2021-08-12T15:04:43.754Z,1628780683.754 [MissionManager](IMPORTANT): Started mission Smear 2021-08-12T15:13:08.774Z,1628781188.774 [Smear:SamplingToWaypoint:TakeSamples:Firing:B](IMPORTANT): Taking sample No. 1.000000 count . 2021-08-12T15:13:09.554Z,1628781189.554 [Smear:InsertWait:A](IMPORTANT): Wait 15.000000 min before firing sample. 2021-08-12T15:28:12.124Z,1628782092.124 [Smear:TriggerSampling:TriggerESP:D](IMPORTANT): Trigger ESP sampling. DepthLogged, TempLogged, ChlLogged = 37.395538 m , 6.140710 degC , 0.416000 ug/l . 2021-08-12T15:28:23.897Z,1628782103.897 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T15:28:24.022Z,1628782104.022 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T15:29:11.463Z,1628782151.463 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T15:29:12.274Z,1628782152.274 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -1 2021-08-12T15:29:12.274Z,1628782152.274 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T15:29:12.709Z,1628782152.709 [Smear:TriggerSampling:TriggerESP:G](IMPORTANT): Wait for ESP sampling to complete. 2021-08-12T15:29:13.083Z,1628782153.083 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T15:29:13.483Z,1628782153.483 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T15:29:15.503Z,1628782155.503 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T15:29:49.844Z,1628782189.844 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@46,aRNAl_bac) 2021-08-12T15:29:51.459Z,1628782191.459 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T15:29:51.864Z,1628782191.864 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=176.399994 issuedCmd_=Cmd.startFiltering 2021-08-12T15:29:52.268Z,1628782192.268 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T15:32:13.269Z,1628782333.269 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T15:32:13.671Z,1628782333.671 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T16:15:40.838Z,1628784940.838 [RudderServo](FAULT): Reading outside of valid range:0.000000 2021-08-12T16:15:41.513Z,1628784941.513 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2021-08-12T16:33:23.292Z,1628786003.292 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@1076.0) 2021-08-12T16:33:23.694Z,1628786003.694 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@1076.0) 2021-08-12T16:33:25.713Z,1628786005.713 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T16:33:26.116Z,1628786006.116 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@46,aRNAl_bac) 2021-08-12T16:35:01.055Z,1628786101.055 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@46,aRNAl_bac) 2021-08-12T16:35:08.332Z,1628786108.332 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T16:35:08.731Z,1628786108.731 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2021-08-12T16:40:08.511Z,1628786408.511 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2021-08-12T16:40:08.915Z,1628786408.915 [ESPComponent](IMPORTANT): [sample #11] ESP log summary report (3 messages): @11:29:26.29 Selecting Cartridge 46 @12:33:20.90 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @12:33:21.10 Sampled 1076.0ml 2021-08-12T16:40:09.758Z,1628786409.758 [Smear:SamplingToWaypoint:TakeSamples:Firing:F](IMPORTANT): Sample No. 1.000000 count completed. 2021-08-12T16:49:21.690Z,1628786961.690 [VerticalControl](CRITICAL): Failure to ascend, stopping mission after 300.206390 seconds, depthRate=-0.001370 m/s, pitch=17.969828 deg. 2021-08-12T16:49:21.786Z,1628786961.786 [CommandLine](FAULT): Scheduling is paused 2021-08-12T16:49:22.508Z,1628786962.508 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T16:49:22.805Z,1628786962.805 [WetLabsBB2FL](FAULT): LCB fault: Software Overcurrent. 2021-08-12T17:01:48.228Z,1628787708.228 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.421654 min 2021-08-12T17:02:50.895Z,1628787770.895 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=44486, MT Status=1, MTMSN=3129 2021-08-12T17:02:51.443Z,1628787771.443 [CommandLine](IMPORTANT): got command failComponent 2021-08-12T17:02:51.443Z,1628787771.443 [CommandLine](IMPORTANT): Failed components: 2021-08-12T17:02:51.443Z,1628787771.443 [CommandLine](IMPORTANT): No failed Components. 2021-08-12T17:02:51.443Z,1628787771.443 [CommandLine](IMPORTANT): got command ibit 2021-08-12T17:02:51.444Z,1628787771.444 [CommandLine](IMPORTANT): got command configSet list 2021-08-12T17:02:51.444Z,1628787771.444 [CommandLine](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2021-08-12T17:02:51.446Z,1628787771.446 [CommandLine](IMPORTANT): BPC1.batteryMissingStickThreshold=2 count; 2021-08-12T17:02:51.446Z,1628787771.446 [CommandLine](IMPORTANT): CBIT.abortDepth=210 meter; 2021-08-12T17:02:51.447Z,1628787771.447 [CommandLine](IMPORTANT): CBIT.gf24Offset=148.5 microampere; 2021-08-12T17:02:51.447Z,1628787771.447 [CommandLine](IMPORTANT): CBIT.stopDepth=200 meter; 2021-08-12T17:02:51.447Z,1628787771.447 [CommandLine](IMPORTANT): CTD_Seabird.minSalinityBound=0.01 practical_salinity_unit; 2021-08-12T17:02:51.447Z,1628787771.447 [CommandLine](IMPORTANT): DAT.loadAtStartup=0 bool; 2021-08-12T17:02:51.447Z,1628787771.447 [CommandLine](IMPORTANT): DVL_micro.loadAtStartup=1 bool; 2021-08-12T17:02:51.447Z,1628787771.447 [CommandLine](IMPORTANT): DataOverHttps.timeout=1 minute; 2021-08-12T17:02:51.447Z,1628787771.447 [CommandLine](IMPORTANT): DeadReckonUsingMultipleVelocitySources.loadAtStartup=0 bool; 2021-08-12T17:02:51.447Z,1628787771.447 [CommandLine](IMPORTANT): DeadReckonUsingSpeedCalculator.loadAtStartup=1 bool; 2021-08-12T17:02:51.449Z,1628787771.449 [CommandLine](IMPORTANT): Depth_Keller.offset=-0.35 decibar; 2021-08-12T17:02:51.449Z,1628787771.449 [CommandLine](IMPORTANT): Express none WetLabsBB2FL.bin_mean_mass_concentration_of_chlorophyll_in_sea_water; 2021-08-12T17:02:51.449Z,1628787771.449 [CommandLine](IMPORTANT): Express linearApproximation WetLabsBB2FL.bin_median_mass_concentration_of_chlorophyll_in_sea_water 0.400000 microgram_per_liter; 2021-08-12T17:02:51.449Z,1628787771.449 [CommandLine](IMPORTANT): Express linearApproximation height_above_sea_floor 5.000000 meter; 2021-08-12T17:02:51.450Z,1628787771.450 [CommandLine](IMPORTANT): VerticalControl.buoyancyNeutral=305.870803 cubic_centimeter; 2021-08-12T17:02:51.450Z,1628787771.450 [CommandLine](IMPORTANT): VerticalControl.massDefault=7.218472 millimeter; 2021-08-12T17:02:51.450Z,1628787771.450 [CommandLine](IMPORTANT): VerticalControl.surfaceThreshold=0.5 meter; 2021-08-12T17:02:51.490Z,1628787771.490 [IBIT](IMPORTANT): Beginning Initiated BIT 2021-08-12T17:02:51.490Z,1628787771.490 [IBIT](IMPORTANT): Beginning control surface checks. 2021-08-12T17:02:51.499Z,1628787771.499 [CBIT](IMPORTANT): Beginning ground fault scan 2021-08-12T17:03:02.528Z,1628787782.528 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.106905 CHAN A1 (24V): 0.001542 CHAN A2 (12V): -0.015717 CHAN A3 (5V): -0.009728 CHAN B0 (3.3V): -0.001519 CHAN B1 (3.15aV): 0.022551 CHAN B2 (3.15bV): -0.001031 CHAN B3 (GND): -0.000357 OPEN: 0.012224 Full Scale Calc: 4.765 mA, -1.589 mA 2021-08-12T17:03:08.156Z,1628787788.156 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=44487, MT Status=1, MTMSN=3130