2021-08-11T21:40:41.676Z,1628718041.676 [CommandLine](IMPORTANT): got command restart logs 2021-08-11T21:46:28.859Z,1628718388.859 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.298712 min 2021-08-11T21:50:50.292Z,1628718650.292 [DataOverHttps](IMPORTANT): SBD MTMSN=20210811T215051 2021-08-11T21:50:58.247Z,1628718658.247 [CommandLine](IMPORTANT): got command ! onESPclient methane -- Cmd.runStandard 2021-08-11T21:50:58.622Z,1628718658.622 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-11UTC215058_onESPclient 2021-08-11T21:52:03.560Z,1628718723.560 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.877028 min 2021-08-11T21:58:13.112Z,1628719093.112 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 32.036104 min 2021-08-11T22:03:50.456Z,1628719430.456 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 37.659827 min 2021-08-11T22:09:30.458Z,1628719770.458 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 43.325037 min 2021-08-11T22:15:06.080Z,1628720106.080 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 48.919202 min 2021-08-11T22:20:39.540Z,1628720439.540 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 54.476945 min 2021-08-11T22:26:13.479Z,1628720773.479 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 60.042232 min 2021-08-11T22:31:51.852Z,1628721111.852 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 65.681535 min 2021-08-11T22:37:31.786Z,1628721451.786 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 71.347306 min 2021-08-11T22:43:09.728Z,1628721789.728 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 76.979687 min 2021-08-11T22:48:45.627Z,1628722125.627 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 82.578442 min 2021-08-11T22:54:19.420Z,1628722459.420 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 88.141748 min 2021-08-11T22:59:55.655Z,1628722795.655 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 93.746411 min 2021-08-11T22:59:57.509Z,1628722797.509 [DataOverHttps](IMPORTANT): SBD MTMSN=20210811T225958 2021-08-11T23:00:06.007Z,1628722806.007 [CommandLine](IMPORTANT): got command schedule asap "load Science/sci2.xml;set sci2.MissionTimeout 4 h;set sci2.NeedCommsTime 30 min;set sci2.Lat1 41.8358 degree;set sci2.Lon1 -83.3527 degree;set sci2.Speed 0.8 m/s;set sci2.YoYoMinDepth 1.5 m;set sci2.YoYoMaxDepth 2.5 m" 2md7g 1 2.000000 2021-08-11T23:00:06.008Z,1628722806.008 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2md7g 2021-08-11T23:00:06.009Z,1628722806.009 [CommandLine](IMPORTANT): Scheduled #42 (#1 of 2 with id='2md7g'): "load Science/sci2.xml;set sci2.MissionTimeout 4 h;set sci2.NeedCommsTime 30 min;set sci2.Lat1 41.8358 degree;set sci2.Lon1 -83.3527 degree;set sci2.Speed 0.8 m/s;set sci2.YoYoMinDepth 1.5 m;set sci2.YoYoMaxDepth 2.5 m" ASAP 2021-08-11T23:00:07.193Z,1628722807.193 [DataOverHttps](IMPORTANT): SBD MOMSN=15870814, MTMSN=20210811T230007 2021-08-11T23:00:15.311Z,1628722815.311 [CommandLine](IMPORTANT): got command schedule asap "set sci2.YoYoMinAltitude 2 m;set sci2.YoYoUpPitch 12 degree;set sci2.YoYoDownPitch -12 degree;set sci2.MinAltitude 0.5 m;set sci2.MaxDepth 4 m;set sci2.MinOffshore 1 km;run" 2md7g 2 2.000000 2021-08-11T23:00:15.312Z,1628722815.312 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2md7g 2021-08-11T23:00:15.313Z,1628722815.313 [CommandLine](IMPORTANT): Scheduled #43 (#2 of 2 with id='2md7g'): "set sci2.YoYoMinAltitude 2 m;set sci2.YoYoUpPitch 12 degree;set sci2.YoYoDownPitch -12 degree;set sci2.MinAltitude 0.5 m;set sci2.MaxDepth 4 m;set sci2.MinOffshore 1 km;run" ASAP 2021-08-11T23:00:15.402Z,1628722815.402 [CommandLine](IMPORTANT): got command load ./Missions/Science/sci2.xml 2021-08-11T23:00:19.355Z,1628722819.355 [CommandLine](IMPORTANT): Loaded ./Missions/Science/sci2.xml 2021-08-11T23:00:49.626Z,1628722849.626 [CommandLine](IMPORTANT): got command set sci2.MissionTimeout 4.000000 hour 2021-08-11T23:00:49.627Z,1628722849.627 [CommandLine](IMPORTANT): got command set sci2.NeedCommsTime 30.000000 minute 2021-08-11T23:00:49.628Z,1628722849.628 [CommandLine](IMPORTANT): got command set sci2.Lat1 41.835800 degree 2021-08-11T23:00:49.629Z,1628722849.629 [CommandLine](IMPORTANT): got command set sci2.Lon1 -83.352699 degree 2021-08-11T23:00:49.630Z,1628722849.630 [CommandLine](IMPORTANT): got command set sci2.Speed 0.800000 meter_per_second 2021-08-11T23:00:49.631Z,1628722849.631 [CommandLine](IMPORTANT): got command set sci2.YoYoMinDepth 1.500000 meter 2021-08-11T23:00:49.631Z,1628722849.631 [CommandLine](IMPORTANT): got command set sci2.YoYoMaxDepth 2.500000 meter 2021-08-11T23:01:13.323Z,1628722873.323 [CommandLine](IMPORTANT): got command set sci2.YoYoMinAltitude 2.000000 meter 2021-08-11T23:01:13.324Z,1628722873.324 [CommandLine](IMPORTANT): got command set sci2.YoYoUpPitch 12.000000 degree 2021-08-11T23:01:13.325Z,1628722873.325 [CommandLine](IMPORTANT): got command set sci2.YoYoDownPitch -12.000000 degree 2021-08-11T23:01:13.326Z,1628722873.326 [CommandLine](IMPORTANT): got command set sci2.MinAltitude 0.500000 meter 2021-08-11T23:01:13.326Z,1628722873.326 [CommandLine](IMPORTANT): got command set sci2.MaxDepth 4.000000 meter 2021-08-11T23:01:13.391Z,1628722873.391 [CommandLine](IMPORTANT): got command set sci2.MinOffshore 1.000000 kilometer 2021-08-11T23:01:13.391Z,1628722873.391 [CommandLine](IMPORTANT): got command run 2021-08-11T23:01:13.393Z,1628722873.393 [CommandLine](IMPORTANT): Running 2021-08-11T23:01:13.431Z,1628722873.431 [MissionManager](IMPORTANT): Started mission sci2 2021-08-11T23:51:08.381Z,1628725868.381 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached Waypoint: 41.835800,-83.352698 2021-08-11T23:52:19.888Z,1628725939.888 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T00:00:32.403Z,1628726432.403 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T000033 2021-08-12T00:00:40.539Z,1628726440.539 [CommandLine](IMPORTANT): got command ! start onESP upsync 2021-08-12T00:00:40.894Z,1628726440.894 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC000040_onESP 2021-08-12T00:00:42.324Z,1628726442.324 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.367339 min 2021-08-12T00:03:10.068Z,1628726590.068 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T000310 2021-08-12T00:03:17.703Z,1628726597.703 [CommandLine](IMPORTANT): got command ! check 2021-08-12T00:03:17.869Z,1628726597.869 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Thu Aug 12 00:01:16 UTC 2021 -- ssh esp@10.89.11.3 upsync Uploading real real.* *.spr* *.pcr* *.tif* CTD-*.hex .././messages in /var/log/esp to ESPhonu@ESPshore:/home/ftp/honu/esp shutting ESP off 2021-08-12T00:06:40.026Z,1628726800.026 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 14.329101 min 2021-08-12T00:08:12.460Z,1628726892.460 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T000813 2021-08-12T00:08:23.964Z,1628726903.964 [CommandLine](IMPORTANT): got command ! onESPclient methane -- Cartridge.release 54 2021-08-12T00:08:24.618Z,1628726904.618 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC000824_onESPclient 2021-08-12T00:11:55.143Z,1628727115.143 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T001155 2021-08-12T00:12:06.112Z,1628727126.112 [CommandLine](IMPORTANT): got command ! check 2021-08-12T00:12:06.362Z,1628727126.362 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Thu Aug 12 00:08:59 UTC 2021 -- ssh esp@10.89.11.3 espclient methane 1 shutting ESP off 2021-08-12T00:12:30.881Z,1628727150.881 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.176520 min 2021-08-12T00:18:36.732Z,1628727516.732 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 26.273855 min 2021-08-12T00:24:08.364Z,1628727848.364 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.801166 min 2021-08-12T00:29:45.549Z,1628728185.549 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 37.420667 min 2021-08-12T00:35:24.674Z,1628728524.674 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 43.072994 min 2021-08-12T00:40:59.396Z,1628728859.396 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 48.651729 min 2021-08-12T00:44:46.301Z,1628729086.301 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T004447 2021-08-12T00:44:54.368Z,1628729094.368 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" 2mi2a 1 4.000000 2021-08-12T00:44:54.369Z,1628729094.369 [CommandLine](IMPORTANT): Scheduling command #1 of 4 with id=2mi2a 2021-08-12T00:44:54.370Z,1628729094.370 [CommandLine](IMPORTANT): Scheduled #44 (#1 of 4 with id='2mi2a'): "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" ASAP 2021-08-12T00:44:55.340Z,1628729095.340 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T004456 2021-08-12T00:45:03.308Z,1628729103.308 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m;set esp_sample_at_threshold.YoYoMinAltitude 2 m" 2mi2a 2 4.000000 2021-08-12T00:45:03.309Z,1628729103.309 [CommandLine](IMPORTANT): Scheduling command #2 of 4 with id=2mi2a 2021-08-12T00:45:03.309Z,1628729103.309 [CommandLine](IMPORTANT): Scheduled #45 (#2 of 4 with id='2mi2a'): "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m;set esp_sample_at_threshold.YoYoMinAltitude 2 m" ASAP 2021-08-12T00:45:04.281Z,1628729104.281 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T004505 2021-08-12T00:45:12.355Z,1628729112.355 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m;set esp_sample_at_threshold.ESPCartridgeType_2 -18 count" 2mi2a 3 4.000000 2021-08-12T00:45:12.356Z,1628729112.356 [CommandLine](IMPORTANT): Scheduling command #3 of 4 with id=2mi2a 2021-08-12T00:45:12.357Z,1628729112.357 [CommandLine](IMPORTANT): Scheduled #46 (#3 of 4 with id='2mi2a'): "set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m;set esp_sample_at_threshold.ESPCartridgeType_2 -18 count" ASAP 2021-08-12T00:45:13.300Z,1628729113.300 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T004514 2021-08-12T00:45:21.364Z,1628729121.364 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" 2mi2a 4 4.000000 2021-08-12T00:45:21.365Z,1628729121.365 [CommandLine](IMPORTANT): Scheduling command #4 of 4 with id=2mi2a 2021-08-12T00:45:21.366Z,1628729121.366 [CommandLine](IMPORTANT): Scheduled #47 (#4 of 4 with id='2mi2a'): "set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" ASAP 2021-08-12T00:45:21.482Z,1628729121.482 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T00:45:24.346Z,1628729124.346 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T00:45:40.442Z,1628729140.442 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllLowerBound 6.000000 microgram_per_liter 2021-08-12T00:45:40.443Z,1628729140.443 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllUpperBound 100.000000 microgram_per_liter 2021-08-12T00:45:40.444Z,1628729140.444 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lat1 41.837132 degree 2021-08-12T00:46:12.860Z,1628729172.860 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lon1 -83.351646 degree 2021-08-12T00:46:12.861Z,1628729172.861 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinDepth 1.500000 meter 2021-08-12T00:46:12.861Z,1628729172.861 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMaxDepth 2.500000 meter 2021-08-12T00:46:12.866Z,1628729172.866 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinAltitude 2.000000 meter 2021-08-12T00:46:33.626Z,1628729193.626 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 54.221322 min 2021-08-12T00:46:41.141Z,1628729201.141 [DataOverHttps](IMPORTANT): SBD MOMSN=15871306, MTMSN=20210812T004641 2021-08-12T00:46:50.907Z,1628729210.907 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoUpPitch 12.000000 degree 2021-08-12T00:46:50.908Z,1628729210.908 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoDownPitch -12.000000 degree 2021-08-12T00:46:50.909Z,1628729210.909 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.TargetDepth 2.500000 meter 2021-08-12T00:46:50.910Z,1628729210.910 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.ESPCartridgeType_2 -18.000000 count 2021-08-12T00:47:02.143Z,1628729222.143 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MinAltitude 0.500000 meter 2021-08-12T00:47:02.144Z,1628729222.144 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MaxDepth 4.500000 meter 2021-08-12T00:47:02.145Z,1628729222.145 [CommandLine](IMPORTANT): got command run 2021-08-12T00:47:02.174Z,1628729222.174 [CommandLine](IMPORTANT): Running 2021-08-12T00:47:02.182Z,1628729222.182 [CommandLine](IMPORTANT): got command stop 2021-08-12T00:47:02.321Z,1628729222.321 [MissionManager](IMPORTANT): Started mission esp_sample_at_threshold 2021-08-12T00:47:02.858Z,1628729222.858 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T00:49:45.529Z,1628729385.529 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T004946 2021-08-12T00:49:54.112Z,1628729394.112 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" 2miaw 1 4.000000 2021-08-12T00:49:54.113Z,1628729394.113 [CommandLine](IMPORTANT): Scheduling command #1 of 4 with id=2miaw 2021-08-12T00:49:54.114Z,1628729394.114 [CommandLine](IMPORTANT): Scheduled #48 (#1 of 4 with id='2miaw'): "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" ASAP 2021-08-12T00:49:55.016Z,1628729395.016 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T004955 2021-08-12T00:50:03.023Z,1628729403.023 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" 2miaw 2 4.000000 2021-08-12T00:50:03.023Z,1628729403.023 [CommandLine](IMPORTANT): Scheduling command #2 of 4 with id=2miaw 2021-08-12T00:50:03.024Z,1628729403.024 [CommandLine](IMPORTANT): Scheduled #49 (#2 of 4 with id='2miaw'): "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" ASAP 2021-08-12T00:50:04.016Z,1628729404.016 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T005004 2021-08-12T00:50:12.176Z,1628729412.176 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" 2miaw 3 4.000000 2021-08-12T00:50:12.176Z,1628729412.176 [CommandLine](IMPORTANT): Scheduling command #3 of 4 with id=2miaw 2021-08-12T00:50:12.177Z,1628729412.177 [CommandLine](IMPORTANT): Scheduled #50 (#3 of 4 with id='2miaw'): "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" ASAP 2021-08-12T00:50:13.132Z,1628729413.132 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T005013 2021-08-12T00:50:25.252Z,1628729425.252 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" 2miaw 4 4.000000 2021-08-12T00:50:25.252Z,1628729425.252 [CommandLine](IMPORTANT): Scheduling command #4 of 4 with id=2miaw 2021-08-12T00:50:25.253Z,1628729425.253 [CommandLine](IMPORTANT): Scheduled #51 (#4 of 4 with id='2miaw'): "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" ASAP 2021-08-12T00:50:25.382Z,1628729425.382 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T00:50:30.566Z,1628729430.566 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T00:50:48.295Z,1628729448.295 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllLowerBound 6.000000 microgram_per_liter 2021-08-12T00:50:48.296Z,1628729448.296 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllUpperBound 100.000000 microgram_per_liter 2021-08-12T00:50:48.297Z,1628729448.297 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lat1 41.837132 degree 2021-08-12T00:51:10.672Z,1628729470.672 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lon1 -83.351646 degree 2021-08-12T00:51:10.673Z,1628729470.673 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Speed 0.800000 meter_per_second 2021-08-12T00:51:10.674Z,1628729470.674 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinDepth 1.500000 meter 2021-08-12T00:51:10.691Z,1628729470.691 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMaxDepth 2.500000 meter 2021-08-12T00:51:36.065Z,1628729496.065 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinAltitude 2.000000 meter 2021-08-12T00:51:36.066Z,1628729496.066 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoUpPitch 12.000000 degree 2021-08-12T00:51:36.066Z,1628729496.066 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoDownPitch -12.000000 degree 2021-08-12T00:51:36.075Z,1628729496.075 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.TargetDepth 2.500000 meter 2021-08-12T00:51:59.230Z,1628729519.230 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.ESPCartridgeType_2 -18.000000 count 2021-08-12T00:51:59.231Z,1628729519.231 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MinAltitude 0.500000 meter 2021-08-12T00:51:59.231Z,1628729519.231 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MaxDepth 4.500000 meter 2021-08-12T00:51:59.232Z,1628729519.232 [CommandLine](IMPORTANT): got command run 2021-08-12T00:51:59.234Z,1628729519.234 [CommandLine](IMPORTANT): Running 2021-08-12T00:51:59.396Z,1628729519.396 [MissionManager](IMPORTANT): Started mission esp_sample_at_threshold 2021-08-12T01:55:47.745Z,1628733347.745 [ESPComponent](CRITICAL): [sample #8] ESP sampling sequence terminated with error: In state S_WAITING_FOR_PROCESSED got exception while awaiting completion of cmd='Cmd.startProcessing' exception=<>. Elapsed time since command issued: 205.6193 2021-08-12T00:56:49.989Z,1628729809.989 [esp_sample_at_threshold:SampleESP:sample:A](IMPORTANT): At 3.509304 m , settling for 3.000000 min 2021-08-12T00:59:50.989Z,1628729990.989 [esp_sample_at_threshold:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -4.000000 count 2021-08-12T01:00:02.799Z,1628730002.799 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T01:00:03.039Z,1628730003.039 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T01:00:40.782Z,1628730040.782 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T01:00:41.605Z,1628730041.605 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -4 2021-08-12T01:00:41.606Z,1628730041.606 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T01:00:42.020Z,1628730042.020 [esp_sample_at_threshold:SampleESP:sample:F](IMPORTANT): ESP sampling at 3.631555 m in 25.342249 degC water with 7.341640 ug/l chlorophyll fluorescence. 2021-08-12T01:00:42.406Z,1628730042.406 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T01:00:42.810Z,1628730042.810 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T01:00:44.829Z,1628730044.829 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T01:01:24.421Z,1628730084.421 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@26,archiveHiBiomass_bac) 2021-08-12T01:01:26.038Z,1628730086.038 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T01:01:26.452Z,1628730086.452 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=93.070000 issuedCmd_=Cmd.startFiltering 2021-08-12T01:01:26.847Z,1628730086.847 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T01:03:41.029Z,1628730221.029 [CBIT](IMPORTANT): Beginning ground fault scan 2021-08-12T01:03:51.935Z,1628730231.935 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.011448 CHAN A1 (24V): 0.018032 CHAN A2 (12V): 0.007380 CHAN A3 (5V): 0.002417 CHAN B0 (3.3V): 0.002079 CHAN B1 (3.15aV): 0.001948 CHAN B2 (3.15bV): 0.002296 CHAN B3 (GND): 0.000384 OPEN: 0.007924 Full Scale Calc: 4.765 mA, -1.589 mA 2021-08-12T01:04:12.097Z,1628730252.097 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T01:04:12.505Z,1628730252.505 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T01:16:38.294Z,1628730998.294 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@191.9) 2021-08-12T01:16:38.694Z,1628730998.694 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@191.9) 2021-08-12T01:16:40.718Z,1628731000.718 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T01:16:41.141Z,1628731001.141 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@26,archiveHiBiomass_bac) 2021-08-12T01:29:28.360Z,1628731768.360 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@26,archiveHiBiomass_bac) 2021-08-12T01:29:37.666Z,1628731777.666 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T01:29:38.050Z,1628731778.050 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2021-08-12T01:34:37.839Z,1628732077.839 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2021-08-12T01:34:38.239Z,1628732078.239 [ESPComponent](IMPORTANT): [sample #7] ESP log summary report (3 messages): @21:00:55.42 Selecting Cartridge 26 @21:16:35.38 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @21:16:35.57 Sampled 191.9ml 2021-08-12T01:34:39.125Z,1628732079.125 [esp_sample_at_threshold:SampleESP:sample:A](IMPORTANT): At 2.447917 m , settling for 3.000000 min 2021-08-12T01:37:40.112Z,1628732260.112 [esp_sample_at_threshold:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -18.000000 count 2021-08-12T01:37:51.899Z,1628732271.899 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T01:37:52.071Z,1628732272.071 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T01:38:29.834Z,1628732309.834 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T01:38:30.650Z,1628732310.650 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -18 2021-08-12T01:38:30.651Z,1628732310.651 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T01:38:31.089Z,1628732311.089 [esp_sample_at_threshold:SampleESP:sample:F](IMPORTANT): ESP sampling at 2.304929 m in 25.322992 degC water with 7.255800 ug/l chlorophyll fluorescence. 2021-08-12T01:38:31.460Z,1628732311.460 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T01:38:31.860Z,1628732311.860 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T01:38:33.874Z,1628732313.874 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T01:39:14.279Z,1628732354.279 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@23,microSPR_phyto) 2021-08-12T01:39:15.891Z,1628732355.891 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T01:39:16.299Z,1628732356.299 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=14.740000 issuedCmd_=Cmd.startFiltering 2021-08-12T01:39:16.703Z,1628732356.703 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T01:42:48.824Z,1628732568.824 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T01:42:49.226Z,1628732569.226 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T01:46:51.232Z,1628732811.232 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@75.0) 2021-08-12T01:46:51.610Z,1628732811.610 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@75.0) 2021-08-12T01:46:53.639Z,1628732813.639 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T01:46:54.039Z,1628732814.039 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@21,microSPR_phyto) 2021-08-12T01:50:17.255Z,1628733017.255 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@21,microSPR_phyto) 2021-08-12T01:50:47.563Z,1628733047.563 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T01:50:47.991Z,1628733047.991 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2021-08-12T01:55:47.877Z,1628733347.877 [CommandLine](FAULT): Scheduling is paused 2021-08-12T01:55:48.147Z,1628733348.147 [ESPComponent](IMPORTANT): [sample #8] ESP log summary report (9 messages): @21:38:44.39 Selecting Cartridge 23 @21:42:09.25 Cartridge::Sampler::Leaky in FILTERING -- Isolated cartridge pressure fell by 25% to 17.1psia @21:42:09.27 Retry #1 of 2 @21:42:09.29 Cmd::SpareCartridge in FILTERING -- 0 microSPR_phyto spares remaining @21:42:09.34 Selecting Cartridge 21 @21:46:48.96 Sampled 75.0ml @21:50:12.63 Slide::Error in PROCESSING -- Archive Syringe (ArS) positionErr at 78uL (sensed 103uL) @21:50:22.14 Slide::Error in LRAUV -- Archive Syringe (ArS) positionErr at 958uL (sensed 933uL) @21:50:22.15 Retry #1 of 2 2021-08-12T01:55:48.795Z,1628733348.795 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T02:03:51.324Z,1628733831.324 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.035275 min 2021-08-12T02:09:30.812Z,1628734170.812 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 13.693727 min 2021-08-12T02:15:08.040Z,1628734508.040 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 19.314288 min 2021-08-12T02:20:40.628Z,1628734840.628 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 24.857440 min 2021-08-12T02:26:13.211Z,1628735173.211 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 30.399672 min 2021-08-12T02:31:52.035Z,1628735512.035 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 36.047331 min 2021-08-12T02:37:29.120Z,1628735849.120 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 41.665312 min 2021-08-12T02:43:06.269Z,1628736186.269 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 47.284627 min 2021-08-12T02:48:39.738Z,1628736519.738 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 52.842069 min 2021-08-12T02:54:12.343Z,1628736852.343 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 58.387297 min 2021-08-12T02:59:49.528Z,1628737189.528 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 64.005448 min 2021-08-12T03:05:32.588Z,1628737532.588 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 69.722884 min 2021-08-12T03:11:09.290Z,1628737869.290 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 75.334798 min 2021-08-12T03:12:59.280Z,1628737979.280 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T031300 2021-08-12T03:13:07.284Z,1628737987.284 [CommandLine](IMPORTANT): got command ! onESPclient methane -- spare PhytoSampler: 400 2021-08-12T03:13:07.647Z,1628737987.647 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC031307_onESPclient 2021-08-12T03:16:42.395Z,1628738202.395 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 80.886898 min 2021-08-12T03:17:42.748Z,1628738262.748 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T031743 2021-08-12T03:17:53.645Z,1628738273.645 [CommandLine](IMPORTANT): got command ! check 2021-08-12T03:17:53.984Z,1628738273.984 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Thu Aug 12 03:13:43 UTC 2021 -- ssh esp@10.89.11.3 espclient methane {PhytoSampler: 400, archiveHiBiomass_bac: 1, microSPR_phyto: 0} shutting ESP off 2021-08-12T03:22:36.476Z,1628738556.476 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 86.787467 min 2021-08-12T03:26:11.904Z,1628738771.904 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T032612 2021-08-12T03:26:23.859Z,1628738783.859 [CommandLine](IMPORTANT): got command ! onESPclient methane -- slot microSPR_phyto:1 2021-08-12T03:26:24.510Z,1628738784.510 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC032624_onESPclient 2021-08-12T03:28:09.298Z,1628738889.298 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 92.334253 min 2021-08-12T03:28:44.825Z,1628738924.825 [DataOverHttps](IMPORTANT): SBD MOMSN=15871922, MTMSN=20210812T032845 2021-08-12T03:28:55.951Z,1628738935.951 [CommandLine](IMPORTANT): got command ! check 2021-08-12T03:28:56.239Z,1628738936.239 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Thu Aug 12 03:26:59 UTC 2021 -- ssh esp@10.89.11.3 espclient methane {microSPR_phyto: 1} shutting ESP off 2021-08-12T03:34:46.784Z,1628739286.784 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 98.959741 min 2021-08-12T03:34:48.144Z,1628739288.144 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T033448 2021-08-12T03:34:56.317Z,1628739296.317 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" 2mpxi 1 4.000000 2021-08-12T03:34:56.318Z,1628739296.318 [CommandLine](IMPORTANT): Scheduling command #1 of 4 with id=2mpxi 2021-08-12T03:34:56.319Z,1628739296.319 [CommandLine](IMPORTANT): Scheduled #52 (#1 of 4 with id='2mpxi'): "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" ASAP 2021-08-12T03:34:57.558Z,1628739297.558 [DataOverHttps](IMPORTANT): SBD MOMSN=15871942, MTMSN=20210812T033458 2021-08-12T03:35:05.520Z,1628739305.520 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" 2mpxi 2 4.000000 2021-08-12T03:35:05.520Z,1628739305.520 [CommandLine](IMPORTANT): Scheduling command #2 of 4 with id=2mpxi 2021-08-12T03:35:05.521Z,1628739305.521 [CommandLine](IMPORTANT): Scheduled #53 (#2 of 4 with id='2mpxi'): "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" ASAP 2021-08-12T03:35:06.726Z,1628739306.726 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T033507 2021-08-12T03:35:14.439Z,1628739314.439 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" 2mpxi 3 4.000000 2021-08-12T03:35:14.440Z,1628739314.440 [CommandLine](IMPORTANT): Scheduling command #3 of 4 with id=2mpxi 2021-08-12T03:35:14.441Z,1628739314.441 [CommandLine](IMPORTANT): Scheduled #54 (#3 of 4 with id='2mpxi'): "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" ASAP 2021-08-12T03:35:15.697Z,1628739315.697 [DataOverHttps](IMPORTANT): SBD MOMSN=15871949, MTMSN=20210812T033516 2021-08-12T03:35:23.475Z,1628739323.475 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" 2mpxi 4 4.000000 2021-08-12T03:35:23.476Z,1628739323.476 [CommandLine](IMPORTANT): Scheduling command #4 of 4 with id=2mpxi 2021-08-12T03:35:23.477Z,1628739323.477 [CommandLine](IMPORTANT): Scheduled #55 (#4 of 4 with id='2mpxi'): "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" ASAP 2021-08-12T03:35:23.677Z,1628739323.677 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T03:35:25.068Z,1628739325.068 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T033525 2021-08-12T03:35:26.823Z,1628739326.823 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T03:35:47.621Z,1628739347.621 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllLowerBound 6.000000 microgram_per_liter 2021-08-12T03:35:47.622Z,1628739347.622 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllUpperBound 100.000000 microgram_per_liter 2021-08-12T03:35:47.624Z,1628739347.624 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lat1 41.837132 degree 2021-08-12T03:35:47.695Z,1628739347.695 [CommandLine](IMPORTANT): got command schedule asap 2021-08-12T03:35:47.695Z,1628739347.695 [CommandLine](FAULT): Incomplete syntax. Try: help schedule 2021-08-12T03:35:47.795Z,1628739347.795 [CommandLine](IMPORTANT): got command maintain 2021-08-12T03:35:48.417Z,1628739348.417 [DataOverHttps](IMPORTANT): SBD MOMSN=15871953, MTMSN=20210812T033538 2021-08-12T03:36:23.020Z,1628739383.020 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lon1 -83.351646 degree 2021-08-12T03:36:23.022Z,1628739383.022 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Speed 0.800000 meter_per_second 2021-08-12T03:36:23.023Z,1628739383.023 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinDepth 1.500000 meter 2021-08-12T03:36:23.024Z,1628739383.024 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMaxDepth 2.500000 meter 2021-08-12T03:36:23.039Z,1628739383.039 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" 2mpye 2 4.000000 2021-08-12T03:36:23.047Z,1628739383.047 [CommandLine](IMPORTANT): Scheduling command #2 of 4 with id=2mpye 2021-08-12T03:36:23.048Z,1628739383.048 [CommandLine](IMPORTANT): Scheduled #56 (#2 of 4 with id='2mpye'): "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" ASAP 2021-08-12T03:36:23.886Z,1628739383.886 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T033608 2021-08-12T03:36:53.336Z,1628739413.336 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinAltitude 2.000000 meter 2021-08-12T03:36:53.337Z,1628739413.337 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoUpPitch 12.000000 degree 2021-08-12T03:36:53.338Z,1628739413.338 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoDownPitch -12.000000 degree 2021-08-12T03:36:53.339Z,1628739413.339 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.TargetDepth 2.500000 meter 2021-08-12T03:36:53.347Z,1628739413.347 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" 2mpye 3 4.000000 2021-08-12T03:36:53.355Z,1628739413.355 [CommandLine](IMPORTANT): Scheduling command #3 of 4 with id=2mpye 2021-08-12T03:36:53.356Z,1628739413.356 [CommandLine](IMPORTANT): Scheduled #57 (#3 of 4 with id='2mpye'): "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" ASAP 2021-08-12T03:36:53.746Z,1628739413.746 [DataOverHttps](IMPORTANT): SBD MOMSN=15871957, MTMSN=20210812T033641 2021-08-12T03:37:13.027Z,1628739433.027 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.ESPCartridgeType_2 -18.000000 count 2021-08-12T03:37:13.028Z,1628739433.028 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MinAltitude 0.500000 meter 2021-08-12T03:37:13.028Z,1628739433.028 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MaxDepth 4.500000 meter 2021-08-12T03:37:13.029Z,1628739433.029 [CommandLine](IMPORTANT): got command run 2021-08-12T03:37:13.031Z,1628739433.031 [CommandLine](IMPORTANT): Running 2021-08-12T03:37:13.039Z,1628739433.039 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" 2mpye 4 4.000000 2021-08-12T03:37:13.047Z,1628739433.047 [CommandLine](IMPORTANT): Scheduling command #4 of 4 with id=2mpye 2021-08-12T03:37:13.048Z,1628739433.048 [CommandLine](IMPORTANT): Scheduled #58 (#4 of 4 with id='2mpye'): "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" ASAP 2021-08-12T03:37:13.346Z,1628739433.346 [MissionManager](IMPORTANT): Started mission esp_sample_at_threshold 2021-08-12T03:42:16.311Z,1628739736.311 [DataOverHttps](IMPORTANT): SBD MOMSN=15872053, MTMSN=20210812T034217 2021-08-12T03:42:30.948Z,1628739750.948 [CommandLine](IMPORTANT): got command stop 2021-08-12T03:42:30.949Z,1628739750.949 [CommandLine](IMPORTANT): Scheduling is paused for 3 commands 2021-08-12T03:42:31.536Z,1628739751.536 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T03:43:54.732Z,1628739834.732 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T034355 2021-08-12T03:44:02.776Z,1628739842.776 [CommandLine](IMPORTANT): got command schedule clear 2021-08-12T03:44:02.776Z,1628739842.776 [CommandLine](IMPORTANT): #Cleared scheduled commands:3 2021-08-12T03:44:02.777Z,1628739842.777 [CommandLine](IMPORTANT): got command schedule resume 2021-08-12T03:44:02.777Z,1628739842.777 [CommandLine](IMPORTANT): Scheduling is resumed 2021-08-12T03:44:38.813Z,1628739878.813 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T034439 2021-08-12T03:44:46.819Z,1628739886.819 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" 2mqe4 1 4.000000 2021-08-12T03:44:46.820Z,1628739886.820 [CommandLine](IMPORTANT): Scheduling command #1 of 4 with id=2mqe4 2021-08-12T03:44:46.821Z,1628739886.821 [CommandLine](IMPORTANT): Scheduled #59 (#1 of 4 with id='2mqe4'): "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" ASAP 2021-08-12T03:44:47.769Z,1628739887.769 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T034448 2021-08-12T03:44:55.782Z,1628739895.782 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" 2mqe4 2 4.000000 2021-08-12T03:44:55.783Z,1628739895.783 [CommandLine](IMPORTANT): Scheduling command #2 of 4 with id=2mqe4 2021-08-12T03:44:55.784Z,1628739895.784 [CommandLine](IMPORTANT): Scheduled #60 (#2 of 4 with id='2mqe4'): "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" ASAP 2021-08-12T03:44:56.761Z,1628739896.761 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T034458 2021-08-12T03:45:04.999Z,1628739904.999 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" 2mqe4 3 4.000000 2021-08-12T03:45:04.000Z,1628739905.000 [CommandLine](IMPORTANT): Scheduling command #3 of 4 with id=2mqe4 2021-08-12T03:45:05.001Z,1628739905.001 [CommandLine](IMPORTANT): Scheduled #61 (#3 of 4 with id='2mqe4'): "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" ASAP 2021-08-12T03:45:05.933Z,1628739905.933 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T034506 2021-08-12T03:45:14.028Z,1628739914.028 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" 2mqe4 4 4.000000 2021-08-12T03:45:14.029Z,1628739914.029 [CommandLine](IMPORTANT): Scheduling command #4 of 4 with id=2mqe4 2021-08-12T03:45:14.030Z,1628739914.030 [CommandLine](IMPORTANT): Scheduled #62 (#4 of 4 with id='2mqe4'): "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" ASAP 2021-08-12T03:45:14.132Z,1628739914.132 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T03:45:16.984Z,1628739916.984 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T03:45:33.638Z,1628739933.638 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllLowerBound 6.000000 microgram_per_liter 2021-08-12T03:45:33.639Z,1628739933.639 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllUpperBound 100.000000 microgram_per_liter 2021-08-12T03:45:33.640Z,1628739933.640 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lat1 41.837132 degree 2021-08-12T03:46:04.743Z,1628739964.743 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lon1 -83.351646 degree 2021-08-12T03:46:04.744Z,1628739964.744 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Speed 0.800000 meter_per_second 2021-08-12T03:46:04.744Z,1628739964.744 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinDepth 1.500000 meter 2021-08-12T03:46:04.745Z,1628739964.745 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMaxDepth 2.500000 meter 2021-08-12T03:46:38.199Z,1628739998.199 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinAltitude 2.000000 meter 2021-08-12T03:46:38.200Z,1628739998.200 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoUpPitch 12.000000 degree 2021-08-12T03:46:38.201Z,1628739998.201 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoDownPitch -12.000000 degree 2021-08-12T03:46:38.202Z,1628739998.202 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.TargetDepth 2.500000 meter 2021-08-12T03:46:53.240Z,1628740013.240 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.ESPCartridgeType_2 -18.000000 count 2021-08-12T03:46:53.241Z,1628740013.241 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MinAltitude 0.500000 meter 2021-08-12T03:46:53.242Z,1628740013.242 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MaxDepth 4.500000 meter 2021-08-12T03:46:53.242Z,1628740013.242 [CommandLine](IMPORTANT): got command run 2021-08-12T03:46:53.244Z,1628740013.244 [CommandLine](IMPORTANT): Running 2021-08-12T03:46:53.324Z,1628740013.324 [MissionManager](IMPORTANT): Started mission esp_sample_at_threshold 2021-08-12T04:53:21.758Z,1628744001.758 [ESPComponent](CRITICAL): [sample #10] ESP sampling sequence terminated with error: Timeout while awaiting completion of cmd='Cmd.startFiltering' 2021-08-12T03:51:23.058Z,1628740283.058 [Radio_Surface](FAULT): LCB fault: Software Overcurrent. 2021-08-12T03:51:53.966Z,1628740313.966 [esp_sample_at_threshold:SampleESP:sample:A](IMPORTANT): At 3.699376 m , settling for 3.000000 min 2021-08-12T03:54:54.944Z,1628740494.944 [esp_sample_at_threshold:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -4.000000 count 2021-08-12T03:55:06.752Z,1628740506.752 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T03:55:06.899Z,1628740506.899 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T03:55:45.024Z,1628740545.024 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T03:55:45.839Z,1628740545.839 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -4 2021-08-12T03:55:45.839Z,1628740545.839 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T03:55:46.248Z,1628740546.248 [esp_sample_at_threshold:SampleESP:sample:F](IMPORTANT): ESP sampling at 3.238017 m in 25.244135 degC water with 8.387960 ug/l chlorophyll fluorescence. 2021-08-12T03:55:46.642Z,1628740546.642 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T03:55:47.038Z,1628740547.038 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T03:55:49.066Z,1628740549.066 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T03:56:28.650Z,1628740588.650 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@24,archiveHiBiomass_bac) 2021-08-12T03:56:30.270Z,1628740590.270 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T03:56:30.673Z,1628740590.673 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=93.070000 issuedCmd_=Cmd.startFiltering 2021-08-12T03:56:31.115Z,1628740591.115 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T03:59:17.541Z,1628740757.541 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T03:59:17.949Z,1628740757.949 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T04:10:47.606Z,1628741447.606 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@189.2) 2021-08-12T04:10:47.000Z,1628741448.000 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@189.2) 2021-08-12T04:10:50.030Z,1628741450.030 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T04:10:50.435Z,1628741450.435 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@24,archiveHiBiomass_bac) 2021-08-12T04:23:37.659Z,1628742217.659 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@24,archiveHiBiomass_bac) 2021-08-12T04:23:46.935Z,1628742226.935 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T04:23:47.354Z,1628742227.354 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2021-08-12T04:28:47.118Z,1628742527.118 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2021-08-12T04:28:47.528Z,1628742527.528 [ESPComponent](IMPORTANT): [sample #9] ESP log summary report (3 messages): @23:55:59.46 Selecting Cartridge 24 @00:10:44.57 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @00:10:44.77 Sampled 189.2ml 2021-08-12T04:28:48.389Z,1628742528.389 [esp_sample_at_threshold:SampleESP:sample:A](IMPORTANT): At 2.492842 m , settling for 3.000000 min 2021-08-12T04:31:49.341Z,1628742709.341 [esp_sample_at_threshold:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -18.000000 count 2021-08-12T04:32:01.146Z,1628742721.146 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T04:32:01.251Z,1628742721.251 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T04:32:40.218Z,1628742760.218 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T04:32:41.057Z,1628742761.057 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -18 2021-08-12T04:32:41.058Z,1628742761.058 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T04:32:41.483Z,1628742761.483 [esp_sample_at_threshold:SampleESP:sample:F](IMPORTANT): ESP sampling at 2.629782 m in 25.229975 degC water with 8.201200 ug/l chlorophyll fluorescence. 2021-08-12T04:32:41.852Z,1628742761.852 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T04:32:42.253Z,1628742762.253 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T04:32:44.265Z,1628742764.265 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T04:33:24.698Z,1628742804.698 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@19,microSPR_phyto) 2021-08-12T04:33:26.313Z,1628742806.313 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T04:33:26.732Z,1628742806.732 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=14.740000 issuedCmd_=Cmd.startFiltering 2021-08-12T04:33:27.115Z,1628742807.115 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T04:36:15.577Z,1628742975.577 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T04:36:15.988Z,1628742975.988 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T04:48:09.061Z,1628743689.061 [ESPComponent](FAULT): Timeout while awaiting completion of cmd='Cmd.startFiltering' 2021-08-12T04:48:09.463Z,1628743689.463 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@0.0) 2021-08-12T04:48:21.580Z,1628743701.580 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T04:48:21.987Z,1628743701.987 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2021-08-12T04:53:21.817Z,1628744001.817 [CommandLine](FAULT): Scheduling is paused 2021-08-12T04:53:22.162Z,1628744002.162 [ESPComponent](IMPORTANT): [sample #10] ESP log summary report (1 messages): @00:32:54.59 Selecting Cartridge 19 2021-08-12T04:53:22.715Z,1628744002.715 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T05:01:45.018Z,1628744505.018 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.364676 min 2021-08-12T05:03:17.004Z,1628744597.004 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T050317 2021-08-12T05:03:24.987Z,1628744604.987 [CommandLine](IMPORTANT): got command ! start onESP upsync 2021-08-12T05:03:25.218Z,1628744605.218 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC050325_onESP 2021-08-12T05:07:23.370Z,1628744843.370 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 14.004277 min 2021-08-12T05:09:26.479Z,1628744966.479 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T050927 2021-08-12T05:09:34.883Z,1628744974.883 [CommandLine](IMPORTANT): got command ! check 2021-08-12T05:09:35.049Z,1628744975.049 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Thu Aug 12 05:04:00 UTC 2021 -- ssh esp@10.89.11.3 upsync Uploading real real.* *.spr* *.pcr* *.tif* CTD-*.hex .././messages in /var/log/esp to ESPhonu@ESPshore:/home/ftp/honu/esp shutting ESP off 2021-08-12T05:12:58.057Z,1628745178.057 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 19.582424 min 2021-08-12T05:18:46.938Z,1628745526.938 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.397292 min 2021-08-12T05:22:24.292Z,1628745744.292 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T052225 2021-08-12T05:22:33.547Z,1628745753.547 [CommandLine](IMPORTANT): got command ! onESPclient methane -- spare PhytoSampler: 75 2021-08-12T05:22:33.916Z,1628745753.916 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC052233_onESPclient 2021-08-12T05:24:18.355Z,1628745858.355 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 30.921556 min 2021-08-12T05:25:01.560Z,1628745901.560 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T052502 2021-08-12T05:25:09.623Z,1628745909.623 [CommandLine](IMPORTANT): got command ! check 2021-08-12T05:25:09.865Z,1628745909.865 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Thu Aug 12 05:23:09 UTC 2021 -- ssh esp@10.89.11.3 espclient methane {PhytoSampler: 75, archiveHiBiomass_bac: 1, microSPR_phyto: 1} shutting ESP off 2021-08-12T05:29:55.052Z,1628746195.052 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 36.532568 min 2021-08-12T05:35:43.949Z,1628746543.949 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T053544 2021-08-12T05:35:52.820Z,1628746552.820 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" 2mvjg 1 4.000000 2021-08-12T05:35:52.820Z,1628746552.820 [CommandLine](IMPORTANT): Scheduling command #1 of 4 with id=2mvjg 2021-08-12T05:35:52.821Z,1628746552.821 [CommandLine](IMPORTANT): Scheduled #63 (#1 of 4 with id='2mvjg'): "load Science/esp_sample_at_threshold.xml;set esp_sample_at_threshold.chlorophyllLowerBound 6 ug/l;set esp_sample_at_threshold.chlorophyllUpperBound 100 ug/l;set esp_sample_at_threshold.Lat1 41.837133 degree" ASAP 2021-08-12T05:35:53.784Z,1628746553.784 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T053554 2021-08-12T05:35:54.840Z,1628746554.840 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 42.528357 min 2021-08-12T05:36:05.271Z,1628746565.271 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" 2mvjg 2 4.000000 2021-08-12T05:36:05.272Z,1628746565.272 [CommandLine](IMPORTANT): Scheduling command #2 of 4 with id=2mvjg 2021-08-12T05:36:05.273Z,1628746565.273 [CommandLine](IMPORTANT): Scheduled #64 (#2 of 4 with id='2mvjg'): "set esp_sample_at_threshold.Lon1 -83.351648 degree;set esp_sample_at_threshold.Speed 0.8 m/s;set esp_sample_at_threshold.YoYoMinDepth 1.5 m;set esp_sample_at_threshold.YoYoMaxDepth 2.5 m" ASAP 2021-08-12T05:36:05.850Z,1628746565.850 [DataOverHttps](IMPORTANT): SBD MOMSN=15872495, MTMSN=20210812T053606 2021-08-12T05:36:13.787Z,1628746573.787 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" 2mvjg 3 4.000000 2021-08-12T05:36:13.788Z,1628746573.788 [CommandLine](IMPORTANT): Scheduling command #3 of 4 with id=2mvjg 2021-08-12T05:36:13.790Z,1628746573.790 [CommandLine](IMPORTANT): Scheduled #65 (#3 of 4 with id='2mvjg'): "set esp_sample_at_threshold.YoYoMinAltitude 2 m;set esp_sample_at_threshold.YoYoUpPitch 12 degree;set esp_sample_at_threshold.YoYoDownPitch -12 degree;set esp_sample_at_threshold.TargetDepth 2.5 m" ASAP 2021-08-12T05:36:15.002Z,1628746575.002 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T053615 2021-08-12T05:36:27.197Z,1628746587.197 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" 2mvjg 4 4.000000 2021-08-12T05:36:27.197Z,1628746587.197 [CommandLine](IMPORTANT): Scheduling command #4 of 4 with id=2mvjg 2021-08-12T05:36:27.198Z,1628746587.198 [CommandLine](IMPORTANT): Scheduled #66 (#4 of 4 with id='2mvjg'): "set esp_sample_at_threshold.ESPCartridgeType_2 -18 count;set esp_sample_at_threshold.MinAltitude 0.5 m;set esp_sample_at_threshold.MaxDepth 4.5 m;run" ASAP 2021-08-12T05:36:27.501Z,1628746587.501 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T05:36:32.517Z,1628746592.517 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_at_threshold.xml 2021-08-12T05:36:57.517Z,1628746617.517 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllLowerBound 6.000000 microgram_per_liter 2021-08-12T05:36:57.523Z,1628746617.523 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.chlorophyllUpperBound 100.000000 microgram_per_liter 2021-08-12T05:36:57.523Z,1628746617.523 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lat1 41.837132 degree 2021-08-12T05:37:25.257Z,1628746645.257 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Lon1 -83.351646 degree 2021-08-12T05:37:25.257Z,1628746645.257 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.Speed 0.800000 meter_per_second 2021-08-12T05:37:25.262Z,1628746645.262 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinDepth 1.500000 meter 2021-08-12T05:37:25.263Z,1628746645.263 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMaxDepth 2.500000 meter 2021-08-12T05:37:59.122Z,1628746679.122 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoMinAltitude 2.000000 meter 2021-08-12T05:37:59.127Z,1628746679.127 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoUpPitch 12.000000 degree 2021-08-12T05:37:59.128Z,1628746679.128 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.YoYoDownPitch -12.000000 degree 2021-08-12T05:37:59.129Z,1628746679.129 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.TargetDepth 2.500000 meter 2021-08-12T05:38:36.609Z,1628746716.609 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.ESPCartridgeType_2 -18.000000 count 2021-08-12T05:38:36.615Z,1628746716.615 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MinAltitude 0.500000 meter 2021-08-12T05:38:36.616Z,1628746716.616 [CommandLine](IMPORTANT): got command set esp_sample_at_threshold.MaxDepth 4.500000 meter 2021-08-12T05:38:36.616Z,1628746716.616 [CommandLine](IMPORTANT): got command run 2021-08-12T05:38:36.647Z,1628746716.647 [CommandLine](IMPORTANT): Running 2021-08-12T05:38:36.812Z,1628746716.812 [MissionManager](IMPORTANT): Started mission esp_sample_at_threshold 2021-08-12T05:46:06.953Z,1628747166.953 [esp_sample_at_threshold:SampleESP:sample:A](IMPORTANT): At 3.647969 m , settling for 3.000000 min 2021-08-12T05:49:07.908Z,1628747347.908 [esp_sample_at_threshold:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -4.000000 count 2021-08-12T05:49:19.707Z,1628747359.707 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T05:49:19.831Z,1628747359.831 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T05:49:57.989Z,1628747397.989 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T05:49:58.811Z,1628747398.811 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -4 2021-08-12T05:49:58.812Z,1628747398.812 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T05:49:59.230Z,1628747399.230 [esp_sample_at_threshold:SampleESP:sample:F](IMPORTANT): ESP sampling at 3.471288 m in 25.200373 degC water with 7.853200 ug/l chlorophyll fluorescence. 2021-08-12T05:49:59.602Z,1628747399.602 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T05:50:00.019Z,1628747400.019 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T05:50:02.031Z,1628747402.031 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T05:50:41.653Z,1628747441.653 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@22,archiveHiBiomass_bac) 2021-08-12T05:50:43.251Z,1628747443.251 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T05:50:43.657Z,1628747443.657 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=93.070000 issuedCmd_=Cmd.startFiltering 2021-08-12T05:50:44.055Z,1628747444.055 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T05:53:30.902Z,1628747610.902 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T05:53:31.307Z,1628747611.307 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T06:09:39.357Z,1628748579.357 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@261.5) 2021-08-12T06:09:39.746Z,1628748579.746 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@261.5) 2021-08-12T06:09:41.771Z,1628748581.771 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T06:09:42.170Z,1628748582.170 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@22,archiveHiBiomass_bac) 2021-08-12T06:22:29.399Z,1628749349.399 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@22,archiveHiBiomass_bac) 2021-08-12T06:22:39.078Z,1628749359.078 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T06:22:39.501Z,1628749359.501 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2021-08-12T06:27:39.255Z,1628749659.255 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2021-08-12T06:27:39.650Z,1628749659.650 [ESPComponent](IMPORTANT): [sample #11] ESP log summary report (3 messages): @01:50:12.89 Selecting Cartridge 22 @02:09:36.73 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @02:09:36.93 Sampled 261.5ml 2021-08-12T06:27:40.472Z,1628749660.472 [esp_sample_at_threshold:SampleESP:sample:A](IMPORTANT): At 2.429772 m , settling for 3.000000 min 2021-08-12T06:30:41.462Z,1628749841.462 [esp_sample_at_threshold:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -18.000000 count 2021-08-12T06:30:53.267Z,1628749853.267 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T06:30:53.397Z,1628749853.397 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T06:31:32.362Z,1628749892.362 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T06:31:33.179Z,1628749893.179 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -18 2021-08-12T06:31:33.180Z,1628749893.180 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T06:31:33.603Z,1628749893.603 [esp_sample_at_threshold:SampleESP:sample:F](IMPORTANT): ESP sampling at 2.444027 m in 25.190088 degC water with 6.195560 ug/l chlorophyll fluorescence. 2021-08-12T06:31:33.982Z,1628749893.982 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T06:31:34.391Z,1628749894.391 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T06:31:36.400Z,1628749896.400 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T06:32:16.803Z,1628749936.803 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@17,microSPR_phyto) 2021-08-12T06:32:18.417Z,1628749938.417 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T06:32:18.849Z,1628749938.849 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=14.740000 issuedCmd_=Cmd.startFiltering 2021-08-12T06:32:19.230Z,1628749939.230 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T06:35:08.094Z,1628750108.094 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T06:35:08.494Z,1628750108.494 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T06:38:58.367Z,1628750338.367 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PAUSED (@75.0) 2021-08-12T06:38:58.778Z,1628750338.778 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PROCESSING (@75.0) 2021-08-12T06:39:00.800Z,1628750340.800 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T06:39:01.198Z,1628750341.198 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PROCESSED (@17,microSPR_phyto) 2021-08-12T07:03:52.133Z,1628751832.133 [CBIT](IMPORTANT): Beginning ground fault scan 2021-08-12T07:04:03.085Z,1628751843.085 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): 0.049289 CHAN A1 (24V): 0.032693 CHAN A2 (12V): 0.010516 CHAN A3 (5V): 0.005066 CHAN B0 (3.3V): 0.004918 CHAN B1 (3.15aV): 0.005468 CHAN B2 (3.15bV): 0.009446 CHAN B3 (GND): -0.000075 OPEN: 0.007669 Full Scale Calc: 4.765 mA, -1.589 mA 2021-08-12T07:31:59.239Z,1628753519.239 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@17,microSPR_phyto) 2021-08-12T07:32:10.550Z,1628753530.550 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T07:32:10.947Z,1628753530.947 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED 2021-08-12T07:37:10.728Z,1628753830.728 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence completed normally. 2021-08-12T07:37:11.127Z,1628753831.127 [ESPComponent](IMPORTANT): [sample #12] ESP log summary report (2 messages): @02:31:47.48 Selecting Cartridge 17 @02:38:56.86 Sampled 75.0ml 2021-08-12T07:37:12.354Z,1628753832.354 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T07:44:28.920Z,1628754268.920 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T074429 2021-08-12T07:44:36.611Z,1628754276.611 [CommandLine](IMPORTANT): got command ! start onESP upsync 2021-08-12T07:44:36.909Z,1628754276.909 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC074436_onESP 2021-08-12T07:45:39.324Z,1628754339.324 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.442945 min 2021-08-12T07:47:32.544Z,1628754452.544 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T074733 2021-08-12T07:47:40.610Z,1628754460.610 [CommandLine](IMPORTANT): got command ! check 2021-08-12T07:47:41.034Z,1628754461.034 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Thu Aug 12 07:45:12 UTC 2021 -- ssh esp@10.89.11.3 upsync Uploading real real.* *.spr* *.pcr* *.tif* CTD-*.hex .././messages in /var/log/esp to ESPhonu@ESPshore:/home/ftp/honu/esp shutting ESP off 2021-08-12T07:51:41.554Z,1628754701.554 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 14.480081 min 2021-08-12T07:57:34.443Z,1628755054.443 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T075735 2021-08-12T07:57:37.353Z,1628755057.353 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.410046 min 2021-08-12T07:57:42.155Z,1628755062.155 [CommandLine](IMPORTANT): got command ! onESPclient methane -- Cmd.runStandard 2021-08-12T07:57:42.585Z,1628755062.585 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC075742_onESPclient 2021-08-12T08:02:11.933Z,1628755331.933 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T080212 2021-08-12T08:02:23.191Z,1628755343.191 [CommandLine](IMPORTANT): got command schedule asap "load Science/sci2.xml;set sci2.MissionTimeout 10 h;set sci2.NeedCommsTime 30 min;set sci2.Lat1 41.735083333333 degree;set sci2.Lon1 -83.30071 degree;set sci2.Speed 0.8 m/s;set sci2.YoYoMinDepth 1.5 m;set sci2.YoYoMaxDepth 2.5 m" 2n2b2 1 2.000000 2021-08-12T08:02:23.193Z,1628755343.193 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2n2b2 2021-08-12T08:02:23.194Z,1628755343.194 [CommandLine](IMPORTANT): Scheduled #67 (#1 of 2 with id='2n2b2'): "load Science/sci2.xml;set sci2.MissionTimeout 10 h;set sci2.NeedCommsTime 30 min;set sci2.Lat1 41.735083333333 degree;set sci2.Lon1 -83.30071 degree;set sci2.Speed 0.8 m/s;set sci2.YoYoMinDepth 1.5 m;set sci2.YoYoMaxDepth 2.5 m" ASAP 2021-08-12T08:02:24.093Z,1628755344.093 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T080224 2021-08-12T08:02:33.884Z,1628755353.884 [CommandLine](IMPORTANT): got command schedule asap "set sci2.YoYoMinAltitude 2 m;set sci2.YoYoUpPitch 12 degree;set sci2.YoYoDownPitch -12 degree;set sci2.MinAltitude 0.5 m;set sci2.MaxDepth 4 m;set sci2.MinOffshore 1 km;run" 2n2b2 2 2.000000 2021-08-12T08:02:33.885Z,1628755353.885 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2n2b2 2021-08-12T08:02:33.886Z,1628755353.886 [CommandLine](IMPORTANT): Scheduled #68 (#2 of 2 with id='2n2b2'): "set sci2.YoYoMinAltitude 2 m;set sci2.YoYoUpPitch 12 degree;set sci2.YoYoDownPitch -12 degree;set sci2.MinAltitude 0.5 m;set sci2.MaxDepth 4 m;set sci2.MinOffshore 1 km;run" ASAP 2021-08-12T08:02:33.959Z,1628755353.959 [CommandLine](IMPORTANT): got command load ./Missions/Science/sci2.xml 2021-08-12T08:02:36.418Z,1628755356.418 [CommandLine](IMPORTANT): Loaded ./Missions/Science/sci2.xml 2021-08-12T08:02:50.937Z,1628755370.937 [CommandLine](IMPORTANT): got command set sci2.MissionTimeout 10.000000 hour 2021-08-12T08:02:50.938Z,1628755370.938 [CommandLine](IMPORTANT): got command set sci2.NeedCommsTime 30.000000 minute 2021-08-12T08:02:50.939Z,1628755370.939 [CommandLine](IMPORTANT): got command set sci2.Lat1 41.735085 degree 2021-08-12T08:02:50.940Z,1628755370.940 [CommandLine](IMPORTANT): got command set sci2.Lon1 -83.300713 degree 2021-08-12T08:02:50.940Z,1628755370.940 [CommandLine](IMPORTANT): got command set sci2.Speed 0.800000 meter_per_second 2021-08-12T08:02:50.941Z,1628755370.941 [CommandLine](IMPORTANT): got command set sci2.YoYoMinDepth 1.500000 meter 2021-08-12T08:02:50.942Z,1628755370.942 [CommandLine](IMPORTANT): got command set sci2.YoYoMaxDepth 2.500000 meter 2021-08-12T08:03:08.711Z,1628755388.711 [CommandLine](IMPORTANT): got command set sci2.YoYoMinAltitude 2.000000 meter 2021-08-12T08:03:08.712Z,1628755388.712 [CommandLine](IMPORTANT): got command set sci2.YoYoUpPitch 12.000000 degree 2021-08-12T08:03:08.713Z,1628755388.713 [CommandLine](IMPORTANT): got command set sci2.YoYoDownPitch -12.000000 degree 2021-08-12T08:03:08.714Z,1628755388.714 [CommandLine](IMPORTANT): got command set sci2.MinAltitude 0.500000 meter 2021-08-12T08:03:08.714Z,1628755388.714 [CommandLine](IMPORTANT): got command set sci2.MaxDepth 4.000000 meter 2021-08-12T08:03:08.739Z,1628755388.739 [CommandLine](IMPORTANT): got command set sci2.MinOffshore 1.000000 kilometer 2021-08-12T08:03:08.739Z,1628755388.739 [CommandLine](IMPORTANT): got command run 2021-08-12T08:03:08.741Z,1628755388.741 [CommandLine](IMPORTANT): Running 2021-08-12T08:03:09.055Z,1628755389.055 [MissionManager](IMPORTANT): Started mission sci2 2021-08-12T10:20:32.063Z,1628763632.063 [ElevatorServo](FAULT): Reading outside of valid range:0.000000 2021-08-12T10:20:32.444Z,1628763632.444 [ElevatorServo](FAULT): Command Overflow Error 2021-08-12T10:20:32.864Z,1628763632.864 [ElevatorServo](FAULT): Invalid EZ Servo response:"&\246\323\r" 2021-08-12T10:23:03.655Z,1628763783.655 [Radio_Surface](FAULT): LCB fault: Invalid Command. Hardware Overcurrent Shutdown. Software Overcurrent. 2021-08-12T12:00:35.199Z,1628769635.199 [Radio_Surface](FAULT): LCB fault: Hardware Overcurrent Shutdown. Software Overcurrent. 2021-08-12T13:04:03.115Z,1628773443.115 [CBIT](IMPORTANT): Beginning ground fault scan 2021-08-12T13:04:13.999Z,1628773453.999 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): 0.011838 CHAN A1 (24V): 0.017231 CHAN A2 (12V): 0.009163 CHAN A3 (5V): 0.003021 CHAN B0 (3.3V): 0.001885 CHAN B1 (3.15aV): 0.002272 CHAN B2 (3.15bV): 0.002275 CHAN B3 (GND): 0.000749 OPEN: 0.007895 Full Scale Calc: 4.765 mA, -1.589 mA 2021-08-12T13:24:27.502Z,1628774667.502 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached Waypoint: 41.735085,-83.300714 2021-08-12T13:24:30.331Z,1628774670.331 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T13:32:09.319Z,1628775129.319 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 7.643416 min 2021-08-12T13:37:37.116Z,1628775457.116 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T133737 2021-08-12T13:37:45.568Z,1628775465.568 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 13.247702 min 2021-08-12T13:37:48.691Z,1628775468.691 [CommandLine](IMPORTANT): got command ! onESPclient methane -- spare PhytoSampler: 75 2021-08-12T13:37:49.743Z,1628775469.743 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC133749_onESPclient 2021-08-12T13:43:55.593Z,1628775835.593 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 19.414278 min 2021-08-12T13:47:09.871Z,1628776029.871 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T134710 2021-08-12T13:47:17.623Z,1628776037.623 [CommandLine](IMPORTANT): got command ! check 2021-08-12T13:47:17.933Z,1628776037.933 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Thu Aug 12 13:38:26 UTC 2021 -- ssh esp@10.89.11.3 espclient methane {PhytoSampler: 75, archiveHiBiomass_bac: 1, microSPR_phyto: 1} shutting ESP off 2021-08-12T13:49:32.336Z,1628776172.336 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.026685 min 2021-08-12T13:55:46.165Z,1628776546.165 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 31.256954 min 2021-08-12T13:55:46.773Z,1628776546.773 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T135547 2021-08-12T13:55:57.275Z,1628776557.275 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 3.5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ApproachSpeed 0 m/s" 2nioa 1 2.000000 2021-08-12T13:55:57.276Z,1628776557.276 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2nioa 2021-08-12T13:55:57.278Z,1628776557.278 [CommandLine](IMPORTANT): Scheduled #69 (#1 of 2 with id='2nioa'): "load Science/esp_sample_at_depth.xml;set esp_sample_at_depth.MissionTimeout 3.5 h;set esp_sample_at_depth.TargetDepth 2.0 m;set esp_sample_at_depth.ESPCartridgeType_2 -18 count;set esp_sample_at_depth.ApproachSpeed 0 m/s" ASAP 2021-08-12T13:55:58.719Z,1628776558.719 [DataOverHttps](IMPORTANT): SBD MOMSN=15873318, MTMSN=20210812T135559 2021-08-12T13:56:08.173Z,1628776568.173 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_at_depth.MinAltitude 1 m;set esp_sample_at_depth.MaxDepth 4.8 m;set esp_sample_at_depth:StandardEnvelopes.MinAltitude 3 m;run" 2nioa 2 2.000000 2021-08-12T13:56:08.174Z,1628776568.174 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2nioa 2021-08-12T13:56:08.175Z,1628776568.175 [CommandLine](IMPORTANT): Scheduled #70 (#2 of 2 with id='2nioa'): "set esp_sample_at_depth.MinAltitude 1 m;set esp_sample_at_depth.MaxDepth 4.8 m;set esp_sample_at_depth:StandardEnvelopes.MinAltitude 3 m;run" ASAP 2021-08-12T13:56:08.271Z,1628776568.271 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_at_depth.xml 2021-08-12T13:56:10.584Z,1628776570.584 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_at_depth.xml 2021-08-12T13:56:47.416Z,1628776607.416 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MissionTimeout 3.500000 hour 2021-08-12T13:56:47.417Z,1628776607.417 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.TargetDepth 2.000000 meter 2021-08-12T13:56:47.417Z,1628776607.417 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ESPCartridgeType_2 -18.000000 count 2021-08-12T13:56:47.418Z,1628776607.418 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.ApproachSpeed 0.000000 meter_per_second 2021-08-12T13:57:08.226Z,1628776628.226 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MinAltitude 1.000000 meter 2021-08-12T13:57:08.227Z,1628776628.227 [CommandLine](IMPORTANT): got command set esp_sample_at_depth.MaxDepth 4.800000 meter 2021-08-12T13:57:08.228Z,1628776628.228 [CommandLine](IMPORTANT): got command set esp_sample_at_depth:StandardEnvelopes.MinAltitude 3.000000 meter 2021-08-12T13:57:08.229Z,1628776628.229 [CommandLine](IMPORTANT): got command run 2021-08-12T13:57:08.231Z,1628776628.231 [CommandLine](IMPORTANT): Running 2021-08-12T13:57:08.352Z,1628776628.352 [MissionManager](IMPORTANT): Started mission esp_sample_at_depth 2021-08-12T14:03:08.154Z,1628776988.154 [esp_sample_at_depth:SampleESP:sample:A](IMPORTANT): At 2.831115 m , settling for 3.000000 min 2021-08-12T14:06:09.148Z,1628777169.148 [esp_sample_at_depth:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -4.000000 count 2021-08-12T14:06:20.963Z,1628777180.963 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T14:06:21.105Z,1628777181.105 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T14:06:58.825Z,1628777218.825 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T14:06:59.654Z,1628777219.654 [ESPComponent](IMPORTANT): [sample #13] Setting cartridge argument to -4 2021-08-12T14:06:59.655Z,1628777219.655 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T14:07:00.061Z,1628777220.061 [esp_sample_at_depth:SampleESP:sample:F](IMPORTANT): ESP sampling at 2.914057 m in 24.997095 degC water with 1.240040 ug/l chlorophyll fluorescence. 2021-08-12T14:07:00.442Z,1628777220.442 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T14:07:00.843Z,1628777220.843 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T14:07:02.863Z,1628777222.863 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T14:07:42.470Z,1628777262.470 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PRIMING (@20,archiveHiBiomass_bac) 2021-08-12T14:07:44.078Z,1628777264.078 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T14:07:44.477Z,1628777264.477 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=93.070000 issuedCmd_=Cmd.startFiltering 2021-08-12T14:07:44.882Z,1628777264.882 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T14:10:31.745Z,1628777431.745 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T14:10:32.142Z,1628777432.142 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T14:26:03.035Z,1628778363.035 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PAUSED (@242.3) 2021-08-12T14:26:03.386Z,1628778363.386 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PROCESSING (@242.3) 2021-08-12T14:26:05.408Z,1628778365.408 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T14:26:05.810Z,1628778365.810 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PROCESSED (@20,archiveHiBiomass_bac) 2021-08-12T14:38:53.016Z,1628779133.016 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPING (@20,archiveHiBiomass_bac) 2021-08-12T14:39:02.307Z,1628779142.307 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T14:39:02.715Z,1628779142.715 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPED 2021-08-12T14:44:02.481Z,1628779442.481 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence completed normally. 2021-08-12T14:44:02.889Z,1628779442.889 [ESPComponent](IMPORTANT): [sample #13] ESP log summary report (3 messages): @10:07:13.37 Selecting Cartridge 20 @10:25:59.82 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @10:26:00.01 Sampled 242.3ml 2021-08-12T14:44:03.731Z,1628779443.731 [esp_sample_at_depth:SampleESP:sample:A](IMPORTANT): At 2.206025 m , settling for 3.000000 min 2021-08-12T14:47:04.730Z,1628779624.730 [esp_sample_at_depth:SampleESP:sample:C](IMPORTANT): Requesting ESP cartridge type: -18.000000 count 2021-08-12T14:47:16.526Z,1628779636.526 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-08-12T14:47:16.643Z,1628779636.643 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-08-12T14:47:55.195Z,1628779675.195 [ESPComponent](IMPORTANT): ESP has connected as client 2021-08-12T14:47:56.004Z,1628779676.004 [ESPComponent](IMPORTANT): [sample #14] Setting cartridge argument to -18 2021-08-12T14:47:56.005Z,1628779676.005 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-08-12T14:47:56.442Z,1628779676.442 [esp_sample_at_depth:SampleESP:sample:F](IMPORTANT): ESP sampling at 2.223736 m in 25.091516 degC water with 1.323560 ug/l chlorophyll fluorescence. 2021-08-12T14:47:56.820Z,1628779676.820 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_LOG 2021-08-12T14:47:57.218Z,1628779677.218 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-08-12T14:47:59.238Z,1628779679.238 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_LOADING_CARTRIDGE 2021-08-12T14:48:39.649Z,1628779719.649 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PRIMING (@15,microSPR_phyto) 2021-08-12T14:48:41.271Z,1628779721.271 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PRIMING (@100.0) 2021-08-12T14:48:41.655Z,1628779721.655 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=14.740000 issuedCmd_=Cmd.startFiltering 2021-08-12T14:48:42.068Z,1628779722.068 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-08-12T14:51:31.344Z,1628779891.344 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_FILTERING (@0.0) 2021-08-12T14:51:31.763Z,1628779891.763 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-08-12T14:55:32.122Z,1628780132.122 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PAUSED (@75.0) 2021-08-12T14:55:32.531Z,1628780132.531 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PROCESSING (@75.0) 2021-08-12T14:55:34.547Z,1628780134.547 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing 2021-08-12T14:55:34.950Z,1628780134.950 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PROCESSED (@15,microSPR_phyto) 2021-08-12T15:24:50.335Z,1628781890.335 [MassServo](FAULT): Mass Shifter uart error: serial timeout 2021-08-12T15:48:52.663Z,1628783332.663 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPING (@15,microSPR_phyto) 2021-08-12T15:49:03.977Z,1628783343.977 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2021-08-12T15:49:04.385Z,1628783344.385 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPED 2021-08-12T15:54:04.170Z,1628783644.170 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence completed normally. 2021-08-12T15:54:04.570Z,1628783644.570 [ESPComponent](IMPORTANT): [sample #14] ESP log summary report (2 messages): @10:48:10.13 Selecting Cartridge 15 @10:55:30.15 Sampled 75.0ml 2021-08-12T15:54:05.810Z,1628783645.810 [MissionManager](IMPORTANT): Started mission Default 2021-08-12T16:02:03.472Z,1628784123.472 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T160204 2021-08-12T16:02:11.164Z,1628784131.164 [CommandLine](IMPORTANT): got command ! start onESP upsync 2021-08-12T16:02:11.442Z,1628784131.442 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-08-12UTC160211_onESP 2021-08-12T16:02:19.304Z,1628784139.304 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.218429 min 2021-08-12T16:07:56.536Z,1628784476.536 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 13.838746 min 2021-08-12T16:09:53.827Z,1628784593.827 [DataOverHttps](IMPORTANT): SBD MTMSN=20210812T160954 2021-08-12T16:10:04.191Z,1628784604.191 [CommandLine](IMPORTANT): got command ! check 2021-08-12T16:10:04.387Z,1628784604.387 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected. Thu Aug 12 16:02:46 UTC 2021 -- ssh esp@10.89.11.3 upsync Uploading real real.* *.spr* *.pcr* *.tif* CTD-*.hex .././messages in /var/log/esp to ESPhonu@ESPshore:/home/ftp/honu/esp shutting ESP off 2021-08-12T16:13:24.575Z,1628784804.575 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 19.306026 min 2021-08-12T16:19:19.512Z,1628785159.512 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.222229 min 2021-08-12T16:24:45.263Z,1628785485.263 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 30.650616 min 2021-08-12T16:30:17.872Z,1628785817.872 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 36.194755 min 2021-08-12T16:35:51.716Z,1628786151.716 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 41.758687 min 2021-08-12T16:41:31.649Z,1628786491.649 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 47.423759 min 2021-08-12T16:47:07.136Z,1628786827.136 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 53.015837 min