2021-04-20T18:52:22.359Z,1618944742.359 [CommandLine](IMPORTANT): got command restart logs 2021-04-20T18:53:52.868Z,1618944832.868 [DataOverHttps](IMPORTANT): SBD MTMSN=20210420T185352 2021-04-20T18:54:00.477Z,1618944840.477 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_station.xml 2021-04-20T18:54:05.038Z,1618944845.038 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_station.xml 2021-04-20T18:54:26.318Z,1618944866.318 [CommandLine](IMPORTANT): got command set esp_sample_station.MissionTimeout 48.000000 hour 2021-04-20T18:54:26.319Z,1618944866.319 [CommandLine](IMPORTANT): got command set esp_sample_station.LapRepeat 4.000000 count 2021-04-20T18:54:26.321Z,1618944866.321 [CommandLine](IMPORTANT): got command set esp_sample_station.TransitYoYoMinDepth 3.000000 meter 2021-04-20T18:54:26.321Z,1618944866.321 [CommandLine](IMPORTANT): got command set esp_sample_station.SampleYoYoMinDepth 3.000000 meter 2021-04-20T18:54:26.322Z,1618944866.322 [CommandLine](IMPORTANT): got command run 2021-04-20T18:54:26.325Z,1618944866.325 [CommandLine](IMPORTANT): Running 2021-04-20T18:54:26.516Z,1618944866.516 [MissionManager](IMPORTANT): Started mission esp_sample_station 2021-04-20T18:55:35.777Z,1618944935.777 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2021-04-20T18:55:36.241Z,1618944936.241 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2021-04-20T18:55:36.577Z,1618944936.577 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2021-04-20T18:55:36.980Z,1618944936.980 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2021-04-20T18:55:37.353Z,1618944937.353 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2021-04-20T18:55:41.836Z,1618944941.836 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 2021-04-20T19:00:36.716Z,1618945236.716 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2021-04-20T19:23:42.374Z,1618946622.374 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000 2021-04-20T19:28:54.295Z,1618946934.295 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2021-04-20T19:28:55.870Z,1618946935.870 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::' 2021-04-20T19:29:06.018Z,1618946946.018 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-04-20T19:29:06.115Z,1618946946.115 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-04-20T19:29:53.245Z,1618946993.245 [ESPComponent](IMPORTANT): ESP has connected as client 2021-04-20T19:29:54.070Z,1618946994.070 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -6 2021-04-20T19:29:54.071Z,1618946994.071 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-04-20T19:29:54.483Z,1618946994.483 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 17.163319 m in 11.214106 degC water with 11.298400 ug/l chlorophyll fluorescence. 2021-04-20T19:29:55.265Z,1618946995.265 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2021-04-20T19:29:55.673Z,1618946995.673 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-04-20T19:29:57.285Z,1618946997.285 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2021-04-20T19:30:31.642Z,1618947031.642 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@12,archive_bac) 2021-04-20T19:30:33.240Z,1618947033.240 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2021-04-20T19:30:34.072Z,1618947034.072 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-04-20T19:33:07.188Z,1618947187.188 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2021-04-20T19:33:07.572Z,1618947187.572 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-04-20T20:04:28.273Z,1618949068.273 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@499.4) 2021-04-20T20:04:28.673Z,1618949068.673 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@499.4) 2021-04-20T20:04:31.104Z,1618949071.104 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@12,archive_bac) 2021-04-20T20:16:34.677Z,1618949794.677 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@12,archive_bac) 2021-04-20T20:16:47.199Z,1618949807.199 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2021-04-20T20:16:47.610Z,1618949807.610 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2021-04-20T20:21:47.379Z,1618950107.379 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2021-04-20T20:21:47.805Z,1618950107.805 [ESPComponent](IMPORTANT): [sample #1] ESP log summary report (3 messages): @12:30:07.37 Selecting Cartridge 12 @13:04:25.25 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @13:04:25.44 Sampled 499.4ml 2021-04-20T20:27:36.965Z,1618950456.965 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2021-04-20T22:18:10.151Z,1618957090.151 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600 2021-04-20T22:22:59.773Z,1618957379.773 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2021-04-20T22:23:11.570Z,1618957391.570 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2021-04-20T22:23:11.698Z,1618957391.698 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2021-04-20T22:23:59.616Z,1618957439.616 [ESPComponent](IMPORTANT): ESP has connected as client 2021-04-20T22:24:00.441Z,1618957440.441 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -6 2021-04-20T22:24:00.442Z,1618957440.442 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2021-04-20T22:24:00.930Z,1618957440.930 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 21.244041 m in 9.490808 degC water with 0.475600 ug/l chlorophyll fluorescence. 2021-04-20T22:24:01.236Z,1618957441.236 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2021-04-20T22:24:01.645Z,1618957441.645 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2021-04-20T22:24:03.675Z,1618957443.675 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2021-04-20T22:24:37.997Z,1618957477.997 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@11,archive_bac) 2021-04-20T22:24:39.622Z,1618957479.622 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2021-04-20T22:24:40.458Z,1618957480.458 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2021-04-20T22:27:11.935Z,1618957631.935 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2021-04-20T22:27:12.325Z,1618957632.325 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2021-04-20T22:52:52.602Z,1618959172.602 [ElevatorServo](FAULT): Elevator uart error: serial timeout 2021-04-20T23:21:14.099Z,1618960874.099 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@1000.0) 2021-04-20T23:21:14.496Z,1618960874.496 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@1000.0) 2021-04-20T23:21:16.936Z,1618960876.936 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@11,archive_bac) 2021-04-20T23:30:32.682Z,1618961432.682 [Radio_Surface](FAULT): LCB fault: LCB Watchdog Reset. 2021-04-20T23:33:20.521Z,1618961600.521 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@11,archive_bac) 2021-04-20T23:33:33.046Z,1618961613.046 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2021-04-20T23:33:33.445Z,1618961613.445 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2021-04-20T23:38:33.222Z,1618961913.222 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2021-04-20T23:38:33.620Z,1618961913.620 [ESPComponent](IMPORTANT): [sample #2] ESP log summary report (2 messages): @15:24:13.91 Selecting Cartridge 11 @16:21:12.67 Sampled 1000.0ml 2021-04-20T23:42:14.221Z,1618962134.221 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=2729, MT Status=1, MTMSN=197 2021-04-20T23:42:18.763Z,1618962138.763 [CommandLine](IMPORTANT): got command configSet Express none WetLabsBB2FL.bin_mean_mass_concentration_of_chlorophyll_in_sea_water 2021-04-20T23:42:18.765Z,1618962138.765 [CommandLine](IMPORTANT): got command configSet Express linearApproximation WetLabsBB2FL.bin_median_mass_concentration_of_chlorophyll_in_sea_water 0.200000 microgram_per_liter persist 2021-04-20T23:45:11.774Z,1618962311.774 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 2021-04-20T23:55:03.262Z,1618962903.262 [CBIT](IMPORTANT): Beginning ground fault scan 2021-04-20T23:55:14.245Z,1618962914.245 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.004388 CHAN A1 (24V): -0.001094 CHAN A2 (12V): -0.000175 CHAN A3 (5V): -0.000834 CHAN B0 (3.3V): 0.000337 CHAN B1 (3.15aV): 0.000282 CHAN B2 (3.15bV): -0.000359 CHAN B3 (GND): 0.000237 OPEN: 0.007696 Full Scale Calc: 4.765 mA, -1.589 mA 2021-04-21T01:25:31.966Z,1618968331.966 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=2739, MT Status=1, MTMSN=198