2020-10-08T15:47:06.313Z,1602172026.313 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg 2020-10-08T15:47:09.864Z,1602172029.864 [SBIT](IMPORTANT): git: 2020-10-07_A 2020-10-08T15:47:09.865Z,1602172029.865 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8 2020-10-08T15:47:10.561Z,1602172030.561 [CommandLine](IMPORTANT): got command ! echo 1 > /dev/loadB2 2020-10-08T15:47:11.518Z,1602172031.518 [AMEcho](IMPORTANT): Setting 'enabled' to 0 2020-10-08T15:48:26.288Z,1602172106.288 [SBIT](IMPORTANT): Beginning Startup BIT 2020-10-08T15:48:26.292Z,1602172106.292 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-08T15:48:37.224Z,1602172117.224 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.001657 CHAN A1 (24V): 0.008665 CHAN A2 (12V): -0.001265 CHAN A3 (5V): -0.001157 CHAN B0 (3.3V): -0.000246 CHAN B1 (3.15aV): -0.000251 CHAN B2 (3.15bV): -0.000525 CHAN B3 (GND): 0.000121 OPEN: 0.005239 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-08T15:49:20.024Z,1602172160.024 [SBIT](IMPORTANT): SBIT PASSED 2020-10-08T15:49:20.025Z,1602172160.025 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2020-10-08T15:49:20.026Z,1602172160.026 [SBIT](IMPORTANT): AMEcho.enabled=0 bool; 2020-10-08T15:49:20.026Z,1602172160.026 [SBIT](IMPORTANT): Aanderaa_O2.loadAtStartup=0 bool; 2020-10-08T15:49:20.026Z,1602172160.026 [SBIT](IMPORTANT): BPC1.batteryMissingStickThreshold=2 count; 2020-10-08T15:49:20.026Z,1602172160.026 [SBIT](IMPORTANT): CANONSampler.loadAtStartup=1 bool; 2020-10-08T15:49:20.026Z,1602172160.026 [SBIT](IMPORTANT): CANONSampler.rotateOnly=0 bool; 2020-10-08T15:49:20.026Z,1602172160.026 [SBIT](IMPORTANT): CANONSampler.simulateHardware=0 bool; 2020-10-08T15:49:20.026Z,1602172160.026 [SBIT](IMPORTANT): CBIT.gf24Offset=148.5 microampere; 2020-10-08T15:49:20.026Z,1602172160.026 [SBIT](IMPORTANT): DVL_micro.loadAtStartup=0 bool; 2020-10-08T15:49:20.027Z,1602172160.027 [SBIT](IMPORTANT): ESPComponent.loadAtStartup=1 bool; 2020-10-08T15:49:20.027Z,1602172160.027 [SBIT](IMPORTANT): ESPComponent.sampleTimeout=25 minute; 2020-10-08T15:49:20.027Z,1602172160.027 [SBIT](IMPORTANT): ESPComponent.simulateHardware=0 bool; 2020-10-08T15:49:20.027Z,1602172160.027 [SBIT](IMPORTANT): Express none WetLabsBB2FL.bin_mean_mass_concentration_of_chlorophyll_in_sea_water; 2020-10-08T15:49:20.027Z,1602172160.027 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=280 cubic_centimeter; 2020-10-08T15:49:20.027Z,1602172160.027 [SBIT](IMPORTANT): VerticalControl.massDefault=14 millimeter; 2020-10-08T15:49:20.420Z,1602172160.420 [MissionManager](IMPORTANT): Started mission Startup 2020-10-08T15:50:23.471Z,1602172223.471 [MissionManager](IMPORTANT): Started mission Default 2020-10-08T15:50:46.399Z,1602172246.399 [DataOverHttps](IMPORTANT): SBD MTMSN=20201008T155045 2020-10-08T15:50:53.744Z,1602172253.744 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_station.xml 2020-10-08T15:50:56.209Z,1602172256.209 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_station.xml 2020-10-08T15:51:00.453Z,1602172260.453 [CommandLine](IMPORTANT): got command set esp_sample_station.NeedCommsTime 60.000000 minute 2020-10-08T15:51:00.454Z,1602172260.454 [CommandLine](IMPORTANT): got command set esp_sample_station.Speed 0.750000 meter_per_second 2020-10-08T15:51:00.454Z,1602172260.454 [CommandLine](IMPORTANT): got command run 2020-10-08T15:51:00.457Z,1602172260.457 [CommandLine](IMPORTANT): Running 2020-10-08T15:51:00.759Z,1602172260.759 [MissionManager](IMPORTANT): Started mission esp_sample_station 2020-10-08T15:52:31.930Z,1602172351.930 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-08T15:52:32.328Z,1602172352.328 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-08T15:52:32.741Z,1602172352.741 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-08T15:52:33.174Z,1602172353.174 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-08T15:52:33.567Z,1602172353.567 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-08T15:52:37.565Z,1602172357.565 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 2020-10-08T15:57:33.581Z,1602172653.581 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2020-10-08T16:42:51.845Z,1602175371.845 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000 2020-10-08T16:44:45.861Z,1602175485.861 [NAL9602](FAULT): received: +CSQ:0 OK283, 2, 0, 0, 0 OK 2020-10-08T16:49:32.633Z,1602175772.633 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-08T16:49:34.202Z,1602175774.202 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::' 2020-10-08T16:49:44.345Z,1602175784.345 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-08T16:49:44.630Z,1602175784.630 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-08T16:50:22.761Z,1602175822.761 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-08T16:50:23.676Z,1602175823.676 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -6 2020-10-08T16:50:23.677Z,1602175823.677 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-08T16:50:24.009Z,1602175824.009 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 14.365960 m in 12.962915 degC water with 2.124400 ug/l chlorophyll fluorescence. 2020-10-08T16:50:24.781Z,1602175824.781 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-08T16:50:25.186Z,1602175825.186 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-08T16:50:26.802Z,1602175826.802 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-08T16:50:59.934Z,1602175859.934 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@58,archive_bac) 2020-10-08T16:51:01.550Z,1602175861.550 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2020-10-08T16:51:03.167Z,1602175863.167 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-08T16:53:31.439Z,1602176011.439 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2020-10-08T16:53:31.842Z,1602176011.842 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-08T17:51:44.673Z,1602179504.673 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@1000.0) 2020-10-08T17:51:45.078Z,1602179505.078 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@1000.0) 2020-10-08T17:51:47.906Z,1602179507.906 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@58,archive_bac) 2020-10-08T18:03:51.518Z,1602180231.518 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@58,archive_bac) 2020-10-08T18:04:16.975Z,1602180256.975 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-08T18:04:17.380Z,1602180257.380 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2020-10-08T18:09:17.165Z,1602180557.165 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2020-10-08T18:09:17.570Z,1602180557.570 [ESPComponent](IMPORTANT): [sample #1] ESP log summary report (2 messages): @09:50:36.75 Selecting Cartridge 58 @10:51:43.19 Sampled 1000.0ml 2020-10-08T18:15:03.717Z,1602180903.717 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2020-10-08T18:15:36.921Z,1602180936.921 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:19:54.238Z,1602181194.238 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:21:03.317Z,1602181263.317 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:21:43.329Z,1602181303.329 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:22:02.725Z,1602181322.725 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:22:30.585Z,1602181350.585 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:25:13.413Z,1602181513.413 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:25:34.973Z,1602181534.973 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:25:53.040Z,1602181553.040 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:25:59.912Z,1602181559.912 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:26:06.765Z,1602181566.765 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:26:13.681Z,1602181573.681 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:26:20.565Z,1602181580.565 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:26:20.569Z,1602181580.569 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T18:26:20.900Z,1602181580.900 [CommandLine](FAULT): Scheduling is paused 2020-10-08T18:26:21.651Z,1602181581.651 [MissionManager](IMPORTANT): Started mission Default 2020-10-08T18:26:56.097Z,1602181616.097 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:27:02.961Z,1602181622.961 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:27:04.953Z,1602181624.953 [BuoyancyServo](FAULT): Buoyancy getPosition uart error.serial timeout 2020-10-08T18:27:09.437Z,1602181629.437 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:27:16.251Z,1602181636.251 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:35:05.977Z,1602182105.977 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.732123 min 2020-10-08T18:37:12.994Z,1602182232.994 [DataOverHttps](IMPORTANT): SBD MTMSN=20201008T183711 2020-10-08T18:37:20.236Z,1602182240.236 [CommandLine](IMPORTANT): got command load ./Missions/Transport/keepstation.xml 2020-10-08T18:37:20.785Z,1602182240.785 [CommandLine](IMPORTANT): Loaded ./Missions/Transport/keepstation.xml 2020-10-08T18:37:27.856Z,1602182247.856 [CommandLine](IMPORTANT): got command set keepstation.MissionTimeout 20.000000 minute 2020-10-08T18:37:27.857Z,1602182247.857 [CommandLine](IMPORTANT): got command set keepstation.NeedCommsTime 20.000000 minute 2020-10-08T18:37:27.858Z,1602182247.858 [CommandLine](IMPORTANT): got command set keepstation.Latitude 36.801468 degree 2020-10-08T18:37:27.858Z,1602182247.858 [CommandLine](IMPORTANT): got command set keepstation.Longitude -121.842949 degree 2020-10-08T18:37:27.859Z,1602182247.859 [CommandLine](IMPORTANT): got command run 2020-10-08T18:37:27.864Z,1602182247.864 [CommandLine](IMPORTANT): Running 2020-10-08T18:37:28.191Z,1602182248.191 [MissionManager](IMPORTANT): Started mission keepstation 2020-10-08T18:38:24.473Z,1602182304.473 [ThrusterServo](FAULT): Overload Error 2020-10-08T18:38:24.477Z,1602182304.477 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T18:38:25.563Z,1602182305.563 [MissionManager](IMPORTANT): Started mission Default 2020-10-08T18:44:11.796Z,1602182651.796 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 5.763749 min 2020-10-08T18:44:22.456Z,1602182662.456 [DataOverHttps](IMPORTANT): SBD MOMSN=12825381, MTMSN=20201008T184421 2020-10-08T18:44:29.718Z,1602182669.718 [CommandLine](IMPORTANT): got command load ./Missions/Maintenance/ballast_and_trim.xml 2020-10-08T18:44:31.988Z,1602182671.988 [CommandLine](IMPORTANT): Loaded ./Missions/Maintenance/ballast_and_trim.xml 2020-10-08T18:44:36.923Z,1602182676.923 [CommandLine](IMPORTANT): got command set ballast_and_trim.MissionTimeout 30.000000 minute 2020-10-08T18:44:36.924Z,1602182676.924 [CommandLine](IMPORTANT): got command set ballast_and_trim.ApproachSpeed 0.000000 meter_per_second 2020-10-08T18:44:36.924Z,1602182676.924 [CommandLine](IMPORTANT): got command run 2020-10-08T18:44:36.928Z,1602182676.928 [CommandLine](IMPORTANT): Running 2020-10-08T18:44:37.263Z,1602182677.263 [MissionManager](IMPORTANT): Started mission ballast_and_trim 2020-10-08T19:14:37.743Z,1602184477.743 [MissionManager](IMPORTANT): Started mission Default 2020-10-08T19:14:42.321Z,1602184482.321 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:14:49.261Z,1602184489.261 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:14:56.037Z,1602184496.037 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:15:02.585Z,1602184502.585 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:15:09.337Z,1602184509.337 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:15:09.345Z,1602184509.345 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T19:15:52.541Z,1602184552.541 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:16:02.649Z,1602184562.649 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:16:11.921Z,1602184571.921 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:16:19.197Z,1602184579.197 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:16:26.069Z,1602184586.069 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:16:26.073Z,1602184586.073 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T19:17:01.633Z,1602184621.633 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:17:08.489Z,1602184628.489 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:17:15.357Z,1602184635.357 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:17:22.221Z,1602184642.221 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:17:29.097Z,1602184649.097 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:17:29.100Z,1602184649.100 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T19:18:04.620Z,1602184684.620 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:18:13.916Z,1602184693.916 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:18:14.693Z,1602184694.693 [ThrusterServo](FAULT): LCB fault: Software Overcurrent. 2020-10-08T19:18:23.616Z,1602184703.616 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:18:34.228Z,1602184714.228 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:18:45.833Z,1602184725.833 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:18:45.837Z,1602184725.837 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T19:19:33.089Z,1602184773.089 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:19:39.953Z,1602184779.953 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:19:46.908Z,1602184786.908 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:19:53.709Z,1602184793.709 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:20:00.589Z,1602184800.589 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:20:00.593Z,1602184800.593 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T19:20:36.116Z,1602184836.116 [ThrusterServo](FAULT): Overload Error 2020-10-08T19:26:47.569Z,1602185207.569 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.156964 min 2020-10-08T19:32:16.430Z,1602185536.430 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 17.638371 min 2020-10-08T19:33:01.490Z,1602185581.490 [DataOverHttps](IMPORTANT): SBD MTMSN=20201008T193300 2020-10-08T19:33:08.897Z,1602185588.897 [CommandLine](IMPORTANT): got command load ./Missions/Maintenance/ballast_and_trim.xml 2020-10-08T19:33:10.646Z,1602185590.646 [CommandLine](IMPORTANT): Loaded ./Missions/Maintenance/ballast_and_trim.xml 2020-10-08T19:33:15.103Z,1602185595.103 [CommandLine](IMPORTANT): got command set ballast_and_trim.MissionTimeout 30.000000 minute 2020-10-08T19:33:15.104Z,1602185595.104 [CommandLine](IMPORTANT): got command set ballast_and_trim.ApproachSpeed 0.000000 meter_per_second 2020-10-08T19:33:15.104Z,1602185595.104 [CommandLine](IMPORTANT): got command run 2020-10-08T19:33:15.106Z,1602185595.106 [CommandLine](IMPORTANT): Running 2020-10-08T19:33:15.449Z,1602185595.449 [MissionManager](IMPORTANT): Started mission ballast_and_trim 2020-10-08T20:03:16.131Z,1602187396.131 [MissionManager](IMPORTANT): Started mission Default 2020-10-08T20:03:27.188Z,1602187407.188 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:03:34.012Z,1602187414.012 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:03:40.941Z,1602187420.941 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:03:47.389Z,1602187427.389 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:03:47.393Z,1602187427.393 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T20:04:22.457Z,1602187462.457 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:04:28.911Z,1602187468.911 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:04:35.377Z,1602187475.377 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:04:42.246Z,1602187482.246 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:04:48.713Z,1602187488.713 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:04:48.717Z,1602187488.717 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T20:05:23.872Z,1602187523.872 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:05:30.325Z,1602187530.325 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:05:36.783Z,1602187536.783 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:05:43.241Z,1602187543.241 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:05:49.713Z,1602187549.713 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:05:49.717Z,1602187549.717 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T20:06:24.956Z,1602187584.956 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:06:31.309Z,1602187591.309 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:06:37.761Z,1602187597.761 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:06:44.241Z,1602187604.241 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:06:50.705Z,1602187610.705 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:06:50.722Z,1602187610.722 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T20:07:25.872Z,1602187645.872 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:07:32.421Z,1602187652.421 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:07:38.757Z,1602187658.757 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:07:45.233Z,1602187665.233 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:07:52.097Z,1602187672.097 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:07:52.100Z,1602187672.100 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T20:08:27.660Z,1602187707.660 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:08:34.916Z,1602187714.916 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:08:41.777Z,1602187721.777 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:08:48.649Z,1602187728.649 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:08:55.524Z,1602187735.524 [ThrusterServo](FAULT): Overload Error 2020-10-08T20:08:55.545Z,1602187735.545 [CBIT](CRITICAL): Hardware Fault in component: ThrusterServo 2020-10-08T20:09:36.690Z,1602187776.690 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=39316, MT Status=1, MTMSN=2770 2020-10-08T20:09:37.218Z,1602187777.218 [CommandLine](IMPORTANT): got command stop 2020-10-08T20:09:37.218Z,1602187777.218 [CommandLine](IMPORTANT): Scheduling is paused 2020-10-08T20:15:50.005Z,1602188150.005 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.557815 min 2020-10-08T20:21:20.487Z,1602188480.487 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 18.065774 min 2020-10-08T20:23:27.751Z,1602188607.751 [DataOverHttps](IMPORTANT): SBD MTMSN=20201008T202326 2020-10-08T20:23:35.090Z,1602188615.090 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_station.xml;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.StartWaypoint 2 count;set esp_sample_station.TransitYoYoMinDepth 3 meter;set esp_sample_station.SampleYoYoMinDepth 3 meter" 297ag 1 2.000000 2020-10-08T20:23:35.091Z,1602188615.091 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=297ag 2020-10-08T20:23:35.092Z,1602188615.092 [CommandLine](IMPORTANT): Scheduled #1 (#1 of 2 with id='297ag'): "load Science/esp_sample_station.xml;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.StartWaypoint 2 count;set esp_sample_station.TransitYoYoMinDepth 3 meter;set esp_sample_station.SampleYoYoMinDepth 3 meter" ASAP 2020-10-08T20:23:36.067Z,1602188616.067 [DataOverHttps](IMPORTANT): SBD MTMSN=20201008T202334 2020-10-08T20:23:43.494Z,1602188623.494 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_station.Speed 0.75 meter_per_second;run " 297ag 2 2.000000 2020-10-08T20:23:43.495Z,1602188623.495 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=297ag 2020-10-08T20:23:43.496Z,1602188623.496 [CommandLine](IMPORTANT): Scheduled #2 (#2 of 2 with id='297ag'): "set esp_sample_station.Speed 0.75 meter_per_second;run " ASAP 2020-10-08T20:23:43.739Z,1602188623.739 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_station.xml 2020-10-08T20:23:45.443Z,1602188625.443 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_station.xml 2020-10-08T20:23:55.664Z,1602188635.664 [CommandLine](IMPORTANT): got command set esp_sample_station.NeedCommsTime 60.000000 minute 2020-10-08T20:23:55.665Z,1602188635.665 [CommandLine](IMPORTANT): got command set esp_sample_station.StartWaypoint 2.000000 count 2020-10-08T20:23:55.677Z,1602188635.677 [CommandLine](IMPORTANT): got command set esp_sample_station.TransitYoYoMinDepth 3.000000 meter 2020-10-08T20:23:55.678Z,1602188635.678 [CommandLine](IMPORTANT): got command set esp_sample_station.SampleYoYoMinDepth 3.000000 meter 2020-10-08T20:23:57.632Z,1602188637.632 [CommandLine](IMPORTANT): got command set esp_sample_station.Speed 0.750000 meter_per_second 2020-10-08T20:23:57.632Z,1602188637.632 [CommandLine](IMPORTANT): got command run 2020-10-08T20:23:57.648Z,1602188637.648 [CommandLine](IMPORTANT): Running 2020-10-08T20:23:57.885Z,1602188637.885 [MissionManager](IMPORTANT): Started mission esp_sample_station 2020-10-08T20:25:02.575Z,1602188702.575 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-08T20:25:03.007Z,1602188703.007 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-08T20:25:03.386Z,1602188703.386 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-08T20:25:03.760Z,1602188703.760 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-08T20:25:04.184Z,1602188704.184 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-08T20:25:08.199Z,1602188708.199 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2020-10-08T21:48:37.447Z,1602193717.447 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-08T21:48:48.371Z,1602193728.371 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.004872 CHAN A1 (24V): 0.004421 CHAN A2 (12V): -0.004058 CHAN A3 (5V): -0.002340 CHAN B0 (3.3V): -0.001022 CHAN B1 (3.15aV): -0.001311 CHAN B2 (3.15bV): -0.000834 CHAN B3 (GND): 0.000014 OPEN: 0.005342 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-08T22:08:57.967Z,1602194937.967 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600 2020-10-08T22:17:45.737Z,1602195465.737 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-08T22:17:57.517Z,1602195477.517 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-08T22:17:57.609Z,1602195477.609 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-08T22:18:35.792Z,1602195515.792 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-08T22:18:36.616Z,1602195516.616 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -6 2020-10-08T22:18:36.660Z,1602195516.660 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-08T22:18:37.055Z,1602195517.055 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 20.767452 m in 12.002008 degC water with 0.508500 ug/l chlorophyll fluorescence. 2020-10-08T22:18:37.406Z,1602195517.406 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-08T22:18:37.810Z,1602195517.810 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-08T22:18:39.829Z,1602195519.829 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-08T22:19:12.957Z,1602195552.957 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@57,archive_bac) 2020-10-08T22:19:14.573Z,1602195554.573 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2020-10-08T22:19:16.190Z,1602195556.190 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-08T22:21:44.865Z,1602195704.865 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2020-10-08T22:21:45.269Z,1602195705.269 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-08T23:09:17.670Z,1602198557.670 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@1000.0) 2020-10-08T23:09:18.074Z,1602198558.074 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@1000.0) 2020-10-08T23:09:20.498Z,1602198560.498 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@57,archive_bac) 2020-10-08T23:21:24.105Z,1602199284.105 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@57,archive_bac) 2020-10-08T23:21:49.562Z,1602199309.562 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-08T23:21:49.967Z,1602199309.967 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2020-10-08T23:26:49.765Z,1602199609.765 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2020-10-08T23:26:50.169Z,1602199610.169 [ESPComponent](IMPORTANT): [sample #2] ESP log summary report (2 messages): @15:18:49.54 Selecting Cartridge 57 @16:09:16.16 Sampled 1000.0ml 2020-10-08T23:27:52.009Z,1602199672.009 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=39340, MT Status=1, MTMSN=2771 2020-10-08T23:27:52.593Z,1602199672.593 [CommandLine](IMPORTANT): got command ! echo 1 > /dev/loadB2;echo 1 > /dev/loadB3 2020-10-08T23:36:00.898Z,1602200160.898 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 2020-10-08T23:36:05.457Z,1602200165.457 [BuoyancyServo](FAULT): Buoyancy getPosition uart error.serial timeout 2020-10-09T00:38:19.561Z,1602203899.561 [BuoyancyServo](FAULT): Buoyancy getPosition uart error.serial timeout 2020-10-09T00:38:21.816Z,1602203901.816 [BuoyancyServo](FAULT): LCB fault: Current Limiter Activated. 2020-10-09T03:09:52.297Z,1602212992.297 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900 2020-10-09T03:13:46.063Z,1602213226.063 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-09T03:13:57.845Z,1602213237.845 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-09T03:13:57.952Z,1602213237.952 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-09T03:14:37.333Z,1602213277.333 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-09T03:14:38.144Z,1602213278.144 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6 2020-10-09T03:14:38.145Z,1602213278.145 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-09T03:14:38.592Z,1602213278.592 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 16.449467 m in 12.252893 degC water with 0.519800 ug/l chlorophyll fluorescence. 2020-10-09T03:14:38.951Z,1602213278.951 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-09T03:14:39.353Z,1602213279.353 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-09T03:14:41.374Z,1602213281.374 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-09T03:15:14.502Z,1602213314.502 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@56,archive_bac) 2020-10-09T03:15:16.118Z,1602213316.118 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2020-10-09T03:15:16.928Z,1602213316.928 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-09T03:16:49.985Z,1602213409.985 [ThrusterServo](FAULT): getVelocity uart error serial timeout 2020-10-09T03:16:49.985Z,1602213409.985 [ThrusterServo](FAULT): Thruster uart error: serial timeout 2020-10-09T03:17:46.017Z,1602213466.017 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2020-10-09T03:17:46.422Z,1602213466.422 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-09T03:48:48.700Z,1602215328.700 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-09T03:48:59.555Z,1602215339.555 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.004055 CHAN A1 (24V): 0.006769 CHAN A2 (12V): -0.002950 CHAN A3 (5V): -0.001824 CHAN B0 (3.3V): -0.001045 CHAN B1 (3.15aV): -0.001196 CHAN B2 (3.15bV): -0.001094 CHAN B3 (GND): 0.000117 OPEN: 0.007519 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-09T04:10:31.526Z,1602216631.526 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1000.0) 2020-10-09T04:10:31.931Z,1602216631.931 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1000.0) 2020-10-09T04:10:34.353Z,1602216634.353 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@56,archive_bac) 2020-10-09T04:22:38.361Z,1602217358.361 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@56,archive_bac) 2020-10-09T04:23:03.814Z,1602217383.814 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-09T04:23:04.218Z,1602217384.218 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2020-10-09T04:28:04.009Z,1602217684.009 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2020-10-09T04:28:04.414Z,1602217684.414 [ESPComponent](IMPORTANT): [sample #3] ESP log summary report (2 messages): @20:14:50.73 Selecting Cartridge 56 @21:10:29.42 Sampled 1000.0ml 2020-10-09T04:35:18.915Z,1602218118.915 [MissionManager](IMPORTANT): Started mission Default 2020-10-09T04:41:03.589Z,1602218463.589 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 5.737892 min 2020-10-09T04:43:41.002Z,1602218621.002 [DataOverHttps](IMPORTANT): SBD MTMSN=20201009T044339 2020-10-09T04:43:48.254Z,1602218628.254 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_station.xml;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.TransitYoYoMinDepth 3 meter;set esp_sample_station.SampleYoYoMinDepth 3 meter" 29ug1 1 2.000000 2020-10-09T04:43:48.254Z,1602218628.254 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=29ug1 2020-10-09T04:43:48.255Z,1602218628.255 [CommandLine](IMPORTANT): Scheduled #3 (#1 of 2 with id='29ug1'): "load Science/esp_sample_station.xml;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.TransitYoYoMinDepth 3 meter;set esp_sample_station.SampleYoYoMinDepth 3 meter" ASAP 2020-10-09T04:43:49.158Z,1602218629.158 [DataOverHttps](IMPORTANT): SBD MTMSN=20201009T044347 2020-10-09T04:43:56.657Z,1602218636.657 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_station.Speed 0.75 meter_per_second;run" 29ug1 2 2.000000 2020-10-09T04:43:56.658Z,1602218636.658 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=29ug1 2020-10-09T04:43:56.658Z,1602218636.658 [CommandLine](IMPORTANT): Scheduled #4 (#2 of 2 with id='29ug1'): "set esp_sample_station.Speed 0.75 meter_per_second;run" ASAP 2020-10-09T04:43:56.717Z,1602218636.717 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_station.xml 2020-10-09T04:43:58.404Z,1602218638.404 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_station.xml 2020-10-09T04:44:05.726Z,1602218645.726 [CommandLine](IMPORTANT): got command set esp_sample_station.NeedCommsTime 60.000000 minute 2020-10-09T04:44:05.726Z,1602218645.726 [CommandLine](IMPORTANT): got command set esp_sample_station.TransitYoYoMinDepth 3.000000 meter 2020-10-09T04:44:05.727Z,1602218645.727 [CommandLine](IMPORTANT): got command set esp_sample_station.SampleYoYoMinDepth 3.000000 meter 2020-10-09T04:44:07.840Z,1602218647.840 [CommandLine](IMPORTANT): got command set esp_sample_station.Speed 0.750000 meter_per_second 2020-10-09T04:44:07.845Z,1602218647.845 [CommandLine](IMPORTANT): got command run 2020-10-09T04:44:07.847Z,1602218647.847 [CommandLine](IMPORTANT): Running 2020-10-09T04:44:07.856Z,1602218647.856 [MissionManager](IMPORTANT): Started mission esp_sample_station 2020-10-09T04:49:17.902Z,1602218957.902 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2020-10-09T04:49:19.922Z,1602218959.922 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-09T04:49:20.342Z,1602218960.342 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-09T04:49:20.736Z,1602218960.736 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-09T04:49:21.157Z,1602218961.157 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-09T04:49:21.543Z,1602218961.543 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-09T04:49:25.588Z,1602218965.588 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 2020-10-09T06:29:54.776Z,1602224994.776 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000 2020-10-09T06:35:28.433Z,1602225328.433 [ThrusterServo](FAULT): Invalid EZ Servo response:"" 2020-10-09T06:35:43.301Z,1602225343.301 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-09T06:35:55.085Z,1602225355.085 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-09T06:35:55.381Z,1602225355.381 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-09T06:38:25.470Z,1602225505.470 [ESPComponent](FAULT): Timeout while waiting for ESP to connect (elapsed=150.0888) 2020-10-09T06:38:37.293Z,1602225517.293 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-09T06:38:37.395Z,1602225517.395 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-09T06:39:15.970Z,1602225555.970 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-09T06:39:16.780Z,1602225556.780 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6 2020-10-09T06:39:16.785Z,1602225556.785 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-09T06:39:17.218Z,1602225557.218 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 49.364197 m in 11.128992 degC water with 0.248600 ug/l chlorophyll fluorescence. 2020-10-09T06:39:17.585Z,1602225557.585 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-09T06:39:17.991Z,1602225557.991 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-09T06:39:20.013Z,1602225560.013 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-09T06:39:53.141Z,1602225593.141 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@55,archive_bac) 2020-10-09T06:39:54.758Z,1602225594.758 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2020-10-09T06:39:55.566Z,1602225595.566 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-09T06:42:23.842Z,1602225743.842 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2020-10-09T06:42:24.246Z,1602225744.246 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-09T07:35:25.099Z,1602228925.099 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@1000.0) 2020-10-09T07:35:25.502Z,1602228925.502 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@1000.0) 2020-10-09T07:35:27.927Z,1602228927.927 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@55,archive_bac) 2020-10-09T07:47:31.542Z,1602229651.542 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@55,archive_bac) 2020-10-09T07:47:56.994Z,1602229676.994 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-09T07:47:57.398Z,1602229677.398 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2020-10-09T07:52:57.185Z,1602229977.185 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2020-10-09T07:52:57.590Z,1602229977.590 [ESPComponent](IMPORTANT): [sample #4] ESP log summary report (2 messages): @23:39:29.97 Selecting Cartridge 55 @00:35:23.68 Sampled 1000.0ml 2020-10-09T07:58:26.604Z,1602230306.604 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2020-10-09T09:48:59.765Z,1602236939.765 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-09T09:49:10.695Z,1602236950.695 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005129 CHAN A1 (24V): 0.004174 CHAN A2 (12V): -0.003359 CHAN A3 (5V): -0.002263 CHAN B0 (3.3V): -0.000839 CHAN B1 (3.15aV): -0.000825 CHAN B2 (3.15bV): -0.000820 CHAN B3 (GND): 0.000047 OPEN: 0.007631 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-09T10:05:33.215Z,1602237933.215 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600 2020-10-09T10:11:50.587Z,1602238310.587 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-09T10:12:02.377Z,1602238322.377 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-09T10:12:02.517Z,1602238322.517 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-09T10:12:41.096Z,1602238361.096 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-09T10:12:41.904Z,1602238361.904 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6 2020-10-09T10:12:41.905Z,1602238361.905 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-09T10:12:42.361Z,1602238362.361 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 10.182357 m in 13.269007 degC water with 2.847600 ug/l chlorophyll fluorescence. 2020-10-09T10:12:42.710Z,1602238362.710 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-09T10:12:43.116Z,1602238363.116 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-09T10:12:45.133Z,1602238365.133 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-09T10:13:18.262Z,1602238398.262 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@54,archive_bac) 2020-10-09T10:13:19.878Z,1602238399.878 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2020-10-09T10:13:20.686Z,1602238400.686 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-09T10:15:49.771Z,1602238549.771 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2020-10-09T10:15:50.173Z,1602238550.173 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-09T11:18:08.984Z,1602242288.984 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0) 2020-10-09T11:18:09.386Z,1602242289.386 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0) 2020-10-09T11:18:11.809Z,1602242291.809 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@54,archive_bac) 2020-10-09T11:30:15.430Z,1602243015.430 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@54,archive_bac) 2020-10-09T11:30:40.882Z,1602243040.882 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-09T11:30:41.286Z,1602243041.286 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2020-10-09T11:35:41.070Z,1602243341.070 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2020-10-09T11:35:41.473Z,1602243341.473 [ESPComponent](IMPORTANT): [sample #5] ESP log summary report (2 messages): @03:12:55.04 Selecting Cartridge 54 @04:18:07.33 Sampled 1000.0ml 2020-10-09T11:40:19.367Z,1602243619.367 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 2020-10-09T14:40:26.574Z,1602254426.574 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900 2020-10-09T14:45:18.860Z,1602254718.860 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-09T14:45:30.629Z,1602254730.629 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-09T14:45:30.766Z,1602254730.766 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-09T14:46:08.904Z,1602254768.904 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-09T14:46:09.717Z,1602254769.717 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -6 2020-10-09T14:46:09.718Z,1602254769.718 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-09T14:46:10.164Z,1602254770.164 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 10.610500 m in 13.034296 degC water with 2.169600 ug/l chlorophyll fluorescence. 2020-10-09T14:46:10.517Z,1602254770.517 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-09T14:46:10.922Z,1602254770.922 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-09T14:46:12.941Z,1602254772.941 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-09T14:46:46.070Z,1602254806.070 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@53,archive_bac) 2020-10-09T14:46:47.690Z,1602254807.690 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2020-10-09T14:46:49.301Z,1602254809.301 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-09T14:49:17.991Z,1602254957.991 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2020-10-09T14:49:18.393Z,1602254958.393 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-09T15:43:12.242Z,1602258192.242 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1000.0) 2020-10-09T15:43:12.645Z,1602258192.645 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1000.0) 2020-10-09T15:43:15.075Z,1602258195.075 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@53,archive_bac) 2020-10-09T15:49:10.701Z,1602258550.701 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-09T15:49:21.607Z,1602258561.607 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003919 CHAN A1 (24V): 0.005953 CHAN A2 (12V): -0.003159 CHAN A3 (5V): -0.001987 CHAN B0 (3.3V): -0.000763 CHAN B1 (3.15aV): -0.001232 CHAN B2 (3.15bV): -0.000884 CHAN B3 (GND): -0.000221 OPEN: 0.007462 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-09T15:55:18.681Z,1602258918.681 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@53,archive_bac) 2020-10-09T15:55:44.138Z,1602258944.138 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-09T15:55:44.538Z,1602258944.538 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2020-10-09T16:00:44.334Z,1602259244.334 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2020-10-09T16:00:44.738Z,1602259244.738 [ESPComponent](IMPORTANT): [sample #6] ESP log summary report (2 messages): @07:46:22.57 Selecting Cartridge 53 @08:43:10.20 Sampled 1000.0ml 2020-10-09T16:07:25.449Z,1602259645.449 [MissionManager](IMPORTANT): Started mission Default 2020-10-09T16:13:04.903Z,1602259984.903 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 5.650945 min 2020-10-09T16:18:34.190Z,1602260314.190 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 11.139168 min 2020-10-09T16:24:04.681Z,1602260644.681 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 16.646980 min 2020-10-09T16:29:44.903Z,1602260984.903 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 22.317108 min 2020-10-09T16:32:23.345Z,1602261143.345 [DataOverHttps](IMPORTANT): SBD MTMSN=20201009T163221 2020-10-09T16:32:30.653Z,1602261150.653 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_station.xml;set esp_sample_station.MissionTimeout 110 hour;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.LapRepeat 8 count;set esp_sample_station.StartWaypoint 2 count" 2ar9b 1 2.000000 2020-10-09T16:32:30.654Z,1602261150.654 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2ar9b 2020-10-09T16:32:30.655Z,1602261150.655 [CommandLine](IMPORTANT): Scheduled #5 (#1 of 2 with id='2ar9b'): "load Science/esp_sample_station.xml;set esp_sample_station.MissionTimeout 110 hour;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.LapRepeat 8 count;set esp_sample_station.StartWaypoint 2 count" ASAP 2020-10-09T16:32:31.591Z,1602261151.591 [DataOverHttps](IMPORTANT): SBD MTMSN=20201009T163230 2020-10-09T16:32:39.057Z,1602261159.057 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_station.TransitYoYoMinDepth 3 meter;set esp_sample_station.SampleYoYoMinDepth 3 meter;set esp_sample_station.Speed 0.75 meter_per_second;run " 2ar9b 2 2.000000 2020-10-09T16:32:39.058Z,1602261159.058 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2ar9b 2020-10-09T16:32:39.059Z,1602261159.059 [CommandLine](IMPORTANT): Scheduled #6 (#2 of 2 with id='2ar9b'): "set esp_sample_station.TransitYoYoMinDepth 3 meter;set esp_sample_station.SampleYoYoMinDepth 3 meter;set esp_sample_station.Speed 0.75 meter_per_second;run " ASAP 2020-10-09T16:32:39.207Z,1602261159.207 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_station.xml 2020-10-09T16:32:40.954Z,1602261160.954 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_station.xml 2020-10-09T16:32:54.938Z,1602261174.938 [CommandLine](IMPORTANT): got command set esp_sample_station.MissionTimeout 110.000000 hour 2020-10-09T16:32:54.962Z,1602261174.962 [CommandLine](IMPORTANT): got command set esp_sample_station.NeedCommsTime 60.000000 minute 2020-10-09T16:32:54.963Z,1602261174.963 [CommandLine](IMPORTANT): got command set esp_sample_station.LapRepeat 8.000000 count 2020-10-09T16:32:54.963Z,1602261174.963 [CommandLine](IMPORTANT): got command set esp_sample_station.StartWaypoint 2.000000 count 2020-10-09T16:33:02.155Z,1602261182.155 [CommandLine](IMPORTANT): got command set esp_sample_station.TransitYoYoMinDepth 3.000000 meter 2020-10-09T16:33:02.156Z,1602261182.156 [CommandLine](IMPORTANT): got command set esp_sample_station.SampleYoYoMinDepth 3.000000 meter 2020-10-09T16:33:02.156Z,1602261182.156 [CommandLine](IMPORTANT): got command set esp_sample_station.Speed 0.750000 meter_per_second 2020-10-09T16:33:02.157Z,1602261182.157 [CommandLine](IMPORTANT): got command run 2020-10-09T16:33:02.166Z,1602261182.166 [CommandLine](IMPORTANT): Running 2020-10-09T16:33:02.499Z,1602261182.499 [MissionManager](IMPORTANT): Started mission esp_sample_station 2020-10-09T16:34:15.977Z,1602261255.977 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-09T16:34:16.374Z,1602261256.374 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-09T16:34:16.785Z,1602261256.785 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-09T16:34:17.191Z,1602261257.191 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-09T16:34:17.574Z,1602261257.574 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-09T16:34:21.611Z,1602261261.611 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2020-10-09T17:39:22.415Z,1602265162.415 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=39486, MT Status=1, MTMSN=2772 2020-10-09T17:39:22.905Z,1602265162.905 [CommandLine](IMPORTANT): got command stop 2020-10-09T17:39:23.233Z,1602265163.233 [MissionManager](IMPORTANT): Started mission Default 2020-10-09T17:39:23.473Z,1602265163.473 [WetLabsBB2FL](FAULT): LCB fault: Software Overcurrent. 2020-10-09T17:40:02.488Z,1602265202.488 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=39487, MT Status=1, MTMSN=2773 2020-10-09T17:40:03.259Z,1602265203.259 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_station.xml;set esp_sample_station.MissionTimeout 110 hour;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.LapRepeat 8 count;set esp_sample_station.TransitYoYoMinDepth 3 meter" 2as5q 1 2.000000 2020-10-09T17:40:03.259Z,1602265203.259 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2as5q 2020-10-09T17:40:03.260Z,1602265203.260 [CommandLine](IMPORTANT): Scheduled #7 (#1 of 2 with id='2as5q'): "load Science/esp_sample_station.xml;set esp_sample_station.MissionTimeout 110 hour;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.LapRepeat 8 count;set esp_sample_station.TransitYoYoMinDepth 3 meter" ASAP 2020-10-09T17:40:24.930Z,1602265224.930 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=39488, MT Status=1, MTMSN=2774 2020-10-09T17:40:25.535Z,1602265225.535 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_station.SampleYoYoMinDepth 3 meter;set esp_sample_station.Speed 0.75 meter_per_second;run " 2as5q 2 2.000000 2020-10-09T17:40:25.536Z,1602265225.536 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2as5q 2020-10-09T17:40:25.536Z,1602265225.536 [CommandLine](IMPORTANT): Scheduled #8 (#2 of 2 with id='2as5q'): "set esp_sample_station.SampleYoYoMinDepth 3 meter;set esp_sample_station.Speed 0.75 meter_per_second;run " ASAP 2020-10-09T17:40:25.647Z,1602265225.647 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_station.xml 2020-10-09T17:40:27.322Z,1602265227.322 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_station.xml 2020-10-09T17:40:37.006Z,1602265237.006 [CommandLine](IMPORTANT): got command set esp_sample_station.MissionTimeout 110.000000 hour 2020-10-09T17:40:37.007Z,1602265237.007 [CommandLine](IMPORTANT): got command set esp_sample_station.NeedCommsTime 60.000000 minute 2020-10-09T17:40:37.007Z,1602265237.007 [CommandLine](IMPORTANT): got command set esp_sample_station.LapRepeat 8.000000 count 2020-10-09T17:40:37.008Z,1602265237.008 [CommandLine](IMPORTANT): got command set esp_sample_station.TransitYoYoMinDepth 3.000000 meter 2020-10-09T17:40:42.214Z,1602265242.214 [CommandLine](IMPORTANT): got command set esp_sample_station.SampleYoYoMinDepth 3.000000 meter 2020-10-09T17:40:42.215Z,1602265242.215 [CommandLine](IMPORTANT): got command set esp_sample_station.Speed 0.750000 meter_per_second 2020-10-09T17:40:42.215Z,1602265242.215 [CommandLine](IMPORTANT): got command run 2020-10-09T17:40:42.217Z,1602265242.217 [CommandLine](IMPORTANT): Running 2020-10-09T17:40:42.577Z,1602265242.577 [MissionManager](IMPORTANT): Started mission esp_sample_station 2020-10-09T17:42:46.784Z,1602265366.784 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-09T17:42:47.219Z,1602265367.219 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-09T17:42:47.767Z,1602265367.767 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-09T17:42:47.974Z,1602265367.974 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-09T17:42:48.428Z,1602265368.428 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-09T17:42:52.453Z,1602265372.453 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 2020-10-09T18:17:09.286Z,1602267429.286 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000 2020-10-09T18:22:47.368Z,1602267767.368 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-09T18:22:59.161Z,1602267779.161 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-09T18:22:59.296Z,1602267779.296 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-09T18:23:38.654Z,1602267818.654 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-09T18:23:39.464Z,1602267819.464 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -6 2020-10-09T18:23:39.465Z,1602267819.465 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-09T18:23:39.906Z,1602267819.906 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 12.847443 m in 12.214014 degC water with 0.565000 ug/l chlorophyll fluorescence. 2020-10-09T18:23:40.269Z,1602267820.269 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-09T18:23:40.674Z,1602267820.674 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-09T18:23:42.693Z,1602267822.693 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-09T18:24:15.825Z,1602267855.825 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@52,archive_bac) 2020-10-09T18:24:17.441Z,1602267857.441 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2020-10-09T18:24:19.057Z,1602267859.057 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-09T18:26:57.433Z,1602268017.433 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2020-10-09T18:26:57.838Z,1602268017.838 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-09T19:05:08.646Z,1602270308.646 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@673.296) 2020-10-09T19:05:09.053Z,1602270309.053 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@673.296) 2020-10-09T19:05:11.475Z,1602270311.475 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@52,archive_bac) 2020-10-09T19:17:15.078Z,1602271035.078 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@52,archive_bac) 2020-10-09T19:17:40.530Z,1602271060.530 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-09T19:17:40.934Z,1602271060.934 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2020-10-09T19:22:40.724Z,1602271360.724 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2020-10-09T19:22:41.126Z,1602271361.126 [ESPComponent](IMPORTANT): [sample #7] ESP log summary report (5 messages): @11:23:52.55 Selecting Cartridge 52 @11:24:30.53 Cartridge::Sampler::PossibleLeak in FILTERING -- Retained only 37.3 of initial 45.9psia @11:24:30.54 Retry #1 of 2 @12:05:06.42 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @12:05:06.61 Sampled 673.3ml 2020-10-09T19:27:56.181Z,1602271676.181 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2020-10-09T21:27:41.911Z,1602278861.911 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600 2020-10-09T21:31:46.393Z,1602279106.393 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-09T21:31:58.177Z,1602279118.177 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-09T21:31:58.301Z,1602279118.301 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-09T21:32:37.269Z,1602279157.269 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-09T21:32:38.080Z,1602279158.080 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -6 2020-10-09T21:32:38.081Z,1602279158.081 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-09T21:32:38.531Z,1602279158.531 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 11.139071 m in 13.408808 degC water with 3.740300 ug/l chlorophyll fluorescence. 2020-10-09T21:32:38.886Z,1602279158.886 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-09T21:32:39.289Z,1602279159.289 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-09T21:32:41.309Z,1602279161.309 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-09T21:33:14.437Z,1602279194.437 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@51,archive_bac) 2020-10-09T21:33:16.053Z,1602279196.053 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2020-10-09T21:33:17.670Z,1602279197.670 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-09T21:35:49.594Z,1602279349.594 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2020-10-09T21:35:50.006Z,1602279350.006 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-09T21:49:21.765Z,1602280161.765 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-09T21:49:32.675Z,1602280172.675 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.004968 CHAN A1 (24V): -0.175343 CHAN A2 (12V): -0.003515 CHAN A3 (5V): -0.001728 CHAN B0 (3.3V): -0.000819 CHAN B1 (3.15aV): -0.001161 CHAN B2 (3.15bV): -0.001215 CHAN B3 (GND): -0.000475 OPEN: 0.005764 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-09T22:32:33.505Z,1602282753.505 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@1000.0) 2020-10-09T22:32:33.912Z,1602282753.912 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@1000.0) 2020-10-09T22:32:36.334Z,1602282756.334 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@51,archive_bac) 2020-10-09T22:44:39.942Z,1602283479.942 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@51,archive_bac) 2020-10-09T22:45:05.394Z,1602283505.394 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-09T22:45:05.798Z,1602283505.798 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2020-10-09T22:50:05.594Z,1602283805.594 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2020-10-09T22:50:05.999Z,1602283805.999 [ESPComponent](IMPORTANT): [sample #8] ESP log summary report (2 messages): @14:32:51.12 Selecting Cartridge 51 @15:32:31.78 Sampled 1000.0ml 2020-10-09T22:57:12.335Z,1602284232.335 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 2020-10-10T02:10:27.531Z,1602295827.531 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900 2020-10-10T02:19:18.993Z,1602296358.993 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-10T02:19:30.757Z,1602296370.757 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-10T02:19:31.165Z,1602296371.165 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-10T02:20:08.945Z,1602296408.945 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-10T02:20:09.759Z,1602296409.759 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -6 2020-10-10T02:20:09.760Z,1602296409.760 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-10T02:20:10.219Z,1602296410.219 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 17.431276 m in 12.115686 degC water with 0.711900 ug/l chlorophyll fluorescence. 2020-10-10T02:20:10.561Z,1602296410.561 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-10T02:20:10.966Z,1602296410.966 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-10T02:20:12.991Z,1602296412.991 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-10T02:20:46.114Z,1602296446.114 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@50,archive_bac) 2020-10-10T02:20:47.729Z,1602296447.729 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2020-10-10T02:20:48.538Z,1602296448.538 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-10T02:23:16.805Z,1602296596.805 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2020-10-10T02:23:17.210Z,1602296597.210 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-10T03:16:45.078Z,1602299805.078 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@867.132) 2020-10-10T03:16:45.485Z,1602299805.485 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@867.132) 2020-10-10T03:16:47.905Z,1602299807.905 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@50,archive_bac) 2020-10-10T03:28:51.494Z,1602300531.494 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@50,archive_bac) 2020-10-10T03:29:16.946Z,1602300556.946 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-10T03:29:17.350Z,1602300557.350 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2020-10-10T03:34:17.121Z,1602300857.121 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2020-10-10T03:34:17.526Z,1602300857.526 [ESPComponent](IMPORTANT): [sample #9] ESP log summary report (3 messages): @19:20:22.36 Selecting Cartridge 50 @20:16:42.40 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @20:16:42.60 Sampled 867.1ml 2020-10-10T03:38:53.470Z,1602301133.470 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-10T03:38:53.872Z,1602301133.872 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-10T03:38:54.357Z,1602301134.357 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-10T03:38:54.727Z,1602301134.727 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-10T03:38:55.090Z,1602301135.090 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-10T03:38:59.152Z,1602301139.152 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 2020-10-10T03:49:32.696Z,1602301772.696 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-10T03:49:43.599Z,1602301783.599 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.004741 CHAN A1 (24V): -0.175306 CHAN A2 (12V): -0.003896 CHAN A3 (5V): -0.002369 CHAN B0 (3.3V): -0.000901 CHAN B1 (3.15aV): -0.000847 CHAN B2 (3.15bV): -0.000993 CHAN B3 (GND): 0.000159 OPEN: 0.005332 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-10T04:48:05.271Z,1602305285.271 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000 2020-10-10T04:54:56.823Z,1602305696.823 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-10T04:55:08.613Z,1602305708.613 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-10T04:55:08.721Z,1602305708.721 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-10T04:55:47.290Z,1602305747.290 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-10T04:55:48.100Z,1602305748.100 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -6 2020-10-10T04:55:48.101Z,1602305748.101 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-10T04:55:48.532Z,1602305748.532 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 14.450325 m in 12.179010 degC water with 0.666700 ug/l chlorophyll fluorescence. 2020-10-10T04:55:48.905Z,1602305748.905 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-10T04:55:49.309Z,1602305749.309 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-10T04:55:51.335Z,1602305751.335 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-10T04:56:24.461Z,1602305784.461 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@49,archive_bac) 2020-10-10T04:56:26.078Z,1602305786.078 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2020-10-10T04:56:26.889Z,1602305786.889 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-10T04:58:55.168Z,1602305935.168 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2020-10-10T04:58:55.570Z,1602305935.570 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-10T05:35:35.893Z,1602308135.893 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@629.421) 2020-10-10T05:35:36.294Z,1602308136.294 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@629.421) 2020-10-10T05:35:38.718Z,1602308138.718 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,archive_bac) 2020-10-10T05:47:42.725Z,1602308862.725 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@49,archive_bac) 2020-10-10T05:48:08.178Z,1602308888.178 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-10T05:48:08.582Z,1602308888.582 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2020-10-10T05:53:08.361Z,1602309188.361 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2020-10-10T05:53:08.771Z,1602309188.771 [ESPComponent](IMPORTANT): [sample #10] ESP log summary report (3 messages): @21:56:01.24 Selecting Cartridge 49 @22:35:33.45 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @22:35:33.65 Sampled 629.4ml 2020-10-10T05:58:34.440Z,1602309514.440 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2020-10-10T08:04:14.795Z,1602317054.795 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600 2020-10-10T08:10:25.443Z,1602317425.443 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-10T08:10:37.221Z,1602317437.221 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-10T08:10:37.534Z,1602317437.534 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-10T08:11:16.046Z,1602317476.046 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-10T08:11:16.862Z,1602317476.862 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -6 2020-10-10T08:11:16.863Z,1602317476.863 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-10T08:11:17.293Z,1602317477.293 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 10.162161 m in 13.064600 degC water with 2.180900 ug/l chlorophyll fluorescence. 2020-10-10T08:11:17.662Z,1602317477.662 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-10T08:11:18.066Z,1602317478.066 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-10T08:11:20.088Z,1602317480.088 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-10T08:11:53.214Z,1602317513.214 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@48,archive_bac) 2020-10-10T08:11:54.829Z,1602317514.829 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2020-10-10T08:11:55.638Z,1602317515.638 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-10T08:14:32.794Z,1602317672.794 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2020-10-10T08:14:33.198Z,1602317673.198 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-10T08:33:11.195Z,1602318791.195 [BPC1](FAULT): Battery stick #38 (s/n: 0139) reported TERMINATE_DISCHARGE_ALARM. Status code: 0x4AD0. 2020-10-10T08:44:46.969Z,1602319486.969 [ThrusterServo](FAULT): Invalid EZ Servo response:"\377\r" 2020-10-10T08:44:47.770Z,1602319487.770 [ThrusterServo](FAULT): LCB fault: Software Overcurrent. 2020-10-10T09:12:03.157Z,1602321123.157 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@1000.0) 2020-10-10T09:12:03.560Z,1602321123.560 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@1000.0) 2020-10-10T09:12:05.982Z,1602321125.982 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@48,archive_bac) 2020-10-10T09:24:09.569Z,1602321849.569 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@48,archive_bac) 2020-10-10T09:24:35.025Z,1602321875.025 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-10T09:24:35.426Z,1602321875.426 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2020-10-10T09:29:35.207Z,1602322175.207 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2020-10-10T09:29:35.610Z,1602322175.610 [ESPComponent](IMPORTANT): [sample #11] ESP log summary report (4 messages): @01:11:29.73 Selecting Cartridge 48 @01:12:05.78 Cartridge::Sampler::PossibleLeak in FILTERING -- Retained only 32.1 of initial 42.1psia @01:12:05.80 Retry #1 of 2 @02:12:01.52 Sampled 1000.0ml 2020-10-10T09:36:25.830Z,1602322585.830 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 2020-10-10T09:49:43.824Z,1602323383.824 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-10T09:49:54.724Z,1602323394.724 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.005855 CHAN A1 (24V): -0.175432 CHAN A2 (12V): -0.003925 CHAN A3 (5V): -0.002078 CHAN B0 (3.3V): -0.000814 CHAN B1 (3.15aV): -0.001328 CHAN B2 (3.15bV): -0.001052 CHAN B3 (GND): 0.000290 OPEN: 0.005408 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-10T12:35:11.345Z,1602333311.345 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900 2020-10-10T12:41:12.656Z,1602333672.656 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-10T12:41:24.433Z,1602333684.433 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-10T12:41:24.696Z,1602333684.696 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-10T12:42:03.225Z,1602333723.225 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-10T12:42:04.036Z,1602333724.036 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -6 2020-10-10T12:42:04.037Z,1602333724.037 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-10T12:42:04.472Z,1602333724.472 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 13.405704 m in 13.017908 degC water with 2.429500 ug/l chlorophyll fluorescence. 2020-10-10T12:42:04.841Z,1602333724.841 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-10T12:42:05.245Z,1602333725.245 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-10T12:42:07.265Z,1602333727.265 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-10T12:42:40.399Z,1602333760.399 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@47,archive_bac) 2020-10-10T12:42:42.014Z,1602333762.014 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@100.0) 2020-10-10T12:42:42.821Z,1602333762.821 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-10T12:45:24.034Z,1602333924.034 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_FILTERING (@0.0) 2020-10-10T12:45:24.438Z,1602333924.438 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-10T12:46:08.573Z,1602333968.573 [RudderServo](FAULT): Reading outside of valid range:0.000000 2020-10-10T12:46:09.241Z,1602333969.241 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2020-10-10T13:43:19.854Z,1602337399.854 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PAUSED (@955.161) 2020-10-10T13:43:20.262Z,1602337400.262 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PROCESSING (@955.161) 2020-10-10T13:43:22.681Z,1602337402.681 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PROCESSED (@47,archive_bac) 2020-10-10T13:55:26.305Z,1602338126.305 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@47,archive_bac) 2020-10-10T13:55:51.758Z,1602338151.758 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-10T13:55:52.163Z,1602338152.163 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED 2020-10-10T14:00:51.953Z,1602338451.953 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence completed normally. 2020-10-10T14:00:52.358Z,1602338452.358 [ESPComponent](IMPORTANT): [sample #12] ESP log summary report (5 messages): @05:42:16.93 Selecting Cartridge 47 @05:42:56.64 Cartridge::Sampler::PossibleLeak in FILTERING -- Retained only 38.6 of initial 47.4psia @05:42:56.66 Retry #1 of 2 @06:43:17.42 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @06:43:17.61 Sampled 955.2ml 2020-10-10T14:06:20.367Z,1602338780.367 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-10T14:06:20.775Z,1602338780.775 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-10T14:06:21.178Z,1602338781.178 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-10T14:06:21.587Z,1602338781.587 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-10T14:06:21.979Z,1602338781.979 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-10T14:06:26.033Z,1602338786.033 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 2020-10-10T15:16:22.527Z,1602342982.527 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000 2020-10-10T15:24:00.274Z,1602343440.274 [DataOverHttps](IMPORTANT): SBD MTMSN=20201010T152358 2020-10-10T15:24:08.441Z,1602343448.441 [CommandLine](IMPORTANT): got command configSet DAT.loadAtStartup 1.000000 bool persist 2020-10-10T15:24:08.444Z,1602343448.444 [CommandLine](IMPORTANT): configSet DAT.loadAtStartup requires a restart to take effect. 2020-10-10T15:25:32.203Z,1602343532.203 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-10T15:25:43.993Z,1602343543.993 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-10T15:25:44.125Z,1602343544.125 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-10T15:26:21.905Z,1602343581.905 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-10T15:26:22.719Z,1602343582.719 [ESPComponent](IMPORTANT): [sample #13] Setting cartridge argument to -6 2020-10-10T15:26:22.719Z,1602343582.719 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-10T15:26:23.156Z,1602343583.156 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 18.548569 m in 12.090296 degC water with 0.745800 ug/l chlorophyll fluorescence. 2020-10-10T15:26:23.521Z,1602343583.521 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-10T15:26:23.926Z,1602343583.926 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-10T15:26:25.945Z,1602343585.945 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-10T15:26:59.080Z,1602343619.080 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PRIMING (@46,archive_bac) 2020-10-10T15:27:00.693Z,1602343620.693 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PRIMING (@100.0) 2020-10-10T15:27:01.502Z,1602343621.502 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-10T15:29:29.769Z,1602343769.769 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_FILTERING (@0.0) 2020-10-10T15:29:30.173Z,1602343770.173 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-10T15:39:14.129Z,1602344354.129 [ThrusterServo](FAULT): Invalid EZ Servo response:"" 2020-10-10T15:49:54.849Z,1602344994.849 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-10T15:50:05.790Z,1602345005.790 [CBIT](IMPORTANT): Ground fault detected mA: CHAN A0 (Batt): -0.005238 CHAN A1 (24V): -0.175173 CHAN A2 (12V): -0.004280 CHAN A3 (5V): -0.002611 CHAN B0 (3.3V): -0.000969 CHAN B1 (3.15aV): -0.001184 CHAN B2 (3.15bV): -0.000922 CHAN B3 (GND): 0.000274 OPEN: 0.005221 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-10T16:20:29.036Z,1602346829.036 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PAUSED (@1000.0) 2020-10-10T16:20:29.437Z,1602346829.437 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PROCESSING (@1000.0) 2020-10-10T16:20:31.861Z,1602346831.861 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PROCESSED (@46,archive_bac) 2020-10-10T16:32:35.467Z,1602347555.467 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPING (@46,archive_bac) 2020-10-10T16:33:00.919Z,1602347580.919 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-10T16:33:01.322Z,1602347581.322 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPED 2020-10-10T16:38:01.099Z,1602347881.099 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence completed normally. 2020-10-10T16:38:01.503Z,1602347881.503 [ESPComponent](IMPORTANT): [sample #13] ESP log summary report (2 messages): @08:26:35.66 Selecting Cartridge 46 @09:20:27.42 Sampled 1000.0ml 2020-10-10T16:41:36.876Z,1602348096.876 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=39674, MT Status=1, MTMSN=2775 2020-10-10T16:41:37.414Z,1602348097.414 [CommandLine](IMPORTANT): got command ! echo i > /dev/loadB2;echo i > /dev/loadB3 2020-10-10T16:41:50.806Z,1602348110.806 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=39675, MT Status=1, MTMSN=2776 2020-10-10T16:41:51.327Z,1602348111.327 [CommandLine](IMPORTANT): got command restart application 2020-10-10T16:41:52.843Z,1602348112.843 [MissionManager](IMPORTANT): Started mission Default