2020-10-10T16:50:15.635Z,1602348615.635 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg 2020-10-10T16:50:19.348Z,1602348619.348 [SBIT](IMPORTANT): git: 2020-10-07_A 2020-10-10T16:50:19.348Z,1602348619.348 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8 2020-10-10T16:50:19.959Z,1602348619.959 [CommandLine](IMPORTANT): got command ! echo 1 > /dev/loadB2 2020-10-10T16:50:20.975Z,1602348620.975 [AMEcho](IMPORTANT): Setting 'enabled' to 0 2020-10-10T16:50:40.937Z,1602348640.937 [MassServo](FAULT): Invalid EZ Servo response:"\r" 2020-10-10T16:50:51.283Z,1602348651.283 [DAT](FAULT): failed to enter command mode 2020-10-10T16:51:36.135Z,1602348696.135 [SBIT](IMPORTANT): Beginning Startup BIT 2020-10-10T16:51:36.139Z,1602348696.139 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-10T16:51:47.088Z,1602348707.088 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.006272 CHAN A1 (24V): 0.003338 CHAN A2 (12V): -0.003948 CHAN A3 (5V): -0.002522 CHAN B0 (3.3V): -0.000967 CHAN B1 (3.15aV): -0.000911 CHAN B2 (3.15bV): -0.000973 CHAN B3 (GND): -0.000030 OPEN: 0.007448 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-10T16:52:29.887Z,1602348749.887 [SBIT](IMPORTANT): SBIT PASSED 2020-10-10T16:52:29.887Z,1602348749.887 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2020-10-10T16:52:29.888Z,1602348749.888 [SBIT](IMPORTANT): AMEcho.enabled=0 bool; 2020-10-10T16:52:29.888Z,1602348749.888 [SBIT](IMPORTANT): Aanderaa_O2.loadAtStartup=0 bool; 2020-10-10T16:52:29.888Z,1602348749.888 [SBIT](IMPORTANT): BPC1.batteryMissingStickThreshold=2 count; 2020-10-10T16:52:29.888Z,1602348749.888 [SBIT](IMPORTANT): CANONSampler.loadAtStartup=1 bool; 2020-10-10T16:52:29.888Z,1602348749.888 [SBIT](IMPORTANT): CANONSampler.rotateOnly=0 bool; 2020-10-10T16:52:29.888Z,1602348749.888 [SBIT](IMPORTANT): CANONSampler.simulateHardware=0 bool; 2020-10-10T16:52:29.888Z,1602348749.888 [SBIT](IMPORTANT): CBIT.gf24Offset=148.5 microampere; 2020-10-10T16:52:29.889Z,1602348749.889 [SBIT](IMPORTANT): DAT.loadAtStartup=1 bool; 2020-10-10T16:52:29.889Z,1602348749.889 [SBIT](IMPORTANT): DVL_micro.loadAtStartup=0 bool; 2020-10-10T16:52:29.889Z,1602348749.889 [SBIT](IMPORTANT): ESPComponent.loadAtStartup=1 bool; 2020-10-10T16:52:29.889Z,1602348749.889 [SBIT](IMPORTANT): ESPComponent.sampleTimeout=25 minute; 2020-10-10T16:52:29.889Z,1602348749.889 [SBIT](IMPORTANT): ESPComponent.simulateHardware=0 bool; 2020-10-10T16:52:29.889Z,1602348749.889 [SBIT](IMPORTANT): Express none WetLabsBB2FL.bin_mean_mass_concentration_of_chlorophyll_in_sea_water; 2020-10-10T16:52:29.889Z,1602348749.889 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=280 cubic_centimeter; 2020-10-10T16:52:29.889Z,1602348749.889 [SBIT](IMPORTANT): VerticalControl.massDefault=14 millimeter; 2020-10-10T16:52:30.286Z,1602348750.286 [MissionManager](IMPORTANT): Started mission Startup 2020-10-10T16:53:33.005Z,1602348813.005 [MissionManager](IMPORTANT): Started mission Default 2020-10-10T16:54:12.847Z,1602348852.847 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:54:12.847Z,1602348852.847 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:54:13.139Z,1602348853.139 [CommandLine](FAULT): Scheduling is paused 2020-10-10T16:54:21.199Z,1602348861.199 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:54:21.199Z,1602348861.199 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:54:51.862Z,1602348891.862 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:54:51.863Z,1602348891.863 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:55:22.499Z,1602348922.499 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:55:22.499Z,1602348922.499 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:55:53.014Z,1602348953.014 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:55:53.015Z,1602348953.015 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:56:23.767Z,1602348983.767 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:56:23.767Z,1602348983.767 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:56:54.274Z,1602349014.274 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:56:54.274Z,1602349014.274 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:57:26.398Z,1602349046.398 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:57:26.399Z,1602349046.399 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:57:54.667Z,1602349074.667 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:57:54.667Z,1602349074.667 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:58:24.979Z,1602349104.979 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:58:24.979Z,1602349104.979 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:58:55.367Z,1602349135.367 [DataOverHttps](IMPORTANT): Could not stat file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:58:55.367Z,1602349135.367 [DataOverHttps](CRITICAL): Could not open file Logs/20201010T165009/Courier0004.lzma 2020-10-10T16:59:11.273Z,1602349151.273 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 5.631156 min 2020-10-10T17:05:50.067Z,1602349550.067 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.277844 min 2020-10-10T17:06:00.011Z,1602349560.011 [DataOverHttps](IMPORTANT): SBD MOMSN=12850622, MTMSN=20201010T170558 2020-10-10T17:06:07.393Z,1602349567.393 [CommandLine](IMPORTANT): got command schedule asap "load Science/esp_sample_station.xml;set esp_sample_station.MissionTimeout 110 hour;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.LapRepeat 6 count;set esp_sample_station.StartWaypoint 2 count" 2cnhx 1 2.000000 2020-10-10T17:06:07.394Z,1602349567.394 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=2cnhx 2020-10-10T17:06:07.395Z,1602349567.395 [CommandLine](IMPORTANT): Scheduled #1 (#1 of 2 with id='2cnhx'): "load Science/esp_sample_station.xml;set esp_sample_station.MissionTimeout 110 hour;set esp_sample_station.NeedCommsTime 60 minute;set esp_sample_station.LapRepeat 6 count;set esp_sample_station.StartWaypoint 2 count" ASAP 2020-10-10T17:06:08.493Z,1602349568.493 [DataOverHttps](IMPORTANT): SBD MTMSN=20201010T170606 2020-10-10T17:06:15.802Z,1602349575.802 [CommandLine](IMPORTANT): got command schedule asap "set esp_sample_station.TransitYoYoMinDepth 3 meter;set esp_sample_station.SampleYoYoMinDepth 3 meter;set esp_sample_station.Speed 0.75 meter_per_second;run " 2cnhx 2 2.000000 2020-10-10T17:06:15.802Z,1602349575.802 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=2cnhx 2020-10-10T17:06:15.803Z,1602349575.803 [CommandLine](IMPORTANT): Scheduled #2 (#2 of 2 with id='2cnhx'): "set esp_sample_station.TransitYoYoMinDepth 3 meter;set esp_sample_station.SampleYoYoMinDepth 3 meter;set esp_sample_station.Speed 0.75 meter_per_second;run " ASAP 2020-10-10T17:06:15.925Z,1602349575.925 [CommandLine](IMPORTANT): got command load ./Missions/Science/esp_sample_station.xml 2020-10-10T17:06:18.848Z,1602349578.848 [CommandLine](IMPORTANT): Loaded ./Missions/Science/esp_sample_station.xml 2020-10-10T17:06:30.961Z,1602349590.961 [CommandLine](IMPORTANT): got command set esp_sample_station.MissionTimeout 110.000000 hour 2020-10-10T17:06:30.962Z,1602349590.962 [CommandLine](IMPORTANT): got command set esp_sample_station.NeedCommsTime 60.000000 minute 2020-10-10T17:06:30.963Z,1602349590.963 [CommandLine](IMPORTANT): got command set esp_sample_station.LapRepeat 6.000000 count 2020-10-10T17:06:30.964Z,1602349590.964 [CommandLine](IMPORTANT): got command set esp_sample_station.StartWaypoint 2.000000 count 2020-10-10T17:06:40.800Z,1602349600.800 [CommandLine](IMPORTANT): got command set esp_sample_station.TransitYoYoMinDepth 3.000000 meter 2020-10-10T17:06:40.801Z,1602349600.801 [CommandLine](IMPORTANT): got command set esp_sample_station.SampleYoYoMinDepth 3.000000 meter 2020-10-10T17:06:40.802Z,1602349600.802 [CommandLine](IMPORTANT): got command set esp_sample_station.Speed 0.750000 meter_per_second 2020-10-10T17:06:40.802Z,1602349600.802 [CommandLine](IMPORTANT): got command run 2020-10-10T17:06:40.806Z,1602349600.806 [CommandLine](IMPORTANT): Running 2020-10-10T17:06:40.971Z,1602349600.971 [MissionManager](IMPORTANT): Started mission esp_sample_station 2020-10-10T17:09:03.301Z,1602349743.301 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-10T17:09:03.725Z,1602349743.725 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-10T17:09:04.131Z,1602349744.131 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-10T17:09:04.559Z,1602349744.559 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-10T17:09:04.966Z,1602349744.966 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-10T17:09:09.353Z,1602349749.353 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2020-10-10T19:12:44.086Z,1602357164.086 [BPC1](FAULT): Battery stick #38 (s/n: 0139) reported TERMINATE_DISCHARGE_ALARM. Status code: 0x4AD0. 2020-10-10T19:30:58.046Z,1602358258.046 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600 2020-10-10T19:35:25.094Z,1602358525.094 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-10T19:35:26.666Z,1602358526.666 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::' 2020-10-10T19:35:36.813Z,1602358536.813 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-10T19:35:36.926Z,1602358536.926 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-10T19:36:15.150Z,1602358575.150 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-10T19:36:15.963Z,1602358575.963 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -6 2020-10-10T19:36:15.964Z,1602358575.964 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-10T19:36:16.415Z,1602358576.415 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 14.467344 m in 12.916498 degC water with 1.582000 ug/l chlorophyll fluorescence. 2020-10-10T19:36:17.169Z,1602358577.169 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-10T19:36:17.573Z,1602358577.573 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-10T19:36:19.189Z,1602358579.189 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-10T19:36:52.322Z,1602358612.322 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@45,archive_bac) 2020-10-10T19:36:53.940Z,1602358613.940 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2020-10-10T19:36:55.554Z,1602358615.554 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-10T19:39:23.418Z,1602358763.418 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2020-10-10T19:39:23.821Z,1602358763.821 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-10T20:29:02.674Z,1602361742.674 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@813.499) 2020-10-10T20:29:03.083Z,1602361743.083 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@813.499) 2020-10-10T20:29:05.502Z,1602361745.502 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@45,archive_bac) 2020-10-10T20:41:09.118Z,1602362469.118 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@45,archive_bac) 2020-10-10T20:41:34.570Z,1602362494.570 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-10T20:41:34.975Z,1602362494.975 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2020-10-10T20:46:34.769Z,1602362794.769 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2020-10-10T20:46:35.174Z,1602362795.174 [ESPComponent](IMPORTANT): [sample #1] ESP log summary report (3 messages): @12:36:28.80 Selecting Cartridge 45 @13:28:59.96 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @13:29:00.16 Sampled 813.5ml 2020-10-10T20:51:53.441Z,1602363113.441 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 2020-10-10T22:51:47.404Z,1602370307.404 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-10T22:51:58.748Z,1602370318.748 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005874 CHAN A1 (24V): 0.004039 CHAN A2 (12V): -0.003872 CHAN A3 (5V): -0.002148 CHAN B0 (3.3V): -0.001407 CHAN B1 (3.15aV): -0.000864 CHAN B2 (3.15bV): -0.001073 CHAN B3 (GND): 0.000132 OPEN: 0.007510 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-11T00:40:21.568Z,1602376821.568 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900 2020-10-11T00:47:21.426Z,1602377241.426 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-11T00:47:33.189Z,1602377253.189 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-11T00:47:33.312Z,1602377253.312 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-11T00:48:12.681Z,1602377292.681 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-11T00:48:13.497Z,1602377293.497 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -6 2020-10-11T00:48:13.497Z,1602377293.497 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-11T00:48:13.934Z,1602377293.934 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 13.937010 m in 12.278406 degC water with 0.621500 ug/l chlorophyll fluorescence. 2020-10-11T00:48:14.300Z,1602377294.300 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-11T00:48:14.701Z,1602377294.701 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-11T00:48:16.722Z,1602377296.722 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-11T00:48:49.850Z,1602377329.850 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@44,archive_bac) 2020-10-11T00:48:51.466Z,1602377331.466 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2020-10-11T00:48:52.273Z,1602377332.273 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-11T00:51:21.349Z,1602377481.349 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2020-10-11T00:51:21.754Z,1602377481.754 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-11T01:46:10.072Z,1602380770.072 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@963.627) 2020-10-11T01:46:10.475Z,1602380770.475 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@963.627) 2020-10-11T01:46:12.899Z,1602380772.899 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@44,archive_bac) 2020-10-11T01:58:16.506Z,1602381496.506 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@44,archive_bac) 2020-10-11T01:58:41.966Z,1602381521.966 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-11T01:58:42.361Z,1602381522.361 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2020-10-11T02:03:42.150Z,1602381822.150 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2020-10-11T02:03:42.554Z,1602381822.554 [ESPComponent](IMPORTANT): [sample #2] ESP log summary report (3 messages): @17:48:26.75 Selecting Cartridge 44 @18:46:07.62 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @18:46:07.82 Sampled 963.6ml 2020-10-11T02:11:03.118Z,1602382263.118 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 2020-10-11T03:25:33.053Z,1602386733.053 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000 2020-10-11T03:30:07.702Z,1602387007.702 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-11T03:30:19.473Z,1602387019.473 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-11T03:30:19.597Z,1602387019.597 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-11T03:30:58.553Z,1602387058.553 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-11T03:30:59.365Z,1602387059.365 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6 2020-10-11T03:30:59.365Z,1602387059.365 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-11T03:30:59.802Z,1602387059.802 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 11.169608 m in 13.208612 degC water with 4.779900 ug/l chlorophyll fluorescence. 2020-10-11T03:31:00.169Z,1602387060.169 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-11T03:31:00.573Z,1602387060.573 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-11T03:31:02.594Z,1602387062.594 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-11T03:31:35.726Z,1602387095.726 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@43,archive_bac) 2020-10-11T03:31:37.341Z,1602387097.341 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2020-10-11T03:31:38.150Z,1602387098.150 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-11T03:34:51.273Z,1602387291.273 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2020-10-11T03:34:51.682Z,1602387291.682 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-11T04:31:11.701Z,1602390671.701 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@952.165) 2020-10-11T04:31:12.108Z,1602390672.108 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@952.165) 2020-10-11T04:31:14.529Z,1602390674.529 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@42,archive_bac) 2020-10-11T04:43:18.115Z,1602391398.115 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@42,archive_bac) 2020-10-11T04:43:43.566Z,1602391423.566 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-11T04:43:43.970Z,1602391423.970 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2020-10-11T04:48:43.765Z,1602391723.765 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2020-10-11T04:48:44.170Z,1602391724.170 [ESPComponent](IMPORTANT): [sample #3] ESP log summary report (6 messages): @20:31:12.56 Selecting Cartridge 43 @20:34:20.43 Cartridge::Sampler::Leak in FILTERING -- Isolated cartridge pressure fell by 15% to 71.2psia @20:34:20.45 Retry #1 of 2 @20:34:20.49 Selecting Cartridge 42 @21:31:09.35 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @21:31:09.55 Sampled 952.2ml 2020-10-11T04:51:59.087Z,1602391919.087 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-11T04:52:11.137Z,1602391931.137 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005030 CHAN A1 (24V): 0.003700 CHAN A2 (12V): -0.004039 CHAN A3 (5V): -0.002096 CHAN B0 (3.3V): -0.000548 CHAN B1 (3.15aV): -0.001312 CHAN B2 (3.15bV): -0.001279 CHAN B3 (GND): 0.000021 OPEN: 0.007056 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-11T04:54:47.372Z,1602392087.372 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-11T04:54:47.834Z,1602392087.834 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-11T04:54:48.210Z,1602392088.210 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-11T04:54:48.600Z,1602392088.600 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-11T04:54:49.006Z,1602392089.006 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-11T04:54:53.531Z,1602392093.531 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2020-10-11T07:23:05.347Z,1602400985.347 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600 2020-10-11T07:27:51.169Z,1602401271.169 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-11T07:28:02.945Z,1602401282.945 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-11T07:28:03.066Z,1602401283.066 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-11T07:28:41.221Z,1602401321.221 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-11T07:28:42.032Z,1602401322.032 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6 2020-10-11T07:28:42.033Z,1602401322.033 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-11T07:28:42.507Z,1602401322.507 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 10.193124 m in 13.625696 degC water with 9.277300 ug/l chlorophyll fluorescence. 2020-10-11T07:28:42.838Z,1602401322.838 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-11T07:28:43.242Z,1602401323.242 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-11T07:28:45.262Z,1602401325.262 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-11T07:29:18.389Z,1602401358.389 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@41,archive_bac) 2020-10-11T07:29:20.005Z,1602401360.005 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2020-10-11T07:29:21.626Z,1602401361.626 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-11T07:31:49.893Z,1602401509.893 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2020-10-11T07:31:50.298Z,1602401510.298 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-11T08:09:36.854Z,1602403776.854 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@608.031) 2020-10-11T08:09:37.259Z,1602403777.259 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@608.031) 2020-10-11T08:09:39.682Z,1602403779.682 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@41,archive_bac) 2020-10-11T08:21:43.298Z,1602404503.298 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@41,archive_bac) 2020-10-11T08:22:08.759Z,1602404528.759 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-11T08:22:09.158Z,1602404529.158 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2020-10-11T08:24:26.261Z,1602404666.261 [ThrusterServo](FAULT): Invalid EZ Servo response:"" 2020-10-11T08:27:08.945Z,1602404828.945 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2020-10-11T08:27:09.349Z,1602404829.349 [ESPComponent](IMPORTANT): [sample #4] ESP log summary report (3 messages): @00:28:55.04 Selecting Cartridge 41 @01:09:34.07 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @01:09:34.28 Sampled 608.0ml 2020-10-11T08:33:56.432Z,1602405236.432 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 2020-10-11T10:52:11.505Z,1602413531.505 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-11T10:52:22.407Z,1602413542.407 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005614 CHAN A1 (24V): 0.003512 CHAN A2 (12V): -0.003871 CHAN A3 (5V): -0.002222 CHAN B0 (3.3V): -0.000702 CHAN B1 (3.15aV): -0.000997 CHAN B2 (3.15bV): -0.000779 CHAN B3 (GND): -0.000135 OPEN: 0.007587 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-11T12:05:35.095Z,1602417935.095 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900 2020-10-11T12:09:43.269Z,1602418183.269 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-11T12:09:55.053Z,1602418195.053 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-11T12:09:55.212Z,1602418195.212 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-11T12:10:33.413Z,1602418233.413 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-11T12:10:34.224Z,1602418234.224 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6 2020-10-11T12:10:34.225Z,1602418234.225 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-11T12:10:34.669Z,1602418234.669 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 14.021374 m in 13.479792 degC water with 4.892900 ug/l chlorophyll fluorescence. 2020-10-11T12:10:35.030Z,1602418235.030 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-11T12:10:35.433Z,1602418235.433 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-11T12:10:37.454Z,1602418237.454 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-11T12:11:10.586Z,1602418270.586 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@40,archive_bac) 2020-10-11T12:11:12.205Z,1602418272.205 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2020-10-11T12:11:13.822Z,1602418273.822 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-11T12:13:53.406Z,1602418433.406 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2020-10-11T12:13:53.813Z,1602418433.813 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-11T13:07:24.151Z,1602421644.151 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@937.276) 2020-10-11T13:07:24.554Z,1602421644.554 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@937.276) 2020-10-11T13:07:26.978Z,1602421646.978 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@40,archive_bac) 2020-10-11T13:19:30.578Z,1602422370.578 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@40,archive_bac) 2020-10-11T13:19:56.034Z,1602422396.034 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-11T13:19:56.438Z,1602422396.438 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2020-10-11T13:24:56.221Z,1602422696.221 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2020-10-11T13:24:56.626Z,1602422696.626 [ESPComponent](IMPORTANT): [sample #5] ESP log summary report (5 messages): @05:10:47.10 Selecting Cartridge 40 @05:11:26.36 Cartridge::Sampler::PossibleLeak in FILTERING -- Retained only 40.0 of initial 48.9psia @05:11:26.38 Retry #1 of 2 @06:07:21.72 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @06:07:21.91 Sampled 937.3ml 2020-10-11T13:30:55.418Z,1602423055.418 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 2020-10-11T14:56:49.258Z,1602428209.258 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000 2020-10-11T14:59:57.842Z,1602428397.842 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-11T15:00:09.605Z,1602428409.605 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-11T15:00:09.710Z,1602428409.710 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-11T15:00:48.309Z,1602428448.309 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-11T15:00:49.121Z,1602428449.121 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -6 2020-10-11T15:00:49.122Z,1602428449.122 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-11T15:00:49.558Z,1602428449.558 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 7.705751 m in 13.870691 degC water with 6.102000 ug/l chlorophyll fluorescence. 2020-10-11T15:00:49.927Z,1602428449.927 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-11T15:00:50.330Z,1602428450.330 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-11T15:00:52.349Z,1602428452.349 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-11T15:01:25.477Z,1602428485.477 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@39,archive_bac) 2020-10-11T15:01:27.094Z,1602428487.094 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2020-10-11T15:01:27.901Z,1602428487.901 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-11T15:03:56.981Z,1602428636.981 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2020-10-11T15:03:57.386Z,1602428637.386 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-11T15:41:02.848Z,1602430862.848 [ThrusterServo](FAULT): getVelocity uart error serial timeout 2020-10-11T15:41:02.848Z,1602430862.848 [ThrusterServo](FAULT): Thruster uart error: serial timeout 2020-10-11T15:51:56.392Z,1602431516.392 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@830.942) 2020-10-11T15:51:56.794Z,1602431516.794 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@830.942) 2020-10-11T15:51:59.217Z,1602431519.217 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@39,archive_bac) 2020-10-11T16:04:02.825Z,1602432242.825 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@39,archive_bac) 2020-10-11T16:04:28.279Z,1602432268.279 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-11T16:04:28.682Z,1602432268.682 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2020-10-11T16:09:28.466Z,1602432568.466 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2020-10-11T16:09:28.869Z,1602432568.869 [ESPComponent](IMPORTANT): [sample #6] ESP log summary report (3 messages): @08:01:02.17 Selecting Cartridge 39 @08:51:53.72 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @08:51:53.92 Sampled 830.9ml 2020-10-11T16:15:41.879Z,1602432941.879 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap 2020-10-11T16:15:42.276Z,1602432942.276 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 2020-10-11T16:15:42.675Z,1602432942.675 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 2020-10-11T16:15:43.087Z,1602432943.087 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 2020-10-11T16:15:43.483Z,1602432943.483 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 2020-10-11T16:15:47.943Z,1602432947.943 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 2020-10-11T16:52:22.601Z,1602435142.601 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-11T16:52:33.515Z,1602435153.515 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005518 CHAN A1 (24V): 0.003985 CHAN A2 (12V): -0.004234 CHAN A3 (5V): -0.002642 CHAN B0 (3.3V): -0.000895 CHAN B1 (3.15aV): -0.000878 CHAN B2 (3.15bV): -0.001359 CHAN B3 (GND): -0.000024 OPEN: 0.007415 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-11T18:19:44.085Z,1602440384.085 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600 2020-10-11T18:27:29.546Z,1602440849.546 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-11T18:27:41.325Z,1602440861.325 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-11T18:27:41.429Z,1602440861.429 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-11T18:28:20.453Z,1602440900.453 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-11T18:28:21.273Z,1602440901.273 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -6 2020-10-11T18:28:21.274Z,1602440901.274 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-11T18:28:21.702Z,1602440901.702 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 13.087934 m in 13.453089 degC water with 3.073600 ug/l chlorophyll fluorescence. 2020-10-11T18:28:22.070Z,1602440902.070 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-11T18:28:22.474Z,1602440902.474 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-11T18:28:24.494Z,1602440904.494 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-11T18:28:57.630Z,1602440937.630 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@38,archive_bac) 2020-10-11T18:28:59.246Z,1602440939.246 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2020-10-11T18:29:00.056Z,1602440940.056 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-11T18:31:27.918Z,1602441087.918 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2020-10-11T18:31:28.322Z,1602441088.322 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-11T18:58:28.421Z,1602442708.421 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2020-10-11T19:18:17.489Z,1602443897.489 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@784.44) 2020-10-11T19:18:17.894Z,1602443897.894 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@784.44) 2020-10-11T19:18:20.317Z,1602443900.317 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@38,archive_bac) 2020-10-11T19:30:23.909Z,1602444623.909 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@38,archive_bac) 2020-10-11T19:30:49.364Z,1602444649.364 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-11T19:30:49.761Z,1602444649.761 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2020-10-11T19:35:49.534Z,1602444949.534 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2020-10-11T19:35:49.938Z,1602444949.938 [ESPComponent](IMPORTANT): [sample #7] ESP log summary report (3 messages): @11:28:34.50 Selecting Cartridge 38 @12:18:14.90 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @12:18:15.10 Sampled 784.4ml 2020-10-11T19:42:57.705Z,1602445377.705 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 2020-10-11T20:24:28.940Z,1602447868.940 [BPC1](FAULT): Failed to receive data from 2 sticks prior to timeout. Missing stick IDs are: 38, 39. 2020-10-11T22:52:33.728Z,1602456753.728 [CBIT](IMPORTANT): Beginning ground fault scan 2020-10-11T22:52:44.616Z,1602456764.616 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.005591 CHAN A1 (24V): 0.003052 CHAN A2 (12V): -0.004364 CHAN A3 (5V): -0.002145 CHAN B0 (3.3V): -0.000492 CHAN B1 (3.15aV): -0.000860 CHAN B2 (3.15bV): -0.001315 CHAN B3 (GND): 0.000002 OPEN: 0.007324 Full Scale Calc: 4.765 mA, -1.589 mA 2020-10-11T23:32:52.215Z,1602459172.215 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900 2020-10-11T23:41:23.177Z,1602459683.177 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-11T23:41:34.949Z,1602459694.949 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-11T23:41:35.224Z,1602459695.224 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-11T23:42:13.349Z,1602459733.349 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-11T23:42:14.164Z,1602459734.164 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -6 2020-10-11T23:42:14.164Z,1602459734.164 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-11T23:42:14.597Z,1602459734.597 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 15.649379 m in 12.322809 degC water with 0.689300 ug/l chlorophyll fluorescence. 2020-10-11T23:42:14.966Z,1602459734.966 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-11T23:42:15.373Z,1602459735.373 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-11T23:42:17.389Z,1602459737.389 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-11T23:42:50.525Z,1602459770.525 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@37,archive_bac) 2020-10-11T23:42:52.138Z,1602459772.138 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2020-10-11T23:42:52.946Z,1602459772.946 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-11T23:45:21.221Z,1602459921.221 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2020-10-11T23:45:21.626Z,1602459921.626 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-12T00:31:14.642Z,1602462674.642 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@737.312) 2020-10-12T00:31:15.045Z,1602462675.045 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@737.312) 2020-10-12T00:31:17.469Z,1602462677.469 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@37,archive_bac) 2020-10-12T00:43:21.086Z,1602463401.086 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@37,archive_bac) 2020-10-12T00:43:46.539Z,1602463426.539 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-12T00:43:46.942Z,1602463426.942 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2020-10-12T00:48:46.718Z,1602463726.718 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2020-10-12T00:48:47.122Z,1602463727.122 [ESPComponent](IMPORTANT): [sample #8] ESP log summary report (3 messages): @16:42:26.96 Selecting Cartridge 37 @17:31:11.96 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @17:31:12.16 Sampled 737.3ml 2020-10-12T01:01:32.569Z,1602464492.569 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 2020-10-12T02:36:33.928Z,1602470193.928 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000 2020-10-12T02:43:41.357Z,1602470621.357 [BPC1](FAULT): Failed to receive data from 2 sticks prior to timeout. Missing stick IDs are: 38, 39. 2020-10-12T02:44:32.272Z,1602470672.272 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 2020-10-12T02:44:44.062Z,1602470684.062 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2020-10-12T02:44:44.187Z,1602470684.187 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2020-10-12T02:45:23.949Z,1602470723.949 [ESPComponent](IMPORTANT): ESP has connected as client 2020-10-12T02:45:24.761Z,1602470724.761 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -6 2020-10-12T02:45:24.761Z,1602470724.761 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2020-10-12T02:45:25.206Z,1602470725.206 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 16.372242 m in 12.900995 degC water with 0.994400 ug/l chlorophyll fluorescence. 2020-10-12T02:45:25.565Z,1602470725.565 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2020-10-12T02:45:25.970Z,1602470725.970 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2020-10-12T02:45:27.989Z,1602470727.989 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2020-10-12T02:46:01.122Z,1602470761.122 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@36,archive_bac) 2020-10-12T02:46:02.737Z,1602470762.737 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2020-10-12T02:46:03.546Z,1602470763.546 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2020-10-12T02:48:32.221Z,1602470912.221 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0) 2020-10-12T02:48:32.626Z,1602470912.626 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2020-10-12T03:36:57.542Z,1602473817.542 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@835.592) 2020-10-12T03:36:57.946Z,1602473817.946 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@835.592) 2020-10-12T03:37:00.370Z,1602473820.370 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@36,archive_bac) 2020-10-12T03:49:03.970Z,1602474543.970 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@36,archive_bac) 2020-10-12T03:49:29.431Z,1602474569.431 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs & 2020-10-12T03:49:29.834Z,1602474569.834 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2020-10-12T03:54:29.622Z,1602474869.622 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally. 2020-10-12T03:54:30.026Z,1602474870.026 [ESPComponent](IMPORTANT): [sample #9] ESP log summary report (3 messages): @19:45:38.01 Selecting Cartridge 36 @20:36:55.26 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged @20:36:55.46 Sampled 835.6ml 2020-10-12T03:59:55.699Z,1602475195.699 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=39982, MT Status=1, MTMSN=2781