2022-10-12T21:17:29.763Z,1665609449.763 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg 2022-10-12T21:17:32.499Z,1665609452.499 [SBIT](IMPORTANT): git: 2022-10-11 2022-10-12T21:17:32.499Z,1665609452.499 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8 2022-10-12T21:18:19.066Z,1665609499.066 [SBIT](IMPORTANT): Beginning Startup BIT 2022-10-12T21:18:19.071Z,1665609499.071 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-12T21:18:30.047Z,1665609510.047 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003426 CHAN A1 (24V): 0.006855 CHAN A2 (12V): -0.004331 CHAN A3 (5V): -0.004212 CHAN B0 (3.3V): -0.001978 CHAN B1 (3.15aV): -0.001478 CHAN B2 (3.15bV): -0.001035 CHAN B3 (GND): -0.000906 OPEN: 0.012238 Full Scale: +/- 1 mA 2022-10-12T21:19:12.798Z,1665609552.798 [SBIT](IMPORTANT): SBIT PASSED 2022-10-12T21:19:12.798Z,1665609552.798 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg 2022-10-12T21:19:12.799Z,1665609552.799 [SBIT](IMPORTANT): BPC1.batteryMissingStickThreshold=2 count; 2022-10-12T21:19:12.800Z,1665609552.800 [SBIT](IMPORTANT): BPC1.loadAtStartup=0 bool; 2022-10-12T21:19:12.802Z,1665609552.802 [SBIT](IMPORTANT): CBIT.runNavErrorCritical=0 bool; 2022-10-12T21:19:12.802Z,1665609552.802 [SBIT](IMPORTANT): DAT.loadAtStartup=1 bool; 2022-10-12T21:19:12.802Z,1665609552.802 [SBIT](IMPORTANT): DAT.sbdAddress=-1 enum; 2022-10-12T21:19:12.802Z,1665609552.802 [SBIT](IMPORTANT): DVL_micro.loadAtStartup=0 bool; 2022-10-12T21:19:12.802Z,1665609552.802 [SBIT](IMPORTANT): Depth_Keller.offset=-0.8 decibar; 2022-10-12T21:19:12.802Z,1665609552.802 [SBIT](IMPORTANT): ElevatorServo.offsetAngle=1 degree; 2022-10-12T21:19:12.803Z,1665609552.803 [SBIT](IMPORTANT): Express linearApproximation NAL9602.sigQuality 1.000000 count; 2022-10-12T21:19:12.803Z,1665609552.803 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff470nm; 2022-10-12T21:19:12.803Z,1665609552.803 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff650nm; 2022-10-12T21:19:12.803Z,1665609552.803 [SBIT](IMPORTANT): RudderServo.offsetAngle=0 degree; 2022-10-12T21:19:12.803Z,1665609552.803 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=86.344810 cubic_centimeter; 2022-10-12T21:19:12.803Z,1665609552.803 [SBIT](IMPORTANT): VerticalControl.massBackOnGoToSurface=0 bool; 2022-10-12T21:19:12.803Z,1665609552.803 [SBIT](IMPORTANT): VerticalControl.massDefault=-6.557592 millimeter; 2022-10-12T21:19:12.803Z,1665609552.803 [SBIT](IMPORTANT): VerticalControl.massPositionLimitAft=-37 millimeter; 2022-10-12T21:19:13.202Z,1665609553.202 [MissionManager](IMPORTANT): Started mission Startup 2022-10-12T21:21:14.001Z,1665609674.001 [MissionManager](IMPORTANT): Started mission Default 2022-10-12T21:23:16.803Z,1665609796.803 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-12T21:31:33.477Z,1665610293.477 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 10.317197 min 2022-10-12T21:36:37.575Z,1665610597.575 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-12T21:37:41.983Z,1665610661.983 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 16.459465 min 2022-10-12T21:42:40.407Z,1665610960.407 [DataOverHttps](IMPORTANT): SBD MTMSN=20221012T214236 2022-10-12T21:42:47.057Z,1665610967.057 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-12T21:42:49.673Z,1665610969.673 [CommandExec](IMPORTANT): got command schedule asap "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 35 count;set LocationDepthSampling.MissionTimeout 90 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s" 409ma 1 3.000000 2022-10-12T21:42:49.673Z,1665610969.673 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=409ma 2022-10-12T21:42:49.674Z,1665610969.674 [CommandExec](IMPORTANT): Scheduled #1 (#1 of 3 with id='409ma'): "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 35 count;set LocationDepthSampling.MissionTimeout 90 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s", ASAP 2022-10-12T21:42:51.582Z,1665610971.582 [DataOverHttps](IMPORTANT): SBD MTMSN=20221012T214247 2022-10-12T21:42:59.143Z,1665610979.143 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m" 409ma 2 3.000000 2022-10-12T21:42:59.144Z,1665610979.144 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=409ma 2022-10-12T21:42:59.145Z,1665610979.145 [CommandExec](IMPORTANT): Scheduled #2 (#2 of 3 with id='409ma'): "set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m", ASAP 2022-10-12T21:43:00.093Z,1665610980.093 [DataOverHttps](IMPORTANT): SBD MTMSN=20221012T214255 2022-10-12T21:43:07.511Z,1665610987.511 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run" 409ma 3 3.000000 2022-10-12T21:43:07.511Z,1665610987.511 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=409ma 2022-10-12T21:43:07.512Z,1665610987.512 [CommandExec](IMPORTANT): Scheduled #3 (#3 of 3 with id='409ma'): "set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run", ASAP 2022-10-12T21:43:07.682Z,1665610987.682 [CommandExec](IMPORTANT): got command load ./Missions/Science/location_depth_sampling.xml 2022-10-12T21:43:17.008Z,1665610997.008 [CommandExec](IMPORTANT): Loaded ./Missions/Science/location_depth_sampling.xml 2022-10-12T21:43:30.518Z,1665611010.518 [CommandExec](IMPORTANT): got command set LocationDepthSampling.NumSamplers 35 count 2022-10-12T21:43:30.519Z,1665611010.519 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MissionTimeout 90 hour 2022-10-12T21:43:30.519Z,1665611010.519 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedTransit 0.8 meter_per_second 2022-10-12T21:43:30.520Z,1665611010.520 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedSampling 0.8 meter_per_second 2022-10-12T21:43:35.160Z,1665611015.160 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 22.345787 min 2022-10-12T21:43:56.850Z,1665611036.850 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MaxDepth 180 meter 2022-10-12T21:43:56.851Z,1665611036.851 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LatSampling 36.80 degree 2022-10-12T21:43:56.856Z,1665611036.856 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LonSampling -122.097 degree 2022-10-12T21:43:56.857Z,1665611036.857 [CommandExec](IMPORTANT): got command set LocationDepthSampling.Dep1 150 meter 2022-10-12T21:44:01.435Z,1665611041.435 [CommandExec](IMPORTANT): got command set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 meter_per_second 2022-10-12T21:44:01.435Z,1665611041.435 [CommandExec](IMPORTANT): got command run 2022-10-12T21:44:01.486Z,1665611041.486 [CommandExec](IMPORTANT): Running 2022-10-12T21:44:01.497Z,1665611041.497 [MissionManager](IMPORTANT): Started mission LocationDepthSampling 2022-10-12T21:44:01.610Z,1665611041.610 [LocationDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2022-10-12T21:45:26.769Z,1665611126.769 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47317, MT Status=1, MTMSN=3522 2022-10-12T21:45:27.556Z,1665611127.556 [CommandExec](IMPORTANT): got command schedule asap "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 35 count;set LocationDepthSampling.MissionTimeout 90 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s" 406yv 1 3.000000 2022-10-12T21:45:27.557Z,1665611127.557 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=406yv 2022-10-12T21:45:27.558Z,1665611127.558 [CommandExec](IMPORTANT): Scheduled #4 (#1 of 3 with id='406yv'): "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 35 count;set LocationDepthSampling.MissionTimeout 90 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s", ASAP 2022-10-12T21:48:17.986Z,1665611297.986 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-12T21:53:56.915Z,1665611636.915 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000 2022-10-12T22:01:55.757Z,1665612115.757 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000 2022-10-12T22:01:58.171Z,1665612118.171 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-12T22:01:58.572Z,1665612118.572 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 150.000000 m . 2022-10-12T22:01:58.982Z,1665612118.982 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-12T22:13:57.711Z,1665612837.711 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.004654 m , settling for 5.000000 min 2022-10-12T22:19:00.270Z,1665613140.270 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:' 2022-10-12T22:19:10.404Z,1665613150.404 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-12T22:19:10.734Z,1665613150.734 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-12T22:20:10.734Z,1665613210.734 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-12T22:20:11.547Z,1665613211.547 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -6 2022-10-12T22:20:11.548Z,1665613211.548 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-12T22:20:11.993Z,1665613211.993 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.387604 m in 9.934351 degC water with 0.067600 ug/l chlorophyll fluorescence. 2022-10-12T22:20:12.755Z,1665613212.755 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-12T22:20:13.158Z,1665613213.158 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-12T22:20:14.773Z,1665613214.773 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-12T22:20:49.115Z,1665613249.115 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@36,archive_bac) 2022-10-12T22:20:51.946Z,1665613251.946 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0) 2022-10-12T22:20:51.947Z,1665613251.947 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-12T22:20:52.350Z,1665613252.350 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-12T22:23:10.923Z,1665613390.923 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0) 2022-10-12T22:23:11.326Z,1665613391.326 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-12T23:08:07.245Z,1665616087.245 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@1000.0) 2022-10-12T23:08:07.650Z,1665616087.650 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-12T23:08:09.670Z,1665616089.670 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-12T23:08:10.073Z,1665616090.073 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@36,archive_bac) 2022-10-12T23:20:13.658Z,1665616813.658 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@36,archive_bac) 2022-10-12T23:20:23.354Z,1665616823.354 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-12T23:20:23.761Z,1665616823.761 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED 2022-10-12T23:25:23.531Z,1665617123.531 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally. 2022-10-12T23:25:24.393Z,1665617124.393 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed. 2022-10-12T23:25:30.864Z,1665617130.864 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-12T23:42:46.707Z,1665618166.707 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-12T23:44:14.941Z,1665618254.941 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000 2022-10-12T23:49:16.791Z,1665618556.791 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-12T23:55:31.315Z,1665618931.315 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000 2022-10-12T23:55:33.760Z,1665618933.760 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-12T23:55:34.137Z,1665618934.137 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Target depth = 5.000000 m . 2022-10-12T23:55:34.546Z,1665618934.546 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-12T23:59:37.353Z,1665619177.353 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.988626 m , settling for 5.000000 min 2022-10-13T00:04:50.113Z,1665619490.113 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T00:04:50.419Z,1665619490.419 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T00:05:51.201Z,1665619551.201 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T00:05:52.092Z,1665619552.092 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -6 2022-10-13T00:05:52.093Z,1665619552.093 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T00:05:52.496Z,1665619552.496 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.116970 m in 13.881403 degC water with 6.466720 ug/l chlorophyll fluorescence. 2022-10-13T00:05:52.820Z,1665619552.820 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T00:05:53.221Z,1665619553.221 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T00:05:55.241Z,1665619555.241 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T00:06:29.582Z,1665619589.582 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@35,archive_bac) 2022-10-13T00:06:31.601Z,1665619591.601 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T00:06:32.007Z,1665619592.007 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T00:06:32.411Z,1665619592.411 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T00:08:50.985Z,1665619730.985 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T00:08:51.393Z,1665619731.393 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T01:12:07.026Z,1665623527.026 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@999.5) 2022-10-13T01:12:07.430Z,1665623527.430 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@999.5) 2022-10-13T01:12:09.450Z,1665623529.450 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T01:12:09.854Z,1665623529.854 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@35,archive_bac) 2022-10-13T01:24:13.438Z,1665624253.438 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@35,archive_bac) 2022-10-13T01:24:23.539Z,1665624263.539 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T01:24:23.945Z,1665624263.945 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED 2022-10-13T01:29:23.718Z,1665624563.718 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally. 2022-10-13T01:29:24.574Z,1665624564.574 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed. 2022-10-13T01:29:31.042Z,1665624571.042 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-13T01:35:00.701Z,1665624900.701 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T01:36:17.031Z,1665624977.031 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000 2022-10-13T01:41:18.779Z,1665625278.779 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T01:49:47.441Z,1665625787.441 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000 2022-10-13T01:49:49.884Z,1665625789.884 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-13T01:49:50.278Z,1665625790.278 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 3.000000 count . Target depth = 150.000000 m . 2022-10-13T01:49:50.667Z,1665625790.667 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T02:01:16.267Z,1665626476.267 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.059799 m , settling for 5.000000 min 2022-10-13T02:06:29.036Z,1665626789.036 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T02:06:29.161Z,1665626789.161 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T02:07:29.961Z,1665626849.961 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T02:07:30.775Z,1665626850.775 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6 2022-10-13T02:07:30.776Z,1665626850.776 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T02:07:31.233Z,1665626851.233 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.423035 m in 10.155115 degC water with 0.087360 ug/l chlorophyll fluorescence. 2022-10-13T02:07:31.578Z,1665626851.578 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T02:07:31.982Z,1665626851.982 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T02:07:34.001Z,1665626854.001 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T02:08:08.342Z,1665626888.342 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@34,archive_bac) 2022-10-13T02:08:11.170Z,1665626891.170 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T02:08:11.171Z,1665626891.171 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T02:08:11.573Z,1665626891.573 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T02:10:30.150Z,1665627030.150 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T02:10:30.553Z,1665627030.553 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T02:54:59.425Z,1665629699.425 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1000.0) 2022-10-13T02:54:59.829Z,1665629699.829 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-13T02:55:01.850Z,1665629701.850 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T02:55:02.254Z,1665629702.254 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@34,archive_bac) 2022-10-13T03:07:05.829Z,1665630425.829 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@34,archive_bac) 2022-10-13T03:07:15.530Z,1665630435.530 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T03:07:15.930Z,1665630435.930 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED 2022-10-13T03:12:15.697Z,1665630735.697 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally. 2022-10-13T03:12:16.579Z,1665630736.579 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 3.000000 count completed. 2022-10-13T03:12:23.020Z,1665630743.020 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-13T03:18:30.387Z,1665631110.387 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-13T03:18:41.259Z,1665631121.259 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002579 CHAN A1 (24V): 0.007663 CHAN A2 (12V): -0.004040 CHAN A3 (5V): -0.003387 CHAN B0 (3.3V): -0.001928 CHAN B1 (3.15aV): -0.001184 CHAN B2 (3.15bV): -0.001338 CHAN B3 (GND): -0.001233 OPEN: 0.012251 Full Scale: +/- 1 mA 2022-10-13T03:30:48.742Z,1665631848.742 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000 2022-10-13T03:35:49.670Z,1665632149.670 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T03:43:04.432Z,1665632584.432 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000 2022-10-13T03:43:06.823Z,1665632586.823 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-13T03:43:07.230Z,1665632587.230 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 4.000000 count . Target depth = 5.000000 m . 2022-10-13T03:43:07.650Z,1665632587.650 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T03:47:06.420Z,1665632826.420 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.004333 m , settling for 5.000000 min 2022-10-13T03:52:19.164Z,1665633139.164 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T03:52:19.471Z,1665633139.471 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T03:53:19.865Z,1665633199.865 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T03:53:20.677Z,1665633200.677 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6 2022-10-13T03:53:20.677Z,1665633200.677 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T03:53:21.121Z,1665633201.121 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.623340 m in 13.837427 degC water with 4.884880 ug/l chlorophyll fluorescence. 2022-10-13T03:53:21.481Z,1665633201.481 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T03:53:21.886Z,1665633201.886 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T03:53:23.908Z,1665633203.908 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T03:53:58.246Z,1665633238.246 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@33,archive_bac) 2022-10-13T03:54:01.074Z,1665633241.074 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T03:54:01.075Z,1665633241.075 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T03:54:01.478Z,1665633241.478 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T03:56:20.050Z,1665633380.050 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T03:56:20.455Z,1665633380.455 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T04:45:27.278Z,1665636327.278 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@824.9) 2022-10-13T04:45:27.681Z,1665636327.681 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@824.9) 2022-10-13T04:45:29.702Z,1665636329.702 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T04:45:30.105Z,1665636330.105 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@33,archive_bac) 2022-10-13T04:57:33.677Z,1665637053.677 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@33,archive_bac) 2022-10-13T04:57:43.780Z,1665637063.780 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T04:57:44.181Z,1665637064.181 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED 2022-10-13T05:02:43.953Z,1665637363.953 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally. 2022-10-13T05:02:44.838Z,1665637364.838 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 4.000000 count completed. 2022-10-13T05:02:51.267Z,1665637371.267 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-13T05:08:20.524Z,1665637700.524 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T05:38:42.974Z,1665639522.974 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000 2022-10-13T06:19:59.171Z,1665641999.171 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000 2022-10-13T06:20:01.601Z,1665642001.601 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-13T06:20:02.003Z,1665642002.003 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 5.000000 count . Target depth = 150.000000 m . 2022-10-13T06:20:02.412Z,1665642002.412 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T06:31:46.612Z,1665642706.612 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.047256 m , settling for 5.000000 min 2022-10-13T06:36:59.356Z,1665643019.356 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T06:36:59.465Z,1665643019.465 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T06:37:59.878Z,1665643079.878 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T06:38:00.693Z,1665643080.693 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6 2022-10-13T06:38:00.694Z,1665643080.694 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T06:38:01.135Z,1665643081.135 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.731110 m in 9.982965 degC water with 0.069680 ug/l chlorophyll fluorescence. 2022-10-13T06:38:01.494Z,1665643081.494 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T06:38:01.897Z,1665643081.897 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T06:38:03.918Z,1665643083.918 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T06:38:38.257Z,1665643118.257 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@32,archive_bac) 2022-10-13T06:38:40.278Z,1665643120.278 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T06:38:40.682Z,1665643120.682 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T06:38:41.086Z,1665643121.086 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T06:41:00.466Z,1665643260.466 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T06:41:00.870Z,1665643260.870 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T07:26:58.213Z,1665646018.213 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0) 2022-10-13T07:26:58.617Z,1665646018.617 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-13T07:27:00.638Z,1665646020.638 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T07:27:01.041Z,1665646021.041 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@32,archive_bac) 2022-10-13T07:39:04.617Z,1665646744.617 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@32,archive_bac) 2022-10-13T07:39:14.317Z,1665646754.317 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T07:39:14.718Z,1665646754.718 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED 2022-10-13T07:44:14.485Z,1665647054.485 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally. 2022-10-13T07:44:15.342Z,1665647055.342 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 5.000000 count completed. 2022-10-13T07:44:21.810Z,1665647061.810 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-13T08:20:07.921Z,1665649207.921 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000 2022-10-13T08:25:08.637Z,1665649508.637 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T08:32:26.213Z,1665649946.213 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000 2022-10-13T08:32:28.615Z,1665649948.615 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-13T08:32:29.020Z,1665649949.020 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 6.000000 count . Target depth = 5.000000 m . 2022-10-13T08:32:29.433Z,1665649949.433 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T08:36:28.229Z,1665650188.229 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.977407 m , settling for 5.000000 min 2022-10-13T08:41:40.961Z,1665650500.961 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T08:41:41.077Z,1665650501.077 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T08:42:41.864Z,1665650561.864 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T08:42:42.677Z,1665650562.677 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -6 2022-10-13T08:42:42.677Z,1665650562.677 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T08:42:43.127Z,1665650563.127 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.407313 m in 13.063593 degC water with 3.502720 ug/l chlorophyll fluorescence. 2022-10-13T08:42:43.481Z,1665650563.481 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T08:42:43.886Z,1665650563.886 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T08:42:45.908Z,1665650565.908 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T08:43:20.246Z,1665650600.246 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@31,archive_bac) 2022-10-13T08:43:22.265Z,1665650602.265 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T08:43:22.673Z,1665650602.673 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T08:43:23.074Z,1665650603.074 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T08:45:42.051Z,1665650742.051 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T08:45:42.456Z,1665650742.456 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T09:18:41.460Z,1665652721.460 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-13T09:18:52.318Z,1665652732.318 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002400 CHAN A1 (24V): 0.007445 CHAN A2 (12V): -0.003993 CHAN A3 (5V): -0.003445 CHAN B0 (3.3V): -0.001613 CHAN B1 (3.15aV): -0.001579 CHAN B2 (3.15bV): -0.001026 CHAN B3 (GND): -0.001156 OPEN: 0.012213 Full Scale: +/- 1 mA 2022-10-13T09:30:11.737Z,1665653411.737 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@760.3) 2022-10-13T09:30:12.142Z,1665653412.142 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@760.3) 2022-10-13T09:30:14.163Z,1665653414.163 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T09:30:14.571Z,1665653414.571 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@31,archive_bac) 2022-10-13T09:42:18.536Z,1665654138.536 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@31,archive_bac) 2022-10-13T09:42:28.230Z,1665654148.230 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T09:42:28.633Z,1665654148.633 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED 2022-10-13T09:47:28.411Z,1665654448.411 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally. 2022-10-13T09:47:29.273Z,1665654449.273 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 6.000000 count completed. 2022-10-13T09:47:35.740Z,1665654455.740 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-13T09:53:03.753Z,1665654783.753 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T09:54:24.793Z,1665654864.793 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000 2022-10-13T09:59:02.430Z,1665655142.430 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000 2022-10-13T09:59:04.884Z,1665655144.884 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-13T09:59:05.273Z,1665655145.273 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 7.000000 count . Target depth = 150.000000 m . 2022-10-13T09:59:05.659Z,1665655145.659 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T09:59:25.845Z,1665655165.845 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T10:10:53.094Z,1665655853.094 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.040512 m , settling for 5.000000 min 2022-10-13T10:16:05.860Z,1665656165.860 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T10:16:06.157Z,1665656166.157 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T10:17:06.941Z,1665656226.941 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T10:17:07.753Z,1665656227.753 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -6 2022-10-13T10:17:07.753Z,1665656227.753 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T10:17:08.206Z,1665656228.206 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.392990 m in 9.825006 degC water with 0.061360 ug/l chlorophyll fluorescence. 2022-10-13T10:17:08.557Z,1665656228.557 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T10:17:08.962Z,1665656228.962 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T10:17:10.985Z,1665656230.985 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T10:17:45.322Z,1665656265.322 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@30,archive_bac) 2022-10-13T10:17:47.745Z,1665656267.745 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T10:17:47.746Z,1665656267.746 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T10:17:48.150Z,1665656268.150 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T10:20:07.132Z,1665656407.132 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T10:20:07.535Z,1665656407.535 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T11:03:20.829Z,1665659000.829 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1000.0) 2022-10-13T11:03:21.233Z,1665659001.233 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-13T11:03:23.254Z,1665659003.254 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T11:03:23.657Z,1665659003.657 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@30,archive_bac) 2022-10-13T11:15:27.233Z,1665659727.233 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@30,archive_bac) 2022-10-13T11:15:36.932Z,1665659736.932 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T11:15:37.334Z,1665659737.334 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED 2022-10-13T11:20:37.102Z,1665660037.102 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally. 2022-10-13T11:20:37.978Z,1665660037.978 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 7.000000 count completed. 2022-10-13T11:20:44.414Z,1665660044.414 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-13T11:38:18.863Z,1665661098.863 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T11:39:30.900Z,1665661170.900 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000 2022-10-13T11:43:11.205Z,1665661391.205 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000 2022-10-13T11:43:13.614Z,1665661393.614 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-13T11:43:14.019Z,1665661394.019 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 8.000000 count . Target depth = 5.000000 m . 2022-10-13T11:43:14.431Z,1665661394.431 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T11:44:32.778Z,1665661472.778 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T11:46:52.983Z,1665661612.983 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.976510 m , settling for 5.000000 min 2022-10-13T11:52:05.757Z,1665661925.757 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T11:52:05.889Z,1665661925.889 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T11:53:06.677Z,1665661986.677 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T11:53:07.488Z,1665661987.488 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -6 2022-10-13T11:53:07.488Z,1665661987.488 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T11:53:07.948Z,1665661987.948 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.093809 m in 13.459955 degC water with 4.915040 ug/l chlorophyll fluorescence. 2022-10-13T11:53:08.293Z,1665661988.293 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T11:53:08.699Z,1665661988.699 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T11:53:10.719Z,1665661990.719 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T11:53:45.061Z,1665662025.061 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@29,archive_bac) 2022-10-13T11:53:47.891Z,1665662027.891 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T11:53:47.892Z,1665662027.892 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T11:53:48.294Z,1665662028.294 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T11:56:06.469Z,1665662166.469 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T11:56:06.865Z,1665662166.865 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T12:43:37.525Z,1665665017.525 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@768.9) 2022-10-13T12:43:37.930Z,1665665017.930 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@768.9) 2022-10-13T12:43:39.951Z,1665665019.951 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T12:43:40.356Z,1665665020.356 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@29,archive_bac) 2022-10-13T12:55:43.940Z,1665665743.940 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@29,archive_bac) 2022-10-13T12:55:53.634Z,1665665753.634 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T12:55:54.034Z,1665665754.034 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED 2022-10-13T13:00:53.810Z,1665666053.810 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally. 2022-10-13T13:00:54.700Z,1665666054.700 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 8.000000 count completed. 2022-10-13T13:01:01.130Z,1665666061.130 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-13T13:06:32.818Z,1665666392.818 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T13:07:41.118Z,1665666461.118 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000 2022-10-13T13:12:42.984Z,1665666762.984 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T13:28:22.722Z,1665667702.722 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000 2022-10-13T13:28:25.158Z,1665667705.158 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-13T13:28:25.542Z,1665667705.542 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 9.000000 count . Target depth = 150.000000 m . 2022-10-13T13:28:25.965Z,1665667705.965 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T13:41:28.524Z,1665668488.524 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 149.989395 m , settling for 5.000000 min 2022-10-13T13:46:41.276Z,1665668801.276 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T13:46:41.387Z,1665668801.387 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T13:47:42.609Z,1665668862.609 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T13:47:43.420Z,1665668863.420 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -6 2022-10-13T13:47:43.421Z,1665668863.421 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T13:47:43.879Z,1665668863.879 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.534683 m in 9.915033 degC water with 0.079040 ug/l chlorophyll fluorescence. 2022-10-13T13:47:44.225Z,1665668864.225 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T13:47:44.629Z,1665668864.629 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T13:47:46.649Z,1665668866.649 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T13:48:20.993Z,1665668900.993 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@28,archive_bac) 2022-10-13T13:48:23.821Z,1665668903.821 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T13:48:23.822Z,1665668903.822 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T13:48:24.225Z,1665668904.225 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T13:50:54.110Z,1665669054.110 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@0.0) 2022-10-13T13:51:01.382Z,1665669061.382 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T13:51:01.794Z,1665669061.794 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED 2022-10-13T13:56:01.553Z,1665669361.553 [ESPComponent](CRITICAL): [sample #9] ESP sampling sequence terminated with error: In state S_WAITING_FOR_FILTERING got exception while awaiting completion of cmd='Cmd.startFiltering' exception=<>. Elapsed time since command issued: 153.1167 2022-10-13T13:56:01.706Z,1665669361.706 [CommandExec](FAULT): Scheduling is paused for 1 commands 2022-10-13T13:56:02.713Z,1665669362.713 [MissionManager](IMPORTANT): Started mission Default 2022-10-13T14:02:37.612Z,1665669757.612 [MassServo](FAULT): Overload Error 2022-10-13T14:06:51.921Z,1665670011.921 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T14:16:55.637Z,1665670615.637 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 20.876373 min 2022-10-13T14:16:57.244Z,1665670617.244 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-10-13T14:22:08.102Z,1665670928.102 [DataOverHttps](IMPORTANT): SBD MOMSN=17244444, MTMSN=20221013T142203 2022-10-13T14:22:23.650Z,1665670943.650 [CommandExec](IMPORTANT): got command ! onESPclient chrisp -- spare:2 2022-10-13T14:22:24.213Z,1665670944.213 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-10-13UTC142224_onESPclient 2022-10-13T14:22:41.893Z,1665670961.893 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T14:24:46.569Z,1665671086.569 [DataOverHttps](IMPORTANT): SBD MTMSN=20221013T142442 2022-10-13T14:24:53.971Z,1665671093.971 [CommandExec](IMPORTANT): got command ! onESPclient chrisp -- slot spare:2 2022-10-13T14:24:54.324Z,1665671094.324 [CommandExec](IMPORTANT): /mnt/mmc/started/2022-10-13UTC142454_onESPclient 2022-10-13T14:28:28.777Z,1665671308.777 [DataOverHttps](IMPORTANT): SBD MTMSN=20221013T142824 2022-10-13T14:28:36.198Z,1665671316.198 [CommandExec](IMPORTANT): got command ! check -- carts spare: 2022-10-13T14:28:36.859Z,1665671316.859 [CommandExec](IMPORTANT): Waiting for ESP to boot... Connected. Thu Oct 13 14:26:09 UTC 2022 -- ssh esp@10.89.5.3 espclient chrisp {spare: 2} shutting ESP off 2022-10-13T14:28:53.696Z,1665671333.696 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 32.843062 min 2022-10-13T14:28:55.303Z,1665671335.303 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-10-13T14:34:40.598Z,1665671680.598 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T14:37:26.693Z,1665671846.693 [DataOverHttps](IMPORTANT): SBD MTMSN=20221013T143722 2022-10-13T14:37:34.117Z,1665671854.117 [CommandExec](IMPORTANT): got command resume 2022-10-13T14:37:45.626Z,1665671865.626 [MissionManager](IMPORTANT): Resuming from info in Logs/20221012T211723/Science_location_depth_sampling_xml.state 2022-10-13T14:37:46.188Z,1665671866.188 [CommandExec](IMPORTANT): Resuming mission and schedule 2022-10-13T14:37:46.188Z,1665671866.188 [CommandExec](IMPORTANT): Scheduling is resumed 2022-10-13T14:37:46.189Z,1665671866.189 [CommandExec](IMPORTANT): Scheduled #4 (#1 of 3 with id='406yv'): "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 35 count;set LocationDepthSampling.MissionTimeout 90 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s", ASAP 2022-10-13T14:37:46.316Z,1665671866.316 [MissionManager](IMPORTANT): Started mission LocationDepthSampling 2022-10-13T14:37:46.501Z,1665671866.501 [LocationDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours. 2022-10-13T14:51:10.135Z,1665672670.135 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.046768 m , settling for 5.000000 min 2022-10-13T14:56:22.885Z,1665672982.885 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T14:56:22.993Z,1665672982.993 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T14:57:24.184Z,1665673044.184 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T14:57:24.997Z,1665673044.997 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -6 2022-10-13T14:57:24.998Z,1665673044.998 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T14:57:25.446Z,1665673045.446 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 151.149033 m in 10.312769 degC water with 0.114400 ug/l chlorophyll fluorescence. 2022-10-13T14:57:25.803Z,1665673045.803 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T14:57:26.205Z,1665673046.205 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T14:57:28.226Z,1665673048.226 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T14:58:02.575Z,1665673082.575 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@27,archive_bac) 2022-10-13T14:58:05.397Z,1665673085.397 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T14:58:05.398Z,1665673085.398 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T14:58:05.804Z,1665673085.804 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T15:00:24.375Z,1665673224.375 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T15:00:24.780Z,1665673224.780 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T15:18:52.389Z,1665674332.389 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-13T15:19:03.231Z,1665674343.231 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002869 CHAN A1 (24V): 0.007584 CHAN A2 (12V): -0.004539 CHAN A3 (5V): -0.003338 CHAN B0 (3.3V): -0.002761 CHAN B1 (3.15aV): -0.001323 CHAN B2 (3.15bV): -0.001465 CHAN B3 (GND): -0.001025 OPEN: 0.012100 Full Scale: +/- 1 mA 2022-10-13T15:23:55.764Z,1665674635.764 [ThrusterHE](FAULT): getSpeed error serial timeout 2022-10-13T15:23:55.764Z,1665674635.764 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2022-10-13T15:44:26.573Z,1665675866.573 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@1000.0) 2022-10-13T15:44:26.977Z,1665675866.977 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-13T15:44:28.998Z,1665675868.998 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T15:44:29.401Z,1665675869.401 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@27,archive_bac) 2022-10-13T15:56:32.998Z,1665676592.998 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@27,archive_bac) 2022-10-13T15:56:43.098Z,1665676603.098 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T15:56:43.498Z,1665676603.498 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED 2022-10-13T16:01:43.276Z,1665676903.276 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally. 2022-10-13T16:01:44.151Z,1665676904.151 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 9.000000 count completed. 2022-10-13T16:01:50.627Z,1665676910.627 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-13T16:14:20.819Z,1665677660.819 [NAL9602](FAULT): GPS failed to acquire within timeout. 2022-10-13T16:20:05.679Z,1665678005.679 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T16:22:41.507Z,1665678161.507 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-13T16:27:42.777Z,1665678462.777 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T16:47:15.627Z,1665679635.627 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-13T16:47:18.043Z,1665679638.043 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-13T16:47:18.457Z,1665679638.457 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 10.000000 count . Target depth = 5.000000 m . 2022-10-13T16:47:18.845Z,1665679638.845 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T16:51:27.307Z,1665679887.307 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.996255 m , settling for 5.000000 min 2022-10-13T16:56:40.080Z,1665680200.080 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T16:56:40.245Z,1665680200.245 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T16:57:41.038Z,1665680261.038 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T16:57:41.848Z,1665680261.848 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -6 2022-10-13T16:57:41.849Z,1665680261.849 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T16:57:42.292Z,1665680262.292 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.024527 m in 13.578821 degC water with 4.562480 ug/l chlorophyll fluorescence. 2022-10-13T16:57:42.656Z,1665680262.656 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T16:57:43.057Z,1665680263.057 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T16:57:45.077Z,1665680265.077 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T16:58:19.421Z,1665680299.421 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@26,archive_bac) 2022-10-13T16:58:22.249Z,1665680302.249 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T16:58:22.250Z,1665680302.250 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T16:58:22.654Z,1665680302.654 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T17:00:41.229Z,1665680441.229 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T17:00:41.637Z,1665680441.637 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T18:02:08.997Z,1665684128.997 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@1000.0) 2022-10-13T18:02:09.401Z,1665684129.401 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-13T18:02:11.422Z,1665684131.422 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T18:02:11.831Z,1665684131.831 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@26,archive_bac) 2022-10-13T18:14:15.401Z,1665684855.401 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@26,archive_bac) 2022-10-13T18:14:25.100Z,1665684865.100 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T18:14:25.505Z,1665684865.505 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED 2022-10-13T18:19:25.285Z,1665685165.285 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally. 2022-10-13T18:19:26.141Z,1665685166.141 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 10.000000 count completed. 2022-10-13T18:19:32.605Z,1665685172.605 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-13T18:25:06.717Z,1665685506.717 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T18:26:22.610Z,1665685582.610 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-13T18:31:23.784Z,1665685883.784 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T18:58:56.204Z,1665687536.204 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-13T18:58:58.633Z,1665687538.633 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-13T18:58:59.050Z,1665687539.050 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 11.000000 count . Target depth = 150.000000 m . 2022-10-13T18:58:59.427Z,1665687539.427 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T19:10:43.631Z,1665688243.631 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.037384 m , settling for 5.000000 min 2022-10-13T19:15:56.392Z,1665688556.392 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T19:15:56.511Z,1665688556.511 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T19:16:56.922Z,1665688616.922 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T19:16:57.732Z,1665688617.732 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -6 2022-10-13T19:16:57.733Z,1665688617.733 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T19:16:58.194Z,1665688618.194 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.070099 m in 9.848169 degC water with 0.081120 ug/l chlorophyll fluorescence. 2022-10-13T19:16:58.538Z,1665688618.538 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T19:16:58.941Z,1665688618.941 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T19:17:00.962Z,1665688620.962 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T19:17:35.304Z,1665688655.304 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@25,archive_bac) 2022-10-13T19:17:37.725Z,1665688657.725 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T19:17:37.726Z,1665688657.726 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T19:17:38.129Z,1665688658.129 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T19:19:57.110Z,1665688797.110 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T19:19:57.513Z,1665688797.513 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T20:04:23.536Z,1665691463.536 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PAUSED (@1000.0) 2022-10-13T20:04:23.937Z,1665691463.937 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-13T20:04:25.958Z,1665691465.958 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T20:04:26.362Z,1665691466.362 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PROCESSED (@25,archive_bac) 2022-10-13T20:16:29.964Z,1665692189.964 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@25,archive_bac) 2022-10-13T20:16:39.659Z,1665692199.659 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T20:16:40.067Z,1665692200.067 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED 2022-10-13T20:21:39.837Z,1665692499.837 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence completed normally. 2022-10-13T20:21:40.699Z,1665692500.699 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 11.000000 count completed. 2022-10-13T20:21:47.160Z,1665692507.160 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-13T20:39:33.766Z,1665693573.766 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T20:40:59.554Z,1665693659.554 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-13T20:46:00.951Z,1665693960.951 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T20:55:08.020Z,1665694508.020 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-13T20:55:10.432Z,1665694510.432 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-13T20:55:10.856Z,1665694510.856 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 12.000000 count . Target depth = 5.000000 m . 2022-10-13T20:55:11.253Z,1665694511.253 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T20:59:06.008Z,1665694746.008 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.959009 m , settling for 5.000000 min 2022-10-13T21:04:18.736Z,1665695058.736 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T21:04:18.891Z,1665695058.891 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T21:05:19.670Z,1665695119.670 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T21:05:20.490Z,1665695120.490 [ESPComponent](IMPORTANT): [sample #13] Setting cartridge argument to -6 2022-10-13T21:05:20.490Z,1665695120.490 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T21:05:20.927Z,1665695120.927 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.066261 m in 13.645929 degC water with 6.056960 ug/l chlorophyll fluorescence. 2022-10-13T21:05:21.285Z,1665695121.285 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T21:05:21.689Z,1665695121.689 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T21:05:23.710Z,1665695123.710 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T21:05:58.050Z,1665695158.050 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PRIMING (@24,archive_bac) 2022-10-13T21:06:00.473Z,1665695160.473 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T21:06:00.474Z,1665695160.474 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T21:06:00.878Z,1665695160.878 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T21:08:19.854Z,1665695299.854 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T21:08:20.258Z,1665695300.258 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T21:19:03.593Z,1665695943.593 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-13T21:19:14.423Z,1665695954.423 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003529 CHAN A1 (24V): 0.007284 CHAN A2 (12V): -0.004201 CHAN A3 (5V): -0.003654 CHAN B0 (3.3V): -0.002066 CHAN B1 (3.15aV): -0.001412 CHAN B2 (3.15bV): -0.001056 CHAN B3 (GND): -0.000525 OPEN: 0.012522 Full Scale: +/- 1 mA 2022-10-13T21:40:39.048Z,1665697239.048 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2022-10-13T21:45:14.246Z,1665697514.246 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PAUSED (@596.5) 2022-10-13T21:45:14.649Z,1665697514.649 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PROCESSING (@596.5) 2022-10-13T21:45:16.670Z,1665697516.670 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T21:45:17.077Z,1665697517.077 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PROCESSED (@24,archive_bac) 2022-10-13T21:57:20.653Z,1665698240.653 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPING (@24,archive_bac) 2022-10-13T21:57:30.346Z,1665698250.346 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T21:57:30.754Z,1665698250.754 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPED 2022-10-13T22:02:30.538Z,1665698550.538 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence completed normally. 2022-10-13T22:02:31.404Z,1665698551.404 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 12.000000 count completed. 2022-10-13T22:02:37.870Z,1665698557.870 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-13T22:08:06.691Z,1665698886.691 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T22:11:03.781Z,1665699063.781 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-13T22:16:04.880Z,1665699364.880 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-13T22:23:01.442Z,1665699781.442 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-13T22:23:03.876Z,1665699783.876 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-13T22:23:04.263Z,1665699784.263 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 13.000000 count . Target depth = 150.000000 m . 2022-10-13T22:23:04.669Z,1665699784.669 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-13T22:34:55.351Z,1665700495.351 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.018997 m , settling for 5.000000 min 2022-10-13T22:40:08.120Z,1665700808.120 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-13T22:40:08.276Z,1665700808.276 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-13T22:41:09.501Z,1665700869.501 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-13T22:41:10.312Z,1665700870.312 [ESPComponent](IMPORTANT): [sample #14] Setting cartridge argument to -6 2022-10-13T22:41:10.312Z,1665700870.312 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-13T22:41:10.776Z,1665700870.776 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.029755 m in 9.652613 degC water with 0.064480 ug/l chlorophyll fluorescence. 2022-10-13T22:41:11.117Z,1665700871.117 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-13T22:41:11.521Z,1665700871.521 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-13T22:41:13.541Z,1665700873.541 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-13T22:41:47.885Z,1665700907.885 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PRIMING (@23,archive_bac) 2022-10-13T22:41:49.901Z,1665700909.901 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PRIMING (@100.0) 2022-10-13T22:41:50.306Z,1665700910.306 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-13T22:41:50.710Z,1665700910.710 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-13T22:44:10.089Z,1665701050.089 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_FILTERING (@0.0) 2022-10-13T22:44:10.494Z,1665701050.494 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-13T23:29:08.842Z,1665703748.842 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PAUSED (@1000.0) 2022-10-13T23:29:09.246Z,1665703749.246 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-13T23:29:11.293Z,1665703751.293 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-13T23:29:11.670Z,1665703751.670 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PROCESSED (@23,archive_bac) 2022-10-13T23:41:15.242Z,1665704475.242 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPING (@23,archive_bac) 2022-10-13T23:41:25.347Z,1665704485.347 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-13T23:41:25.751Z,1665704485.751 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPED 2022-10-13T23:46:25.513Z,1665704785.513 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence completed normally. 2022-10-13T23:46:26.391Z,1665704786.391 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 13.000000 count completed. 2022-10-13T23:46:32.851Z,1665704792.851 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-14T00:03:38.198Z,1665705818.198 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T00:05:20.745Z,1665705920.745 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T00:07:27.048Z,1665706047.048 [ThrusterHE](FAULT): getSpeed error serial timeout 2022-10-14T00:07:27.048Z,1665706047.048 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2022-10-14T00:10:21.777Z,1665706221.777 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T00:18:44.398Z,1665706724.398 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T00:18:46.821Z,1665706726.821 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-14T00:18:47.221Z,1665706727.221 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 14.000000 count . Target depth = 5.000000 m . 2022-10-14T00:18:47.612Z,1665706727.612 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T00:22:37.100Z,1665706957.100 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.972022 m , settling for 5.000000 min 2022-10-14T00:27:49.872Z,1665707269.872 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T00:27:50.029Z,1665707270.029 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T00:28:51.633Z,1665707331.633 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T00:28:52.444Z,1665707332.444 [ESPComponent](IMPORTANT): [sample #15] Setting cartridge argument to -6 2022-10-14T00:28:52.444Z,1665707332.444 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T00:28:52.908Z,1665707332.908 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.616160 m in 13.771235 degC water with 5.630560 ug/l chlorophyll fluorescence. 2022-10-14T00:28:53.251Z,1665707333.251 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T00:28:53.653Z,1665707333.653 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T00:28:55.673Z,1665707335.673 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T00:29:30.013Z,1665707370.013 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PRIMING (@22,archive_bac) 2022-10-14T00:29:32.843Z,1665707372.843 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T00:29:32.844Z,1665707372.844 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T00:29:33.247Z,1665707373.247 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T00:31:52.222Z,1665707512.222 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T00:31:52.626Z,1665707512.626 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T01:13:45.557Z,1665710025.557 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PAUSED (@666.4) 2022-10-14T01:13:45.962Z,1665710025.962 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PROCESSING (@666.4) 2022-10-14T01:13:47.982Z,1665710027.982 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T01:13:48.385Z,1665710028.385 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PROCESSED (@22,archive_bac) 2022-10-14T01:25:51.950Z,1665710751.950 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPING (@22,archive_bac) 2022-10-14T01:26:01.646Z,1665710761.646 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T01:26:02.049Z,1665710762.049 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPED 2022-10-14T01:31:01.819Z,1665711061.819 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence completed normally. 2022-10-14T01:31:02.701Z,1665711062.701 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 14.000000 count completed. 2022-10-14T01:31:09.148Z,1665711069.148 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-14T01:36:41.604Z,1665711401.604 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T01:38:01.503Z,1665711481.503 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T01:43:02.724Z,1665711782.724 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T02:03:49.940Z,1665713029.940 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T02:03:52.357Z,1665713032.357 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-14T02:03:52.756Z,1665713032.756 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 15.000000 count . Target depth = 150.000000 m . 2022-10-14T02:03:53.171Z,1665713033.171 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T02:15:46.628Z,1665713746.628 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.028870 m , settling for 5.000000 min 2022-10-14T02:20:59.388Z,1665714059.388 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T02:20:59.523Z,1665714059.523 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T02:22:00.327Z,1665714120.327 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T02:22:01.137Z,1665714121.137 [ESPComponent](IMPORTANT): [sample #16] Setting cartridge argument to -6 2022-10-14T02:22:01.138Z,1665714121.138 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T02:22:01.587Z,1665714121.587 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.293427 m in 10.303736 degC water with 0.116480 ug/l chlorophyll fluorescence. 2022-10-14T02:22:01.942Z,1665714121.942 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T02:22:02.348Z,1665714122.348 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T02:22:04.367Z,1665714124.367 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T02:22:38.707Z,1665714158.707 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PRIMING (@21,archive_bac) 2022-10-14T02:22:41.537Z,1665714161.537 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T02:22:41.538Z,1665714161.538 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T02:22:41.938Z,1665714161.938 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T02:25:00.515Z,1665714300.515 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T02:25:00.918Z,1665714300.918 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T03:09:37.034Z,1665716977.034 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PAUSED (@1000.0) 2022-10-14T03:09:37.437Z,1665716977.437 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-14T03:09:39.458Z,1665716979.458 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T03:09:39.862Z,1665716979.862 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PROCESSED (@21,archive_bac) 2022-10-14T03:19:14.540Z,1665717554.540 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-14T03:19:25.402Z,1665717565.402 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002905 CHAN A1 (24V): 0.008919 CHAN A2 (12V): -0.003112 CHAN A3 (5V): -0.003975 CHAN B0 (3.3V): -0.001952 CHAN B1 (3.15aV): -0.001109 CHAN B2 (3.15bV): -0.001738 CHAN B3 (GND): -0.000812 OPEN: 0.012454 Full Scale: +/- 1 mA 2022-10-14T03:21:43.441Z,1665717703.441 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPING (@21,archive_bac) 2022-10-14T03:21:53.946Z,1665717713.946 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T03:21:54.351Z,1665717714.351 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPED 2022-10-14T03:26:54.124Z,1665718014.124 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence completed normally. 2022-10-14T03:26:54.971Z,1665718014.971 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 15.000000 count completed. 2022-10-14T03:27:01.453Z,1665718021.453 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-14T03:45:56.024Z,1665719156.024 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T03:50:57.679Z,1665719457.679 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T04:16:11.549Z,1665720971.549 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T04:16:13.951Z,1665720973.951 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-14T04:16:14.380Z,1665720974.380 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 16.000000 count . Target depth = 5.000000 m . 2022-10-14T04:16:14.755Z,1665720974.755 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T04:20:19.202Z,1665721219.202 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.978754 m , settling for 5.000000 min 2022-10-14T04:25:31.952Z,1665721531.952 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T04:25:32.117Z,1665721532.117 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T04:26:33.255Z,1665721593.255 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T04:26:34.065Z,1665721594.065 [ESPComponent](IMPORTANT): [sample #17] Setting cartridge argument to -6 2022-10-14T04:26:34.065Z,1665721594.065 [ESPComponent](IMPORTANT): [sample #17] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T04:26:34.521Z,1665721594.521 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.017796 m in 13.885309 degC water with 4.109040 ug/l chlorophyll fluorescence. 2022-10-14T04:26:34.871Z,1665721594.871 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T04:26:35.273Z,1665721595.273 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T04:26:37.294Z,1665721597.294 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T04:27:11.634Z,1665721631.634 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PRIMING (@20,archive_bac) 2022-10-14T04:27:14.463Z,1665721634.463 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T04:27:14.464Z,1665721634.464 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T04:27:14.869Z,1665721634.869 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T04:29:33.440Z,1665721773.440 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T04:29:33.843Z,1665721773.843 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T05:06:28.205Z,1665723988.205 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PAUSED (@589.1) 2022-10-14T05:06:28.610Z,1665723988.610 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PROCESSING (@589.1) 2022-10-14T05:06:30.635Z,1665723990.635 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T05:06:31.035Z,1665723991.035 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PROCESSED (@20,archive_bac) 2022-10-14T05:18:34.606Z,1665724714.606 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_STOPPING (@20,archive_bac) 2022-10-14T05:18:44.302Z,1665724724.302 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T05:18:44.709Z,1665724724.709 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_STOPPED 2022-10-14T05:23:44.481Z,1665725024.481 [ESPComponent](IMPORTANT): [sample #17] ESP sampling sequence completed normally. 2022-10-14T05:23:45.365Z,1665725025.365 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 16.000000 count completed. 2022-10-14T05:23:51.835Z,1665725031.835 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-14T05:29:21.888Z,1665725361.888 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T05:30:36.953Z,1665725436.953 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T05:35:37.836Z,1665725737.836 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T06:03:37.740Z,1665727417.740 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T06:03:40.160Z,1665727420.160 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-14T06:03:40.555Z,1665727420.555 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 17.000000 count . Target depth = 150.000000 m . 2022-10-14T06:03:40.968Z,1665727420.968 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T06:15:46.961Z,1665728146.961 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.102859 m , settling for 5.000000 min 2022-10-14T06:20:59.724Z,1665728459.724 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T06:20:59.883Z,1665728459.883 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T06:22:01.056Z,1665728521.056 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T06:22:01.869Z,1665728521.869 [ESPComponent](IMPORTANT): [sample #18] Setting cartridge argument to -6 2022-10-14T06:22:01.869Z,1665728521.869 [ESPComponent](IMPORTANT): [sample #18] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T06:22:02.337Z,1665728522.337 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 151.762054 m in 10.356623 degC water with 0.092560 ug/l chlorophyll fluorescence. 2022-10-14T06:22:02.673Z,1665728522.673 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T06:22:03.077Z,1665728523.077 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T06:22:05.097Z,1665728525.097 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T06:22:39.441Z,1665728559.441 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PRIMING (@19,archive_bac) 2022-10-14T06:22:42.269Z,1665728562.269 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T06:22:42.270Z,1665728562.270 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T06:22:42.674Z,1665728562.674 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T06:25:01.245Z,1665728701.245 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T06:25:01.649Z,1665728701.649 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T07:09:26.068Z,1665731366.068 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PAUSED (@1000.0) 2022-10-14T07:09:26.469Z,1665731366.469 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-14T07:09:28.490Z,1665731368.490 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T07:09:28.893Z,1665731368.893 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PROCESSED (@19,archive_bac) 2022-10-14T07:21:32.465Z,1665732092.465 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_STOPPING (@19,archive_bac) 2022-10-14T07:21:42.566Z,1665732102.566 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T07:21:42.973Z,1665732102.973 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_STOPPED 2022-10-14T07:26:42.745Z,1665732402.745 [ESPComponent](IMPORTANT): [sample #18] ESP sampling sequence completed normally. 2022-10-14T07:26:43.639Z,1665732403.639 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 17.000000 count completed. 2022-10-14T07:26:50.078Z,1665732410.078 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-14T07:44:34.608Z,1665733474.608 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T07:45:47.065Z,1665733547.065 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T07:50:48.707Z,1665733848.707 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T07:52:10.872Z,1665733930.872 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout 2022-10-14T08:16:40.150Z,1665735400.150 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T08:16:42.575Z,1665735402.575 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-14T08:16:42.988Z,1665735402.988 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 18.000000 count . Target depth = 5.000000 m . 2022-10-14T08:16:43.370Z,1665735403.370 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T08:20:35.694Z,1665735635.694 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.976510 m , settling for 5.000000 min 2022-10-14T08:25:48.432Z,1665735948.432 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T08:25:48.564Z,1665735948.564 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T08:26:49.331Z,1665736009.331 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T08:26:50.144Z,1665736010.144 [ESPComponent](IMPORTANT): [sample #19] Setting cartridge argument to -6 2022-10-14T08:26:50.144Z,1665736010.144 [ESPComponent](IMPORTANT): [sample #19] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T08:26:50.588Z,1665736010.588 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.980549 m in 14.204248 degC water with 3.469440 ug/l chlorophyll fluorescence. 2022-10-14T08:26:50.945Z,1665736010.945 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T08:26:51.349Z,1665736011.349 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T08:26:53.369Z,1665736013.369 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T08:27:27.711Z,1665736047.711 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PRIMING (@18,archive_bac) 2022-10-14T08:27:30.538Z,1665736050.538 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T08:27:30.539Z,1665736050.539 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T08:27:30.941Z,1665736050.941 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T08:29:49.514Z,1665736189.514 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T08:29:49.918Z,1665736189.918 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T09:19:20.598Z,1665739160.598 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PAUSED (@777.1) 2022-10-14T09:19:20.973Z,1665739160.973 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PROCESSING (@777.1) 2022-10-14T09:19:22.994Z,1665739162.994 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T09:19:23.397Z,1665739163.397 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PROCESSED (@18,archive_bac) 2022-10-14T09:19:25.567Z,1665739165.567 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-14T09:19:36.475Z,1665739176.475 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003120 CHAN A1 (24V): 0.008081 CHAN A2 (12V): -0.003746 CHAN A3 (5V): -0.003717 CHAN B0 (3.3V): -0.001408 CHAN B1 (3.15aV): -0.001682 CHAN B2 (3.15bV): -0.000923 CHAN B3 (GND): -0.001290 OPEN: 0.012230 Full Scale: +/- 1 mA 2022-10-14T09:31:26.969Z,1665739886.969 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_STOPPING (@18,archive_bac) 2022-10-14T09:31:36.667Z,1665739896.667 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T09:31:37.070Z,1665739897.070 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_STOPPED 2022-10-14T09:36:36.854Z,1665740196.854 [ESPComponent](IMPORTANT): [sample #19] ESP sampling sequence completed normally. 2022-10-14T09:36:37.715Z,1665740197.715 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 18.000000 count completed. 2022-10-14T09:36:44.178Z,1665740204.178 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-14T09:42:14.624Z,1665740534.624 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T09:43:29.614Z,1665740609.614 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T09:48:31.120Z,1665740911.120 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T10:11:18.723Z,1665742278.723 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T10:11:21.131Z,1665742281.131 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-14T10:11:21.539Z,1665742281.539 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 19.000000 count . Target depth = 150.000000 m . 2022-10-14T10:11:21.931Z,1665742281.931 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T10:22:47.944Z,1665742967.944 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.023483 m , settling for 5.000000 min 2022-10-14T10:28:00.728Z,1665743280.728 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T10:28:00.885Z,1665743280.885 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T10:29:02.494Z,1665743342.494 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T10:29:03.304Z,1665743343.304 [ESPComponent](IMPORTANT): [sample #20] Setting cartridge argument to -6 2022-10-14T10:29:03.304Z,1665743343.304 [ESPComponent](IMPORTANT): [sample #20] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T10:29:03.764Z,1665743343.764 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.953522 m in 9.667994 degC water with 0.076960 ug/l chlorophyll fluorescence. 2022-10-14T10:29:04.109Z,1665743344.109 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T10:29:04.513Z,1665743344.513 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T10:29:06.534Z,1665743346.534 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T10:29:40.873Z,1665743380.873 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PRIMING (@17,archive_bac) 2022-10-14T10:29:43.701Z,1665743383.701 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T10:29:43.702Z,1665743383.702 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T10:29:44.115Z,1665743384.115 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T10:32:02.677Z,1665743522.677 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T10:32:03.082Z,1665743523.082 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T11:15:47.893Z,1665746147.893 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PAUSED (@1000.0) 2022-10-14T11:15:48.298Z,1665746148.298 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-14T11:15:50.317Z,1665746150.317 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T11:15:50.722Z,1665746150.722 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PROCESSED (@17,archive_bac) 2022-10-14T11:27:54.294Z,1665746874.294 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_STOPPING (@17,archive_bac) 2022-10-14T11:28:04.394Z,1665746884.394 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T11:28:04.798Z,1665746884.798 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_STOPPED 2022-10-14T11:33:04.565Z,1665747184.565 [ESPComponent](IMPORTANT): [sample #20] ESP sampling sequence completed normally. 2022-10-14T11:33:05.431Z,1665747185.431 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 19.000000 count completed. 2022-10-14T11:33:11.911Z,1665747191.911 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-14T11:50:43.915Z,1665748243.915 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T11:51:59.902Z,1665748319.902 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T11:57:00.659Z,1665748620.659 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T12:17:29.281Z,1665749849.281 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T12:17:31.685Z,1665749851.685 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-14T12:17:32.109Z,1665749852.109 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 20.000000 count . Target depth = 5.000000 m . 2022-10-14T12:17:32.513Z,1665749852.513 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T12:21:39.349Z,1665750099.349 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 5.002538 m , settling for 5.000000 min 2022-10-14T12:26:52.117Z,1665750412.117 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T12:26:52.300Z,1665750412.300 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T12:27:53.521Z,1665750473.521 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T12:27:54.332Z,1665750474.332 [ESPComponent](IMPORTANT): [sample #21] Setting cartridge argument to -6 2022-10-14T12:27:54.333Z,1665750474.333 [ESPComponent](IMPORTANT): [sample #21] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T12:27:54.806Z,1665750474.806 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.684820 m in 13.874872 degC water with 3.030560 ug/l chlorophyll fluorescence. 2022-10-14T12:27:55.142Z,1665750475.142 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T12:27:55.564Z,1665750475.564 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T12:27:57.568Z,1665750477.568 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T12:28:31.905Z,1665750511.905 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PRIMING (@16,archive_bac) 2022-10-14T12:28:34.736Z,1665750514.736 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T12:28:34.737Z,1665750514.737 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T12:28:35.137Z,1665750515.137 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T12:30:53.710Z,1665750653.710 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T12:30:54.114Z,1665750654.114 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T13:25:01.909Z,1665753901.909 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PAUSED (@862.1) 2022-10-14T13:25:02.313Z,1665753902.313 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PROCESSING (@862.1) 2022-10-14T13:25:04.334Z,1665753904.334 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T13:25:04.737Z,1665753904.737 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PROCESSED (@16,archive_bac) 2022-10-14T13:37:08.313Z,1665754628.313 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_STOPPING (@16,archive_bac) 2022-10-14T13:37:18.010Z,1665754638.010 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T13:37:18.414Z,1665754638.414 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_STOPPED 2022-10-14T13:42:18.186Z,1665754938.186 [ESPComponent](IMPORTANT): [sample #21] ESP sampling sequence completed normally. 2022-10-14T13:42:19.055Z,1665754939.055 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 20.000000 count completed. 2022-10-14T13:42:25.519Z,1665754945.519 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-14T13:47:54.775Z,1665755274.775 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T13:49:08.430Z,1665755348.430 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T13:54:09.769Z,1665755649.769 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T14:05:07.923Z,1665756307.923 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T14:05:10.333Z,1665756310.333 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-14T14:05:10.742Z,1665756310.742 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 21.000000 count . Target depth = 150.000000 m . 2022-10-14T14:05:11.151Z,1665756311.151 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T14:17:30.502Z,1665757050.502 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.006439 m , settling for 5.000000 min 2022-10-14T14:22:43.249Z,1665757363.249 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T14:22:43.396Z,1665757363.396 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T14:23:44.187Z,1665757424.187 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T14:23:44.998Z,1665757424.998 [ESPComponent](IMPORTANT): [sample #22] Setting cartridge argument to -6 2022-10-14T14:23:44.999Z,1665757424.999 [ESPComponent](IMPORTANT): [sample #22] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T14:23:45.453Z,1665757425.453 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.564728 m in 10.314539 degC water with 0.102960 ug/l chlorophyll fluorescence. 2022-10-14T14:23:45.801Z,1665757425.801 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T14:23:46.208Z,1665757426.208 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T14:23:48.225Z,1665757428.225 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T14:24:22.630Z,1665757462.630 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PRIMING (@15,archive_bac) 2022-10-14T14:24:25.458Z,1665757465.458 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T14:24:25.459Z,1665757465.459 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T14:24:25.862Z,1665757465.862 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T14:26:44.433Z,1665757604.433 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T14:26:44.838Z,1665757604.838 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T14:29:52.068Z,1665757792.068 [ThrusterHE](FAULT): getSpeed error serial timeout 2022-10-14T14:29:52.068Z,1665757792.068 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2022-10-14T14:43:55.216Z,1665758635.216 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout 2022-10-14T15:12:05.426Z,1665760325.426 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PAUSED (@1000.0) 2022-10-14T15:12:05.833Z,1665760325.833 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-14T15:12:07.857Z,1665760327.857 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T15:12:08.253Z,1665760328.253 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PROCESSED (@15,archive_bac) 2022-10-14T15:19:36.831Z,1665760776.831 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-14T15:19:47.766Z,1665760787.766 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.002927 CHAN A1 (24V): 0.008354 CHAN A2 (12V): -0.004142 CHAN A3 (5V): -0.004064 CHAN B0 (3.3V): -0.001966 CHAN B1 (3.15aV): -0.001415 CHAN B2 (3.15bV): -0.001556 CHAN B3 (GND): -0.001101 OPEN: 0.012303 Full Scale: +/- 1 mA 2022-10-14T15:24:12.229Z,1665761052.229 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_STOPPING (@15,archive_bac) 2022-10-14T15:24:21.926Z,1665761061.926 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T15:24:22.370Z,1665761062.370 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_STOPPED 2022-10-14T15:29:22.145Z,1665761362.145 [ESPComponent](IMPORTANT): [sample #22] ESP sampling sequence completed normally. 2022-10-14T15:29:22.987Z,1665761362.987 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 21.000000 count completed. 2022-10-14T15:29:29.456Z,1665761369.456 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-14T15:46:28.011Z,1665762388.011 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T15:47:37.350Z,1665762457.350 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T15:52:38.995Z,1665762758.995 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T16:09:32.299Z,1665763772.299 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T16:09:34.715Z,1665763774.715 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-14T16:09:35.120Z,1665763775.120 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 22.000000 count . Target depth = 5.000000 m . 2022-10-14T16:09:35.545Z,1665763775.545 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T16:13:40.345Z,1665764020.345 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.979651 m , settling for 5.000000 min 2022-10-14T16:18:53.109Z,1665764333.109 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T16:18:53.263Z,1665764333.263 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T16:19:54.067Z,1665764394.067 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T16:19:54.879Z,1665764394.879 [ESPComponent](IMPORTANT): [sample #23] Setting cartridge argument to -6 2022-10-14T16:19:54.880Z,1665764394.880 [ESPComponent](IMPORTANT): [sample #23] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T16:19:55.339Z,1665764395.339 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.016000 m in 13.909601 degC water with 3.378960 ug/l chlorophyll fluorescence. 2022-10-14T16:19:55.682Z,1665764395.682 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T16:19:56.086Z,1665764396.086 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T16:19:58.105Z,1665764398.105 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T16:20:32.851Z,1665764432.851 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PRIMING (@14,archive_bac) 2022-10-14T16:20:35.677Z,1665764435.677 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T16:20:35.678Z,1665764435.678 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T16:20:36.082Z,1665764436.082 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T16:22:54.657Z,1665764574.657 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T16:22:55.061Z,1665764575.061 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T17:21:15.357Z,1665768075.357 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PAUSED (@1000.0) 2022-10-14T17:21:15.762Z,1665768075.762 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-14T17:21:17.782Z,1665768077.782 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T17:21:18.185Z,1665768078.185 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PROCESSED (@14,archive_bac) 2022-10-14T17:33:21.766Z,1665768801.766 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_STOPPING (@14,archive_bac) 2022-10-14T17:33:31.463Z,1665768811.463 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T17:33:31.868Z,1665768811.868 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_STOPPED 2022-10-14T17:38:31.656Z,1665769111.656 [ESPComponent](IMPORTANT): [sample #23] ESP sampling sequence completed normally. 2022-10-14T17:38:32.527Z,1665769112.527 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 22.000000 count completed. 2022-10-14T17:38:38.977Z,1665769118.977 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-14T17:44:11.847Z,1665769451.847 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T17:46:12.902Z,1665769572.902 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T17:51:13.931Z,1665769873.931 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T18:19:56.636Z,1665771596.636 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T18:19:59.072Z,1665771599.072 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-14T18:19:59.460Z,1665771599.460 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 23.000000 count . Target depth = 150.000000 m . 2022-10-14T18:19:59.901Z,1665771599.901 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T18:32:07.507Z,1665772327.507 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.011826 m , settling for 5.000000 min 2022-10-14T18:37:20.265Z,1665772640.265 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T18:37:20.398Z,1665772640.398 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T18:38:22.009Z,1665772702.009 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T18:38:22.820Z,1665772702.820 [ESPComponent](IMPORTANT): [sample #24] Setting cartridge argument to -6 2022-10-14T18:38:22.820Z,1665772702.820 [ESPComponent](IMPORTANT): [sample #24] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T18:38:23.289Z,1665772703.289 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.840073 m in 10.139307 degC water with 0.096720 ug/l chlorophyll fluorescence. 2022-10-14T18:38:23.626Z,1665772703.626 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T18:38:24.033Z,1665772704.033 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T18:38:26.052Z,1665772706.052 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T18:39:00.389Z,1665772740.389 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PRIMING (@13,archive_bac) 2022-10-14T18:39:03.222Z,1665772743.222 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T18:39:03.225Z,1665772743.225 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T18:39:03.622Z,1665772743.622 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T18:41:22.601Z,1665772882.601 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T18:41:23.005Z,1665772883.005 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T19:27:35.714Z,1665775655.714 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PAUSED (@1000.0) 2022-10-14T19:27:36.119Z,1665775656.119 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-14T19:27:38.138Z,1665775658.138 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T19:27:38.542Z,1665775658.542 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PROCESSED (@13,archive_bac) 2022-10-14T19:39:42.529Z,1665776382.529 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_STOPPING (@13,archive_bac) 2022-10-14T19:39:52.226Z,1665776392.226 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T19:39:52.631Z,1665776392.631 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_STOPPED 2022-10-14T19:44:52.412Z,1665776692.412 [ESPComponent](IMPORTANT): [sample #24] ESP sampling sequence completed normally. 2022-10-14T19:44:53.271Z,1665776693.271 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 23.000000 count completed. 2022-10-14T19:44:59.720Z,1665776699.720 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-14T20:02:22.085Z,1665777742.085 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T20:03:26.966Z,1665777806.966 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T20:08:28.011Z,1665778108.011 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T20:22:56.250Z,1665778976.250 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T20:22:58.694Z,1665778978.694 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-14T20:22:59.076Z,1665778979.076 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 24.000000 count . Target depth = 5.000000 m . 2022-10-14T20:22:59.484Z,1665778979.484 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T20:26:57.042Z,1665779217.042 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.987279 m , settling for 5.000000 min 2022-10-14T20:32:09.816Z,1665779529.816 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T20:32:09.956Z,1665779529.956 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T20:33:11.581Z,1665779591.581 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T20:33:12.392Z,1665779592.392 [ESPComponent](IMPORTANT): [sample #25] Setting cartridge argument to -6 2022-10-14T20:33:12.393Z,1665779592.393 [ESPComponent](IMPORTANT): [sample #25] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T20:33:12.853Z,1665779592.853 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.216593 m in 14.185755 degC water with 4.853680 ug/l chlorophyll fluorescence. 2022-10-14T20:33:13.197Z,1665779593.197 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T20:33:13.602Z,1665779593.602 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T20:33:15.625Z,1665779595.625 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T20:33:49.962Z,1665779629.962 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_PRIMING (@12,archive_bac) 2022-10-14T20:33:52.385Z,1665779632.385 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T20:33:52.386Z,1665779632.386 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T20:33:52.790Z,1665779632.790 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T20:36:11.771Z,1665779771.771 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T20:36:12.175Z,1665779772.175 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T21:19:47.771Z,1665782387.771 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-14T21:19:58.751Z,1665782398.751 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003071 CHAN A1 (24V): 0.009179 CHAN A2 (12V): -0.003571 CHAN A3 (5V): -0.003943 CHAN B0 (3.3V): -0.002000 CHAN B1 (3.15aV): -0.001540 CHAN B2 (3.15bV): -0.001046 CHAN B3 (GND): -0.000706 OPEN: 0.012384 Full Scale: +/- 1 mA 2022-10-14T21:31:45.210Z,1665783105.210 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PAUSED (@883.0) 2022-10-14T21:31:45.613Z,1665783105.613 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PROCESSING (@883.0) 2022-10-14T21:31:47.634Z,1665783107.634 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T21:31:48.037Z,1665783108.037 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_PROCESSED (@12,archive_bac) 2022-10-14T21:43:51.614Z,1665783831.614 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_STOPPING (@12,archive_bac) 2022-10-14T21:44:01.714Z,1665783841.714 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-14T21:44:02.120Z,1665783842.120 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_STOPPED 2022-10-14T21:49:01.895Z,1665784141.895 [ESPComponent](IMPORTANT): [sample #25] ESP sampling sequence completed normally. 2022-10-14T21:49:02.763Z,1665784142.763 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 24.000000 count completed. 2022-10-14T21:49:09.223Z,1665784149.223 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-14T21:54:05.311Z,1665784445.311 [NAL9602](FAULT): Timeout waiting for SBDI response. 2022-10-14T21:54:49.753Z,1665784489.753 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T21:56:22.241Z,1665784582.241 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-14T22:01:23.631Z,1665784883.631 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-14T22:23:47.636Z,1665786227.636 [ThrusterHE](FAULT): getSpeed error serial timeout 2022-10-14T22:23:47.636Z,1665786227.636 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2022-10-14T22:42:41.423Z,1665787361.423 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-14T22:42:43.831Z,1665787363.831 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-14T22:42:44.236Z,1665787364.236 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 25.000000 count . Target depth = 150.000000 m . 2022-10-14T22:42:44.642Z,1665787364.642 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-14T22:54:16.076Z,1665788056.076 [ThrusterHE](FAULT): getSpeed error serial timeout 2022-10-14T22:54:16.076Z,1665788056.076 [ThrusterHE](FAULT): Thruster uart error: serial timeout 2022-10-14T22:55:14.922Z,1665788114.922 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.005539 m , settling for 5.000000 min 2022-10-14T23:00:27.676Z,1665788427.676 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-14T23:00:27.889Z,1665788427.889 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-14T23:01:29.853Z,1665788489.853 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-14T23:01:30.669Z,1665788490.669 [ESPComponent](IMPORTANT): [sample #26] Setting cartridge argument to -6 2022-10-14T23:01:30.670Z,1665788490.670 [ESPComponent](IMPORTANT): [sample #26] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-14T23:01:31.115Z,1665788491.115 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.584030 m in 9.917994 degC water with 0.084240 ug/l chlorophyll fluorescence. 2022-10-14T23:01:31.469Z,1665788491.469 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-14T23:01:31.874Z,1665788491.874 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-14T23:01:33.895Z,1665788493.895 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-14T23:02:08.234Z,1665788528.234 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_PRIMING (@11,archive_bac) 2022-10-14T23:02:11.063Z,1665788531.063 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PRIMING (@100.0) 2022-10-14T23:02:11.066Z,1665788531.066 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-14T23:02:11.466Z,1665788531.466 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-14T23:04:30.845Z,1665788670.845 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_FILTERING (@0.0) 2022-10-14T23:04:31.250Z,1665788671.250 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-14T23:49:35.249Z,1665791375.249 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PAUSED (@1000.0) 2022-10-14T23:49:35.653Z,1665791375.653 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-14T23:49:37.674Z,1665791377.674 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-14T23:49:38.077Z,1665791378.077 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_PROCESSED (@11,archive_bac) 2022-10-15T00:01:42.053Z,1665792102.053 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_STOPPING (@11,archive_bac) 2022-10-15T00:01:51.750Z,1665792111.750 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-15T00:01:52.154Z,1665792112.154 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_STOPPED 2022-10-15T00:06:51.921Z,1665792411.921 [ESPComponent](IMPORTANT): [sample #26] ESP sampling sequence completed normally. 2022-10-15T00:06:52.769Z,1665792412.769 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 25.000000 count completed. 2022-10-15T00:06:59.241Z,1665792419.241 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-15T00:24:44.573Z,1665793484.573 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-15T00:26:12.591Z,1665793572.591 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-15T00:31:13.910Z,1665793873.910 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-15T00:56:25.305Z,1665795385.305 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-15T00:56:27.741Z,1665795387.741 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 2022-10-15T00:56:28.163Z,1665795388.163 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 26.000000 count . Target depth = 5.000000 m . 2022-10-15T00:56:28.536Z,1665795388.536 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-15T01:00:29.366Z,1665795629.366 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.985934 m , settling for 5.000000 min 2022-10-15T01:05:42.106Z,1665795942.106 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-15T01:05:42.455Z,1665795942.455 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-15T01:06:44.081Z,1665796004.081 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-15T01:06:44.896Z,1665796004.896 [ESPComponent](IMPORTANT): [sample #27] Setting cartridge argument to -6 2022-10-15T01:06:44.897Z,1665796004.897 [ESPComponent](IMPORTANT): [sample #27] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-15T01:06:45.394Z,1665796005.394 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.034399 m in 14.770227 degC water with 5.085600 ug/l chlorophyll fluorescence. 2022-10-15T01:06:45.697Z,1665796005.697 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-15T01:06:46.101Z,1665796006.101 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-15T01:06:48.121Z,1665796008.121 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-15T01:07:22.462Z,1665796042.462 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_WAITING_FOR_PRIMING (@10,archive_bac) 2022-10-15T01:07:25.290Z,1665796045.290 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PRIMING (@100.0) 2022-10-15T01:07:25.291Z,1665796045.291 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-15T01:07:25.695Z,1665796045.695 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-15T01:09:44.269Z,1665796184.269 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_FILTERING (@0.0) 2022-10-15T01:09:44.677Z,1665796184.677 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-15T01:43:52.569Z,1665798232.569 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PAUSED (@534.9) 2022-10-15T01:43:52.976Z,1665798232.976 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PROCESSING (@534.9) 2022-10-15T01:43:54.995Z,1665798234.995 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-15T01:43:55.397Z,1665798235.397 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_WAITING_FOR_PROCESSED (@10,archive_bac) 2022-10-15T01:55:58.986Z,1665798958.986 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_STOPPING (@10,archive_bac) 2022-10-15T01:56:08.686Z,1665798968.686 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-15T01:56:09.086Z,1665798969.086 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_STOPPED 2022-10-15T02:01:08.861Z,1665799268.861 [ESPComponent](IMPORTANT): [sample #27] ESP sampling sequence completed normally. 2022-10-15T02:01:09.734Z,1665799269.734 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 26.000000 count completed. 2022-10-15T02:01:16.186Z,1665799276.186 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min . 2022-10-15T02:06:47.861Z,1665799607.861 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-15T02:08:13.383Z,1665799693.383 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.800001,-122.096994 2022-10-15T02:11:41.005Z,1665799901.005 [Radio_Surface](FAULT): LCB fault: Invalid Command. Current Limiter Activated. 2022-10-15T02:13:14.618Z,1665799994.618 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session. 2022-10-15T02:41:25.459Z,1665801685.459 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.800001,-122.096994 2022-10-15T02:41:27.860Z,1665801687.860 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 2022-10-15T02:41:28.271Z,1665801688.271 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 27.000000 count . Target depth = 150.000000 m . 2022-10-15T02:41:28.691Z,1665801688.691 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 2022-10-15T02:53:05.986Z,1665802385.986 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.041870 m , settling for 5.000000 min 2022-10-15T02:58:18.752Z,1665802698.752 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8 2022-10-15T02:58:18.896Z,1665802698.896 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000) 2022-10-15T02:59:20.093Z,1665802760.093 [ESPComponent](IMPORTANT): ESP has connected as client 2022-10-15T02:59:20.903Z,1665802760.903 [ESPComponent](IMPORTANT): [sample #28] Setting cartridge argument to -6 2022-10-15T02:59:20.904Z,1665802760.904 [ESPComponent](IMPORTANT): [sample #28] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT 2022-10-15T02:59:21.365Z,1665802761.365 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.365631 m in 9.822748 degC water with 0.072800 ug/l chlorophyll fluorescence. 2022-10-15T02:59:21.710Z,1665802761.710 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PREPARING_SHOW_LOG 2022-10-15T02:59:22.111Z,1665802762.111 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PREPARING_SHOW_STATUS 2022-10-15T02:59:24.129Z,1665802764.129 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_LOADING_CARTRIDGE 2022-10-15T02:59:58.469Z,1665802798.469 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_WAITING_FOR_PRIMING (@9,archive_bac) 2022-10-15T03:00:01.297Z,1665802801.297 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PRIMING (@100.0) 2022-10-15T03:00:01.298Z,1665802801.298 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering 2022-10-15T03:00:01.702Z,1665802801.702 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0) 2022-10-15T03:02:20.685Z,1665802940.685 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_FILTERING (@0.0) 2022-10-15T03:02:21.088Z,1665802941.088 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0) 2022-10-15T03:19:58.928Z,1665803998.928 [CBIT](IMPORTANT): Beginning ground fault scan 2022-10-15T03:20:09.778Z,1665804009.778 [CBIT](IMPORTANT): No ground fault detected mA: CHAN A0 (Batt): -0.003474 CHAN A1 (24V): 0.010124 CHAN A2 (12V): -0.003672 CHAN A3 (5V): -0.003667 CHAN B0 (3.3V): -0.001478 CHAN B1 (3.15aV): -0.001381 CHAN B2 (3.15bV): -0.001073 CHAN B3 (GND): -0.001309 OPEN: 0.012366 Full Scale: +/- 1 mA 2022-10-15T03:48:23.301Z,1665805703.301 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PAUSED (@1000.0) 2022-10-15T03:48:23.706Z,1665805703.706 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PROCESSING (@1000.0) 2022-10-15T03:48:25.726Z,1665805705.726 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing 2022-10-15T03:48:26.132Z,1665805706.132 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_WAITING_FOR_PROCESSED (@9,archive_bac) 2022-10-15T04:00:30.114Z,1665806430.114 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_STOPPING (@9,archive_bac) 2022-10-15T04:00:39.814Z,1665806439.814 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs & 2022-10-15T04:00:40.262Z,1665806440.262 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_STOPPED 2022-10-15T04:05:40.029Z,1665806740.029 [ESPComponent](IMPORTANT): [sample #28] ESP sampling sequence completed normally. 2022-10-15T04:05:40.916Z,1665806740.916 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 27.000000 count completed. 2022-10-15T04:05:47.369Z,1665806747.369 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min . 2022-10-15T04:22:44.211Z,1665807764.211 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47325, MT Status=1, MTMSN=3523