2022-10-12T02:06:15.851Z,1665540375.851 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg
2022-10-12T02:06:18.784Z,1665540378.784 [SBIT](IMPORTANT): git: 2022-10-11
2022-10-12T02:06:18.785Z,1665540378.785 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8
2022-10-12T02:06:20.400Z,1665540380.400 [RudderServo](FAULT): Rudder failed to initialize
2022-10-12T02:07:05.342Z,1665540425.342 [SBIT](IMPORTANT): Beginning Startup BIT
2022-10-12T02:07:05.346Z,1665540425.346 [CBIT](IMPORTANT): Beginning ground fault scan
2022-10-12T02:07:16.297Z,1665540436.297 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002526
CHAN A1 (24V): 0.005492
CHAN A2 (12V): -0.005104
CHAN A3 (5V): -0.003897
CHAN B0 (3.3V): -0.002002
CHAN B1 (3.15aV): -0.001693
CHAN B2 (3.15bV): -0.001026
CHAN B3 (GND): -0.001012
OPEN: 0.011925
Full Scale: +/- 1 mA
2022-10-12T02:07:59.477Z,1665540479.477 [SBIT](IMPORTANT): SBIT PASSED
2022-10-12T02:07:59.478Z,1665540479.478 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg
2022-10-12T02:07:59.478Z,1665540479.478 [SBIT](IMPORTANT): BPC1.batteryMissingStickThreshold=2 count;

2022-10-12T02:07:59.479Z,1665540479.479 [SBIT](IMPORTANT): BPC1.loadAtStartup=0 bool;

2022-10-12T02:07:59.479Z,1665540479.479 [SBIT](IMPORTANT): CBIT.runNavErrorCritical=0 bool;

2022-10-12T02:07:59.479Z,1665540479.479 [SBIT](IMPORTANT): DAT.loadAtStartup=1 bool;

2022-10-12T02:07:59.479Z,1665540479.479 [SBIT](IMPORTANT): DAT.sbdAddress=-1 enum;

2022-10-12T02:07:59.479Z,1665540479.479 [SBIT](IMPORTANT): DVL_micro.loadAtStartup=0 bool;

2022-10-12T02:07:59.479Z,1665540479.479 [SBIT](IMPORTANT): Depth_Keller.offset=-0.8 decibar;

2022-10-12T02:07:59.479Z,1665540479.479 [SBIT](IMPORTANT): ElevatorServo.offsetAngle=1 degree;

2022-10-12T02:07:59.480Z,1665540479.480 [SBIT](IMPORTANT): Express linearApproximation NAL9602.sigQuality 1.000000 count;

2022-10-12T02:07:59.480Z,1665540479.480 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff470nm;

2022-10-12T02:07:59.480Z,1665540479.480 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff650nm;

2022-10-12T02:07:59.480Z,1665540479.480 [SBIT](IMPORTANT): RudderServo.offsetAngle=0 degree;

2022-10-12T02:07:59.480Z,1665540479.480 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=86.344810 cubic_centimeter;

2022-10-12T02:07:59.480Z,1665540479.480 [SBIT](IMPORTANT): VerticalControl.massBackOnGoToSurface=0 bool;

2022-10-12T02:07:59.480Z,1665540479.480 [SBIT](IMPORTANT): VerticalControl.massDefault=-6.557592 millimeter;

2022-10-12T02:07:59.480Z,1665540479.480 [SBIT](IMPORTANT): VerticalControl.massPositionLimitAft=-37 millimeter;

2022-10-12T02:07:59.882Z,1665540479.882 [MissionManager](IMPORTANT): Started mission Startup
2022-10-12T02:10:00.676Z,1665540600.676 [MissionManager](IMPORTANT): Started mission Default
2022-10-12T02:12:02.270Z,1665540722.270 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T02:13:06.594Z,1665540786.594 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=47295, MT Status=1, MTMSN=3516
2022-10-12T02:13:07.148Z,1665540787.148 [CommandExec](IMPORTANT): got command failComponent
2022-10-12T02:13:07.148Z,1665540787.148 [CommandExec](IMPORTANT): Failed components:
2022-10-12T02:13:07.150Z,1665540787.150 [CommandExec](IMPORTANT): No failed Components.
2022-10-12T02:15:02.895Z,1665540902.895 [DataOverHttps](IMPORTANT): SBD MOMSN=17242807, MTMSN=20221012T021459
2022-10-12T02:15:17.724Z,1665540917.724 [CommandExec](IMPORTANT): got command schedule asap "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 12 count;set LocationDepthSampling.MissionTimeout 36 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s" 3yrks 1 3.000000
2022-10-12T02:15:17.725Z,1665540917.725 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=3yrks
2022-10-12T02:15:17.726Z,1665540917.726 [CommandExec](IMPORTANT): Scheduled #1 (#1 of 3 with id='3yrks'): "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 12 count;set LocationDepthSampling.MissionTimeout 36 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s", ASAP
2022-10-12T02:15:19.415Z,1665540919.415 [DataOverHttps](IMPORTANT): SBD MTMSN=20221012T021514
2022-10-12T02:15:29.541Z,1665540929.541 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m" 3yrks 2 3.000000
2022-10-12T02:15:29.541Z,1665540929.541 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=3yrks
2022-10-12T02:15:29.542Z,1665540929.542 [CommandExec](IMPORTANT): Scheduled #2 (#2 of 3 with id='3yrks'): "set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m", ASAP
2022-10-12T02:15:30.780Z,1665540930.780 [DataOverHttps](IMPORTANT): SBD MOMSN=17242836, MTMSN=20221012T021526
2022-10-12T02:15:47.403Z,1665540947.403 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run" 3yrks 3 3.000000
2022-10-12T02:15:47.403Z,1665540947.403 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=3yrks
2022-10-12T02:15:47.404Z,1665540947.404 [CommandExec](IMPORTANT): Scheduled #3 (#3 of 3 with id='3yrks'): "set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run", ASAP
2022-10-12T02:15:47.544Z,1665540947.544 [CommandExec](IMPORTANT): got command load ./Missions/Science/location_depth_sampling.xml
2022-10-12T02:15:59.781Z,1665540959.781 [CommandExec](IMPORTANT): Loaded ./Missions/Science/location_depth_sampling.xml
2022-10-12T02:16:31.378Z,1665540991.378 [CommandExec](IMPORTANT): got command set LocationDepthSampling.NumSamplers 12 count
2022-10-12T02:16:31.378Z,1665540991.378 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MissionTimeout 36 hour
2022-10-12T02:16:31.379Z,1665540991.379 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedTransit 0.8 meter_per_second
2022-10-12T02:16:31.380Z,1665540991.380 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedSampling 0.8 meter_per_second
2022-10-12T02:16:55.756Z,1665541015.756 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MaxDepth 180 meter
2022-10-12T02:16:55.757Z,1665541015.757 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LatSampling 36.80 degree
2022-10-12T02:16:55.757Z,1665541015.757 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LonSampling -122.097 degree
2022-10-12T02:16:55.758Z,1665541015.758 [CommandExec](IMPORTANT): got command set LocationDepthSampling.Dep1 150 meter
2022-10-12T02:17:01.550Z,1665541021.550 [CommandExec](IMPORTANT): got command set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 meter_per_second
2022-10-12T02:17:01.550Z,1665541021.550 [CommandExec](IMPORTANT): got command run
2022-10-12T02:17:01.599Z,1665541021.599 [CommandExec](IMPORTANT): Running 
2022-10-12T02:17:01.757Z,1665541021.757 [MissionManager](IMPORTANT): Started mission LocationDepthSampling
2022-10-12T02:17:01.880Z,1665541021.880 [LocationDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2022-10-12T02:17:39.783Z,1665541059.783 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47296, MT Status=1, MTMSN=3517
2022-10-12T02:17:40.616Z,1665541060.616 [CommandExec](IMPORTANT): got command schedule asap "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 12 count;set LocationDepthSampling.MissionTimeout 36 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s" 3yrdr 1 3.000000
2022-10-12T02:17:40.616Z,1665541060.616 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=3yrdr
2022-10-12T02:17:40.617Z,1665541060.617 [CommandExec](IMPORTANT): Scheduled #4 (#1 of 3 with id='3yrdr'): "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 12 count;set LocationDepthSampling.MissionTimeout 36 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s", ASAP
2022-10-12T02:20:21.458Z,1665541221.458 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-12T02:24:12.967Z,1665541452.967 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47297, MT Status=1, MTMSN=3518
2022-10-12T02:24:13.748Z,1665541453.748 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m" 3yrdr 2 3.000000
2022-10-12T02:24:13.749Z,1665541453.749 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=3yrdr
2022-10-12T02:24:13.750Z,1665541453.750 [CommandExec](IMPORTANT): Scheduled #5 (#2 of 3 with id='3yrdr'): "set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m", ASAP
2022-10-12T02:25:17.169Z,1665541517.169 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-12T02:26:21.929Z,1665541581.929 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=47298, MT Status=1, MTMSN=3519
2022-10-12T02:26:22.559Z,1665541582.559 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run" 3yrdr 3 3.000000
2022-10-12T02:26:22.559Z,1665541582.559 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=3yrdr
2022-10-12T02:26:22.560Z,1665541582.560 [CommandExec](IMPORTANT): Scheduled #6 (#3 of 3 with id='3yrdr'): "set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run", ASAP
2022-10-12T02:26:22.703Z,1665541582.703 [CommandExec](IMPORTANT): got command load ./Missions/Science/location_depth_sampling.xml
2022-10-12T02:26:33.466Z,1665541593.466 [CommandExec](IMPORTANT): Loaded ./Missions/Science/location_depth_sampling.xml
2022-10-12T02:26:33.671Z,1665541593.671 [MissionManager](IMPORTANT): Started mission Default
2022-10-12T02:26:57.420Z,1665541617.420 [CommandExec](IMPORTANT): got command set LocationDepthSampling.NumSamplers 12 count
2022-10-12T02:26:57.421Z,1665541617.421 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MissionTimeout 36 hour
2022-10-12T02:26:57.421Z,1665541617.421 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedTransit 0.8 meter_per_second
2022-10-12T02:26:57.422Z,1665541617.422 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedSampling 0.8 meter_per_second
2022-10-12T02:27:19.788Z,1665541639.788 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MaxDepth 180 meter
2022-10-12T02:27:19.788Z,1665541639.788 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LatSampling 36.80 degree
2022-10-12T02:27:19.790Z,1665541639.790 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LonSampling -122.097 degree
2022-10-12T02:27:19.791Z,1665541639.791 [CommandExec](IMPORTANT): got command set LocationDepthSampling.Dep1 150 meter
2022-10-12T02:27:27.992Z,1665541647.992 [CommandExec](IMPORTANT): got command set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 meter_per_second
2022-10-12T02:27:27.992Z,1665541647.992 [CommandExec](IMPORTANT): got command run
2022-10-12T02:27:27.994Z,1665541647.994 [CommandExec](IMPORTANT): Running 
2022-10-12T02:27:28.340Z,1665541648.340 [MissionManager](IMPORTANT): Started mission LocationDepthSampling
2022-10-12T02:27:28.414Z,1665541648.414 [LocationDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2022-10-12T02:32:48.165Z,1665541968.165 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T02:32:54.697Z,1665541974.697 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-12T02:33:55.171Z,1665542035.171 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-12T02:38:56.411Z,1665542336.411 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T03:02:07.054Z,1665543727.054 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-12T03:02:09.467Z,1665543729.467 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 
2022-10-12T03:02:09.895Z,1665543729.895 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 150.000000 m .
2022-10-12T03:02:10.282Z,1665543730.282 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-12T03:13:54.057Z,1665544434.057 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.007782 m , settling for 5.000000 min 
2022-10-12T03:18:56.609Z,1665544736.609 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:'
2022-10-12T03:19:06.744Z,1665544746.744 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-12T03:19:07.065Z,1665544747.065 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-12T03:20:07.438Z,1665544807.438 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-12T03:20:08.251Z,1665544808.251 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -6
2022-10-12T03:20:08.252Z,1665544808.252 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-12T03:20:08.716Z,1665544808.716 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.735138 m in 10.134668 degC water with 0.081120 ug/l chlorophyll fluorescence.
2022-10-12T03:20:09.458Z,1665544809.458 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-12T03:20:09.861Z,1665544809.861 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-12T03:20:11.479Z,1665544811.479 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-12T03:20:45.826Z,1665544845.826 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@45,archive_bac)
2022-10-12T03:20:48.653Z,1665544848.653 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0)
2022-10-12T03:20:48.654Z,1665544848.654 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-12T03:20:49.057Z,1665544849.057 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-12T03:23:12.479Z,1665544992.479 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0)
2022-10-12T03:23:12.882Z,1665544992.882 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-12T04:06:36.293Z,1665547596.293 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@1000.0)
2022-10-12T04:06:36.697Z,1665547596.697 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-12T04:06:38.723Z,1665547598.723 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-12T04:06:39.126Z,1665547599.126 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@45,archive_bac)
2022-10-12T04:18:42.702Z,1665548322.702 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@45,archive_bac)
2022-10-12T04:18:52.398Z,1665548332.398 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-12T04:18:52.802Z,1665548332.802 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED
2022-10-12T04:23:52.577Z,1665548632.577 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally.
2022-10-12T04:23:53.468Z,1665548633.468 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed.
2022-10-12T04:23:59.929Z,1665548639.929 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min .
2022-10-12T04:41:30.304Z,1665549690.304 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T04:46:52.334Z,1665550012.334 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-12T04:51:53.583Z,1665550313.583 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T04:54:45.708Z,1665550485.708 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-12T04:54:48.136Z,1665550488.136 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 
2022-10-12T04:54:48.550Z,1665550488.550 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Target depth = 5.000000 m .
2022-10-12T04:54:48.967Z,1665550488.967 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-12T04:58:45.295Z,1665550725.295 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.979202 m , settling for 5.000000 min 
2022-10-12T05:03:58.056Z,1665551038.056 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-12T05:03:58.178Z,1665551038.178 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-12T05:04:58.581Z,1665551098.581 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-12T05:04:59.404Z,1665551099.404 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -6
2022-10-12T05:04:59.404Z,1665551099.404 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-12T05:04:59.847Z,1665551099.847 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.966638 m in 13.795862 degC water with 3.939520 ug/l chlorophyll fluorescence.
2022-10-12T05:05:00.201Z,1665551100.201 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-12T05:05:00.605Z,1665551100.605 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-12T05:05:02.627Z,1665551102.627 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-12T05:05:36.969Z,1665551136.969 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@44,archive_bac)
2022-10-12T05:05:38.993Z,1665551138.993 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0)
2022-10-12T05:05:39.394Z,1665551139.394 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-12T05:05:39.798Z,1665551139.798 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-12T05:07:58.372Z,1665551278.372 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0)
2022-10-12T05:07:58.773Z,1665551278.773 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-12T06:08:15.471Z,1665554895.471 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@1000.0)
2022-10-12T06:08:15.873Z,1665554895.873 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-12T06:08:17.894Z,1665554897.894 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-12T06:08:18.298Z,1665554898.298 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@44,archive_bac)
2022-10-12T06:20:21.866Z,1665555621.866 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@44,archive_bac)
2022-10-12T06:20:31.562Z,1665555631.562 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-12T06:20:31.966Z,1665555631.966 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED
2022-10-12T06:25:31.740Z,1665555931.740 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally.
2022-10-12T06:25:32.612Z,1665555932.612 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed.
2022-10-12T06:25:39.065Z,1665555939.065 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-12T06:31:10.367Z,1665556270.367 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T06:32:28.435Z,1665556348.435 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-12T06:37:29.357Z,1665556649.357 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T06:48:25.549Z,1665557305.549 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-12T06:48:27.983Z,1665557307.983 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 
2022-10-12T06:48:28.397Z,1665557308.397 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 3.000000 count . Target depth = 150.000000 m .
2022-10-12T06:48:28.801Z,1665557308.801 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-12T07:00:16.593Z,1665558016.593 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.032883 m , settling for 5.000000 min 
2022-10-12T07:05:29.360Z,1665558329.360 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-12T07:05:29.499Z,1665558329.499 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-12T07:06:31.081Z,1665558391.081 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-12T07:06:31.892Z,1665558391.892 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6
2022-10-12T07:06:31.893Z,1665558391.893 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-12T07:06:32.364Z,1665558392.364 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.902390 m in 9.798731 degC water with 0.073840 ug/l chlorophyll fluorescence.
2022-10-12T07:06:32.698Z,1665558392.698 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-12T07:06:33.101Z,1665558393.101 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-12T07:06:35.122Z,1665558395.122 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-12T07:07:09.462Z,1665558429.462 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@42,archive_bac)
2022-10-12T07:07:12.290Z,1665558432.290 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0)
2022-10-12T07:07:12.291Z,1665558432.291 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-12T07:07:12.700Z,1665558432.700 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-12T07:09:31.272Z,1665558571.272 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0)
2022-10-12T07:09:31.674Z,1665558571.674 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-12T07:52:04.206Z,1665561124.206 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1000.0)
2022-10-12T07:52:04.609Z,1665561124.609 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-12T07:52:06.631Z,1665561126.631 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-12T07:52:07.033Z,1665561127.033 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@42,archive_bac)
2022-10-12T08:04:10.622Z,1665561850.622 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@42,archive_bac)
2022-10-12T08:04:20.310Z,1665561860.310 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-12T08:04:20.718Z,1665561860.718 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED
2022-10-12T08:07:16.620Z,1665562036.620 [CBIT](IMPORTANT): Beginning ground fault scan
2022-10-12T08:07:27.467Z,1665562047.467 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.001299
CHAN A1 (24V): 0.010523
CHAN A2 (12V): -0.002983
CHAN A3 (5V): -0.003605
CHAN B0 (3.3V): -0.001423
CHAN B1 (3.15aV): -0.001597
CHAN B2 (3.15bV): -0.001414
CHAN B3 (GND): -0.001053
OPEN: 0.012321
Full Scale: +/- 1 mA
2022-10-12T08:09:20.483Z,1665562160.483 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally.
2022-10-12T08:09:21.350Z,1665562161.350 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 3.000000 count completed.
2022-10-12T08:09:27.795Z,1665562167.795 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min .
2022-10-12T08:26:17.390Z,1665563177.390 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T08:35:28.561Z,1665563728.561 [Power24vConverter](FAULT): LCB fault:  Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated.
2022-10-12T08:35:28.562Z,1665563728.562 [Power24vConverter](FAULT): 24V Power is reporting not powering/powered.
2022-10-12T08:43:28.476Z,1665564208.476 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-12T08:48:29.364Z,1665564509.364 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T08:52:02.294Z,1665564722.294 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-12T08:52:04.732Z,1665564724.732 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 
2022-10-12T08:52:05.122Z,1665564725.122 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 4.000000 count . Target depth = 5.000000 m .
2022-10-12T08:52:05.541Z,1665564725.541 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-12T08:56:06.738Z,1665564966.738 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.946892 m , settling for 5.000000 min 
2022-10-12T09:01:19.481Z,1665565279.481 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-12T09:01:19.837Z,1665565279.837 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-12T09:02:20.241Z,1665565340.241 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-12T09:02:21.053Z,1665565341.053 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6
2022-10-12T09:02:21.053Z,1665565341.053 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-12T09:02:21.531Z,1665565341.531 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.977856 m in 13.895472 degC water with 4.141280 ug/l chlorophyll fluorescence.
2022-10-12T09:02:21.857Z,1665565341.857 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-12T09:02:22.261Z,1665565342.261 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-12T09:02:24.282Z,1665565344.282 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-12T09:02:58.622Z,1665565378.622 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@41,archive_bac)
2022-10-12T09:03:01.451Z,1665565381.451 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0)
2022-10-12T09:03:01.453Z,1665565381.453 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-12T09:03:01.854Z,1665565381.854 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-12T09:05:20.022Z,1665565520.022 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0)
2022-10-12T09:05:20.425Z,1665565520.425 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-12T09:54:36.118Z,1665568476.118 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@781.9)
2022-10-12T09:54:36.521Z,1665568476.521 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@781.9)
2022-10-12T09:54:38.543Z,1665568478.543 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-12T09:54:38.946Z,1665568478.946 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@41,archive_bac)
2022-10-12T10:06:42.518Z,1665569202.518 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@41,archive_bac)
2022-10-12T10:06:52.215Z,1665569212.215 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-12T10:06:52.643Z,1665569212.643 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED
2022-10-12T10:11:52.406Z,1665569512.406 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally.
2022-10-12T10:11:53.254Z,1665569513.254 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 4.000000 count completed.
2022-10-12T10:11:59.719Z,1665569519.719 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-12T10:19:26.158Z,1665569966.158 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-12T10:24:27.306Z,1665570267.306 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T10:30:49.542Z,1665570649.542 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-12T10:30:51.934Z,1665570651.934 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 
2022-10-12T10:30:52.343Z,1665570652.343 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 5.000000 count . Target depth = 150.000000 m .
2022-10-12T10:30:52.731Z,1665570652.731 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-12T10:42:18.745Z,1665571338.745 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.041412 m , settling for 5.000000 min 
2022-10-12T10:47:31.509Z,1665571651.509 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-12T10:47:31.609Z,1665571651.609 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-12T10:48:32.016Z,1665571712.016 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-12T10:48:32.835Z,1665571712.835 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6
2022-10-12T10:48:32.836Z,1665571712.836 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-12T10:48:33.271Z,1665571713.271 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 151.171921 m in 10.046625 degC water with 0.080080 ug/l chlorophyll fluorescence.
2022-10-12T10:48:33.633Z,1665571713.633 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-12T10:48:34.038Z,1665571714.038 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-12T10:48:36.057Z,1665571716.057 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-12T10:49:10.397Z,1665571750.397 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@40,archive_bac)
2022-10-12T10:49:12.821Z,1665571752.821 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0)
2022-10-12T10:49:12.822Z,1665571752.822 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-12T10:49:13.227Z,1665571753.227 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-12T10:51:40.691Z,1665571900.691 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0)
2022-10-12T10:51:41.094Z,1665571901.094 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-12T11:35:49.487Z,1665574549.487 [ThrusterHE](FAULT): Could not get status:serial timeout
2022-10-12T11:37:17.834Z,1665574637.834 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0)
2022-10-12T11:37:18.238Z,1665574638.238 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-12T11:37:20.258Z,1665574640.258 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-12T11:37:20.663Z,1665574640.663 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@40,archive_bac)
2022-10-12T11:49:24.241Z,1665575364.241 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@40,archive_bac)
2022-10-12T11:49:33.938Z,1665575373.938 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-12T11:49:34.343Z,1665575374.343 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED
2022-10-12T11:54:34.121Z,1665575674.121 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally.
2022-10-12T11:54:35.058Z,1665575675.058 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 5.000000 count completed.
2022-10-12T11:54:41.446Z,1665575681.446 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min .
2022-10-12T12:11:34.802Z,1665576694.802 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-12T12:16:36.527Z,1665576996.527 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T12:21:00.013Z,1665577260.013 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-12T12:21:02.446Z,1665577262.446 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 
2022-10-12T12:21:02.833Z,1665577262.833 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 6.000000 count . Target depth = 5.000000 m .
2022-10-12T12:21:03.232Z,1665577263.232 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-12T12:24:59.596Z,1665577499.596 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.978305 m , settling for 5.000000 min 
2022-10-12T12:30:12.344Z,1665577812.344 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-12T12:30:12.513Z,1665577812.513 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-12T12:31:13.669Z,1665577873.669 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-12T12:31:14.481Z,1665577874.481 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -6
2022-10-12T12:31:14.482Z,1665577874.482 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-12T12:31:14.945Z,1665577874.945 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.208964 m in 14.670313 degC water with 3.519360 ug/l chlorophyll fluorescence.
2022-10-12T12:31:15.286Z,1665577875.286 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-12T12:31:15.691Z,1665577875.691 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-12T12:31:17.711Z,1665577877.711 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-12T12:31:52.049Z,1665577912.049 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@39,archive_bac)
2022-10-12T12:31:54.070Z,1665577914.070 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0)
2022-10-12T12:31:54.474Z,1665577914.474 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-12T12:31:54.878Z,1665577914.878 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-12T12:34:13.858Z,1665578053.858 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0)
2022-10-12T12:34:14.262Z,1665578054.262 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-12T13:28:50.342Z,1665581330.342 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@855.8)
2022-10-12T13:28:50.742Z,1665581330.742 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@855.8)
2022-10-12T13:28:52.762Z,1665581332.762 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-12T13:28:53.166Z,1665581333.166 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@39,archive_bac)
2022-10-12T13:40:56.741Z,1665582056.741 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@39,archive_bac)
2022-10-12T13:41:06.438Z,1665582066.438 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-12T13:41:06.842Z,1665582066.842 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED
2022-10-12T13:46:06.633Z,1665582366.633 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally.
2022-10-12T13:46:07.498Z,1665582367.498 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 6.000000 count completed.
2022-10-12T13:46:13.962Z,1665582373.962 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-12T13:51:43.588Z,1665582703.588 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T13:59:53.168Z,1665583193.168 [AHRS_M2](FAULT): Failed to acquire valid data within timeout.
2022-10-12T14:07:27.566Z,1665583647.566 [CBIT](IMPORTANT): Beginning ground fault scan
2022-10-12T14:07:49.889Z,1665583669.889 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002755
CHAN A1 (24V): 0.007437
CHAN A2 (12V): -0.003980
CHAN A3 (5V): -0.002516
CHAN B0 (3.3V): -0.002022
CHAN B1 (3.15aV): -0.000958
CHAN B2 (3.15bV): -0.001328
CHAN B3 (GND): -0.000706
OPEN: 0.012365
Full Scale: +/- 1 mA
2022-10-12T14:13:15.336Z,1665583995.336 [AHRS_M2](FAULT): Failed to acquire valid data within timeout.
2022-10-12T14:21:59.030Z,1665584519.030 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-12T14:23:00.413Z,1665584580.413 [ElevatorServo](FAULT): Reading outside of valid range:0.000000

2022-10-12T14:23:01.084Z,1665584581.084 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout
2022-10-12T14:50:45.398Z,1665586245.398 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-12T14:50:47.833Z,1665586247.833 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 
2022-10-12T14:50:48.219Z,1665586248.219 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 7.000000 count . Target depth = 150.000000 m .
2022-10-12T14:50:48.625Z,1665586248.625 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-12T15:02:32.804Z,1665586952.804 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.019455 m , settling for 5.000000 min 
2022-10-12T15:07:45.568Z,1665587265.568 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-12T15:07:45.706Z,1665587265.706 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-12T15:08:46.900Z,1665587326.900 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-12T15:08:47.715Z,1665587327.715 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -6
2022-10-12T15:08:47.716Z,1665587327.716 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-12T15:08:48.158Z,1665587328.158 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 151.215424 m in 10.069880 degC water with 0.087360 ug/l chlorophyll fluorescence.
2022-10-12T15:08:48.519Z,1665587328.519 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-12T15:08:48.922Z,1665587328.922 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-12T15:08:50.943Z,1665587330.943 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-12T15:09:25.282Z,1665587365.282 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@38,archive_bac)
2022-10-12T15:09:27.302Z,1665587367.302 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0)
2022-10-12T15:09:27.707Z,1665587367.707 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-12T15:09:28.109Z,1665587368.109 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-12T15:11:47.497Z,1665587507.497 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0)
2022-10-12T15:11:47.894Z,1665587507.894 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-12T15:59:20.986Z,1665590360.986 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1000.0)
2022-10-12T15:59:21.392Z,1665590361.392 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-12T15:59:23.413Z,1665590363.413 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-12T15:59:23.813Z,1665590363.813 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@38,archive_bac)
2022-10-12T16:11:27.398Z,1665591087.398 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@38,archive_bac)
2022-10-12T16:11:37.094Z,1665591097.094 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-12T16:11:37.500Z,1665591097.500 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED
2022-10-12T16:16:37.279Z,1665591397.279 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally.
2022-10-12T16:16:38.126Z,1665591398.126 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 7.000000 count completed.
2022-10-12T16:16:44.599Z,1665591404.599 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min .
2022-10-12T16:33:45.512Z,1665592425.512 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T17:03:54.331Z,1665594234.331 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-12T17:15:09.179Z,1665594909.179 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-12T17:15:11.584Z,1665594911.584 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 
2022-10-12T17:15:11.986Z,1665594911.986 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 8.000000 count . Target depth = 5.000000 m .
2022-10-12T17:15:12.404Z,1665594912.404 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-12T17:19:15.200Z,1665595155.200 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.975163 m , settling for 5.000000 min 
2022-10-12T17:24:27.961Z,1665595467.961 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-12T17:24:28.100Z,1665595468.100 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-12T17:25:28.532Z,1665595528.532 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-12T17:25:29.348Z,1665595529.348 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -6
2022-10-12T17:25:29.349Z,1665595529.349 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-12T17:25:29.809Z,1665595529.809 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.129983 m in 14.208307 degC water with 4.487600 ug/l chlorophyll fluorescence.
2022-10-12T17:25:30.150Z,1665595530.150 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-12T17:25:30.554Z,1665595530.554 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-12T17:25:32.572Z,1665595532.572 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-12T17:26:06.915Z,1665595566.915 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@37,archive_bac)
2022-10-12T17:26:09.338Z,1665595569.338 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0)
2022-10-12T17:26:09.339Z,1665595569.339 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-12T17:26:09.750Z,1665595569.750 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-12T17:28:28.322Z,1665595708.322 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0)
2022-10-12T17:28:28.727Z,1665595708.727 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-12T18:15:44.858Z,1665598544.858 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@735.0)
2022-10-12T18:15:45.261Z,1665598545.261 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@735.0)
2022-10-12T18:15:47.283Z,1665598547.283 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-12T18:15:47.686Z,1665598547.686 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@37,archive_bac)
2022-10-12T18:27:51.250Z,1665599271.250 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@37,archive_bac)
2022-10-12T18:28:00.946Z,1665599280.946 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-12T18:28:01.419Z,1665599281.419 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED
2022-10-12T18:33:01.592Z,1665599581.592 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally.
2022-10-12T18:33:02.443Z,1665599582.443 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 8.000000 count completed.
2022-10-12T18:33:08.922Z,1665599588.922 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-12T18:38:41.373Z,1665599921.373 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T18:48:43.772Z,1665600523.772 [AHRS_M2](FAULT): Failed to acquire valid data within timeout.
2022-10-12T18:59:42.176Z,1665601182.176 [AHRS_M2](FAULT): Failed to acquire valid data within timeout.
2022-10-12T19:06:15.045Z,1665601575.045 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=47309, MT Status=1, MTMSN=3520
2022-10-12T19:06:15.589Z,1665601575.589 [CommandExec](IMPORTANT): got command stop
2022-10-12T19:06:15.589Z,1665601575.589 [CommandExec](IMPORTANT): Scheduling is paused
2022-10-12T19:06:16.313Z,1665601576.313 [MissionManager](IMPORTANT): Started mission Default
2022-10-12T19:59:16.864Z,1665604756.864 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=47312, MT Status=1, MTMSN=3521
2022-10-12T19:59:17.410Z,1665604757.410 [CommandExec](IMPORTANT): got command restart system