2022-10-11T02:53:15.881Z,1665456795.881 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg
2022-10-11T02:53:18.562Z,1665456798.562 [SBIT](IMPORTANT): git: 2022-10-10
2022-10-11T02:53:18.562Z,1665456798.562 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8
2022-10-11T02:54:05.111Z,1665456845.111 [SBIT](IMPORTANT): Beginning Startup BIT
2022-10-11T02:54:05.115Z,1665456845.115 [CBIT](IMPORTANT): Beginning ground fault scan
2022-10-11T02:54:16.056Z,1665456856.056 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.004007
CHAN A1 (24V): 0.003544
CHAN A2 (12V): -0.004413
CHAN A3 (5V): -0.003688
CHAN B0 (3.3V): -0.001039
CHAN B1 (3.15aV): -0.001137
CHAN B2 (3.15bV): -0.001128
CHAN B3 (GND): -0.000147
OPEN: 0.012175
Full Scale: +/- 1 mA
2022-10-11T02:54:59.250Z,1665456899.250 [SBIT](IMPORTANT): SBIT PASSED
2022-10-11T02:54:59.250Z,1665456899.250 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg
2022-10-11T02:54:59.251Z,1665456899.251 [SBIT](IMPORTANT): BPC1.batteryMissingStickThreshold=2 count;

2022-10-11T02:54:59.251Z,1665456899.251 [SBIT](IMPORTANT): BPC1.loadAtStartup=0 bool;

2022-10-11T02:54:59.251Z,1665456899.251 [SBIT](IMPORTANT): CBIT.runNavErrorCritical=0 bool;

2022-10-11T02:54:59.252Z,1665456899.252 [SBIT](IMPORTANT): DAT.loadAtStartup=1 bool;

2022-10-11T02:54:59.252Z,1665456899.252 [SBIT](IMPORTANT): DAT.sbdAddress=-1 enum;

2022-10-11T02:54:59.252Z,1665456899.252 [SBIT](IMPORTANT): Depth_Keller.offset=-0.8 decibar;

2022-10-11T02:54:59.252Z,1665456899.252 [SBIT](IMPORTANT): ElevatorServo.offsetAngle=1 degree;

2022-10-11T02:54:59.252Z,1665456899.252 [SBIT](IMPORTANT): Express linearApproximation NAL9602.sigQuality 1.000000 count;

2022-10-11T02:54:59.252Z,1665456899.252 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff470nm;

2022-10-11T02:54:59.252Z,1665456899.252 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff650nm;

2022-10-11T02:54:59.252Z,1665456899.252 [SBIT](IMPORTANT): RudderServo.offsetAngle=0 degree;

2022-10-11T02:54:59.252Z,1665456899.252 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=86.344810 cubic_centimeter;

2022-10-11T02:54:59.253Z,1665456899.253 [SBIT](IMPORTANT): VerticalControl.massBackOnGoToSurface=0 bool;

2022-10-11T02:54:59.253Z,1665456899.253 [SBIT](IMPORTANT): VerticalControl.massDefault=-6.557592 millimeter;

2022-10-11T02:54:59.253Z,1665456899.253 [SBIT](IMPORTANT): VerticalControl.massPositionLimitAft=-37 millimeter;

2022-10-11T02:54:59.645Z,1665456899.645 [MissionManager](IMPORTANT): Started mission Startup
2022-10-11T02:57:00.454Z,1665457020.454 [MissionManager](IMPORTANT): Started mission Default
2022-10-11T02:59:05.670Z,1665457145.670 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T03:08:52.978Z,1665457732.978 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T030850
2022-10-11T03:09:02.008Z,1665457742.008 [CommandExec](IMPORTANT): got command schedule asap "load Maintenance/ballast_and_trim.tl;set ballast_and_trim.MissionTimeout 60 min;set ballast_and_trim.MassEstimationErrorBound 0.5 mm;set ballast_and_trim.BuoyEstimationErrorBound 25 cc;set ballast_and_trim.EstimationConfidence 98 %" 3wzea 1 2.000000
2022-10-11T03:09:02.008Z,1665457742.008 [CommandExec](IMPORTANT): Scheduling command #1 of 2 with id=3wzea
2022-10-11T03:09:02.009Z,1665457742.009 [CommandExec](IMPORTANT): Scheduled #1 (#1 of 2 with id='3wzea'): "load Maintenance/ballast_and_trim.tl;set ballast_and_trim.MissionTimeout 60 min;set ballast_and_trim.MassEstimationErrorBound 0.5 mm;set ballast_and_trim.BuoyEstimationErrorBound 25 cc;set ballast_and_trim.EstimationConfidence 98 %", ASAP
2022-10-11T03:09:02.921Z,1665457742.921 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T030859
2022-10-11T03:09:13.602Z,1665457753.602 [CommandExec](IMPORTANT): got command schedule asap "run" 3wzea 2 2.000000
2022-10-11T03:09:13.602Z,1665457753.602 [CommandExec](IMPORTANT): Scheduling command #2 of 2 with id=3wzea
2022-10-11T03:09:13.603Z,1665457753.603 [CommandExec](IMPORTANT): Scheduled #2 (#2 of 2 with id='3wzea'): "run", ASAP
2022-10-11T03:09:13.754Z,1665457753.754 [CommandExec](IMPORTANT): got command load ./Missions/Maintenance/ballast_and_trim.tl
2022-10-11T03:09:13.755Z,1665457753.755 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Maintenance/ballast_and_trim.tx
2022-10-11T03:09:18.431Z,1665457758.431 [CommandExec](IMPORTANT): Loaded ./Missions/Maintenance/ballast_and_trim.tl
2022-10-11T03:09:33.649Z,1665457773.649 [CommandExec](IMPORTANT): got command set ballast_and_trim.MissionTimeout 60 minute
2022-10-11T03:09:33.650Z,1665457773.650 [CommandExec](IMPORTANT): got command set ballast_and_trim.MassEstimationErrorBound 0.5 millimeter
2022-10-11T03:09:33.651Z,1665457773.651 [CommandExec](IMPORTANT): got command set ballast_and_trim.BuoyEstimationErrorBound 25 cubic_centimeter
2022-10-11T03:09:33.652Z,1665457773.652 [CommandExec](IMPORTANT): got command set ballast_and_trim.EstimationConfidence 98 percent
2022-10-11T03:09:33.653Z,1665457773.653 [CommandExec](IMPORTANT): got command run
2022-10-11T03:09:33.656Z,1665457773.656 [CommandExec](IMPORTANT): Running 
2022-10-11T03:09:33.910Z,1665457773.910 [MissionManager](IMPORTANT): Started mission ballast_and_trim
2022-10-11T03:14:41.488Z,1665458081.488 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T03:34:45.536Z,1665459285.536 [ballast_and_trim:RunBallastAndTrim:BallastAndTrim:GoToTargetDepth:ApproachDepth:C.Pitch:A_Timeout:A](IMPORTANT): Timed out trying to reach the target depth. Stopping mission at current depth of 34.686375 m 
2022-10-11T03:34:45.969Z,1665459285.969 [CommandExec](IMPORTANT): got command stop
2022-10-11T03:34:45.969Z,1665459285.969 [CommandExec](IMPORTANT): Scheduling is paused
2022-10-11T03:34:46.762Z,1665459286.762 [MissionManager](IMPORTANT): Started mission Default
2022-10-11T03:41:14.598Z,1665459674.598 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T03:41:56.380Z,1665459716.380 [AHRS_M2](FAULT): Failed to acquire valid data within timeout.
2022-10-11T04:15:47.156Z,1665461747.156 [AHRS_M2](FAULT): Failed to acquire valid data within timeout.
2022-10-11T04:24:56.812Z,1665462296.812 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 50.161202 min 
2022-10-11T04:24:58.381Z,1665462298.381 [NAL9602](FAULT): GPS failed to acquire within timeout.
2022-10-11T04:30:35.766Z,1665462635.766 [DataOverHttps](IMPORTANT): SBD MOMSN=17240713, MTMSN=20221011T043033
2022-10-11T04:30:43.536Z,1665462643.536 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T04:31:08.539Z,1665462668.539 [CommandExec](IMPORTANT): got command schedule clear
2022-10-11T04:31:08.539Z,1665462668.539 [CommandExec](IMPORTANT): Cleared 0 scheduled commands.
2022-10-11T04:31:08.540Z,1665462668.540 [CommandExec](IMPORTANT): got command schedule resume
2022-10-11T04:31:08.540Z,1665462668.540 [CommandExec](IMPORTANT): Scheduling is resumed
2022-10-11T04:32:54.017Z,1665462774.017 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T043250
2022-10-11T04:33:01.632Z,1665462781.632 [CommandExec](IMPORTANT): got command schedule asap "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 4 count;set LocationDepthSampling.MissionTimeout 10 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s" 3x3ap 1 3.000000
2022-10-11T04:33:01.633Z,1665462781.633 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=3x3ap
2022-10-11T04:33:01.633Z,1665462781.633 [CommandExec](IMPORTANT): Scheduled #3 (#1 of 3 with id='3x3ap'): "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 4 count;set LocationDepthSampling.MissionTimeout 10 h;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s", ASAP
2022-10-11T04:33:02.613Z,1665462782.613 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T043259
2022-10-11T04:33:10.216Z,1665462790.216 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m;set LocationDepthSampling.Dep2 5 m" 3x3ap 2 3.000000
2022-10-11T04:33:10.216Z,1665462790.216 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=3x3ap
2022-10-11T04:33:10.217Z,1665462790.217 [CommandExec](IMPORTANT): Scheduled #4 (#2 of 3 with id='3x3ap'): "set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m;set LocationDepthSampling.Dep2 5 m", ASAP
2022-10-11T04:33:11.157Z,1665462791.157 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T043308
2022-10-11T04:33:18.746Z,1665462798.746 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.WaitDepUndulationCommon 5 min;set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run" 3x3ap 3 3.000000
2022-10-11T04:33:18.748Z,1665462798.748 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=3x3ap
2022-10-11T04:33:18.749Z,1665462798.749 [CommandExec](IMPORTANT): Scheduled #5 (#3 of 3 with id='3x3ap'): "set LocationDepthSampling.WaitDepUndulationCommon 5 min;set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run", ASAP
2022-10-11T04:33:18.893Z,1665462798.893 [CommandExec](IMPORTANT): got command load ./Missions/Science/location_depth_sampling.xml
2022-10-11T04:33:19.717Z,1665462799.717 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T043316
2022-10-11T04:33:38.776Z,1665462818.776 [CommandExec](IMPORTANT): Loaded ./Missions/Science/location_depth_sampling.xml
2022-10-11T04:34:06.044Z,1665462846.044 [CommandExec](IMPORTANT): got command set LocationDepthSampling.NumSamplers 4 count
2022-10-11T04:34:06.045Z,1665462846.045 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MissionTimeout 10 hour
2022-10-11T04:34:06.046Z,1665462846.046 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedTransit 0.8 meter_per_second
2022-10-11T04:34:06.046Z,1665462846.046 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedSampling 0.8 meter_per_second
2022-10-11T04:34:06.144Z,1665462846.144 [CommandExec](IMPORTANT): got command schedule "load Transport/keepstation.tl;set keepstation.MissionTimeout 18 h;set keepstation.NeedCommsTime 60 min;set keepstation.Latitude 36.8 degree;set keepstation.Longitude -122.097 degree;set keepstation.Speed 0.8 m/s;run"
2022-10-11T04:34:06.144Z,1665462846.144 [CommandExec](IMPORTANT): Scheduled #6: "load Transport/keepstation.tl;set keepstation.MissionTimeout 18 h;set keepstation.NeedCommsTime 60 min;set keepstation.Latitude 36.8 degree;set keepstation.Longitude -122.097 degree;set keepstation.Speed 0.8 m/s;run", AFTER MISSION
2022-10-11T04:34:20.773Z,1665462860.773 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MaxDepth 180 meter
2022-10-11T04:34:20.774Z,1665462860.774 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LatSampling 36.80 degree
2022-10-11T04:34:20.774Z,1665462860.774 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LonSampling -122.097 degree
2022-10-11T04:34:20.776Z,1665462860.776 [CommandExec](IMPORTANT): got command set LocationDepthSampling.Dep1 150 meter
2022-10-11T04:34:20.776Z,1665462860.776 [CommandExec](IMPORTANT): got command set LocationDepthSampling.Dep2 5 meter
2022-10-11T04:34:33.312Z,1665462873.312 [CommandExec](IMPORTANT): got command set LocationDepthSampling.WaitDepUndulationCommon 5 minute
2022-10-11T04:34:33.313Z,1665462873.313 [CommandExec](IMPORTANT): got command set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 meter_per_second
2022-10-11T04:34:33.313Z,1665462873.313 [CommandExec](IMPORTANT): got command run
2022-10-11T04:34:33.319Z,1665462873.319 [CommandExec](IMPORTANT): Running 
2022-10-11T04:34:33.569Z,1665462873.569 [MissionManager](IMPORTANT): Started mission LocationDepthSampling
2022-10-11T04:34:33.708Z,1665462873.708 [LocationDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2022-10-11T04:39:39.568Z,1665463179.568 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T04:39:49.386Z,1665463189.386 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-11T04:40:39.387Z,1665463239.387 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-11T04:44:05.947Z,1665463445.947 [Radio_Surface](FAULT): LCB fault:  Invalid Command.
2022-10-11T04:45:41.187Z,1665463541.187 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T05:10:40.929Z,1665465040.929 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-11T05:10:42.926Z,1665465042.926 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 
2022-10-11T05:10:43.326Z,1665465043.326 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 150.000000 m .
2022-10-11T05:10:43.764Z,1665465043.764 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-11T05:22:27.910Z,1665465747.910 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.054871 m , settling for 5.000000 min 
2022-10-11T05:27:30.488Z,1665466050.488 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:'
2022-10-11T05:27:40.635Z,1665466060.635 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-11T05:27:40.950Z,1665466060.950 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-11T05:28:41.372Z,1665466121.372 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-11T05:28:42.286Z,1665466122.286 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -6
2022-10-11T05:28:42.287Z,1665466122.287 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-11T05:28:42.657Z,1665466122.657 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.336945 m in 10.075220 degC water with 0.090480 ug/l chlorophyll fluorescence.
2022-10-11T05:28:43.395Z,1665466123.395 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-11T05:28:43.797Z,1665466123.797 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-11T05:28:45.413Z,1665466125.413 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-11T05:29:19.753Z,1665466159.753 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@53,archive_bac)
2022-10-11T05:29:22.581Z,1665466162.581 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0)
2022-10-11T05:29:22.582Z,1665466162.582 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-11T05:29:22.985Z,1665466162.985 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-11T05:31:41.963Z,1665466301.963 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0)
2022-10-11T05:31:42.365Z,1665466302.365 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-11T05:58:20.294Z,1665467900.294 [LocationDepthSampling:NeedComms:B.GoToSurface](CRITICAL): Surfacing timeout. Returning calc satisfied.
2022-10-11T05:58:20.333Z,1665467900.333 [CommandExec](FAULT): Scheduling is paused for 1 commands
2022-10-11T05:58:21.109Z,1665467901.109 [ESPComponent](IMPORTANT): [sample #1] Stop sampling requested.
2022-10-11T05:58:21.168Z,1665467901.168 [MissionManager](IMPORTANT): Started mission Default
2022-10-11T05:58:21.501Z,1665467901.501 [ESPComponent](IMPORTANT): Stop sampling requested. Issuing Cmd.pauseFiltering
2022-10-11T05:58:27.157Z,1665467907.157 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@617.4)
2022-10-11T05:58:27.562Z,1665467907.562 [ESPComponent](IMPORTANT): In S_PAUSED received result: 617.4

2022-10-11T05:58:27.965Z,1665467907.965 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@617.4)
2022-10-11T05:58:29.985Z,1665467909.985 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-11T05:58:30.389Z,1665467910.389 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@53,archive_bac)
2022-10-11T06:09:01.409Z,1665468541.409 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T06:10:33.781Z,1665468633.781 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@53,archive_bac)
2022-10-11T06:10:43.478Z,1665468643.478 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-11T06:10:43.896Z,1665468643.896 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED
2022-10-11T06:15:02.514Z,1665468902.514 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 16.682349 min 
2022-10-11T06:15:04.126Z,1665468904.126 [NAL9602](FAULT): GPS failed to acquire within timeout.
2022-10-11T06:15:43.680Z,1665468943.680 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally.
2022-10-11T06:20:48.972Z,1665469248.972 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T06:26:01.292Z,1665469561.292 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 27.661599 min 
2022-10-11T06:26:02.865Z,1665469562.865 [NAL9602](FAULT): GPS failed to acquire within timeout.
2022-10-11T06:31:47.371Z,1665469907.371 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T06:32:34.865Z,1665469954.865 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T063231
2022-10-11T06:32:45.523Z,1665469965.523 [CommandExec](IMPORTANT): got command schedule asap "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 4 count;set LocationDepthSampling.MissionTimeout 10 h;set LocationDepthSampling.NeedCommsTimeInTransit 95 min" 3x8u0 1 4.000000
2022-10-11T06:32:45.524Z,1665469965.524 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=3x8u0
2022-10-11T06:32:45.525Z,1665469965.525 [CommandExec](IMPORTANT): Scheduled #7 (#1 of 4 with id='3x8u0'): "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 4 count;set LocationDepthSampling.MissionTimeout 10 h;set LocationDepthSampling.NeedCommsTimeInTransit 95 min", ASAP
2022-10-11T06:32:46.462Z,1665469966.462 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T063243
2022-10-11T06:32:57.603Z,1665469977.603 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s;set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree" 3x8u0 2 4.000000
2022-10-11T06:32:57.604Z,1665469977.604 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=3x8u0
2022-10-11T06:32:57.605Z,1665469977.605 [CommandExec](IMPORTANT): Scheduled #8 (#2 of 4 with id='3x8u0'): "set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s;set LocationDepthSampling.MaxDepth 180 m;set LocationDepthSampling.LatSampling 36.80 degree", ASAP
2022-10-11T06:32:58.585Z,1665469978.585 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T063255
2022-10-11T06:33:11.999Z,1665469991.999 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m;set LocationDepthSampling.Dep2 5 m;set LocationDepthSampling.WaitDepUndulationCommon 5 min" 3x8u0 3 4.000000
2022-10-11T06:33:11.999Z,1665469991.999 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=3x8u0
2022-10-11T06:33:12.001Z,1665469992.001 [CommandExec](IMPORTANT): Scheduled #9 (#3 of 4 with id='3x8u0'): "set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m;set LocationDepthSampling.Dep2 5 m;set LocationDepthSampling.WaitDepUndulationCommon 5 min", ASAP
2022-10-11T06:33:12.956Z,1665469992.956 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T063309
2022-10-11T06:33:20.451Z,1665470000.451 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run" 3x8u0 4 4.000000
2022-10-11T06:33:20.451Z,1665470000.451 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=3x8u0
2022-10-11T06:33:20.452Z,1665470000.452 [CommandExec](IMPORTANT): Scheduled #10 (#4 of 4 with id='3x8u0'): "set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run", ASAP
2022-10-11T06:33:20.593Z,1665470000.593 [CommandExec](IMPORTANT): got command load ./Missions/Science/location_depth_sampling.xml
2022-10-11T06:33:27.280Z,1665470007.280 [CommandExec](IMPORTANT): Loaded ./Missions/Science/location_depth_sampling.xml
2022-10-11T06:33:42.758Z,1665470022.758 [CommandExec](IMPORTANT): got command set LocationDepthSampling.NumSamplers 4 count
2022-10-11T06:33:42.759Z,1665470022.759 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MissionTimeout 10 hour
2022-10-11T06:33:42.761Z,1665470022.761 [CommandExec](IMPORTANT): got command set LocationDepthSampling.NeedCommsTimeInTransit 95 minute
2022-10-11T06:34:04.400Z,1665470044.400 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedTransit 0.8 meter_per_second
2022-10-11T06:34:04.401Z,1665470044.401 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedSampling 0.8 meter_per_second
2022-10-11T06:34:04.401Z,1665470044.401 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MaxDepth 180 meter
2022-10-11T06:34:04.402Z,1665470044.402 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LatSampling 36.80 degree
2022-10-11T06:34:17.700Z,1665470057.700 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LonSampling -122.097 degree
2022-10-11T06:34:17.701Z,1665470057.701 [CommandExec](IMPORTANT): got command set LocationDepthSampling.Dep1 150 meter
2022-10-11T06:34:17.701Z,1665470057.701 [CommandExec](IMPORTANT): got command set LocationDepthSampling.Dep2 5 meter
2022-10-11T06:34:17.702Z,1665470057.702 [CommandExec](IMPORTANT): got command set LocationDepthSampling.WaitDepUndulationCommon 5 minute
2022-10-11T06:34:25.860Z,1665470065.860 [CommandExec](IMPORTANT): got command set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 meter_per_second
2022-10-11T06:34:25.860Z,1665470065.860 [CommandExec](IMPORTANT): got command run
2022-10-11T06:34:25.866Z,1665470065.866 [CommandExec](IMPORTANT): Running 
2022-10-11T06:34:25.936Z,1665470065.936 [MissionManager](IMPORTANT): Started mission LocationDepthSampling
2022-10-11T06:34:25.989Z,1665470065.989 [LocationDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2022-10-11T06:39:31.699Z,1665470371.699 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T06:39:38.489Z,1665470378.489 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-11T06:40:42.293Z,1665470442.293 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-11T06:40:48.535Z,1665470448.535 [BuoyancyServo](FAULT): Buoyancy getPosition uart error.serial timeout
2022-10-11T06:45:43.652Z,1665470743.652 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T06:53:16.186Z,1665471196.186 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-11T06:53:18.171Z,1665471198.171 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 
2022-10-11T06:53:18.588Z,1665471198.588 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 150.000000 m .
2022-10-11T06:53:18.998Z,1665471198.998 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-11T07:04:41.753Z,1665471881.753 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.049927 m , settling for 5.000000 min 
2022-10-11T07:09:54.545Z,1665472194.545 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-11T07:09:54.887Z,1665472194.887 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-11T07:10:54.069Z,1665472254.069 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-11T07:10:54.897Z,1665472254.897 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -6
2022-10-11T07:10:54.898Z,1665472254.898 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-11T07:10:55.332Z,1665472255.332 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 149.462494 m in 10.076136 degC water with 0.091520 ug/l chlorophyll fluorescence.
2022-10-11T07:10:55.684Z,1665472255.684 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-11T07:10:56.088Z,1665472256.088 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-11T07:10:58.109Z,1665472258.109 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-11T07:11:32.449Z,1665472292.449 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@52,archive_bac)
2022-10-11T07:11:35.276Z,1665472295.276 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0)
2022-10-11T07:11:35.279Z,1665472295.279 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-11T07:11:35.681Z,1665472295.681 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-11T07:13:55.060Z,1665472435.060 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0)
2022-10-11T07:13:55.465Z,1665472435.465 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-11T07:59:42.734Z,1665475182.734 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@1000.0)
2022-10-11T07:59:43.137Z,1665475183.137 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-11T07:59:45.158Z,1665475185.158 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-11T07:59:45.561Z,1665475185.561 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@52,archive_bac)
2022-10-11T08:11:49.144Z,1665475909.144 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@52,archive_bac)
2022-10-11T08:11:58.842Z,1665475918.842 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-11T08:11:59.245Z,1665475919.245 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED
2022-10-11T08:16:59.016Z,1665476219.016 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally.
2022-10-11T08:16:59.878Z,1665476219.878 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed.
2022-10-11T08:17:06.339Z,1665476226.339 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min .
2022-10-11T08:34:36.732Z,1665477276.732 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T08:54:16.926Z,1665478456.926 [CBIT](IMPORTANT): Beginning ground fault scan
2022-10-11T08:54:29.914Z,1665478469.914 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.004221
CHAN A1 (24V): 0.005648
CHAN A2 (12V): -0.004454
CHAN A3 (5V): 0.007419
CHAN B0 (3.3V): -0.000755
CHAN B1 (3.15aV): -0.000476
CHAN B2 (3.15bV): -0.001299
CHAN B3 (GND): -0.000035
OPEN: 0.012628
Full Scale: +/- 1 mA
2022-10-11T09:05:03.207Z,1665479103.207 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-11T09:11:49.600Z,1665479509.600 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-11T09:11:51.604Z,1665479511.604 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 
2022-10-11T09:11:52.025Z,1665479512.025 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Target depth = 5.000000 m .
2022-10-11T09:11:52.422Z,1665479512.422 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-11T09:16:08.180Z,1665479768.180 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.959906 m , settling for 5.000000 min 
2022-10-11T09:21:20.915Z,1665480080.915 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-11T09:21:21.112Z,1665480081.112 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-11T09:22:21.484Z,1665480141.484 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-11T09:22:22.296Z,1665480142.296 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6
2022-10-11T09:22:22.296Z,1665480142.296 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-11T09:22:22.761Z,1665480142.761 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.213900 m in 13.337366 degC water with 3.212560 ug/l chlorophyll fluorescence.
2022-10-11T09:22:23.100Z,1665480143.100 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-11T09:22:23.504Z,1665480143.504 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-11T09:22:25.524Z,1665480145.524 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-11T09:22:59.866Z,1665480179.866 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@51,archive_bac)
2022-10-11T09:23:02.698Z,1665480182.698 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0)
2022-10-11T09:23:02.698Z,1665480182.698 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-11T09:23:03.101Z,1665480183.101 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-11T09:25:20.865Z,1665480320.865 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0)
2022-10-11T09:25:21.268Z,1665480321.268 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-11T10:25:01.585Z,1665483901.585 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1000.0)
2022-10-11T10:25:01.989Z,1665483901.989 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-11T10:25:04.017Z,1665483904.017 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-11T10:25:04.413Z,1665483904.413 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@51,archive_bac)
2022-10-11T10:37:08.009Z,1665484628.009 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@51,archive_bac)
2022-10-11T10:37:18.127Z,1665484638.127 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-11T10:37:18.520Z,1665484638.520 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED
2022-10-11T10:42:18.296Z,1665484938.296 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally.
2022-10-11T10:42:19.146Z,1665484939.146 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed.
2022-10-11T10:42:25.609Z,1665484945.609 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-11T10:47:56.472Z,1665485276.472 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T10:50:27.976Z,1665485427.976 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-11T10:55:28.572Z,1665485728.572 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T11:09:54.011Z,1665486594.011 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-11T11:09:56.058Z,1665486596.058 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 
2022-10-11T11:09:56.449Z,1665486596.449 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 3.000000 count . Target depth = 150.000000 m .
2022-10-11T11:09:56.860Z,1665486596.860 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-11T11:21:42.243Z,1665487302.243 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.043671 m , settling for 5.000000 min 
2022-10-11T11:26:54.999Z,1665487614.999 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-11T11:26:55.126Z,1665487615.126 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-11T11:27:55.505Z,1665487675.505 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-11T11:27:56.317Z,1665487676.317 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6
2022-10-11T11:27:56.321Z,1665487676.321 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-11T11:27:56.758Z,1665487676.758 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.195694 m in 10.105646 degC water with 0.101920 ug/l chlorophyll fluorescence.
2022-10-11T11:27:57.121Z,1665487677.121 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-11T11:27:57.524Z,1665487677.524 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-11T11:27:59.545Z,1665487679.545 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-11T11:28:33.885Z,1665487713.885 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@50,archive_bac)
2022-10-11T11:28:36.713Z,1665487716.713 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0)
2022-10-11T11:28:36.714Z,1665487716.714 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-11T11:28:37.117Z,1665487717.117 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-11T11:30:56.097Z,1665487856.097 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0)
2022-10-11T11:30:56.504Z,1665487856.504 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-11T12:15:13.682Z,1665490513.682 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@1000.0)
2022-10-11T12:15:14.081Z,1665490514.081 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-11T12:15:16.101Z,1665490516.101 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-11T12:15:16.505Z,1665490516.505 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@50,archive_bac)
2022-10-11T12:27:20.073Z,1665491240.073 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@50,archive_bac)
2022-10-11T12:27:29.770Z,1665491249.770 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-11T12:27:30.178Z,1665491250.178 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED
2022-10-11T12:32:29.957Z,1665491549.957 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally.
2022-10-11T12:32:30.767Z,1665491550.767 [ESPComponent](FAULT): LCB 1 fault: LCB Watchdog Reset.
2022-10-11T12:32:30.767Z,1665491550.767 [ESPComponent](FAULT): LCB 2 fault:  Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated.
2022-10-11T12:32:30.872Z,1665491550.872 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 3.000000 count completed.
2022-10-11T12:32:37.270Z,1665491557.270 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min .
2022-10-11T12:42:06.972Z,1665492126.972 [LocationDepthSampling:NeedComms:B.GoToSurface](CRITICAL): Surfacing timeout. Returning calc satisfied.
2022-10-11T12:42:07.994Z,1665492127.994 [MissionManager](IMPORTANT): Started mission Default
2022-10-11T12:48:25.672Z,1665492505.672 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T12:54:18.726Z,1665492858.726 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.172029 min 
2022-10-11T12:54:20.311Z,1665492860.311 [NAL9602](FAULT): GPS failed to acquire within timeout.
2022-10-11T13:00:05.979Z,1665493205.979 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T13:05:13.153Z,1665493513.153 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 23.079537 min 
2022-10-11T13:05:14.771Z,1665493514.771 [NAL9602](FAULT): GPS failed to acquire within timeout.
2022-10-11T13:05:14.822Z,1665493514.822 [CBIT](CRITICAL): Data Fault in component: NAL9602
2022-10-11T13:10:56.219Z,1665493856.219 [DataOverHttps](IMPORTANT): SBD MOMSN=17241463, MTMSN=20221011T131053
2022-10-11T13:11:19.783Z,1665493879.783 [CommandExec](IMPORTANT): got command schedule asap "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 4 count;set LocationDepthSampling.StartIndex 2 count;set LocationDepthSampling.MissionTimeout 10 h" 3xr9p 1 4.000000
2022-10-11T13:11:19.784Z,1665493879.784 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=3xr9p
2022-10-11T13:11:19.785Z,1665493879.785 [CommandExec](IMPORTANT): Scheduled #11 (#1 of 4 with id='3xr9p'): "load Science/location_depth_sampling.xml;set LocationDepthSampling.NumSamplers 4 count;set LocationDepthSampling.StartIndex 2 count;set LocationDepthSampling.MissionTimeout 10 h", ASAP
2022-10-11T13:11:20.765Z,1665493880.765 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T131117
2022-10-11T13:11:28.428Z,1665493888.428 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.NeedCommsTimeInTransit 95 min;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s;set LocationDepthSampling.MaxDepth 180 m" 3xr9p 2 4.000000
2022-10-11T13:11:28.428Z,1665493888.428 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=3xr9p
2022-10-11T13:11:28.429Z,1665493888.429 [CommandExec](IMPORTANT): Scheduled #12 (#2 of 4 with id='3xr9p'): "set LocationDepthSampling.NeedCommsTimeInTransit 95 min;set LocationDepthSampling.SpeedTransit 0.8 m/s;set LocationDepthSampling.SpeedSampling 0.8 m/s;set LocationDepthSampling.MaxDepth 180 m", ASAP
2022-10-11T13:11:29.406Z,1665493889.406 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T131126
2022-10-11T13:11:43.147Z,1665493903.147 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m;set LocationDepthSampling.Dep2 5 m" 3xr9p 3 4.000000
2022-10-11T13:11:43.148Z,1665493903.148 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=3xr9p
2022-10-11T13:11:43.148Z,1665493903.148 [CommandExec](IMPORTANT): Scheduled #13 (#3 of 4 with id='3xr9p'): "set LocationDepthSampling.LatSampling 36.80 degree;set LocationDepthSampling.LonSampling -122.097 degree;set LocationDepthSampling.Dep1 150 m;set LocationDepthSampling.Dep2 5 m", ASAP
2022-10-11T13:11:44.077Z,1665493904.077 [DataOverHttps](IMPORTANT): SBD MTMSN=20221011T131140
2022-10-11T13:11:52.207Z,1665493912.207 [CommandExec](IMPORTANT): got command schedule asap "set LocationDepthSampling.WaitDepUndulationCommon 5 min;set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run" 3xr9p 4 4.000000
2022-10-11T13:11:52.207Z,1665493912.207 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=3xr9p
2022-10-11T13:11:52.208Z,1665493912.208 [CommandExec](IMPORTANT): Scheduled #14 (#4 of 4 with id='3xr9p'): "set LocationDepthSampling.WaitDepUndulationCommon 5 min;set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 m/s;run", ASAP
2022-10-11T13:11:52.354Z,1665493912.354 [CommandExec](IMPORTANT): got command load ./Missions/Science/location_depth_sampling.xml
2022-10-11T13:11:58.838Z,1665493918.838 [CommandExec](IMPORTANT): Loaded ./Missions/Science/location_depth_sampling.xml
2022-10-11T13:12:08.913Z,1665493928.913 [CommandExec](IMPORTANT): got command set LocationDepthSampling.NumSamplers 4 count
2022-10-11T13:12:08.914Z,1665493928.914 [CommandExec](IMPORTANT): got command set LocationDepthSampling.StartIndex 2 count
2022-10-11T13:12:08.915Z,1665493928.915 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MissionTimeout 10 hour
2022-10-11T13:12:27.074Z,1665493947.074 [CommandExec](IMPORTANT): got command set LocationDepthSampling.NeedCommsTimeInTransit 95 minute
2022-10-11T13:12:27.075Z,1665493947.075 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedTransit 0.8 meter_per_second
2022-10-11T13:12:27.088Z,1665493947.088 [CommandExec](IMPORTANT): got command set LocationDepthSampling.SpeedSampling 0.8 meter_per_second
2022-10-11T13:12:27.088Z,1665493947.088 [CommandExec](IMPORTANT): got command set LocationDepthSampling.MaxDepth 180 meter
2022-10-11T13:12:29.442Z,1665493949.442 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T13:12:43.216Z,1665493963.216 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LatSampling 36.80 degree
2022-10-11T13:12:43.217Z,1665493963.217 [CommandExec](IMPORTANT): got command set LocationDepthSampling.LonSampling -122.097 degree
2022-10-11T13:12:43.218Z,1665493963.218 [CommandExec](IMPORTANT): got command set LocationDepthSampling.Dep1 150 meter
2022-10-11T13:12:43.219Z,1665493963.219 [CommandExec](IMPORTANT): got command set LocationDepthSampling.Dep2 5 meter
2022-10-11T13:13:00.177Z,1665493980.177 [CommandExec](IMPORTANT): got command set LocationDepthSampling.WaitDepUndulationCommon 5 minute
2022-10-11T13:13:00.178Z,1665493980.178 [CommandExec](IMPORTANT): got command set LocationDepthSampling:NeedComms.SurfaceSpeed 0.8 meter_per_second
2022-10-11T13:13:00.178Z,1665493980.178 [CommandExec](IMPORTANT): got command run
2022-10-11T13:13:00.180Z,1665493980.180 [CommandExec](IMPORTANT): Running 
2022-10-11T13:13:00.317Z,1665493980.317 [MissionManager](IMPORTANT): Started mission LocationDepthSampling
2022-10-11T13:13:00.342Z,1665493980.342 [LocationDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2022-10-11T13:18:06.760Z,1665494286.760 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T13:18:16.213Z,1665494296.213 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min .
2022-10-11T13:19:13.430Z,1665494353.430 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-11T13:24:14.650Z,1665494654.650 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T13:30:46.992Z,1665495046.992 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-11T13:30:48.985Z,1665495048.985 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 
2022-10-11T13:30:49.393Z,1665495049.393 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 5.000000 m .
2022-10-11T13:30:49.798Z,1665495049.798 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-11T13:34:48.141Z,1665495288.141 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.974715 m , settling for 5.000000 min 
2022-10-11T13:40:00.939Z,1665495600.939 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-11T13:40:01.236Z,1665495601.236 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-11T13:41:01.667Z,1665495661.667 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-11T13:41:02.476Z,1665495662.476 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6
2022-10-11T13:41:02.477Z,1665495662.477 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-11T13:41:02.941Z,1665495662.941 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 4.651612 m in 14.434412 degC water with 2.835040 ug/l chlorophyll fluorescence.
2022-10-11T13:41:03.281Z,1665495663.281 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-11T13:41:03.686Z,1665495663.686 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-11T13:41:05.709Z,1665495665.709 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-11T13:41:40.045Z,1665495700.045 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@49,archive_bac)
2022-10-11T13:41:42.874Z,1665495702.874 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0)
2022-10-11T13:41:42.876Z,1665495702.876 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-11T13:41:43.276Z,1665495703.276 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-11T13:44:01.060Z,1665495841.060 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0)
2022-10-11T13:44:01.465Z,1665495841.465 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-11T14:45:48.642Z,1665499548.642 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0)
2022-10-11T14:45:49.046Z,1665499549.046 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-11T14:45:51.066Z,1665499551.066 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-11T14:45:51.469Z,1665499551.469 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@49,archive_bac)
2022-10-11T14:54:29.939Z,1665500069.939 [CBIT](IMPORTANT): Beginning ground fault scan
2022-10-11T14:54:40.854Z,1665500080.854 [CBIT](IMPORTANT): Ground fault detected
mA:
CHAN A0 (Batt): -0.003775
CHAN A1 (24V): 0.005253
CHAN A2 (12V): -0.004558
CHAN A3 (5V): -0.001462
CHAN B0 (3.3V): -0.001425
CHAN B1 (3.15aV): -0.119369
CHAN B2 (3.15bV): -0.000235
CHAN B3 (GND): -0.000748
OPEN: 0.012165
Full Scale: +/- 1 mA
2022-10-11T14:57:55.038Z,1665500275.038 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@49,archive_bac)
2022-10-11T14:58:04.733Z,1665500284.733 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-11T14:58:05.135Z,1665500285.135 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED
2022-10-11T15:03:04.932Z,1665500584.932 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally.
2022-10-11T15:03:05.801Z,1665500585.801 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed.
2022-10-11T15:03:12.272Z,1665500592.272 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-11T15:08:48.885Z,1665500928.885 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T15:10:13.755Z,1665501013.755 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-11T15:15:14.748Z,1665501314.748 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T15:23:02.644Z,1665501782.644 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-11T15:23:04.657Z,1665501784.657 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 
2022-10-11T15:23:05.037Z,1665501785.037 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Target depth = 150.000000 m .
2022-10-11T15:23:05.446Z,1665501785.446 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-11T15:34:46.422Z,1665502486.422 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.060257 m , settling for 5.000000 min 
2022-10-11T15:39:59.168Z,1665502799.168 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-11T15:39:59.315Z,1665502799.315 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-11T15:40:59.709Z,1665502859.709 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-11T15:41:00.537Z,1665502860.537 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -6
2022-10-11T15:41:00.538Z,1665502860.538 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-11T15:41:01.042Z,1665502861.042 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.120804 m in 10.293451 degC water with 0.119600 ug/l chlorophyll fluorescence.
2022-10-11T15:41:01.325Z,1665502861.325 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-11T15:41:01.728Z,1665502861.728 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-11T15:41:03.749Z,1665502863.749 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-11T15:41:38.089Z,1665502898.089 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@48,archive_bac)
2022-10-11T15:41:40.512Z,1665502900.512 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0)
2022-10-11T15:41:40.513Z,1665502900.513 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-11T15:41:40.921Z,1665502900.921 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-11T15:43:59.893Z,1665503039.893 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0)
2022-10-11T15:44:00.298Z,1665503040.298 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-11T16:31:09.948Z,1665505869.948 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1000.0)
2022-10-11T16:31:10.353Z,1665505870.353 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-11T16:31:12.377Z,1665505872.377 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-11T16:31:12.778Z,1665505872.778 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@48,archive_bac)
2022-10-11T16:43:16.377Z,1665506596.377 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@48,archive_bac)
2022-10-11T16:43:26.074Z,1665506606.074 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-11T16:43:26.477Z,1665506606.477 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED
2022-10-11T16:48:26.281Z,1665506906.281 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally.
2022-10-11T16:48:27.178Z,1665506907.178 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed.
2022-10-11T16:48:33.608Z,1665506913.608 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 5.000000 m , 5.000000 min .
2022-10-11T17:05:27.767Z,1665507927.767 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T17:07:03.197Z,1665508023.197 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-11T17:12:03.695Z,1665508323.695 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T17:16:08.965Z,1665508568.965 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-11T17:16:10.970Z,1665508570.970 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 5.000000 m 
2022-10-11T17:16:11.397Z,1665508571.397 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 3.000000 count . Target depth = 5.000000 m .
2022-10-11T17:16:11.785Z,1665508571.785 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-11T17:20:07.316Z,1665508807.316 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 4.976510 m , settling for 5.000000 min 
2022-10-11T17:25:20.080Z,1665509120.080 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-11T17:25:20.240Z,1665509120.240 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-11T17:26:20.649Z,1665509180.649 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-11T17:26:21.461Z,1665509181.461 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -6
2022-10-11T17:26:21.461Z,1665509181.461 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-11T17:26:21.907Z,1665509181.907 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 5.068504 m in 13.224817 degC water with 3.195920 ug/l chlorophyll fluorescence.
2022-10-11T17:26:22.265Z,1665509182.265 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-11T17:26:22.671Z,1665509182.671 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-11T17:26:24.699Z,1665509184.699 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-11T17:26:59.029Z,1665509219.029 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@47,archive_bac)
2022-10-11T17:27:01.856Z,1665509221.856 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0)
2022-10-11T17:27:01.857Z,1665509221.857 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-11T17:27:02.266Z,1665509222.266 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-11T17:29:20.029Z,1665509360.029 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0)
2022-10-11T17:29:20.433Z,1665509360.433 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-11T18:21:52.893Z,1665512512.893 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@1000.0)
2022-10-11T18:21:53.297Z,1665512513.297 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-11T18:21:55.318Z,1665512515.318 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-11T18:21:55.721Z,1665512515.721 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@47,archive_bac)
2022-10-11T18:33:59.298Z,1665513239.298 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@47,archive_bac)
2022-10-11T18:34:08.993Z,1665513248.993 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-11T18:34:09.397Z,1665513249.397 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED
2022-10-11T18:39:09.174Z,1665513549.174 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally.
2022-10-11T18:39:10.033Z,1665513550.033 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 3.000000 count completed.
2022-10-11T18:39:16.520Z,1665513556.520 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-11T18:44:46.557Z,1665513886.557 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T18:46:08.923Z,1665513968.923 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-11T18:51:10.671Z,1665514270.671 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T19:05:32.862Z,1665515132.862 [LocationDepthSampling:GoToLocationToSample:GoToSamplingLocation:LevelRun:G.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-11T19:05:34.866Z,1665515134.866 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:D](IMPORTANT): DepthSet = 150.000000 m 
2022-10-11T19:05:35.266Z,1665515135.266 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:E](IMPORTANT): Taking sample No. 4.000000 count . Target depth = 150.000000 m .
2022-10-11T19:05:35.700Z,1665515135.700 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2022-10-11T19:17:14.199Z,1665515834.199 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 150.033340 m , settling for 5.000000 min 
2022-10-11T19:22:26.944Z,1665516146.944 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2022-10-11T19:22:27.088Z,1665516147.088 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2022-10-11T19:23:27.938Z,1665516207.938 [ESPComponent](IMPORTANT): ESP has connected as client
2022-10-11T19:23:28.748Z,1665516208.748 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -6
2022-10-11T19:23:28.748Z,1665516208.748 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2022-10-11T19:23:29.208Z,1665516209.208 [LocationDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 150.326614 m in 10.131586 degC water with 0.104000 ug/l chlorophyll fluorescence.
2022-10-11T19:23:29.553Z,1665516209.553 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG
2022-10-11T19:23:29.956Z,1665516209.956 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS
2022-10-11T19:23:31.976Z,1665516211.976 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE
2022-10-11T19:24:06.330Z,1665516246.330 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@46,archive_bac)
2022-10-11T19:24:09.157Z,1665516249.157 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0)
2022-10-11T19:24:09.158Z,1665516249.158 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2022-10-11T19:24:09.562Z,1665516249.562 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2022-10-11T19:26:28.537Z,1665516388.537 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0)
2022-10-11T19:26:28.941Z,1665516388.941 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2022-10-11T19:41:55.288Z,1665517315.288 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout
2022-10-11T19:48:44.351Z,1665517724.351 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout
2022-10-11T20:10:30.726Z,1665519030.726 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@1000.0)
2022-10-11T20:10:31.129Z,1665519031.129 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@1000.0)
2022-10-11T20:10:33.152Z,1665519033.152 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2022-10-11T20:10:33.553Z,1665519033.553 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@46,archive_bac)
2022-10-11T20:22:37.136Z,1665519757.136 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@46,archive_bac)
2022-10-11T20:22:46.834Z,1665519766.834 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2022-10-11T20:22:47.234Z,1665519767.234 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED
2022-10-11T20:27:47.020Z,1665520067.020 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally.
2022-10-11T20:27:47.881Z,1665520067.881 [LocationDepthSampling:GoToLocationToSample:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 4.000000 count completed.
2022-10-11T20:27:53.536Z,1665520073.536 [LocationDepthSampling:SetSampleOptions:G](IMPORTANT): Set CartridgeType (if ESP), DepthDesignated, WaitDepthUndulation to -6.000000 count , 150.000000 m , 5.000000 min .
2022-10-11T20:38:05.603Z,1665520685.603 [LocationDepthSampling:NeedComms:B.GoToSurface](CRITICAL): Surfacing timeout. Returning calc satisfied.
2022-10-11T20:38:06.470Z,1665520686.470 [MissionManager](IMPORTANT): Started mission Default
2022-10-11T20:44:49.369Z,1665521089.369 [MassServo](FAULT): Overload Error
2022-10-11T20:48:49.595Z,1665521329.595 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-11T20:53:45.857Z,1665521625.857 [Power24vConverter](FAULT): LCB fault:  Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated.
2022-10-11T20:53:45.858Z,1665521625.858 [Power24vConverter](FAULT): 24V Power is reporting not powering/powered.
2022-10-11T20:53:46.665Z,1665521626.665 [Power24vConverter](FAULT):  LCB fault:  Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated.
2022-10-11T20:54:41.306Z,1665521681.306 [CBIT](IMPORTANT): Beginning ground fault scan
2022-10-11T20:55:06.753Z,1665521706.753 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.001806
CHAN A1 (24V): 0.011857
CHAN A2 (12V): 0.000075
CHAN A3 (5V): -0.001913
CHAN B0 (3.3V): 0.003910
CHAN B1 (3.15aV): -0.000263
CHAN B2 (3.15bV): -0.000852
CHAN B3 (GND): -0.000407
OPEN: 0.012902
Full Scale: +/- 1 mA
2022-10-11T22:25:09.934Z,1665527109.934 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=47287, MT Status=1, MTMSN=3513
2022-10-11T22:25:10.611Z,1665527110.611 [CommandExec](IMPORTANT): got command load ./Missions/Transport/transit.tl
2022-10-11T22:25:10.612Z,1665527110.612 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Transport/transit.tx
2022-10-11T22:25:11.352Z,1665527111.352 [CommandExec](IMPORTANT): Loaded ./Missions/Transport/transit.tl
2022-10-11T22:25:19.007Z,1665527119.007 [CommandExec](IMPORTANT): got command set transit.MissionTimeout 30 minute
2022-10-11T22:25:19.008Z,1665527119.008 [CommandExec](IMPORTANT): got command set transit.Latitude 36.8 degree
2022-10-11T22:25:19.009Z,1665527119.009 [CommandExec](IMPORTANT): got command set transit.Longitude -122.097 degree
2022-10-11T22:25:19.009Z,1665527119.009 [CommandExec](IMPORTANT): got command run
2022-10-11T22:25:19.011Z,1665527119.011 [CommandExec](IMPORTANT): Running 
2022-10-11T22:25:24.005Z,1665527124.005 [MissionManager](IMPORTANT): Started mission transit
2022-10-11T22:25:27.104Z,1665527127.104 [transit:Transit:Wpt1.Waypoint](IMPORTANT): Navigating to waypoint: 36.799999,-122.097000
2022-10-11T22:25:40.738Z,1665527140.738 [NAL9602](FAULT): GPS failed to acquire within timeout.
2022-10-11T22:26:53.206Z,1665527213.206 [transit:Transit:Wpt1.Waypoint](IMPORTANT): Reached waypoint: 36.799999,-122.097000
2022-10-11T22:30:36.201Z,1665527436.201 [DVL_micro](FAULT): Failed to initialize within timeout.
2022-10-11T22:55:27.211Z,1665528927.211 [MissionManager](IMPORTANT): Started mission Default
2022-10-11T22:59:10.408Z,1665529150.408 [AHRS_M2](FAULT): Failed to acquire valid data within timeout.
2022-10-11T23:22:16.273Z,1665530536.273 [NAL9602](FAULT): LCB fault:  Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated.
2022-10-11T23:55:08.501Z,1665532508.501 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 59.681042 min 
2022-10-12T00:00:12.631Z,1665532812.631 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T00:01:22.567Z,1665532882.567 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 65.915629 min 
2022-10-12T00:06:26.634Z,1665533186.634 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2022-10-12T00:07:28.476Z,1665533248.476 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 72.014738 min 
2022-10-12T00:11:21.761Z,1665533481.761 [DataOverHttps](IMPORTANT): SBD MTMSN=20221012T001118
2022-10-12T00:11:29.356Z,1665533489.356 [CommandExec](IMPORTANT): got command restart system