2023-04-19T17:35:11.857Z,1681925711.857 [Supervisor](IMPORTANT): Reading configuration overrides from Data/persisted.cfg
2023-04-19T17:35:15.717Z,1681925715.717 [StratificationFrontDetector](IMPORTANT): thresholdVTHI set to: 0.050012 degC
2023-04-19T17:35:16.263Z,1681925716.263 [SBIT](IMPORTANT): git: 2023-04-14_A
2023-04-19T17:35:16.264Z,1681925716.264 [SBIT](IMPORTANT): Kernel Release: 2.6.27.8
2023-04-19T17:36:27.884Z,1681925787.884 [SBIT](IMPORTANT): Beginning Startup BIT
2023-04-19T17:36:27.888Z,1681925787.888 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-19T17:36:38.809Z,1681925798.809 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002464
CHAN A1 (24V): 0.007576
CHAN A2 (12V): -0.003765
CHAN A3 (5V): -0.003697
CHAN B0 (3.3V): -0.001599
CHAN B1 (3.15aV): -0.001269
CHAN B2 (3.15bV): -0.001259
CHAN B3 (GND): -0.001549
OPEN: 0.012360
Full Scale: +/- 1 mA
2023-04-19T17:37:22.025Z,1681925842.025 [SBIT](IMPORTANT): SBIT PASSED
2023-04-19T17:37:22.026Z,1681925842.026 [SBIT](IMPORTANT): Listing configuration overrides from Data/persisted.cfg
2023-04-19T17:37:22.026Z,1681925842.026 [SBIT](IMPORTANT): CBIT.stopDepth=60 meter;

2023-04-19T17:37:22.026Z,1681925842.026 [SBIT](IMPORTANT): DVL_micro.loadAtStartup=1 bool;

2023-04-19T17:37:22.027Z,1681925842.027 [SBIT](IMPORTANT): Depth_Keller.offset=-0.8 decibar;

2023-04-19T17:37:22.027Z,1681925842.027 [SBIT](IMPORTANT): ESPComponent.sampleTimeout=4 minute;

2023-04-19T17:37:22.027Z,1681925842.027 [SBIT](IMPORTANT): ElevatorServo.offsetAngle=1 degree;

2023-04-19T17:37:22.027Z,1681925842.027 [SBIT](IMPORTANT): Express linearApproximation VerticalHomogeneityIndexCalculator.vertical_salinity_homogeneity_index 0.050000 practical_salinity_unit;

2023-04-19T17:37:22.027Z,1681925842.027 [SBIT](IMPORTANT): Express linearApproximation VerticalHomogeneityIndexCalculator.vertical_temperature_homogeneity_index 0.050000 celsius;

2023-04-19T17:37:22.027Z,1681925842.027 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff470nm;

2023-04-19T17:37:22.027Z,1681925842.027 [SBIT](IMPORTANT): Express none WetLabsBB2FL.BackscatteringCoeff650nm;

2023-04-19T17:37:22.027Z,1681925842.027 [SBIT](IMPORTANT): StratificationFrontDetector.loadAtStartup=1 bool;

2023-04-19T17:37:22.028Z,1681925842.028 [SBIT](IMPORTANT): StratificationFrontDetector.threshold_VTHI=0.05 celsius;

2023-04-19T17:37:22.028Z,1681925842.028 [SBIT](IMPORTANT): StratificationFrontDetector.verbosity=1 count;

2023-04-19T17:37:22.028Z,1681925842.028 [SBIT](IMPORTANT): VerticalControl.buoyancyNeutral=191.501740 cubic_centimeter;

2023-04-19T17:37:22.028Z,1681925842.028 [SBIT](IMPORTANT): VerticalControl.massDefault=12.944775 millimeter;

2023-04-19T17:37:22.028Z,1681925842.028 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.depth3=20 meter;

2023-04-19T17:37:22.028Z,1681925842.028 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.depth4=30 meter;

2023-04-19T17:37:22.029Z,1681925842.029 [SBIT](IMPORTANT): VerticalHomogeneityIndexCalculator.loadAtStartup=1 bool;

2023-04-19T17:37:22.429Z,1681925842.429 [MissionManager](IMPORTANT): Started mission Startup
2023-04-19T17:38:28.654Z,1681925908.654 [MissionManager](IMPORTANT): Started mission Default
2023-04-19T17:44:51.103Z,1681926291.103 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.367189 min 
2023-04-19T17:45:44.126Z,1681926344.126 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3039, MT Status=1, MTMSN=2580
2023-04-19T17:45:44.852Z,1681926344.852 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 12 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h" 3flas 1 4.000000
2023-04-19T17:45:44.853Z,1681926344.853 [CommandExec](IMPORTANT): Scheduling command #1 of 4 with id=3flas
2023-04-19T17:45:44.854Z,1681926344.854 [CommandExec](IMPORTANT): Scheduled #1 (#1 of 4 with id='3flas'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 12 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h", ASAP
2023-04-19T17:46:07.787Z,1681926367.787 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3040, MT Status=1, MTMSN=2581
2023-04-19T17:46:08.588Z,1681926368.588 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m" 3flas 2 4.000000
2023-04-19T17:46:08.589Z,1681926368.589 [CommandExec](IMPORTANT): Scheduling command #2 of 4 with id=3flas
2023-04-19T17:46:08.590Z,1681926368.590 [CommandExec](IMPORTANT): Scheduled #2 (#2 of 4 with id='3flas'): "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m", ASAP
2023-04-19T17:46:27.961Z,1681926387.961 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3041, MT Status=1, MTMSN=2582
2023-04-19T17:46:28.696Z,1681926388.696 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.DeepBoundChl 15 m;set IsothermDepthSampling.TransitYoYoMinDepth 2 m;set IsothermDepthSampling.TransitYoYoMaxDepth 50 m;set IsothermDepthSampling.CartridgeType2 -10 count" 3flas 3 4.000000
2023-04-19T17:46:28.696Z,1681926388.696 [CommandExec](IMPORTANT): Scheduling command #3 of 4 with id=3flas
2023-04-19T17:46:28.697Z,1681926388.697 [CommandExec](IMPORTANT): Scheduled #3 (#3 of 4 with id='3flas'): "set IsothermDepthSampling.DeepBoundChl 15 m;set IsothermDepthSampling.TransitYoYoMinDepth 2 m;set IsothermDepthSampling.TransitYoYoMaxDepth 50 m;set IsothermDepthSampling.CartridgeType2 -10 count", ASAP
2023-04-19T17:46:55.031Z,1681926415.031 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3042, MT Status=1, MTMSN=2583
2023-04-19T17:46:55.675Z,1681926415.675 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 3flas 4 4.000000
2023-04-19T17:46:55.676Z,1681926415.676 [CommandExec](IMPORTANT): Scheduling command #4 of 4 with id=3flas
2023-04-19T17:46:55.677Z,1681926415.677 [CommandExec](IMPORTANT): Scheduled #4 (#4 of 4 with id='3flas'): "set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP
2023-04-19T17:46:55.807Z,1681926415.807 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml
2023-04-19T17:47:19.079Z,1681926439.079 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml
2023-04-19T17:47:38.172Z,1681926458.172 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.NumSamplers 2 count
2023-04-19T17:47:38.173Z,1681926458.173 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 12 hour
2023-04-19T17:47:38.173Z,1681926458.173 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour
2023-04-19T17:48:02.792Z,1681926482.792 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool
2023-04-19T17:48:02.793Z,1681926482.793 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool
2023-04-19T17:48:02.793Z,1681926482.793 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool
2023-04-19T17:48:02.794Z,1681926482.794 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter
2023-04-19T17:48:24.411Z,1681926504.411 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 15 meter
2023-04-19T17:48:24.412Z,1681926504.412 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TransitYoYoMinDepth 2 meter
2023-04-19T17:48:24.413Z,1681926504.413 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.TransitYoYoMaxDepth 50 meter
2023-04-19T17:48:24.413Z,1681926504.413 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType2 -10 count
2023-04-19T17:48:37.158Z,1681926517.158 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 meter
2023-04-19T17:48:37.158Z,1681926517.158 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute
2023-04-19T17:48:37.159Z,1681926517.159 [CommandExec](IMPORTANT): got command run
2023-04-19T17:48:37.188Z,1681926517.188 [CommandExec](IMPORTANT): Running 
2023-04-19T17:48:37.232Z,1681926517.232 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling
2023-04-19T17:48:37.344Z,1681926517.344 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2023-04-19T17:49:45.338Z,1681926585.338 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.795745 arcdeg , longitude = -121.980882 arcdeg . Distance from shore = 16191.760742 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side. MinDistFromPolygonSides = 200.000000 m .
2023-04-19T17:49:47.340Z,1681926587.340 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.177029 m 
2023-04-19T17:50:56.874Z,1681926656.874 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min .
2023-04-19T17:53:46.549Z,1681926826.549 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 18.086897 m 
2023-04-19T17:55:34.053Z,1681926934.053 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 12.653206 m , 9.255999 ug/l , 3.010286 m , 10.805506 degC , 13.956800 ug/l 
2023-04-19T17:55:34.426Z,1681926934.426 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 3.010286 m .
2023-04-19T17:55:34.808Z,1681926934.808 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2023-04-19T17:58:23.332Z,1681927103.332 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.002433 m , settling for 10.000000 min 
2023-04-19T18:08:25.701Z,1681927705.701 [ESPComponent](IMPORTANT): Setting regex for ESP Log summary: 'Selecting Cartridge|Sampled|Error|Underpressure|Overpressure|Fail|Retry|Cmd::|Sampler::|summary:'
2023-04-19T18:08:35.856Z,1681927715.856 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-04-19T18:08:35.996Z,1681927715.996 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-04-19T18:09:29.133Z,1681927769.133 [ESPComponent](IMPORTANT): ESP has connected as client
2023-04-19T18:09:29.946Z,1681927769.946 [ESPComponent](IMPORTANT): [sample #1] Setting cartridge argument to -6
2023-04-19T18:09:29.947Z,1681927769.947 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-04-19T18:09:30.434Z,1681927770.434 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 3.005574 m in 10.960413 degC water with 10.942880 ug/l chlorophyll fluorescence.
2023-04-19T18:09:31.154Z,1681927771.154 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_LOG
2023-04-19T18:09:31.557Z,1681927771.557 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-04-19T18:09:33.173Z,1681927773.173 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_LOADING_CARTRIDGE
2023-04-19T18:10:09.957Z,1681927809.957 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PRIMING (@20,archive_bac)
2023-04-19T18:10:12.786Z,1681927812.786 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PRIMING (@100.0)
2023-04-19T18:10:12.787Z,1681927812.787 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-04-19T18:10:13.193Z,1681927813.193 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-04-19T18:12:27.317Z,1681927947.317 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_FILTERING (@0.0)
2023-04-19T18:12:27.726Z,1681927947.726 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-04-19T18:41:12.184Z,1681929672.184 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-04-19T19:02:53.502Z,1681930973.502 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PAUSED (@801.2)
2023-04-19T19:02:53.905Z,1681930973.905 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_PROCESSING (@801.2)
2023-04-19T19:02:55.927Z,1681930975.927 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-04-19T19:02:56.330Z,1681930976.330 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_WAITING_FOR_PROCESSED (@20,archive_bac)
2023-04-19T19:15:00.802Z,1681931700.802 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPING (@20,archive_bac)
2023-04-19T19:15:11.306Z,1681931711.306 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-04-19T19:15:11.768Z,1681931711.768 [ESPComponent](IMPORTANT): [sample #1] ESP sampling state: S_STOPPED
2023-04-19T19:20:11.537Z,1681932011.537 [ESPComponent](IMPORTANT): [sample #1] ESP sampling sequence completed normally.
2023-04-19T19:20:12.398Z,1681932012.398 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed.
2023-04-19T19:20:28.990Z,1681932028.990 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 5.000000 min .
2023-04-19T19:20:32.629Z,1681932032.629 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Target depth = 3.010286 m .
2023-04-19T19:20:33.028Z,1681932033.028 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 
2023-04-19T19:20:40.295Z,1681932040.295 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 3.005125 m , settling for 5.000000 min 
2023-04-19T19:25:53.040Z,1681932353.040 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-04-19T19:25:53.228Z,1681932353.228 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-04-19T19:26:46.085Z,1681932406.085 [ESPComponent](IMPORTANT): ESP has connected as client
2023-04-19T19:26:46.904Z,1681932406.904 [ESPComponent](IMPORTANT): [sample #2] Setting cartridge argument to -10
2023-04-19T19:26:46.904Z,1681932406.904 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-04-19T19:26:47.372Z,1681932407.372 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 2.886204 m in 11.068995 degC water with 13.037440 ug/l chlorophyll fluorescence.
2023-04-19T19:26:47.694Z,1681932407.694 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_LOG
2023-04-19T19:26:48.106Z,1681932408.106 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-04-19T19:26:50.119Z,1681932410.119 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_LOADING_CARTRIDGE
2023-04-19T19:27:27.694Z,1681932447.694 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PRIMING (@33,daSPR_phyto)
2023-04-19T19:27:30.522Z,1681932450.522 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PRIMING (@100.0)
2023-04-19T19:27:30.523Z,1681932450.523 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-04-19T19:27:30.926Z,1681932450.926 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-04-19T19:30:35.962Z,1681932635.962 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_FILTERING (@0.0)
2023-04-19T19:30:36.366Z,1681932636.366 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-04-19T19:31:41.261Z,1681932701.261 [ThrusterHE](FAULT): getSpeed error serial timeout
2023-04-19T19:31:41.261Z,1681932701.261 [ThrusterHE](FAULT): Thruster uart error: serial timeout
2023-04-19T20:14:28.150Z,1681935268.150 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PAUSED (@672.1)
2023-04-19T20:14:28.553Z,1681935268.553 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_PROCESSING (@672.1)
2023-04-19T20:14:30.575Z,1681935270.575 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing
2023-04-19T20:14:30.978Z,1681935270.978 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_WAITING_FOR_PROCESSED (@31,daSPR_phyto)
2023-04-19T21:10:32.538Z,1681938632.538 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPING (@31,daSPR_phyto)
2023-04-19T21:10:43.850Z,1681938643.850 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-04-19T21:10:44.253Z,1681938644.253 [ESPComponent](IMPORTANT): [sample #2] ESP sampling state: S_STOPPED
2023-04-19T21:15:44.028Z,1681938944.028 [ESPComponent](IMPORTANT): [sample #2] ESP sampling sequence completed normally.
2023-04-19T21:15:44.970Z,1681938944.970 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed.
2023-04-19T21:15:48.118Z,1681938948.118 [MissionManager](IMPORTANT): Started mission Default
2023-04-19T21:16:24.362Z,1681938984.362 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3053, MT Status=1, MTMSN=2584
2023-04-19T21:16:24.938Z,1681938984.938 [CommandExec](IMPORTANT): got command ! echo i > /dev/loadA6;echo i > /dev/loadA7
2023-04-19T21:17:48.782Z,1681939068.782 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3054, MT Status=1, MTMSN=2585
2023-04-19T21:17:49.536Z,1681939069.536 [CommandExec](IMPORTANT): got command configSet CBIT.gfScanTimeout 2 hour persist
2023-04-19T21:17:49.635Z,1681939069.635 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-19T21:18:02.075Z,1681939082.075 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.003310
CHAN A1 (24V): 0.005624
CHAN A2 (12V): -0.004172
CHAN A3 (5V): -0.002899
CHAN B0 (3.3V): -0.001864
CHAN B1 (3.15aV): -0.001756
CHAN B2 (3.15bV): -0.001252
CHAN B3 (GND): -0.001084
OPEN: 0.012163
Full Scale: +/- 1 mA
2023-04-19T21:50:32.523Z,1681941032.523 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 34.733476 min 
2023-04-19T21:50:51.161Z,1681941051.161 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3103, MT Status=1, MTMSN=2586
2023-04-19T21:50:51.948Z,1681941051.948 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.tl
2023-04-19T21:50:51.952Z,1681941051.952 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Science/sci2.tx
2023-04-19T21:50:54.721Z,1681941054.721 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.tl
2023-04-19T21:51:10.573Z,1681941070.573 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 6 hour
2023-04-19T21:51:10.574Z,1681941070.574 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 45 minute
2023-04-19T21:51:10.575Z,1681941070.575 [CommandExec](IMPORTANT): got command set sci2.Lat1 36.755 degree
2023-04-19T21:51:10.577Z,1681941070.577 [CommandExec](IMPORTANT): got command set sci2.Lon1 -122.02 degree
2023-04-19T21:51:10.577Z,1681941070.577 [CommandExec](IMPORTANT): got command set sci2.Speed 0.8 meter_per_second
2023-04-19T21:51:10.578Z,1681941070.578 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 30 meter
2023-04-19T21:51:10.579Z,1681941070.579 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 40 meter
2023-04-19T21:51:10.579Z,1681941070.579 [CommandExec](IMPORTANT): got command run
2023-04-19T21:51:10.582Z,1681941070.582 [CommandExec](IMPORTANT): Running 
2023-04-19T21:51:10.801Z,1681941070.801 [MissionManager](IMPORTANT): Started mission sci2
2023-04-19T21:51:14.808Z,1681941074.808 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3105, MT Status=1, MTMSN=2587
2023-04-19T21:51:15.365Z,1681941075.365 [CommandExec](IMPORTANT): got command stop
2023-04-19T21:51:15.365Z,1681941075.365 [CommandExec](IMPORTANT): Scheduling is paused
2023-04-19T21:51:16.058Z,1681941076.058 [MissionManager](IMPORTANT): Started mission Default
2023-04-19T21:58:01.234Z,1681941481.234 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 6.746332 min 
2023-04-19T21:58:23.103Z,1681941503.103 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3113, MT Status=1, MTMSN=2588
2023-04-19T21:58:24.461Z,1681941504.461 [CommandExec](IMPORTANT): got command configSet Express linearApproximation longitude 0.005 degree persist
2023-04-19T21:58:44.588Z,1681941524.588 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3114, MT Status=1, MTMSN=2589
2023-04-19T21:58:45.751Z,1681941525.751 [CommandExec](IMPORTANT): got command configSet Express linearApproximation latitude 0.005 degree persist
2023-04-19T21:58:55.088Z,1681941535.088 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3115, MT Status=1, MTMSN=2590
2023-04-19T21:58:56.056Z,1681941536.056 [CommandExec](IMPORTANT): got command configSet Express linearApproximation depth 5 meter persist
2023-04-19T22:04:56.395Z,1681941896.395 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 13.665446 min 
2023-04-19T22:05:50.655Z,1681941950.655 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3120, MT Status=1, MTMSN=2591
2023-04-19T22:05:51.461Z,1681941951.461 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.tl
2023-04-19T22:05:51.462Z,1681941951.462 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Science/sci2.tx
2023-04-19T22:05:53.542Z,1681941953.542 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.tl
2023-04-19T22:06:08.113Z,1681941968.113 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 6 hour
2023-04-19T22:06:08.113Z,1681941968.113 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 45 minute
2023-04-19T22:06:08.114Z,1681941968.114 [CommandExec](IMPORTANT): got command set sci2.Lat1 36.755 degree
2023-04-19T22:06:08.115Z,1681941968.115 [CommandExec](IMPORTANT): got command set sci2.Lon1 -122.02 degree
2023-04-19T22:06:08.115Z,1681941968.115 [CommandExec](IMPORTANT): got command set sci2.Speed 0.8 meter_per_second
2023-04-19T22:06:08.116Z,1681941968.116 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 30 meter
2023-04-19T22:06:08.117Z,1681941968.117 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 40 meter
2023-04-19T22:06:08.117Z,1681941968.117 [CommandExec](IMPORTANT): got command run
2023-04-19T22:06:08.119Z,1681941968.119 [CommandExec](IMPORTANT): Running 
2023-04-19T22:06:08.316Z,1681941968.316 [MissionManager](IMPORTANT): Started mission sci2
2023-04-19T22:07:32.843Z,1681942052.843 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.755001,-122.019997
2023-04-19T22:11:39.315Z,1681942299.315 [StratificationFrontDetector](IMPORTANT): Initial vertical homogeneity condition (0 = stratified, 1 = homogeneous): 0
2023-04-19T22:31:34.451Z,1681943494.451 [StratificationFrontDetector](IMPORTANT): Front detection is now enabled. Present vertical homogeneity condition (0 = stratified, 1 = homogeneous), VTHI, VSHI, countSameWaterType_, countOppositeWaterType_, frontDetectionEnabled_ = 0, 0.817682, nan, 5, 0, 1
2023-04-19T22:56:06.697Z,1681944966.697 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3128, MT Status=1, MTMSN=2592
2023-04-19T22:56:07.270Z,1681944967.270 [CommandExec](IMPORTANT): got command ! echo i > /dev/loadA6;echo i > /dev/loadA7
2023-04-19T23:18:02.427Z,1681946282.427 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-19T23:18:13.255Z,1681946293.255 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.004020
CHAN A1 (24V): 0.005113
CHAN A2 (12V): -0.004396
CHAN A3 (5V): -0.006276
CHAN B0 (3.3V): -0.002000
CHAN B1 (3.15aV): -0.001525
CHAN B2 (3.15bV): -0.001703
CHAN B3 (GND): -0.000637
OPEN: 0.012253
Full Scale: +/- 1 mA
2023-04-19T23:52:21.645Z,1681948341.645 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3146, MT Status=1, MTMSN=2593
2023-04-19T23:52:22.308Z,1681948342.308 [CommandExec](IMPORTANT): got command ! onESPclient cmp -- slot spare:2;! echo i > /dev/loadA6;echo i > /dev/loadA7
2023-04-19T23:52:22.891Z,1681948342.891 [CommandExec](IMPORTANT): /mnt/mmc/started/2023-04-19UTC235222_onESPclient
  
2023-04-20T00:24:28.108Z,1681950268.108 [ElevatorServo](FAULT): Overload Error
2023-04-20T00:48:21.346Z,1681951701.346 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3163, MT Status=1, MTMSN=2594
2023-04-20T00:48:21.910Z,1681951701.910 [CommandExec](IMPORTANT): got command ! echo i > /dev/loadA6;echo i > /dev/loadA7
2023-04-20T01:14:07.896Z,1681953247.896 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout
2023-04-20T01:18:13.524Z,1681953493.524 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T01:18:24.807Z,1681953504.807 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.003263
CHAN A1 (24V): -0.014245
CHAN A2 (12V): -0.003948
CHAN A3 (5V): -0.003161
CHAN B0 (3.3V): -0.000870
CHAN B1 (3.15aV): -0.001035
CHAN B2 (3.15bV): -0.001259
CHAN B3 (GND): -0.018884
OPEN: 0.010517
Full Scale: +/- 1 mA
2023-04-20T01:41:55.165Z,1681954915.165 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Reached waypoint: 36.755001,-122.019997
2023-04-20T01:42:23.605Z,1681954943.605 [MissionManager](IMPORTANT): Started mission Default
2023-04-20T01:55:03.971Z,1681955703.971 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 12.665768 min 
2023-04-20T02:02:02.887Z,1681956122.887 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 19.648092 min 
2023-04-20T02:08:00.566Z,1681956480.566 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.609619 min 
2023-04-20T02:08:28.894Z,1681956508.894 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3201, MT Status=1, MTMSN=2595
2023-04-20T02:08:29.640Z,1681956509.640 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h" 3g8g2 1 3.000000
2023-04-20T02:08:29.641Z,1681956509.641 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=3g8g2
2023-04-20T02:08:29.641Z,1681956509.641 [CommandExec](IMPORTANT): Scheduled #5 (#1 of 3 with id='3g8g2'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h", ASAP
2023-04-20T02:09:12.615Z,1681956552.615 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3202, MT Status=1, MTMSN=2596
2023-04-20T02:09:13.376Z,1681956553.376 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m" 3g8g2 2 3.000000
2023-04-20T02:09:13.377Z,1681956553.377 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=3g8g2
2023-04-20T02:09:13.378Z,1681956553.378 [CommandExec](IMPORTANT): Scheduled #6 (#2 of 3 with id='3g8g2'): "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m", ASAP
2023-04-20T02:09:25.562Z,1681956565.562 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3203, MT Status=1, MTMSN=2597
2023-04-20T02:09:26.385Z,1681956566.385 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.DeepBoundChl 20 m;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 3g8g2 3 3.000000
2023-04-20T02:09:26.385Z,1681956566.385 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=3g8g2
2023-04-20T02:09:26.386Z,1681956566.386 [CommandExec](IMPORTANT): Scheduled #7 (#3 of 3 with id='3g8g2'): "set IsothermDepthSampling.DeepBoundChl 20 m;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP
2023-04-20T02:09:26.515Z,1681956566.515 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml
2023-04-20T02:09:42.862Z,1681956582.862 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml
2023-04-20T02:09:59.611Z,1681956599.611 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.NumSamplers 2 count
2023-04-20T02:09:59.612Z,1681956599.612 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 6 hour
2023-04-20T02:09:59.612Z,1681956599.612 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour
2023-04-20T02:10:25.029Z,1681956625.029 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool
2023-04-20T02:10:25.030Z,1681956625.030 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool
2023-04-20T02:10:25.031Z,1681956625.031 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool
2023-04-20T02:10:25.032Z,1681956625.032 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter
2023-04-20T02:10:49.278Z,1681956649.278 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 20 meter
2023-04-20T02:10:49.278Z,1681956649.278 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType2 -10 count
2023-04-20T02:10:49.279Z,1681956649.279 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 meter
2023-04-20T02:10:49.280Z,1681956649.280 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute
2023-04-20T02:10:49.281Z,1681956649.281 [CommandExec](IMPORTANT): got command run
2023-04-20T02:10:49.283Z,1681956649.283 [CommandExec](IMPORTANT): Running 
2023-04-20T02:10:49.423Z,1681956649.423 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling
2023-04-20T02:10:49.536Z,1681956649.536 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2023-04-20T02:14:18.919Z,1681956858.919 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.746764 arcdeg , longitude = -122.017687 arcdeg . Distance from shore = 13824.878906 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m .
2023-04-20T02:14:20.914Z,1681956860.914 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.247486 m 
2023-04-20T02:15:30.415Z,1681956930.415 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min .
2023-04-20T02:18:34.272Z,1681957114.272 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 23.065996 m 
2023-04-20T02:19:46.191Z,1681957186.191 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 20.293356 m , 7.248800 ug/l , 6.676032 m , 10.294688 degC , 8.859764 ug/l 
2023-04-20T02:19:46.582Z,1681957186.582 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 6.676032 m .
2023-04-20T02:19:46.955Z,1681957186.955 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2023-04-20T02:23:15.019Z,1681957395.019 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 6.605063 m , settling for 10.000000 min 
2023-04-20T02:33:27.940Z,1681958007.940 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-04-20T02:33:28.124Z,1681958008.124 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-04-20T02:34:19.657Z,1681958059.657 [ESPComponent](IMPORTANT): ESP has connected as client
2023-04-20T02:34:20.461Z,1681958060.461 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6
2023-04-20T02:34:20.461Z,1681958060.461 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-04-20T02:34:20.951Z,1681958060.951 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 8.022219 m in 10.273676 degC water with 7.990320 ug/l chlorophyll fluorescence.
2023-04-20T02:34:21.261Z,1681958061.261 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG
2023-04-20T02:34:21.667Z,1681958061.667 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-04-20T02:34:23.685Z,1681958063.685 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE
2023-04-20T02:35:00.569Z,1681958100.569 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@18,archive_bac)
2023-04-20T02:35:02.590Z,1681958102.590 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0)
2023-04-20T02:35:02.994Z,1681958102.994 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-04-20T02:35:03.397Z,1681958103.397 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-04-20T02:37:18.350Z,1681958238.350 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0)
2023-04-20T02:37:18.754Z,1681958238.754 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-04-20T03:18:25.197Z,1681960705.197 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T03:18:36.442Z,1681960716.442 [CBIT](IMPORTANT): Ground fault detected
mA:
CHAN A0 (Batt): 0.431241
CHAN A1 (24V): 0.551642
CHAN A2 (12V): 0.238748
CHAN A3 (5V): 0.080912
CHAN B0 (3.3V): 0.049520
CHAN B1 (3.15aV): 0.049810
CHAN B2 (3.15bV): 0.048877
CHAN B3 (GND): -0.010341
OPEN: 0.017992
Full Scale: +/- 1 mA
2023-04-20T03:34:28.225Z,1681961668.225 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1000.0)
2023-04-20T03:34:28.633Z,1681961668.633 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1000.0)
2023-04-20T03:34:30.653Z,1681961670.653 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-04-20T03:34:31.053Z,1681961671.053 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@18,archive_bac)
2023-04-20T03:46:35.441Z,1681962395.441 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@18,archive_bac)
2023-04-20T03:46:45.546Z,1681962405.546 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-04-20T03:46:46.104Z,1681962406.104 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED
2023-04-20T03:51:45.881Z,1681962705.881 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally.
2023-04-20T03:51:46.762Z,1681962706.762 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed.
2023-04-20T03:52:03.345Z,1681962723.345 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 5.000000 min .
2023-04-20T03:52:06.948Z,1681962726.948 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Target depth = 6.676032 m .
2023-04-20T03:52:07.359Z,1681962727.359 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 
2023-04-20T03:52:26.343Z,1681962746.343 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 6.546725 m , settling for 5.000000 min 
2023-04-20T03:57:39.096Z,1681963059.096 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-04-20T03:57:39.264Z,1681963059.264 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-04-20T03:58:27.537Z,1681963107.537 [ESPComponent](IMPORTANT): ESP has connected as client
2023-04-20T03:58:28.348Z,1681963108.348 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -10
2023-04-20T03:58:28.349Z,1681963108.349 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-04-20T03:58:28.817Z,1681963108.817 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 6.291384 m in 10.231989 degC water with 6.666400 ug/l chlorophyll fluorescence.
2023-04-20T03:58:29.154Z,1681963109.154 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG
2023-04-20T03:58:29.557Z,1681963109.557 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-04-20T03:58:31.578Z,1681963111.578 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE
2023-04-20T03:59:09.181Z,1681963149.181 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@29,daSPR_phyto)
2023-04-20T03:59:12.009Z,1681963152.009 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0)
2023-04-20T03:59:12.010Z,1681963152.010 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-04-20T03:59:12.414Z,1681963152.414 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-04-20T04:01:32.198Z,1681963292.198 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0)
2023-04-20T04:01:32.601Z,1681963292.601 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-04-20T05:01:22.681Z,1681966882.681 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@1000.0)
2023-04-20T05:01:23.083Z,1681966883.083 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@1000.0)
2023-04-20T05:01:25.102Z,1681966885.102 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing
2023-04-20T05:01:25.506Z,1681966885.506 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@29,daSPR_phyto)
2023-04-20T05:18:36.834Z,1681967916.834 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T05:18:47.655Z,1681967927.655 [CBIT](IMPORTANT): Ground fault detected
mA:
CHAN A0 (Batt): 0.549749
CHAN A1 (24V): 0.641425
CHAN A2 (12V): 0.273728
CHAN A3 (5V): 0.083136
CHAN B0 (3.3V): 0.049076
CHAN B1 (3.15aV): 0.065087
CHAN B2 (3.15bV): 0.048250
CHAN B3 (GND): -0.010641
OPEN: 0.012625
Full Scale: +/- 1 mA
2023-04-20T06:00:39.269Z,1681970439.269 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@29,daSPR_phyto)
2023-04-20T06:00:50.583Z,1681970450.583 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-04-20T06:00:51.003Z,1681970451.003 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED
2023-04-20T06:05:50.794Z,1681970750.794 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally.
2023-04-20T06:05:51.693Z,1681970751.693 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed.
2023-04-20T06:05:54.790Z,1681970754.790 [MissionManager](IMPORTANT): Started mission Default
2023-04-20T06:06:37.099Z,1681970797.099 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=3213, MT Status=1, MTMSN=2598
2023-04-20T06:06:37.715Z,1681970797.715 [CommandExec](IMPORTANT): got command ! echo i > /dev/loadA6;echo i > /dev/loadA7
2023-04-20T06:07:42.125Z,1681970862.125 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3214, MT Status=1, MTMSN=2599
2023-04-20T06:07:42.929Z,1681970862.929 [CommandExec](IMPORTANT): got command load ./Missions/Science/profile_station.xml
2023-04-20T06:07:45.400Z,1681970865.400 [CommandExec](IMPORTANT): Loaded ./Missions/Science/profile_station.xml
2023-04-20T06:08:05.745Z,1681970885.745 [CommandExec](IMPORTANT): got command set profile_station.MissionTimeout 12 hour
2023-04-20T06:08:05.746Z,1681970885.746 [CommandExec](IMPORTANT): got command set profile_station.Lat 36.658 degree
2023-04-20T06:08:05.748Z,1681970885.748 [CommandExec](IMPORTANT): got command set profile_station.Lon -121.874 degree
2023-04-20T06:08:05.749Z,1681970885.749 [CommandExec](IMPORTANT): got command set profile_station.YoYoMaxDepth 40 meter
2023-04-20T06:08:05.750Z,1681970885.750 [CommandExec](IMPORTANT): got command set profile_station.MaxDepth 50 meter
2023-04-20T06:08:05.750Z,1681970885.750 [CommandExec](IMPORTANT): got command run
2023-04-20T06:08:05.755Z,1681970885.755 [CommandExec](IMPORTANT): Running 
2023-04-20T06:08:06.122Z,1681970886.122 [MissionManager](IMPORTANT): Started mission profile_station
2023-04-20T07:18:47.712Z,1681975127.712 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T07:18:58.578Z,1681975138.578 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.003498
CHAN A1 (24V): 0.008325
CHAN A2 (12V): -0.005211
CHAN A3 (5V): -0.003591
CHAN B0 (3.3V): 0.001150
CHAN B1 (3.15aV): -0.001702
CHAN B2 (3.15bV): -0.000844
CHAN B3 (GND): -0.000868
OPEN: 0.012522
Full Scale: +/- 1 mA
2023-04-20T07:37:41.468Z,1681976261.468 [ElevatorServo](FAULT): Overload Error
2023-04-20T07:57:47.436Z,1681977467.436 [ElevatorServo](FAULT): Overload Error
2023-04-20T09:18:58.609Z,1681982338.609 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T09:19:09.555Z,1681982349.555 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.003977
CHAN A1 (24V): 0.003652
CHAN A2 (12V): -0.003877
CHAN A3 (5V): -0.003872
CHAN B0 (3.3V): 0.004548
CHAN B1 (3.15aV): -0.001003
CHAN B2 (3.15bV): -0.001011
CHAN B3 (GND): -0.000691
OPEN: 0.012484
Full Scale: +/- 1 mA
2023-04-20T09:41:09.440Z,1681983669.440 [ThrusterHE](FAULT): getSpeed error serial timeout
2023-04-20T09:41:09.440Z,1681983669.440 [ThrusterHE](FAULT): Thruster uart error: serial timeout
2023-04-20T09:59:09.692Z,1681984749.692 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-04-20T11:19:09.934Z,1681989549.934 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T11:19:20.785Z,1681989560.785 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.003420
CHAN A1 (24V): 0.004914
CHAN A2 (12V): -0.004867
CHAN A3 (5V): -0.003501
CHAN B0 (3.3V): -0.001076
CHAN B1 (3.15aV): -0.001662
CHAN B2 (3.15bV): -0.000955
CHAN B3 (GND): -0.000980
OPEN: 0.012331
Full Scale: +/- 1 mA
2023-04-20T12:06:00.988Z,1681992360.988 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout
2023-04-20T13:02:35.152Z,1681995755.152 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-04-20T13:19:21.067Z,1681996761.067 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T13:19:32.327Z,1681996772.327 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.003416
CHAN A1 (24V): 0.004009
CHAN A2 (12V): -0.005125
CHAN A3 (5V): -0.003240
CHAN B0 (3.3V): -0.002002
CHAN B1 (3.15aV): -0.000158
CHAN B2 (3.15bV): -0.000782
CHAN B3 (GND): -0.000637
OPEN: 0.012671
Full Scale: +/- 1 mA
2023-04-20T14:03:45.378Z,1681999425.378 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3304, MT Status=1, MTMSN=2600
2023-04-20T14:03:45.893Z,1681999425.893 [CommandExec](IMPORTANT): got command stop
2023-04-20T14:03:46.597Z,1681999426.597 [MissionManager](IMPORTANT): Started mission Default
2023-04-20T14:08:31.634Z,1681999711.634 [DataOverHttps](IMPORTANT): SBD MTMSN=20230420T140830
2023-04-20T14:08:42.140Z,1681999722.140 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h" 3h5y2 1 3.000000
2023-04-20T14:08:42.140Z,1681999722.140 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=3h5y2
2023-04-20T14:08:42.141Z,1681999722.141 [CommandExec](IMPORTANT): Scheduled #8 (#1 of 3 with id='3h5y2'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h", ASAP
2023-04-20T14:08:43.126Z,1681999723.126 [DataOverHttps](IMPORTANT): SBD MTMSN=20230420T140842
2023-04-20T14:08:51.296Z,1681999731.296 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m" 3h5y2 2 3.000000
2023-04-20T14:08:51.297Z,1681999731.297 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=3h5y2
2023-04-20T14:08:51.298Z,1681999731.298 [CommandExec](IMPORTANT): Scheduled #9 (#2 of 3 with id='3h5y2'): "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m", ASAP
2023-04-20T14:08:52.278Z,1681999732.278 [DataOverHttps](IMPORTANT): SBD MTMSN=20230420T140851
2023-04-20T14:08:59.853Z,1681999739.853 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.DeepBoundChl 20 m;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 3h5y2 3 3.000000
2023-04-20T14:08:59.854Z,1681999739.854 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=3h5y2
2023-04-20T14:08:59.855Z,1681999739.855 [CommandExec](IMPORTANT): Scheduled #10 (#3 of 3 with id='3h5y2'): "set IsothermDepthSampling.DeepBoundChl 20 m;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP
2023-04-20T14:08:59.980Z,1681999739.980 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml
2023-04-20T14:09:15.650Z,1681999755.650 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml
2023-04-20T14:09:39.802Z,1681999779.802 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.NumSamplers 2 count
2023-04-20T14:09:39.802Z,1681999779.802 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 6 hour
2023-04-20T14:09:39.803Z,1681999779.803 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour
2023-04-20T14:10:15.259Z,1681999815.259 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool
2023-04-20T14:10:15.260Z,1681999815.260 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool
2023-04-20T14:10:15.261Z,1681999815.261 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool
2023-04-20T14:10:15.262Z,1681999815.262 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter
2023-04-20T14:10:51.860Z,1681999851.860 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 20 meter
2023-04-20T14:10:51.861Z,1681999851.861 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType2 -10 count
2023-04-20T14:10:51.862Z,1681999851.862 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 meter
2023-04-20T14:10:51.863Z,1681999851.863 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute
2023-04-20T14:10:51.863Z,1681999851.863 [CommandExec](IMPORTANT): got command run
2023-04-20T14:10:51.866Z,1681999851.866 [CommandExec](IMPORTANT): Running 
2023-04-20T14:10:52.084Z,1681999852.084 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling
2023-04-20T14:10:52.177Z,1681999852.177 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2023-04-20T14:12:02.488Z,1681999922.488 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.660030 arcdeg , longitude = -121.870424 arcdeg . Distance from shore = 3676.004883 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m .
2023-04-20T14:12:04.495Z,1681999924.495 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.200816 m 
2023-04-20T14:13:14.013Z,1681999994.013 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -6.000000 count , nan K , 0.000000 m , nan degC , nan m , 10.000000 min .
2023-04-20T14:16:34.805Z,1682000194.805 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoUp:B](IMPORTANT): Go up from 23.061684 m 
2023-04-20T14:18:18.643Z,1682000298.643 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:SetChlPeakTempDepth:L](IMPORTANT): Set peak-chl depth and temp. DepthLogged, ChlLogged, PeakChlDepthSet, PeakChlTemperatureSet, PeakChlSet = 17.746527 m , 1.778400 ug/l , 8.376822 m , 11.016248 degC , 14.034294 ug/l 
2023-04-20T14:18:19.052Z,1682000299.052 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 1.000000 count . Target depth = 8.376822 m .
2023-04-20T14:18:19.427Z,1682000299.427 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -6.000000 count 
2023-04-20T14:21:10.753Z,1682000470.753 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 8.361089 m , settling for 10.000000 min 
2023-04-20T14:31:23.640Z,1682001083.640 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-04-20T14:31:23.984Z,1682001083.984 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-04-20T14:32:16.749Z,1682001136.749 [ESPComponent](IMPORTANT): ESP has connected as client
2023-04-20T14:32:17.565Z,1682001137.565 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6
2023-04-20T14:32:17.565Z,1682001137.565 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-04-20T14:32:18.056Z,1682001138.056 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 9.046780 m in 10.695215 degC water with 11.967280 ug/l chlorophyll fluorescence.
2023-04-20T14:32:18.366Z,1682001138.366 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG
2023-04-20T14:32:18.769Z,1682001138.769 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-04-20T14:32:20.788Z,1682001140.788 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE
2023-04-20T14:32:57.554Z,1682001177.554 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@16,archive_bac)
2023-04-20T14:32:59.573Z,1682001179.573 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0)
2023-04-20T14:32:59.983Z,1682001179.983 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-04-20T14:33:00.381Z,1682001180.381 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-04-20T14:35:14.510Z,1682001314.510 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0)
2023-04-20T14:35:14.913Z,1682001314.913 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-04-20T15:19:32.575Z,1682003972.575 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T15:19:43.555Z,1682003983.555 [CBIT](IMPORTANT): Ground fault detected
mA:
CHAN A0 (Batt): 0.198140
CHAN A1 (24V): 0.315982
CHAN A2 (12V): 0.159695
CHAN A3 (5V): 0.047279
CHAN B0 (3.3V): 0.027966
CHAN B1 (3.15aV): 0.049640
CHAN B2 (3.15bV): 0.027985
CHAN B3 (GND): -0.010871
OPEN: 0.016556
Full Scale: +/- 1 mA
2023-04-20T15:30:28.578Z,1682004628.578 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0)
2023-04-20T15:30:28.981Z,1682004628.981 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0)
2023-04-20T15:30:31.003Z,1682004631.003 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-04-20T15:30:31.406Z,1682004631.406 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@16,archive_bac)
2023-04-20T15:42:35.798Z,1682005355.798 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@16,archive_bac)
2023-04-20T15:42:45.898Z,1682005365.898 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-04-20T15:42:46.344Z,1682005366.344 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED
2023-04-20T15:47:46.113Z,1682005666.113 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally.
2023-04-20T15:47:46.999Z,1682005666.999 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 1.000000 count completed.
2023-04-20T15:48:03.554Z,1682005683.554 [IsothermDepthSampling:SetSampleOptions:N](IMPORTANT): Set CartridgeType (if ESP), TempDiffFromPeakChl, DepDiffFromPeakChl, TempDesignated, DepthDesignated, WaitDepthUndulation to -10.000000 count , nan K , 0.000000 m , nan degC , nan m , 5.000000 min .
2023-04-20T15:48:07.203Z,1682005687.203 [IsothermDepthSampling:TakeSamples:FiringOnDepth:E](IMPORTANT): Taking sample No. 2.000000 count . Target depth = 8.376822 m .
2023-04-20T15:48:07.607Z,1682005687.607 [IsothermDepthSampling:TakeSamples:FiringOnDepth:F](IMPORTANT): If ESP, requesting ESP cartridge type: -10.000000 count 
2023-04-20T15:49:10.234Z,1682005750.234 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 8.412694 m , settling for 5.000000 min 
2023-04-20T15:54:22.973Z,1682006062.973 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-04-20T15:54:23.148Z,1682006063.148 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-04-20T15:55:11.805Z,1682006111.805 [ESPComponent](IMPORTANT): ESP has connected as client
2023-04-20T15:55:12.616Z,1682006112.616 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -10
2023-04-20T15:55:12.616Z,1682006112.616 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-04-20T15:55:13.102Z,1682006113.102 [IsothermDepthSampling:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 8.335059 m in 10.832575 degC water with 16.021200 ug/l chlorophyll fluorescence.
2023-04-20T15:55:13.421Z,1682006113.421 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG
2023-04-20T15:55:13.828Z,1682006113.828 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-04-20T15:55:15.845Z,1682006115.845 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE
2023-04-20T15:55:53.418Z,1682006153.418 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@27,daSPR_phyto)
2023-04-20T15:55:56.246Z,1682006156.246 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0)
2023-04-20T15:55:56.247Z,1682006156.247 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-04-20T15:55:56.649Z,1682006156.649 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-04-20T15:58:15.231Z,1682006295.231 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0)
2023-04-20T15:58:15.631Z,1682006295.631 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-04-20T16:55:11.929Z,1682009711.929 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1000.0)
2023-04-20T16:55:12.334Z,1682009712.334 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1000.0)
2023-04-20T16:55:14.354Z,1682009714.354 [ESPComponent](IMPORTANT): Processing timeoutMinutes=120.000000 issuedCmd_=Cmd.startProcessing
2023-04-20T16:55:14.758Z,1682009714.758 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@27,daSPR_phyto)
2023-04-20T17:19:43.896Z,1682011183.896 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T17:19:54.769Z,1682011194.769 [CBIT](IMPORTANT): Ground fault detected
mA:
CHAN A0 (Batt): 0.216831
CHAN A1 (24V): 0.283600
CHAN A2 (12V): 0.110172
CHAN A3 (5V): 0.025329
CHAN B0 (3.3V): 0.014612
CHAN B1 (3.15aV): 0.014164
CHAN B2 (3.15bV): 0.014985
CHAN B3 (GND): -0.011057
OPEN: 0.014956
Full Scale: +/- 1 mA
2023-04-20T17:50:40.953Z,1682013040.953 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@27,daSPR_phyto)
2023-04-20T17:50:52.266Z,1682013052.266 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-04-20T17:50:52.670Z,1682013052.670 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED
2023-04-20T17:55:52.441Z,1682013352.441 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally.
2023-04-20T17:55:53.250Z,1682013353.250 [ESPComponent](FAULT): LCB 1 fault: LCB Watchdog Reset. Hardware Overcurrent Shutdown. Current Limiter Activated.
2023-04-20T17:55:53.332Z,1682013353.332 [IsothermDepthSampling:TakeSamples:FiringOnDepth:TriggerOnDepth:C](IMPORTANT): Sample No. 2.000000 count completed.
2023-04-20T17:55:56.297Z,1682013356.297 [MissionManager](IMPORTANT): Started mission Default
2023-04-20T17:57:59.878Z,1682013479.878 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3321, MT Status=1, MTMSN=2601
2023-04-20T17:58:00.466Z,1682013480.466 [CommandExec](IMPORTANT): got command ! echo i > /dev/loadA6;echo i > /dev/loadA7
2023-04-20T18:06:09.918Z,1682013969.918 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 10.220250 min 
2023-04-20T18:11:35.058Z,1682014295.058 [DataOverHttps](IMPORTANT): SBD MTMSN=20230420T181134
2023-04-20T18:11:45.374Z,1682014305.374 [CommandExec](IMPORTANT): got command load ./Missions/Science/sci2.tl
2023-04-20T18:11:45.376Z,1682014305.376 [MissionManager](IMPORTANT): Loading Compiled TethysL script from file ./Missions/Science/sci2.tx
2023-04-20T18:11:47.461Z,1682014307.461 [CommandExec](IMPORTANT): Loaded ./Missions/Science/sci2.tl
2023-04-20T18:11:53.050Z,1682014313.050 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 15.939037 min 
2023-04-20T18:12:07.102Z,1682014327.102 [CommandExec](IMPORTANT): got command set sci2.MissionTimeout 6 hour
2023-04-20T18:12:07.103Z,1682014327.103 [CommandExec](IMPORTANT): got command set sci2.NeedCommsTime 30 minute
2023-04-20T18:12:07.104Z,1682014327.104 [CommandExec](IMPORTANT): got command set sci2.Lat1 36.725 degree
2023-04-20T18:12:07.105Z,1682014327.105 [CommandExec](IMPORTANT): got command set sci2.Lon1 -121.85 degree
2023-04-20T18:12:07.105Z,1682014327.105 [CommandExec](IMPORTANT): got command set sci2.Speed 0.8 meter_per_second
2023-04-20T18:12:07.106Z,1682014327.106 [CommandExec](IMPORTANT): got command set sci2.YoYoMaxDepth 30 meter
2023-04-20T18:12:07.108Z,1682014327.108 [CommandExec](IMPORTANT): got command set sci2.MaxDepth 40 meter
2023-04-20T18:12:07.108Z,1682014327.108 [CommandExec](IMPORTANT): got command run
2023-04-20T18:12:07.110Z,1682014327.110 [CommandExec](IMPORTANT): Running 
2023-04-20T18:12:07.322Z,1682014327.322 [MissionManager](IMPORTANT): Started mission sci2
2023-04-20T18:14:35.708Z,1682014475.708 [sci2:Lap:Wpt1:A.Waypoint](IMPORTANT): Navigating to waypoint: 36.724998,-121.849998
2023-04-20T18:19:37.444Z,1682014777.444 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-04-20T19:19:54.775Z,1682018394.775 [CBIT](IMPORTANT): Beginning ground fault scan
2023-04-20T19:20:05.715Z,1682018405.715 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.003299
CHAN A1 (24V): 0.020427
CHAN A2 (12V): -0.004872
CHAN A3 (5V): -0.004193
CHAN B0 (3.3V): 0.000699
CHAN B1 (3.15aV): -0.001487
CHAN B2 (3.15bV): -0.001272
CHAN B3 (GND): -0.000466
OPEN: 0.012186
Full Scale: +/- 1 mA
2023-04-20T20:02:02.412Z,1682020922.412 [MassServo](FAULT): Mass Shifter uart error: serial timeout
2023-04-20T20:02:44.070Z,1682020964.070 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=3337, MT Status=1, MTMSN=2602
2023-04-20T20:02:44.553Z,1682020964.553 [CommandExec](IMPORTANT): got command stop
2023-04-20T20:02:44.943Z,1682020964.943 [MissionManager](IMPORTANT): Started mission Default
2023-04-20T20:10:49.346Z,1682021449.346 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 8.066735 min 
2023-04-20T20:16:31.039Z,1682021791.039 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 13.761341 min 
2023-04-20T20:22:12.738Z,1682022132.738 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 19.455977 min 
2023-04-20T20:27:55.866Z,1682022475.866 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 25.175104 min 
2023-04-20T20:29:11.478Z,1682022551.478 [DataOverHttps](IMPORTANT): SBD MTMSN=20230420T202910
2023-04-20T20:29:18.932Z,1682022558.932 [CommandExec](IMPORTANT): got command schedule asap "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h" 3hnk4 1 3.000000
2023-04-20T20:29:18.932Z,1682022558.932 [CommandExec](IMPORTANT): Scheduling command #1 of 3 with id=3hnk4
2023-04-20T20:29:18.933Z,1682022558.933 [CommandExec](IMPORTANT): Scheduled #11 (#1 of 3 with id='3hnk4'): "load Science/isotherm_depth_sampling.xml;set IsothermDepthSampling.NumSamplers 2 count;set IsothermDepthSampling.MissionTimeout 6 h;set IsothermDepthSampling.MaxWaitNoFiring 1 h", ASAP
2023-04-20T20:29:19.906Z,1682022559.906 [DataOverHttps](IMPORTANT): SBD MTMSN=20230420T202918
2023-04-20T20:29:27.461Z,1682022567.461 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m" 3hnk4 2 3.000000
2023-04-20T20:29:27.461Z,1682022567.461 [CommandExec](IMPORTANT): Scheduling command #2 of 3 with id=3hnk4
2023-04-20T20:29:27.462Z,1682022567.462 [CommandExec](IMPORTANT): Scheduled #12 (#2 of 3 with id='3hnk4'): "set IsothermDepthSampling.SendSampleStatusAndData 0 bool;set IsothermDepthSampling.SendDataToMultipleModems 0 bool;set IsothermDepthSampling.SampleAtPeakChl 1 bool;set IsothermDepthSampling.ShallowBoundChl 3 m", ASAP
2023-04-20T20:29:28.402Z,1682022568.402 [DataOverHttps](IMPORTANT): SBD MTMSN=20230420T202927
2023-04-20T20:29:38.797Z,1682022578.797 [CommandExec](IMPORTANT): got command schedule asap "set IsothermDepthSampling.DeepBoundChl 25 m;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run" 3hnk4 3 3.000000
2023-04-20T20:29:38.797Z,1682022578.797 [CommandExec](IMPORTANT): Scheduling command #3 of 3 with id=3hnk4
2023-04-20T20:29:38.798Z,1682022578.798 [CommandExec](IMPORTANT): Scheduled #13 (#3 of 3 with id='3hnk4'): "set IsothermDepthSampling.DeepBoundChl 25 m;set IsothermDepthSampling.CartridgeType2 -10 count;set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 m;set IsothermDepthSampling.WaitDepUndulation1 10 min;run", ASAP
2023-04-20T20:29:38.955Z,1682022578.955 [CommandExec](IMPORTANT): got command load ./Missions/Science/isotherm_depth_sampling.xml
2023-04-20T20:29:59.891Z,1682022599.891 [CommandExec](IMPORTANT): Loaded ./Missions/Science/isotherm_depth_sampling.xml
2023-04-20T20:30:20.522Z,1682022620.522 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.NumSamplers 2 count
2023-04-20T20:30:20.523Z,1682022620.523 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MissionTimeout 6 hour
2023-04-20T20:30:20.524Z,1682022620.524 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.MaxWaitNoFiring 1 hour
2023-04-20T20:30:58.825Z,1682022658.825 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendSampleStatusAndData 0 bool
2023-04-20T20:30:58.826Z,1682022658.826 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SendDataToMultipleModems 0 bool
2023-04-20T20:30:58.826Z,1682022658.826 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.SampleAtPeakChl 1 bool
2023-04-20T20:30:58.827Z,1682022658.827 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.ShallowBoundChl 3 meter
2023-04-20T20:31:27.490Z,1682022687.490 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DeepBoundChl 25 meter
2023-04-20T20:31:27.492Z,1682022687.492 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.CartridgeType2 -10 count
2023-04-20T20:31:27.493Z,1682022687.493 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.DepDiffFromPeakChlCommon 0 meter
2023-04-20T20:31:27.493Z,1682022687.493 [CommandExec](IMPORTANT): got command set IsothermDepthSampling.WaitDepUndulation1 10 minute
2023-04-20T20:31:27.494Z,1682022687.494 [CommandExec](IMPORTANT): got command run
2023-04-20T20:31:27.497Z,1682022687.497 [CommandExec](IMPORTANT): Running 
2023-04-20T20:31:27.601Z,1682022687.601 [MissionManager](IMPORTANT): Started mission IsothermDepthSampling
2023-04-20T20:31:27.693Z,1682022687.693 [IsothermDepthSampling:AbortDrift:B.AbortDrift](IMPORTANT): Acoustic timeout set to: 240.00 hours.
2023-04-20T20:32:48.532Z,1682022768.532 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout
2023-04-20T20:32:48.836Z,1682022768.836 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout
2023-04-20T20:32:49.777Z,1682022769.777 [ElevatorServo](FAULT): Initialization error opening port: cannot open serial port
2023-04-20T20:32:49.778Z,1682022769.778 [RudderServo](FAULT): Initialization error opening port: cannot open serial port
2023-04-20T20:32:51.308Z,1682022771.308 [ElevatorServo](FAULT): Elevator failed to initialize
2023-04-20T20:32:51.617Z,1682022771.617 [RudderServo](FAULT): Rudder failed to initialize
2023-04-20T20:32:51.809Z,1682022771.809 [IsothermDepthSampling:StayAwayFromWallsAndShore:FarEnoughFromWallsAndShore:A](IMPORTANT): Far enough from walls and shore. Latitude = 36.697143 arcdeg , longitude = -121.855037 arcdeg . Distance from shore = 3442.770752 m . OffshoreDistanceTurnBack (MinOffshore + MarginMinOffshore) = 2000.000000 m + 100.000000 m . Shortest distance nan m to side 0 count . MinDistFromPolygonSides = 200.000000 m .
2023-04-20T20:32:52.617Z,1682022772.617 [RudderServo](FAULT): Initialization error opening port: cannot open serial port
2023-04-20T20:32:53.804Z,1682022773.804 [RudderServo](FAULT): Rudder failed to initialize
2023-04-20T20:32:53.819Z,1682022773.819 [CBIT](CRITICAL): Communications Fault in component: RudderServo
2023-04-20T20:32:54.055Z,1682022774.055 [IsothermDepthSampling:FindPeakChlVertical:FindPeak:LogGoDown:B](IMPORTANT): Go down from 0.262296 m 
2023-04-20T20:32:54.141Z,1682022774.141 [CommandExec](FAULT): Scheduling is paused
2023-04-20T20:32:55.367Z,1682022775.367 [MissionManager](IMPORTANT): Started mission Default