2023-05-22T01:52:19.698Z,1684720339.698 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027897 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 25.000000 count 
2023-05-22T01:58:21.302Z,1684720701.302 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.755600 m , settling for 5.000000 min 
2023-05-22T02:03:34.125Z,1684721014.125 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-22T02:03:34.289Z,1684721014.289 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-22T02:04:25.757Z,1684721065.757 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-22T02:04:26.574Z,1684721066.574 [ESPComponent](IMPORTANT): [sample #25] Setting cartridge argument to -1
2023-05-22T02:04:26.574Z,1684721066.574 [ESPComponent](IMPORTANT): [sample #25] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-22T02:04:26.995Z,1684721066.995 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.832542 m in 9.224451 degC water with 0.034800 ug/l chlorophyll fluorescence.
2023-05-22T02:04:27.379Z,1684721067.379 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-22T02:04:27.777Z,1684721067.777 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-22T02:04:29.798Z,1684721069.798 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-22T02:05:01.720Z,1684721101.720 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_PRIMING (@32,aRNAl_bac)
2023-05-22T02:05:04.544Z,1684721104.544 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PRIMING (@100.0)
2023-05-22T02:05:04.545Z,1684721104.545 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-22T02:05:04.947Z,1684721104.947 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-22T02:07:47.357Z,1684721267.357 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_FILTERING (@0.0)
2023-05-22T02:07:47.767Z,1684721267.767 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-22T02:51:58.091Z,1684723918.091 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PAUSED (@1000.0)
2023-05-22T02:51:58.502Z,1684723918.502 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-22T02:52:00.532Z,1684723920.532 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-22T02:52:00.919Z,1684723920.919 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_WAITING_FOR_PROCESSED (@32,aRNAl_bac)
2023-05-22T02:53:36.262Z,1684724016.262 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_STOPPING (@32,aRNAl_bac)
2023-05-22T02:53:43.939Z,1684724023.939 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-22T02:53:44.343Z,1684724024.343 [ESPComponent](IMPORTANT): [sample #25] ESP sampling state: S_STOPPED
2023-05-22T02:58:44.130Z,1684724324.130 [ESPComponent](IMPORTANT): [sample #25] ESP sampling sequence completed normally.
2023-05-22T02:58:46.580Z,1684724326.580 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 25.000000 count 
2023-05-22T02:58:46.963Z,1684724326.963 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-22T02:58:47.367Z,1684724327.367 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-22T02:58:47.775Z,1684724327.775 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-22T02:58:48.190Z,1684724328.190 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 101.130615 m 
2023-05-22T02:58:48.613Z,1684724328.613 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.239313 degC 
2023-05-22T02:58:48.987Z,1684724328.987 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-22T02:58:49.804Z,1684724329.804 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 25.000000 count completed.
2023-05-22T02:58:56.022Z,1684724336.022 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T02:59:06.357Z,1684724346.357 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T03:03:53.635Z,1684724633.635 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-22T03:32:48.521Z,1684726368.521 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout
2023-05-22T03:52:22.493Z,1684727542.493 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.032878 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 26.000000 count 
2023-05-22T03:59:27.919Z,1684727967.919 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.810493 m , settling for 5.000000 min 
2023-05-22T04:04:40.689Z,1684728280.689 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-22T04:04:40.817Z,1684728280.817 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-22T04:05:28.729Z,1684728328.729 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-22T04:05:29.542Z,1684728329.542 [ESPComponent](IMPORTANT): [sample #26] Setting cartridge argument to -1
2023-05-22T04:05:29.543Z,1684728329.543 [ESPComponent](IMPORTANT): [sample #26] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-22T04:05:29.986Z,1684728329.986 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.343964 m in 9.254450 degC water with 0.031320 ug/l chlorophyll fluorescence.
2023-05-22T04:05:30.351Z,1684728330.351 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-22T04:05:30.755Z,1684728330.755 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-22T04:05:32.771Z,1684728332.771 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-22T04:06:04.687Z,1684728364.687 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_PRIMING (@31,aRNAl_bac)
2023-05-22T04:06:07.116Z,1684728367.116 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PRIMING (@100.0)
2023-05-22T04:06:07.117Z,1684728367.117 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-22T04:06:07.519Z,1684728367.519 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-22T04:08:50.336Z,1684728530.336 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_FILTERING (@0.0)
2023-05-22T04:08:50.735Z,1684728530.735 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-22T04:25:02.521Z,1684729502.521 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-22T04:53:15.266Z,1684731195.266 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PAUSED (@1000.0)
2023-05-22T04:53:15.676Z,1684731195.676 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-22T04:53:17.690Z,1684731197.690 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-22T04:53:18.099Z,1684731198.099 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_WAITING_FOR_PROCESSED (@31,aRNAl_bac)
2023-05-22T04:54:53.442Z,1684731293.442 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_STOPPING (@31,aRNAl_bac)
2023-05-22T04:55:01.151Z,1684731301.151 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-22T04:55:01.559Z,1684731301.559 [ESPComponent](IMPORTANT): [sample #26] ESP sampling state: S_STOPPED
2023-05-22T05:00:01.367Z,1684731601.367 [ESPComponent](IMPORTANT): [sample #26] ESP sampling sequence completed normally.
2023-05-22T05:00:02.257Z,1684731602.257 [ESPComponent](FAULT): LCB 1 fault:  Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated.
2023-05-22T05:00:03.889Z,1684731603.889 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 26.000000 count 
2023-05-22T05:00:04.303Z,1684731604.303 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-22T05:00:04.720Z,1684731604.720 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-22T05:00:05.137Z,1684731605.137 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-22T05:00:05.505Z,1684731605.505 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.729790 m 
2023-05-22T05:00:05.918Z,1684731605.918 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.339106 degC 
2023-05-22T05:00:06.307Z,1684731606.307 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 
2023-05-22T05:00:07.116Z,1684731607.116 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 26.000000 count completed.
2023-05-22T05:00:13.361Z,1684731613.361 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T05:00:23.697Z,1684731623.697 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T05:00:53.966Z,1684731653.966 [DAT](FAULT): failed to enter command mode
2023-05-22T05:05:10.976Z,1684731910.976 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-22T05:52:25.141Z,1684734745.141 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030290 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 27.000000 count 
2023-05-22T05:58:36.001Z,1684735116.001 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.788536 m , settling for 5.000000 min 
2023-05-22T06:03:48.829Z,1684735428.829 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-22T06:03:48.958Z,1684735428.958 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-22T06:04:40.858Z,1684735480.858 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-22T06:04:41.680Z,1684735481.680 [ESPComponent](IMPORTANT): [sample #27] Setting cartridge argument to -1
2023-05-22T06:04:41.681Z,1684735481.681 [ESPComponent](IMPORTANT): [sample #27] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-22T06:04:42.115Z,1684735482.115 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.922028 m in 9.306879 degC water with 0.031320 ug/l chlorophyll fluorescence.
2023-05-22T06:04:42.474Z,1684735482.474 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-22T06:04:42.887Z,1684735482.887 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-22T06:04:44.908Z,1684735484.908 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-22T06:05:16.813Z,1684735516.813 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_WAITING_FOR_PRIMING (@30,aRNAl_bac)
2023-05-22T06:05:19.647Z,1684735519.647 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PRIMING (@100.0)
2023-05-22T06:05:19.648Z,1684735519.648 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-22T06:05:20.051Z,1684735520.051 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-22T06:08:02.454Z,1684735682.454 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_FILTERING (@0.0)
2023-05-22T06:08:02.865Z,1684735682.865 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-22T06:24:47.801Z,1684736687.801 [CBIT](IMPORTANT): Beginning ground fault scan
2023-05-22T06:24:58.664Z,1684736698.664 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002385
CHAN A1 (24V): 0.000048
CHAN A2 (12V): 0.000287
CHAN A3 (5V): -0.000437
CHAN B0 (3.3V): -0.000026
CHAN B1 (3.15aV): -0.000285
CHAN B2 (3.15bV): 0.000474
CHAN B3 (GND): -0.000077
OPEN: 0.007702
Full Scale: +/- 1 mA
2023-05-22T06:43:29.521Z,1684737809.521 [ThrusterHE](FAULT): getSpeed error serial timeout
2023-05-22T06:43:29.521Z,1684737809.521 [ThrusterHE](FAULT): Thruster uart error: serial timeout
2023-05-22T06:52:33.811Z,1684738353.811 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PAUSED (@1000.0)
2023-05-22T06:52:34.219Z,1684738354.219 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-22T06:52:36.233Z,1684738356.233 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-22T06:52:36.644Z,1684738356.644 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_WAITING_FOR_PROCESSED (@30,aRNAl_bac)
2023-05-22T06:54:11.981Z,1684738451.981 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_STOPPING (@30,aRNAl_bac)
2023-05-22T06:54:19.667Z,1684738459.667 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-22T06:54:20.062Z,1684738460.062 [ESPComponent](IMPORTANT): [sample #27] ESP sampling state: S_STOPPED
2023-05-22T06:59:19.838Z,1684738759.838 [ESPComponent](IMPORTANT): [sample #27] ESP sampling sequence completed normally.
2023-05-22T06:59:22.325Z,1684738762.325 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 27.000000 count 
2023-05-22T06:59:22.707Z,1684738762.707 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-22T06:59:23.094Z,1684738763.094 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-22T06:59:23.502Z,1684738763.502 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-22T06:59:23.911Z,1684738763.911 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.758766 m 
2023-05-22T06:59:24.330Z,1684738764.330 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.330988 degC 
2023-05-22T06:59:24.706Z,1684738764.706 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 
2023-05-22T06:59:25.526Z,1684738765.526 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 27.000000 count completed.
2023-05-22T06:59:31.817Z,1684738771.817 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T06:59:42.169Z,1684738782.169 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T07:04:29.364Z,1684739069.364 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-22T07:52:27.631Z,1684741947.631 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027905 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 28.000000 count 
2023-05-22T07:59:18.096Z,1684742358.096 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.668068 m , settling for 5.000000 min 
2023-05-22T08:04:30.878Z,1684742670.878 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-22T08:04:31.005Z,1684742671.005 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-22T08:05:22.919Z,1684742722.919 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-22T08:05:23.784Z,1684742723.784 [ESPComponent](IMPORTANT): [sample #28] Setting cartridge argument to -1
2023-05-22T08:05:23.784Z,1684742723.784 [ESPComponent](IMPORTANT): [sample #28] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-22T08:05:24.166Z,1684742724.166 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.767914 m in 9.450800 degC water with 0.037120 ug/l chlorophyll fluorescence.
2023-05-22T08:05:24.550Z,1684742724.550 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-22T08:05:24.935Z,1684742724.935 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-22T08:05:26.955Z,1684742726.955 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-22T08:05:58.870Z,1684742758.870 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_WAITING_FOR_PRIMING (@29,aRNAl_bac)
2023-05-22T08:06:00.891Z,1684742760.891 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PRIMING (@100.0)
2023-05-22T08:06:01.301Z,1684742761.301 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-22T08:06:01.698Z,1684742761.698 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-22T08:08:44.120Z,1684742924.120 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_FILTERING (@0.0)
2023-05-22T08:08:44.519Z,1684742924.519 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-22T08:46:01.425Z,1684745161.425 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-22T08:54:39.509Z,1684745679.509 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PAUSED (@1000.0)
2023-05-22T08:54:39.919Z,1684745679.919 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-22T08:54:41.943Z,1684745681.943 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-22T08:54:42.338Z,1684745682.338 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_WAITING_FOR_PROCESSED (@29,aRNAl_bac)
2023-05-22T08:56:17.359Z,1684745777.359 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_STOPPING (@29,aRNAl_bac)
2023-05-22T08:56:25.039Z,1684745785.039 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-22T08:56:25.444Z,1684745785.444 [ESPComponent](IMPORTANT): [sample #28] ESP sampling state: S_STOPPED
2023-05-22T09:01:25.211Z,1684746085.211 [ESPComponent](IMPORTANT): [sample #28] ESP sampling sequence completed normally.
2023-05-22T09:01:27.676Z,1684746087.676 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 28.000000 count 
2023-05-22T09:01:28.054Z,1684746088.054 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-22T09:01:28.475Z,1684746088.475 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-22T09:01:28.863Z,1684746088.863 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-22T09:01:29.265Z,1684746089.265 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.171730 m 
2023-05-22T09:01:29.670Z,1684746089.670 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.445612 degC 
2023-05-22T09:01:30.086Z,1684746090.086 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 
2023-05-22T09:01:30.880Z,1684746090.880 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 28.000000 count completed.
2023-05-22T09:01:37.158Z,1684746097.158 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T09:01:47.494Z,1684746107.494 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T09:06:34.828Z,1684746394.828 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-22T09:52:30.143Z,1684749150.143 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028784 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 29.000000 count 
2023-05-22T09:58:24.877Z,1684749504.877 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.743698 m , settling for 5.000000 min 
2023-05-22T10:03:37.658Z,1684749817.658 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-22T10:03:37.794Z,1684749817.794 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-22T10:04:29.696Z,1684749869.696 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-22T10:04:30.509Z,1684749870.509 [ESPComponent](IMPORTANT): [sample #29] Setting cartridge argument to -1
2023-05-22T10:04:30.510Z,1684749870.510 [ESPComponent](IMPORTANT): [sample #29] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-22T10:04:30.931Z,1684749870.931 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.705254 m in 9.460566 degC water with 0.034800 ug/l chlorophyll fluorescence.
2023-05-22T10:04:31.312Z,1684749871.312 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-22T10:04:31.720Z,1684749871.720 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-22T10:04:33.737Z,1684749873.737 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-22T10:05:05.650Z,1684749905.650 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_WAITING_FOR_PRIMING (@28,aRNAl_bac)
2023-05-22T10:05:08.483Z,1684749908.483 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_PRIMING (@100.0)
2023-05-22T10:05:08.485Z,1684749908.485 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-22T10:05:08.878Z,1684749908.878 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-22T10:07:51.689Z,1684750071.689 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_FILTERING (@0.0)
2023-05-22T10:07:52.102Z,1684750072.102 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-22T10:53:00.221Z,1684752780.221 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_PAUSED (@1000.0)
2023-05-22T10:53:00.631Z,1684752780.631 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-22T10:53:02.647Z,1684752782.647 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-22T10:53:03.051Z,1684752783.051 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_WAITING_FOR_PROCESSED (@28,aRNAl_bac)
2023-05-22T10:54:38.007Z,1684752878.007 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_STOPPING (@28,aRNAl_bac)
2023-05-22T10:54:45.680Z,1684752885.680 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-22T10:54:46.080Z,1684752886.080 [ESPComponent](IMPORTANT): [sample #29] ESP sampling state: S_STOPPED
2023-05-22T10:59:45.869Z,1684753185.869 [ESPComponent](IMPORTANT): [sample #29] ESP sampling sequence completed normally.
2023-05-22T10:59:48.298Z,1684753188.298 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 29.000000 count 
2023-05-22T10:59:48.703Z,1684753188.703 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-22T10:59:49.117Z,1684753189.117 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-22T10:59:49.518Z,1684753189.518 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-22T10:59:49.910Z,1684753189.910 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.450851 m 
2023-05-22T10:59:50.317Z,1684753190.317 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.349390 degC 
2023-05-22T10:59:50.725Z,1684753190.725 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 
2023-05-22T10:59:51.522Z,1684753191.522 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 29.000000 count completed.
2023-05-22T10:59:57.662Z,1684753197.662 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T11:00:07.997Z,1684753207.997 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T11:00:38.254Z,1684753238.254 [DAT](FAULT): failed to enter command mode
2023-05-22T11:04:55.343Z,1684753495.343 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-22T11:52:32.862Z,1684756352.862 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.031950 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 30.000000 count 
2023-05-22T11:58:32.434Z,1684756712.434 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.770912 m , settling for 5.000000 min 
2023-05-22T12:03:45.233Z,1684757025.233 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-22T12:03:45.360Z,1684757025.360 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-22T12:04:33.207Z,1684757073.207 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-22T12:04:34.028Z,1684757074.028 [ESPComponent](IMPORTANT): [sample #30] Setting cartridge argument to -1
2023-05-22T12:04:34.028Z,1684757074.028 [ESPComponent](IMPORTANT): [sample #30] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-22T12:04:34.440Z,1684757074.440 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.160141 m in 9.294764 degC water with 0.029000 ug/l chlorophyll fluorescence.
2023-05-22T12:04:34.819Z,1684757074.819 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-22T12:04:35.227Z,1684757075.227 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-22T12:04:37.251Z,1684757077.251 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-22T12:05:09.162Z,1684757109.162 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_WAITING_FOR_PRIMING (@27,aRNAl_bac)
2023-05-22T12:05:11.994Z,1684757111.994 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_PRIMING (@100.0)
2023-05-22T12:05:11.995Z,1684757111.995 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-22T12:05:12.403Z,1684757112.403 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-22T12:07:54.811Z,1684757274.811 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_FILTERING (@0.0)
2023-05-22T12:07:55.223Z,1684757275.223 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-22T12:24:58.793Z,1684758298.793 [CBIT](IMPORTANT): Beginning ground fault scan
2023-05-22T12:25:10.191Z,1684758310.191 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.001939
CHAN A1 (24V): 0.000024
CHAN A2 (12V): 0.000060
CHAN A3 (5V): -0.000393
CHAN B0 (3.3V): 0.000425
CHAN B1 (3.15aV): -0.000271
CHAN B2 (3.15bV): -0.001065
CHAN B3 (GND): 0.000234
OPEN: 0.007452
Full Scale: +/- 1 mA
2023-05-22T12:52:32.191Z,1684759952.191 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_PAUSED (@1000.0)
2023-05-22T12:52:32.592Z,1684759952.592 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-22T12:52:34.616Z,1684759954.616 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-22T12:52:35.014Z,1684759955.014 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_WAITING_FOR_PROCESSED (@27,aRNAl_bac)
2023-05-22T12:54:09.951Z,1684760049.951 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_STOPPING (@27,aRNAl_bac)
2023-05-22T12:54:17.632Z,1684760057.632 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-22T12:54:18.031Z,1684760058.031 [ESPComponent](IMPORTANT): [sample #30] ESP sampling state: S_STOPPED
2023-05-22T12:55:39.517Z,1684760139.517 [ThrusterHE](FAULT): getSpeed error serial timeout
2023-05-22T12:55:39.517Z,1684760139.517 [ThrusterHE](FAULT): Thruster uart error: serial timeout
2023-05-22T12:59:17.799Z,1684760357.799 [ESPComponent](IMPORTANT): [sample #30] ESP sampling sequence completed normally.
2023-05-22T12:59:20.241Z,1684760360.241 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 30.000000 count 
2023-05-22T12:59:20.646Z,1684760360.646 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-22T12:59:21.051Z,1684760361.051 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-22T12:59:21.467Z,1684760361.467 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-22T12:59:21.876Z,1684760361.876 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.395996 m 
2023-05-22T12:59:22.261Z,1684760362.261 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.241816 degC 
2023-05-22T12:59:22.671Z,1684760362.671 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-22T12:59:23.485Z,1684760363.485 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 30.000000 count completed.
2023-05-22T12:59:29.609Z,1684760369.609 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T12:59:39.949Z,1684760379.949 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T13:04:27.304Z,1684760667.304 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-22T13:52:35.550Z,1684763555.550 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030965 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 31.000000 count 
2023-05-22T13:58:50.864Z,1684763930.864 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.783417 m , settling for 5.000000 min 
2023-05-22T14:02:10.258Z,1684764130.258 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout
2023-05-22T14:04:03.857Z,1684764243.857 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-22T14:04:03.995Z,1684764243.995 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-22T14:05:06.746Z,1684764306.746 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-22T14:05:07.552Z,1684764307.552 [ESPComponent](IMPORTANT): [sample #31] Setting cartridge argument to -1
2023-05-22T14:05:07.552Z,1684764307.552 [ESPComponent](IMPORTANT): [sample #31] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-22T14:05:07.998Z,1684764307.998 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.515175 m in 9.232752 degC water with 0.027840 ug/l chlorophyll fluorescence.
2023-05-22T14:05:08.357Z,1684764308.357 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-22T14:05:08.770Z,1684764308.770 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-22T14:05:10.777Z,1684764310.777 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-22T14:05:42.695Z,1684764342.695 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_WAITING_FOR_PRIMING (@26,aRNAl_bac)
2023-05-22T14:05:45.118Z,1684764345.118 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_PRIMING (@100.0)
2023-05-22T14:05:45.119Z,1684764345.119 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-22T14:05:45.538Z,1684764345.538 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-22T14:08:27.943Z,1684764507.943 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_FILTERING (@0.0)
2023-05-22T14:08:28.342Z,1684764508.342 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-22T14:31:05.501Z,1684765865.501 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout
2023-05-22T14:53:44.498Z,1684767224.498 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_PAUSED (@1000.0)
2023-05-22T14:53:44.899Z,1684767224.899 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-22T14:53:46.914Z,1684767226.914 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-22T14:53:47.317Z,1684767227.317 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_WAITING_FOR_PROCESSED (@26,aRNAl_bac)
2023-05-22T14:55:22.257Z,1684767322.257 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_STOPPING (@26,aRNAl_bac)
2023-05-22T14:55:29.936Z,1684767329.936 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-22T14:55:30.339Z,1684767330.339 [ESPComponent](IMPORTANT): [sample #31] ESP sampling state: S_STOPPED
2023-05-22T15:00:30.116Z,1684767630.116 [ESPComponent](IMPORTANT): [sample #31] ESP sampling sequence completed normally.
2023-05-22T15:00:32.570Z,1684767632.570 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 31.000000 count 
2023-05-22T15:00:32.957Z,1684767632.957 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-22T15:00:33.367Z,1684767633.367 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-22T15:00:33.770Z,1684767633.770 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-22T15:00:34.191Z,1684767634.191 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.134201 m 
2023-05-22T15:00:34.579Z,1684767634.579 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.242701 degC 
2023-05-22T15:00:34.979Z,1684767634.979 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-22T15:00:35.798Z,1684767635.798 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 31.000000 count completed.
2023-05-22T15:00:42.045Z,1684767642.045 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T15:00:52.383Z,1684767652.383 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T15:01:12.806Z,1684767672.806 [DAT](FAULT): failed to enter command mode
2023-05-22T15:05:39.627Z,1684767939.627 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-22T15:28:02.167Z,1684769282.167 [CommandExec](IMPORTANT): got command failComponent hardware DAT
2023-05-22T15:28:02.168Z,1684769282.168 [CommandExec](IMPORTANT): DAT failureMode is Hardware Fault
2023-05-22T15:28:23.579Z,1684769303.579 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T15:52:37.977Z,1684770757.977 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027116 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 32.000000 count 
2023-05-22T15:59:37.796Z,1684771177.796 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.726822 m , settling for 5.000000 min 
2023-05-22T16:04:50.529Z,1684771490.529 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-22T16:04:50.668Z,1684771490.668 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-22T16:05:50.646Z,1684771550.646 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-22T16:05:51.470Z,1684771551.470 [ESPComponent](IMPORTANT): [sample #32] Setting cartridge argument to -1
2023-05-22T16:05:51.470Z,1684771551.470 [ESPComponent](IMPORTANT): [sample #32] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-22T16:05:51.896Z,1684771551.896 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.753189 m in 9.246515 degC water with 0.027840 ug/l chlorophyll fluorescence.
2023-05-22T16:05:52.261Z,1684771552.261 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-22T16:05:52.667Z,1684771552.667 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-22T16:05:54.687Z,1684771554.687 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-22T16:06:26.599Z,1684771586.599 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_WAITING_FOR_PRIMING (@25,aRNAl_bac)
2023-05-22T16:06:29.432Z,1684771589.432 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_PRIMING (@100.0)
2023-05-22T16:06:29.433Z,1684771589.433 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-22T16:06:29.831Z,1684771589.831 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-22T16:09:12.243Z,1684771752.243 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_FILTERING (@0.0)
2023-05-22T16:09:12.643Z,1684771752.643 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-22T16:32:17.215Z,1684773137.215 [CommandExec](IMPORTANT): got command configSet IBIT.batteryCapacityThreshold 15 ampere_hour persist
2023-05-22T16:32:26.706Z,1684773146.706 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T16:32:41.829Z,1684773161.829 [DAT](FAULT): failed to set remote address
2023-05-22T16:32:57.214Z,1684773177.214 [DAT](FAULT): failed to set remote address
2023-05-22T16:33:12.599Z,1684773192.599 [DAT](FAULT): failed to set remote address
2023-05-22T16:33:27.971Z,1684773207.971 [DAT](FAULT): failed to set remote address
2023-05-22T16:33:43.366Z,1684773223.366 [DAT](FAULT): failed to set remote address
2023-05-22T16:33:48.034Z,1684773228.034 [CommandExec](IMPORTANT): got command failComponent hardware DAT
2023-05-22T16:33:48.035Z,1684773228.035 [CommandExec](IMPORTANT): DAT failureMode is Hardware Fault
2023-05-22T16:34:23.181Z,1684773263.181 [DAT](FAULT): failed to enter command mode
2023-05-22T16:34:24.947Z,1684773264.947 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T16:35:33.711Z,1684773333.711 [CommandExec](IMPORTANT): got command configSet IBIT.batteryCapacityThreshold 15 ampere_hour persist
2023-05-22T16:36:18.559Z,1684773378.559 [CommandExec](IMPORTANT): got command set circle_sample.SpeedSampling 0.8 meter_per_second
2023-05-22T16:36:59.485Z,1684773419.485 [DAT](FAULT): failed to enter command mode
2023-05-22T16:37:17.242Z,1684773437.242 [CommandExec](IMPORTANT): got command failComponent hardware IBIT
2023-05-22T16:37:17.242Z,1684773437.242 [CommandExec](IMPORTANT): IBIT failureMode is Hardware Fault
2023-05-22T16:37:17.468Z,1684773437.468 [CBIT](CRITICAL): Hardware Fault in component: IBIT
2023-05-22T16:37:17.842Z,1684773437.842 [CommandExec](FAULT): Scheduling is paused
2023-05-22T16:37:18.676Z,1684773438.676 [ESPComponent](IMPORTANT): [sample #32] Stop sampling requested.
2023-05-22T16:37:18.693Z,1684773438.693 [MissionManager](IMPORTANT): Started mission Default
2023-05-22T16:37:19.067Z,1684773439.067 [ESPComponent](IMPORTANT): Stop sampling requested. Issuing Cmd.pauseFiltering
2023-05-22T16:37:24.731Z,1684773444.731 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_PAUSED (@651.4)
2023-05-22T16:37:25.137Z,1684773445.137 [ESPComponent](IMPORTANT): In S_PAUSED received result: 651.4

2023-05-22T16:37:25.535Z,1684773445.535 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_PROCESSING (@651.4)
2023-05-22T16:37:27.571Z,1684773447.571 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-22T16:37:27.959Z,1684773447.959 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_WAITING_FOR_PROCESSED (@25,aRNAl_bac)
2023-05-22T16:39:02.895Z,1684773542.895 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_STOPPING (@25,aRNAl_bac)
2023-05-22T16:39:10.575Z,1684773550.575 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-22T16:39:10.984Z,1684773550.984 [ESPComponent](IMPORTANT): [sample #32] ESP sampling state: S_STOPPED
2023-05-22T16:42:49.678Z,1684773769.678 [CommandExec](IMPORTANT): got command failComponent hardware DAT
2023-05-22T16:42:49.679Z,1684773769.679 [CommandExec](IMPORTANT): DAT failureMode is Hardware Fault
2023-05-22T16:44:10.741Z,1684773850.741 [ESPComponent](IMPORTANT): [sample #32] ESP sampling sequence completed normally.
2023-05-22T16:45:49.285Z,1684773949.285 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2023-05-22T17:49:20.137Z,1684777760.137 [NAL9602](FAULT): Verify xmit timeout failure.
2023-05-22T17:51:50.903Z,1684777910.903 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=19277, MT Status=1, MTMSN=1329
2023-05-22T17:51:51.422Z,1684777911.422 [CommandExec](IMPORTANT): got command restart application