2023-05-21T01:51:09.958Z,1684633869.958 [CommandExec](IMPORTANT): got command restart logs
2023-05-21T01:51:48.415Z,1684633908.415 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028857 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 13.000000 count 
2023-05-21T01:58:10.238Z,1684634290.238 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.673363 m , settling for 5.000000 min 
2023-05-21T02:03:23.001Z,1684634603.001 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T02:03:23.114Z,1684634603.114 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T02:04:14.675Z,1684634654.675 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T02:04:15.484Z,1684634655.484 [ESPComponent](IMPORTANT): [sample #13] Setting cartridge argument to -1
2023-05-21T02:04:15.492Z,1684634655.492 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T02:04:15.933Z,1684634655.933 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.969254 m in 9.116693 degC water with 0.024360 ug/l chlorophyll fluorescence.
2023-05-21T02:04:16.290Z,1684634656.290 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T02:04:16.695Z,1684634656.695 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T02:04:18.720Z,1684634658.720 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T02:04:50.727Z,1684634690.727 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PRIMING (@44,aRNAl_bac)
2023-05-21T02:04:53.559Z,1684634693.559 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T02:04:53.560Z,1684634693.560 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T02:04:53.958Z,1684634693.958 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T02:07:36.772Z,1684634856.772 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T02:07:37.176Z,1684634857.176 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T02:19:43.741Z,1684635583.741 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-21T02:51:51.513Z,1684637511.513 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T02:51:51.924Z,1684637511.924 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T02:51:53.952Z,1684637513.952 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T02:51:54.358Z,1684637514.358 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_WAITING_FOR_PROCESSED (@44,aRNAl_bac)
2023-05-21T02:53:29.296Z,1684637609.296 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPING (@44,aRNAl_bac)
2023-05-21T02:53:36.974Z,1684637616.974 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T02:53:37.364Z,1684637617.364 [ESPComponent](IMPORTANT): [sample #13] ESP sampling state: S_STOPPED
2023-05-21T02:58:37.139Z,1684637917.139 [ESPComponent](IMPORTANT): [sample #13] ESP sampling sequence completed normally.
2023-05-21T02:58:39.598Z,1684637919.598 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 13.000000 count 
2023-05-21T02:58:39.993Z,1684637919.993 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T02:58:40.387Z,1684637920.387 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T02:58:40.791Z,1684637920.791 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T02:58:41.218Z,1684637921.218 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.766663 m 
2023-05-21T02:58:41.623Z,1684637921.623 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.114008 degC 
2023-05-21T02:58:42.005Z,1684637922.005 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 
2023-05-21T02:58:42.802Z,1684637922.802 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 13.000000 count completed.
2023-05-21T02:58:48.953Z,1684637928.953 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T02:58:59.290Z,1684637939.290 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T02:59:14.414Z,1684637954.414 [DAT](FAULT): failed to enter command mode
2023-05-21T03:03:46.658Z,1684638226.658 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T03:51:51.196Z,1684641111.196 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.032682 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 14.000000 count 
2023-05-21T03:57:41.073Z,1684641461.073 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.678581 m , settling for 5.000000 min 
2023-05-21T04:02:53.845Z,1684641773.845 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T04:02:53.966Z,1684641773.966 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T04:03:41.411Z,1684641821.411 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T04:03:42.228Z,1684641822.228 [ESPComponent](IMPORTANT): [sample #14] Setting cartridge argument to -1
2023-05-21T04:03:42.229Z,1684641822.229 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T04:03:42.672Z,1684641822.672 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.067810 m in 9.235712 degC water with 0.025520 ug/l chlorophyll fluorescence.
2023-05-21T04:03:43.023Z,1684641823.023 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T04:03:43.435Z,1684641823.435 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T04:03:45.462Z,1684641825.462 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T04:04:17.364Z,1684641857.364 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PRIMING (@43,aRNAl_bac)
2023-05-21T04:04:20.195Z,1684641860.195 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T04:04:20.196Z,1684641860.196 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T04:04:20.594Z,1684641860.594 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T04:07:03.406Z,1684642023.406 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T04:07:03.811Z,1684642023.811 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T04:13:46.301Z,1684642426.301 [ElevatorServo](FAULT): Reading outside of valid range:0.000000

2023-05-21T04:13:46.961Z,1684642426.961 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout
2023-05-21T04:38:34.805Z,1684643914.805 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-21T04:53:46.495Z,1684644826.495 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T04:53:46.898Z,1684644826.898 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T04:53:48.919Z,1684644828.919 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T04:53:49.333Z,1684644829.333 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_WAITING_FOR_PROCESSED (@43,aRNAl_bac)
2023-05-21T04:55:24.684Z,1684644924.684 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPING (@43,aRNAl_bac)
2023-05-21T04:55:32.343Z,1684644932.343 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T04:55:32.763Z,1684644932.763 [ESPComponent](IMPORTANT): [sample #14] ESP sampling state: S_STOPPED
2023-05-21T05:00:32.522Z,1684645232.522 [ESPComponent](IMPORTANT): [sample #14] ESP sampling sequence completed normally.
2023-05-21T05:00:33.353Z,1684645233.353 [ESPComponent](FAULT): LCB 1 fault: LCB Watchdog Reset. Hardware Overcurrent Shutdown. Current Limiter Activated.
2023-05-21T05:00:34.991Z,1684645234.991 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 14.000000 count 
2023-05-21T05:00:35.408Z,1684645235.408 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T05:00:35.785Z,1684645235.785 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T05:00:36.194Z,1684645236.194 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T05:00:36.606Z,1684645236.606 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 101.176315 m 
2023-05-21T05:00:37.024Z,1684645237.024 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.309809 degC 
2023-05-21T05:00:37.407Z,1684645237.407 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 
2023-05-21T05:00:38.224Z,1684645238.224 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 14.000000 count completed.
2023-05-21T05:00:44.313Z,1684645244.313 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T05:00:49.606Z,1684645249.606 [DAT](FAULT): failed to set remote address
2023-05-21T05:00:54.645Z,1684645254.645 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T05:01:41.770Z,1684645301.770 [DAT](FAULT): failed to set remote address
2023-05-21T05:02:27.154Z,1684645347.154 [DAT](FAULT): failed to set remote address
2023-05-21T05:03:12.273Z,1684645392.273 [DAT](FAULT): failed to set remote address
2023-05-21T05:03:57.653Z,1684645437.653 [DAT](FAULT): failed to set remote address
2023-05-21T05:04:42.777Z,1684645482.777 [DAT](FAULT): failed to set remote address
2023-05-21T05:05:28.174Z,1684645528.174 [DAT](FAULT): failed to set remote address
2023-05-21T05:05:42.062Z,1684645542.062 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T05:06:13.294Z,1684645573.294 [DAT](FAULT): failed to set remote address
2023-05-21T05:06:58.435Z,1684645618.435 [DAT](FAULT): failed to set remote address
2023-05-21T05:07:43.821Z,1684645663.821 [DAT](FAULT): failed to set remote address
2023-05-21T05:08:28.966Z,1684645708.966 [DAT](FAULT): failed to set remote address
2023-05-21T05:09:14.382Z,1684645754.382 [DAT](FAULT): failed to set remote address
2023-05-21T05:09:59.534Z,1684645799.534 [DAT](FAULT): failed to set remote address
2023-05-21T05:10:44.682Z,1684645844.682 [DAT](FAULT): failed to set remote address
2023-05-21T05:11:30.074Z,1684645890.074 [DAT](FAULT): failed to set remote address
2023-05-21T05:13:00.565Z,1684645980.565 [DAT](FAULT): failed to set remote address
2023-05-21T05:13:45.697Z,1684646025.697 [DAT](FAULT): failed to set remote address
2023-05-21T05:14:48.484Z,1684646088.484 [DAT](FAULT): failed to set local address
2023-05-21T05:51:53.750Z,1684648313.750 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028833 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 15.000000 count 
2023-05-21T05:56:38.195Z,1684648598.195 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.815178 m , settling for 5.000000 min 
2023-05-21T06:01:50.962Z,1684648910.962 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T06:01:51.110Z,1684648911.110 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T06:02:50.707Z,1684648970.707 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T06:02:51.545Z,1684648971.545 [ESPComponent](IMPORTANT): [sample #15] Setting cartridge argument to -1
2023-05-21T06:02:51.546Z,1684648971.546 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T06:02:51.940Z,1684648971.940 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.689468 m in 9.107691 degC water with 0.025520 ug/l chlorophyll fluorescence.
2023-05-21T06:02:52.327Z,1684648972.327 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T06:02:52.732Z,1684648972.732 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T06:02:54.768Z,1684648974.768 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T06:03:26.668Z,1684649006.668 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PRIMING (@42,aRNAl_bac)
2023-05-21T06:03:28.685Z,1684649008.685 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T06:03:29.084Z,1684649009.084 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T06:03:29.485Z,1684649009.485 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T06:06:13.106Z,1684649173.106 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T06:06:13.511Z,1684649173.511 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T06:24:02.992Z,1684650242.992 [CBIT](IMPORTANT): Beginning ground fault scan
2023-05-21T06:24:13.932Z,1684650253.932 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002670
CHAN A1 (24V): -0.000026
CHAN A2 (12V): 0.000155
CHAN A3 (5V): -0.000519
CHAN B0 (3.3V): -0.000058
CHAN B1 (3.15aV): 0.000222
CHAN B2 (3.15bV): 0.000012
CHAN B3 (GND): -0.000317
OPEN: 0.007754
Full Scale: +/- 1 mA
2023-05-21T06:53:11.547Z,1684651991.547 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T06:53:11.950Z,1684651991.950 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T06:53:13.980Z,1684651993.980 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T06:53:14.383Z,1684651994.383 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_WAITING_FOR_PROCESSED (@42,aRNAl_bac)
2023-05-21T06:54:49.722Z,1684652089.722 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPING (@42,aRNAl_bac)
2023-05-21T06:54:57.406Z,1684652097.406 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T06:54:57.808Z,1684652097.808 [ESPComponent](IMPORTANT): [sample #15] ESP sampling state: S_STOPPED
2023-05-21T06:59:57.578Z,1684652397.578 [ESPComponent](IMPORTANT): [sample #15] ESP sampling sequence completed normally.
2023-05-21T07:00:00.032Z,1684652400.032 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 15.000000 count 
2023-05-21T07:00:00.441Z,1684652400.441 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T07:00:00.853Z,1684652400.853 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T07:00:01.233Z,1684652401.233 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T07:00:01.643Z,1684652401.643 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.348259 m 
2023-05-21T07:00:02.058Z,1684652402.058 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.163385 degC 
2023-05-21T07:00:02.461Z,1684652402.461 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-21T07:00:03.255Z,1684652403.255 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 15.000000 count completed.
2023-05-21T07:00:09.407Z,1684652409.407 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T07:00:19.741Z,1684652419.741 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T07:05:07.089Z,1684652707.089 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T07:51:56.197Z,1684655516.197 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.027352 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 16.000000 count 
2023-05-21T07:57:33.148Z,1684655853.148 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.693886 m , settling for 5.000000 min 
2023-05-21T08:02:45.974Z,1684656165.974 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T08:02:46.198Z,1684656166.198 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T08:03:37.362Z,1684656217.362 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T08:03:38.178Z,1684656218.178 [ESPComponent](IMPORTANT): [sample #16] Setting cartridge argument to -1
2023-05-21T08:03:38.179Z,1684656218.179 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T08:03:38.603Z,1684656218.603 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.731911 m in 9.179407 degC water with 0.025520 ug/l chlorophyll fluorescence.
2023-05-21T08:03:38.985Z,1684656218.985 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T08:03:39.382Z,1684656219.382 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T08:03:41.407Z,1684656221.407 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T08:04:13.321Z,1684656253.321 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PRIMING (@41,aRNAl_bac)
2023-05-21T08:04:16.155Z,1684656256.155 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T08:04:16.156Z,1684656256.156 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T08:04:16.555Z,1684656256.555 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T08:06:59.374Z,1684656419.374 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T08:06:59.767Z,1684656419.767 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T08:26:11.766Z,1684657571.766 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-21T08:51:58.715Z,1684659118.715 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T08:51:59.118Z,1684659119.118 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T08:52:01.135Z,1684659121.135 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T08:52:01.540Z,1684659121.540 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_WAITING_FOR_PROCESSED (@41,aRNAl_bac)
2023-05-21T08:53:36.483Z,1684659216.483 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPING (@41,aRNAl_bac)
2023-05-21T08:53:44.156Z,1684659224.156 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T08:53:44.567Z,1684659224.567 [ESPComponent](IMPORTANT): [sample #16] ESP sampling state: S_STOPPED
2023-05-21T08:58:44.342Z,1684659524.342 [ESPComponent](IMPORTANT): [sample #16] ESP sampling sequence completed normally.
2023-05-21T08:58:46.800Z,1684659526.800 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 16.000000 count 
2023-05-21T08:58:47.215Z,1684659527.215 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T08:58:47.595Z,1684659527.595 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T08:58:48.003Z,1684659528.003 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T08:58:48.416Z,1684659528.416 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.715523 m 
2023-05-21T08:58:48.824Z,1684659528.824 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.376001 degC 
2023-05-21T08:58:49.210Z,1684659529.210 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-21T08:58:50.042Z,1684659530.042 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 16.000000 count completed.
2023-05-21T08:58:56.154Z,1684659536.154 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T08:59:06.485Z,1684659546.485 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T08:59:31.202Z,1684659571.202 [DAT](FAULT): failed to enter command mode
2023-05-21T09:03:53.848Z,1684659833.848 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T09:51:58.852Z,1684662718.852 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030461 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 17.000000 count 
2023-05-21T09:53:07.133Z,1684662787.133 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.584641 m , settling for 5.000000 min 
2023-05-21T09:57:18.505Z,1684663038.505 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout
2023-05-21T09:58:19.901Z,1684663099.901 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T09:58:20.014Z,1684663100.014 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T09:59:11.569Z,1684663151.569 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T09:59:12.391Z,1684663152.391 [ESPComponent](IMPORTANT): [sample #17] Setting cartridge argument to -1
2023-05-21T09:59:12.391Z,1684663152.391 [ESPComponent](IMPORTANT): [sample #17] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T09:59:12.814Z,1684663152.814 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.812309 m in 9.337275 degC water with 0.025520 ug/l chlorophyll fluorescence.
2023-05-21T09:59:13.186Z,1684663153.186 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T09:59:13.706Z,1684663153.706 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T09:59:15.609Z,1684663155.609 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T09:59:47.531Z,1684663187.531 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PRIMING (@40,aRNAl_bac)
2023-05-21T09:59:49.551Z,1684663189.551 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T09:59:49.963Z,1684663189.963 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T09:59:50.358Z,1684663190.358 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T10:02:33.175Z,1684663353.175 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T10:02:33.569Z,1684663353.569 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T10:35:41.645Z,1684665341.645 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout
2023-05-21T10:46:23.312Z,1684665983.312 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T10:46:23.706Z,1684665983.706 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T10:46:25.728Z,1684665985.728 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T10:46:26.131Z,1684665986.131 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_WAITING_FOR_PROCESSED (@40,aRNAl_bac)
2023-05-21T10:48:01.474Z,1684666081.474 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_STOPPING (@40,aRNAl_bac)
2023-05-21T10:48:09.157Z,1684666089.157 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T10:48:09.569Z,1684666089.569 [ESPComponent](IMPORTANT): [sample #17] ESP sampling state: S_STOPPED
2023-05-21T10:53:09.335Z,1684666389.335 [ESPComponent](IMPORTANT): [sample #17] ESP sampling sequence completed normally.
2023-05-21T10:53:11.774Z,1684666391.774 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 17.000000 count 
2023-05-21T10:53:12.195Z,1684666392.195 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T10:53:12.602Z,1684666392.602 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T10:53:12.990Z,1684666392.990 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T10:53:13.394Z,1684666393.394 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.650757 m 
2023-05-21T10:53:13.799Z,1684666393.799 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.243311 degC 
2023-05-21T10:53:14.216Z,1684666394.216 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 
2023-05-21T10:53:14.998Z,1684666394.998 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 17.000000 count completed.
2023-05-21T10:53:21.081Z,1684666401.081 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T10:53:31.413Z,1684666411.413 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T10:58:18.847Z,1684666698.847 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T11:52:01.624Z,1684669921.624 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.032926 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 18.000000 count 
2023-05-21T11:55:32.108Z,1684670132.108 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.683174 m , settling for 5.000000 min 
2023-05-21T12:00:44.929Z,1684670444.929 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T12:00:45.050Z,1684670445.050 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T12:01:36.567Z,1684670496.567 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T12:01:37.372Z,1684670497.372 [ESPComponent](IMPORTANT): [sample #18] Setting cartridge argument to -1
2023-05-21T12:01:37.372Z,1684670497.372 [ESPComponent](IMPORTANT): [sample #18] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T12:01:37.808Z,1684670497.808 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.562706 m in 9.222712 degC water with 0.030160 ug/l chlorophyll fluorescence.
2023-05-21T12:01:38.177Z,1684670498.177 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T12:01:38.582Z,1684670498.582 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T12:01:40.601Z,1684670500.601 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T12:02:12.518Z,1684670532.518 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PRIMING (@39,aRNAl_bac)
2023-05-21T12:02:15.347Z,1684670535.347 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T12:02:15.348Z,1684670535.348 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T12:02:15.756Z,1684670535.756 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T12:04:58.162Z,1684670698.162 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T12:04:58.570Z,1684670698.570 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T12:15:45.605Z,1684671345.605 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-21T12:24:14.242Z,1684671854.242 [CBIT](IMPORTANT): Beginning ground fault scan
2023-05-21T12:24:25.552Z,1684671865.552 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002552
CHAN A1 (24V): -0.000105
CHAN A2 (12V): 0.000646
CHAN A3 (5V): -0.000108
CHAN B0 (3.3V): 0.000496
CHAN B1 (3.15aV): 0.000413
CHAN B2 (3.15bV): 0.000251
CHAN B3 (GND): 0.000075
OPEN: 0.007645
Full Scale: +/- 1 mA
2023-05-21T12:49:28.457Z,1684673368.457 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T12:49:28.871Z,1684673368.871 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T12:49:30.880Z,1684673370.880 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T12:49:31.284Z,1684673371.284 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_WAITING_FOR_PROCESSED (@39,aRNAl_bac)
2023-05-21T12:51:06.664Z,1684673466.664 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_STOPPING (@39,aRNAl_bac)
2023-05-21T12:51:14.302Z,1684673474.302 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T12:51:14.702Z,1684673474.702 [ESPComponent](IMPORTANT): [sample #18] ESP sampling state: S_STOPPED
2023-05-21T12:56:14.475Z,1684673774.475 [ESPComponent](IMPORTANT): [sample #18] ESP sampling sequence completed normally.
2023-05-21T12:56:16.939Z,1684673776.939 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 18.000000 count 
2023-05-21T12:56:17.355Z,1684673777.355 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T12:56:17.734Z,1684673777.734 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T12:56:18.134Z,1684673778.134 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T12:56:18.550Z,1684673778.550 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 98.960045 m 
2023-05-21T12:56:18.958Z,1684673778.958 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.056910 degC 
2023-05-21T12:56:19.342Z,1684673779.342 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 
2023-05-21T12:56:20.155Z,1684673780.155 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 18.000000 count completed.
2023-05-21T12:56:26.337Z,1684673786.337 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T12:56:36.686Z,1684673796.686 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T13:01:24.010Z,1684674084.010 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T13:49:05.290Z,1684676945.290 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-21T13:52:04.160Z,1684677124.160 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028548 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 19.000000 count 
2023-05-21T13:52:59.916Z,1684677179.916 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.611046 m , settling for 5.000000 min 
2023-05-21T13:58:12.733Z,1684677492.733 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T13:58:12.862Z,1684677492.862 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T13:59:04.410Z,1684677544.410 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T13:59:05.225Z,1684677545.225 [ESPComponent](IMPORTANT): [sample #19] Setting cartridge argument to -1
2023-05-21T13:59:05.226Z,1684677545.226 [ESPComponent](IMPORTANT): [sample #19] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T13:59:05.648Z,1684677545.648 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.795952 m in 9.188257 degC water with 0.025520 ug/l chlorophyll fluorescence.
2023-05-21T13:59:06.027Z,1684677546.027 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T13:59:06.429Z,1684677546.429 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T13:59:08.450Z,1684677548.450 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T13:59:40.367Z,1684677580.367 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PRIMING (@38,aRNAl_bac)
2023-05-21T13:59:43.198Z,1684677583.198 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T13:59:43.199Z,1684677583.199 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T13:59:43.603Z,1684677583.603 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T14:02:26.008Z,1684677746.008 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T14:02:26.411Z,1684677746.411 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T14:17:40.006Z,1684678660.006 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-21T14:33:50.528Z,1684679630.528 [ThrusterHE](FAULT): getSpeed error serial timeout
2023-05-21T14:33:50.529Z,1684679630.529 [ThrusterHE](FAULT): Thruster uart error: serial timeout
2023-05-21T14:47:32.139Z,1684680452.139 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T14:47:32.716Z,1684680452.716 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T14:47:34.482Z,1684680454.482 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T14:47:34.891Z,1684680454.891 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_WAITING_FOR_PROCESSED (@38,aRNAl_bac)
2023-05-21T14:49:10.245Z,1684680550.245 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_STOPPING (@38,aRNAl_bac)
2023-05-21T14:49:17.916Z,1684680557.916 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T14:49:18.324Z,1684680558.324 [ESPComponent](IMPORTANT): [sample #19] ESP sampling state: S_STOPPED
2023-05-21T14:54:18.083Z,1684680858.083 [ESPComponent](IMPORTANT): [sample #19] ESP sampling sequence completed normally.
2023-05-21T14:54:18.896Z,1684680858.896 [ESPComponent](FAULT): LCB 2 fault: LCB Watchdog Reset. Invalid Command. Hardware Overcurrent Shutdown. Current Limiter Activated.
2023-05-21T14:54:20.531Z,1684680860.531 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 19.000000 count 
2023-05-21T14:54:20.936Z,1684680860.936 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T14:54:21.353Z,1684680861.353 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T14:54:21.744Z,1684680861.744 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T14:54:22.146Z,1684680862.146 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.418831 m 
2023-05-21T14:54:22.555Z,1684680862.555 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.247889 degC 
2023-05-21T14:54:22.978Z,1684680862.978 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-21T14:54:23.755Z,1684680863.755 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 19.000000 count completed.
2023-05-21T14:54:29.970Z,1684680869.970 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T14:54:40.301Z,1684680880.301 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T14:54:55.433Z,1684680895.433 [DAT](FAULT): failed to enter command mode
2023-05-21T14:59:27.578Z,1684681167.578 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T15:52:06.985Z,1684684326.985 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.033211 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 20.000000 count 
2023-05-21T15:55:06.436Z,1684684506.436 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.616264 m , settling for 5.000000 min 
2023-05-21T16:00:19.161Z,1684684819.161 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T16:00:19.265Z,1684684819.265 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T16:01:10.779Z,1684684870.779 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T16:01:11.598Z,1684684871.598 [ESPComponent](IMPORTANT): [sample #20] Setting cartridge argument to -1
2023-05-21T16:01:11.598Z,1684684871.598 [ESPComponent](IMPORTANT): [sample #20] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T16:01:12.054Z,1684684872.054 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.440071 m in 9.274195 degC water with 0.032480 ug/l chlorophyll fluorescence.
2023-05-21T16:01:12.394Z,1684684872.394 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T16:01:12.797Z,1684684872.797 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T16:01:14.822Z,1684684874.822 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T16:01:46.755Z,1684684906.755 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PRIMING (@37,aRNAl_bac)
2023-05-21T16:01:49.587Z,1684684909.587 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T16:01:49.589Z,1684684909.589 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T16:01:49.981Z,1684684909.981 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T16:04:32.398Z,1684685072.398 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T16:04:32.799Z,1684685072.799 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T16:49:15.915Z,1684687755.915 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T16:49:16.313Z,1684687756.313 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T16:49:18.334Z,1684687758.334 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T16:49:18.738Z,1684687758.738 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_WAITING_FOR_PROCESSED (@37,aRNAl_bac)
2023-05-21T16:50:54.087Z,1684687854.087 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_STOPPING (@37,aRNAl_bac)
2023-05-21T16:51:01.758Z,1684687861.758 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T16:51:02.164Z,1684687862.164 [ESPComponent](IMPORTANT): [sample #20] ESP sampling state: S_STOPPED
2023-05-21T16:53:39.117Z,1684688019.117 [ThrusterHE](FAULT): getSpeed error serial timeout
2023-05-21T16:53:39.117Z,1684688019.117 [ThrusterHE](FAULT): Thruster uart error: serial timeout
2023-05-21T16:56:01.935Z,1684688161.935 [ESPComponent](IMPORTANT): [sample #20] ESP sampling sequence completed normally.
2023-05-21T16:56:04.455Z,1684688164.455 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 20.000000 count 
2023-05-21T16:56:04.865Z,1684688164.865 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T16:56:05.243Z,1684688165.243 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T16:56:05.655Z,1684688165.655 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T16:56:06.068Z,1684688166.068 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.018471 m 
2023-05-21T16:56:06.454Z,1684688166.454 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.166193 degC 
2023-05-21T16:56:06.859Z,1684688166.859 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-21T16:56:07.682Z,1684688167.682 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 20.000000 count completed.
2023-05-21T16:56:13.821Z,1684688173.821 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T16:56:24.154Z,1684688184.154 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T16:56:54.150Z,1684688214.150 [DAT](FAULT): failed to enter command mode
2023-05-21T17:01:11.495Z,1684688471.495 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T17:52:09.473Z,1684691529.473 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028198 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 21.000000 count 
2023-05-21T17:56:10.672Z,1684691770.672 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.701843 m , settling for 5.000000 min 
2023-05-21T18:01:23.441Z,1684692083.441 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T18:01:23.582Z,1684692083.582 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T18:02:15.535Z,1684692135.535 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T18:02:16.343Z,1684692136.343 [ESPComponent](IMPORTANT): [sample #21] Setting cartridge argument to -1
2023-05-21T18:02:16.344Z,1684692136.344 [ESPComponent](IMPORTANT): [sample #21] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T18:02:16.783Z,1684692136.783 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 99.907051 m in 9.225000 degC water with 0.026680 ug/l chlorophyll fluorescence.
2023-05-21T18:02:17.147Z,1684692137.147 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T18:02:17.551Z,1684692137.551 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T18:02:19.580Z,1684692139.580 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T18:02:51.489Z,1684692171.489 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PRIMING (@36,aRNAl_bac)
2023-05-21T18:02:53.528Z,1684692173.528 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T18:02:53.911Z,1684692173.911 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T18:02:54.315Z,1684692174.315 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T18:05:37.127Z,1684692337.127 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T18:05:37.535Z,1684692337.535 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T18:24:25.644Z,1684693465.644 [CBIT](IMPORTANT): Beginning ground fault scan
2023-05-21T18:24:36.612Z,1684693476.612 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002033
CHAN A1 (24V): -0.000123
CHAN A2 (12V): 0.000395
CHAN A3 (5V): -0.000490
CHAN B0 (3.3V): -0.000472
CHAN B1 (3.15aV): -0.000135
CHAN B2 (3.15bV): -0.000112
CHAN B3 (GND): -0.000369
OPEN: 0.007608
Full Scale: +/- 1 mA
2023-05-21T18:50:40.725Z,1684695040.725 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T18:50:41.133Z,1684695041.133 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T18:50:43.159Z,1684695043.159 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T18:50:43.554Z,1684695043.554 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_WAITING_FOR_PROCESSED (@36,aRNAl_bac)
2023-05-21T18:52:18.922Z,1684695138.922 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_STOPPING (@36,aRNAl_bac)
2023-05-21T18:52:26.607Z,1684695146.607 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T18:52:27.011Z,1684695147.011 [ESPComponent](IMPORTANT): [sample #21] ESP sampling state: S_STOPPED
2023-05-21T18:57:26.777Z,1684695446.777 [ESPComponent](IMPORTANT): [sample #21] ESP sampling sequence completed normally.
2023-05-21T18:57:29.227Z,1684695449.227 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 21.000000 count 
2023-05-21T18:57:29.627Z,1684695449.627 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T18:57:30.056Z,1684695450.056 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T18:57:30.468Z,1684695450.468 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T18:57:30.847Z,1684695450.847 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 100.336472 m 
2023-05-21T18:57:31.247Z,1684695451.247 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.183588 degC 
2023-05-21T18:57:31.654Z,1684695451.654 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-21T18:57:32.468Z,1684695452.468 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 21.000000 count completed.
2023-05-21T18:57:38.601Z,1684695458.601 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T18:57:48.933Z,1684695468.933 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T19:02:36.298Z,1684695756.298 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T19:52:12.067Z,1684698732.067 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.029582 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 22.000000 count 
2023-05-21T19:58:13.248Z,1684699093.248 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.728706 m , settling for 5.000000 min 
2023-05-21T20:03:26.026Z,1684699406.026 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T20:03:26.200Z,1684699406.200 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T20:04:18.159Z,1684699458.159 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T20:04:18.970Z,1684699458.970 [ESPComponent](IMPORTANT): [sample #22] Setting cartridge argument to -1
2023-05-21T20:04:18.970Z,1684699458.970 [ESPComponent](IMPORTANT): [sample #22] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T20:04:19.427Z,1684699459.427 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.292435 m in 9.114771 degC water with 0.024360 ug/l chlorophyll fluorescence.
2023-05-21T20:04:19.774Z,1684699459.774 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T20:04:20.175Z,1684699460.175 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T20:04:22.199Z,1684699462.199 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T20:04:54.115Z,1684699494.115 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PRIMING (@35,aRNAl_bac)
2023-05-21T20:04:56.543Z,1684699496.543 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T20:04:56.544Z,1684699496.544 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T20:04:56.943Z,1684699496.943 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T20:07:39.759Z,1684699659.759 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T20:07:40.163Z,1684699660.163 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T20:28:54.522Z,1684700934.522 [RudderServo](FAULT): Reading outside of valid range:0.000000

2023-05-21T20:28:55.213Z,1684700935.213 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout
2023-05-21T20:40:46.501Z,1684701646.501 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-21T20:51:57.834Z,1684702317.834 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T20:51:58.239Z,1684702318.239 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T20:52:00.268Z,1684702320.268 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T20:52:00.664Z,1684702320.664 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_WAITING_FOR_PROCESSED (@35,aRNAl_bac)
2023-05-21T20:53:36.010Z,1684702416.010 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_STOPPING (@35,aRNAl_bac)
2023-05-21T20:53:43.689Z,1684702423.689 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T20:53:44.099Z,1684702424.099 [ESPComponent](IMPORTANT): [sample #22] ESP sampling state: S_STOPPED
2023-05-21T20:58:43.867Z,1684702723.867 [ESPComponent](IMPORTANT): [sample #22] ESP sampling sequence completed normally.
2023-05-21T20:58:46.304Z,1684702726.304 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 22.000000 count 
2023-05-21T20:58:46.709Z,1684702726.709 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T20:58:47.186Z,1684702727.186 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T20:58:47.543Z,1684702727.543 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T20:58:47.972Z,1684702727.972 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.805618 m 
2023-05-21T20:58:48.346Z,1684702728.346 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.257288 degC 
2023-05-21T20:58:48.739Z,1684702728.739 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-21T20:58:49.569Z,1684702729.569 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 22.000000 count completed.
2023-05-21T20:58:55.642Z,1684702735.642 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T20:59:05.975Z,1684702745.975 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T20:59:44.053Z,1684702784.053 [DAT](FAULT): failed to enter command mode
2023-05-21T21:03:53.399Z,1684703033.399 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T21:52:14.709Z,1684705934.709 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.030607 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 23.000000 count 
2023-05-21T21:57:39.928Z,1684706259.928 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.883850 m , settling for 5.000000 min 
2023-05-21T22:02:52.725Z,1684706572.725 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-21T22:02:52.873Z,1684706572.873 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-21T22:03:45.530Z,1684706625.530 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-21T22:03:46.345Z,1684706626.345 [ESPComponent](IMPORTANT): [sample #23] Setting cartridge argument to -1
2023-05-21T22:03:46.346Z,1684706626.346 [ESPComponent](IMPORTANT): [sample #23] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-21T22:03:46.782Z,1684706626.782 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 98.800850 m in 9.411829 degC water with 0.035960 ug/l chlorophyll fluorescence.
2023-05-21T22:03:47.147Z,1684706627.147 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-21T22:03:47.551Z,1684706627.551 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-21T22:03:49.584Z,1684706629.584 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-21T22:04:21.487Z,1684706661.487 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PRIMING (@34,aRNAl_bac)
2023-05-21T22:04:24.338Z,1684706664.338 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PRIMING (@100.0)
2023-05-21T22:04:24.339Z,1684706664.339 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-21T22:04:24.720Z,1684706664.720 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-21T22:07:07.132Z,1684706827.132 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_FILTERING (@0.0)
2023-05-21T22:07:07.535Z,1684706827.535 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-21T22:26:09.521Z,1684707969.521 [ThrusterHE](FAULT): Could not get status:serial timeout
2023-05-21T22:51:15.955Z,1684709475.955 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PAUSED (@1000.0)
2023-05-21T22:51:16.357Z,1684709476.357 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-21T22:51:18.384Z,1684709478.384 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-21T22:51:18.807Z,1684709478.807 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_WAITING_FOR_PROCESSED (@34,aRNAl_bac)
2023-05-21T22:52:54.130Z,1684709574.130 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_STOPPING (@34,aRNAl_bac)
2023-05-21T22:53:01.814Z,1684709581.814 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-21T22:53:02.219Z,1684709582.219 [ESPComponent](IMPORTANT): [sample #23] ESP sampling state: S_STOPPED
2023-05-21T22:58:01.986Z,1684709881.986 [ESPComponent](IMPORTANT): [sample #23] ESP sampling sequence completed normally.
2023-05-21T22:58:04.471Z,1684709884.471 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 23.000000 count 
2023-05-21T22:58:04.850Z,1684709884.850 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-21T22:58:05.267Z,1684709885.267 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-21T22:58:05.663Z,1684709885.663 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-21T22:58:06.080Z,1684709886.080 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.396141 m 
2023-05-21T22:58:06.498Z,1684709886.498 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.315912 degC 
2023-05-21T22:58:06.870Z,1684709886.870 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.034800 ug/l 
2023-05-21T22:58:07.698Z,1684709887.698 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 23.000000 count completed.
2023-05-21T22:58:13.966Z,1684709893.966 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T22:58:24.305Z,1684709904.305 [DAT](FAULT): Sending transmit timeout failure.
2023-05-21T23:03:11.503Z,1684710191.503 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-21T23:52:17.236Z,1684713137.236 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:F](IMPORTANT): Elapsed time since start or last firing(= 120.028507 min ) is longer than MaxWaitFirstSample (= 0.000000 min ) for the 1st sample, or longer than MaxWaitMustFire (= 120.000000 min ) for non-1st samples. Firing sample No. 24.000000 count 
2023-05-21T23:55:55.803Z,1684713355.803 [circle_sample:SampleAtDepth:sample:SampleWrapper:B](IMPORTANT): At 99.702438 m , settling for 5.000000 min 
2023-05-22T00:01:08.595Z,1684713668.595 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.11.2:10.89.11.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2023-05-22T00:01:08.722Z,1684713668.722 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2023-05-22T00:01:59.870Z,1684713719.870 [ESPComponent](IMPORTANT): ESP has connected as client
2023-05-22T00:02:00.683Z,1684713720.683 [ESPComponent](IMPORTANT): [sample #24] Setting cartridge argument to -1
2023-05-22T00:02:00.683Z,1684713720.683 [ESPComponent](IMPORTANT): [sample #24] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2023-05-22T00:02:01.120Z,1684713721.120 [circle_sample:SampleAtDepth:sample:SampleWrapper:SampleESP:C](IMPORTANT): ESP sampling at 100.123199 m in 9.363367 degC water with 0.032480 ug/l chlorophyll fluorescence.
2023-05-22T00:02:01.481Z,1684713721.481 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PREPARING_SHOW_LOG
2023-05-22T00:02:01.887Z,1684713721.887 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PREPARING_SHOW_STATUS
2023-05-22T00:02:03.905Z,1684713723.905 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_LOADING_CARTRIDGE
2023-05-22T00:02:35.821Z,1684713755.821 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PRIMING (@33,aRNAl_bac)
2023-05-22T00:02:38.651Z,1684713758.651 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PRIMING (@100.0)
2023-05-22T00:02:38.673Z,1684713758.673 [ESPComponent](IMPORTANT): Filtering timeoutMinutes=92.070000 issuedCmd_=Cmd.startFiltering
2023-05-22T00:02:39.054Z,1684713759.054 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2023-05-22T00:05:21.870Z,1684713921.870 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_FILTERING (@0.0)
2023-05-22T00:05:22.289Z,1684713922.289 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2023-05-22T00:24:36.653Z,1684715076.653 [CBIT](IMPORTANT): Beginning ground fault scan
2023-05-22T00:24:47.584Z,1684715087.584 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.003110
CHAN A1 (24V): -0.000010
CHAN A2 (12V): 0.000504
CHAN A3 (5V): -0.000941
CHAN B0 (3.3V): 0.000380
CHAN B1 (3.15aV): 0.000289
CHAN B2 (3.15bV): 0.000492
CHAN B3 (GND): -0.000171
OPEN: 0.007596
Full Scale: +/- 1 mA
2023-05-22T00:51:48.386Z,1684716708.386 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PAUSED (@1000.0)
2023-05-22T00:51:48.790Z,1684716708.790 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_PROCESSING (@1000.0)
2023-05-22T00:51:50.811Z,1684716710.811 [ESPComponent](IMPORTANT): Processing timeoutMinutes=30.000000 issuedCmd_=Cmd.startProcessing
2023-05-22T00:51:51.220Z,1684716711.220 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_WAITING_FOR_PROCESSED (@33,aRNAl_bac)
2023-05-22T00:53:26.186Z,1684716806.186 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_STOPPING (@33,aRNAl_bac)
2023-05-22T00:53:33.848Z,1684716813.848 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq -e 'ssh -o StrictHostKeyChecking=no' --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr*' /LRAUV/ESPlogs &
2023-05-22T00:53:34.259Z,1684716814.259 [ESPComponent](IMPORTANT): [sample #24] ESP sampling state: S_STOPPED
2023-05-22T00:58:34.017Z,1684717114.017 [ESPComponent](IMPORTANT): [sample #24] ESP sampling sequence completed normally.
2023-05-22T00:58:34.826Z,1684717114.826 [ESPComponent](FAULT): LCB 2 fault:  Invalid Command. Current Limiter Activated.
2023-05-22T00:58:36.463Z,1684717116.463 [circle_sample:SendSampleInfo:SampleInfo:C](IMPORTANT): CntSamples = 24.000000 count 
2023-05-22T00:58:36.916Z,1684717116.916 [circle_sample:SendSampleInfo:SampleInfo:D](IMPORTANT): Science:ESPComponent.sample_number = 0 count 
2023-05-22T00:58:37.299Z,1684717117.299 [circle_sample:SendSampleInfo:SampleInfo:E](IMPORTANT): Science:ESPComponent.cartridge = -1 count 
2023-05-22T00:58:37.698Z,1684717117.698 [circle_sample:SendSampleInfo:SampleInfo:F](IMPORTANT): Science:ESPComponent.sampleVolume = nan ml 
2023-05-22T00:58:38.082Z,1684717118.082 [circle_sample:SendSampleInfo:SampleInfo:G](IMPORTANT): Universal:depth = 99.556023 m 
2023-05-22T00:58:38.490Z,1684717118.490 [circle_sample:SendSampleInfo:SampleInfo:H](IMPORTANT): Universal:sea_water_temperature = 9.207911 degC 
2023-05-22T00:58:38.898Z,1684717118.898 [circle_sample:SendSampleInfo:SampleInfo:I](IMPORTANT): Universal:mass_concentration_of_chlorophyll_in_sea_water = 0.023200 ug/l 
2023-05-22T00:58:39.689Z,1684717119.689 [circle_sample:CircleAndSample:Sampling:FiringOnMaxWaitOrNotReceivingOtherLRAUVSampleStartFirstSample:O](IMPORTANT): Sample No. 24.000000 count completed.
2023-05-22T00:58:45.888Z,1684717125.888 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T00:58:56.221Z,1684717136.221 [DAT](FAULT): Sending transmit timeout failure.
2023-05-22T01:03:43.537Z,1684717423.537 [circle_sample:SetSampleOptions:D](IMPORTANT): Set CartridgeType (if ESP), SettleDuration to -1.000000 count , 5.000000 min .
2023-05-22T01:51:11.036Z,1684720271.036 [circle_sample:CycleLogs:A:B](IMPORTANT): Restarting logs.
2023-05-22T01:51:11.522Z,1684720271.522 [CommandExec](IMPORTANT): got command restart logs