2021-04-21T01:25:32.530Z,1618968332.530 [CommandLine](IMPORTANT): got command restart logs
2021-04-21T01:25:53.421Z,1618968353.421 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=2740, MT Status=1, MTMSN=199
2021-04-21T01:25:56.030Z,1618968356.030 [CommandLine](IMPORTANT): got command configSet Express none WetLabsBB2FL.bin_mean_mass_concentration_of_chlorophyll_in_sea_water persist
2021-04-21T02:00:18.193Z,1618970418.193 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900
2021-04-21T02:03:05.045Z,1618970585.045 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-21T02:03:16.802Z,1618970596.802 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T02:03:16.902Z,1618970596.902 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T02:04:04.426Z,1618970644.426 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-21T02:04:05.250Z,1618970645.250 [ESPComponent](IMPORTANT): [sample #3] Setting cartridge argument to -6
2021-04-21T02:04:05.251Z,1618970645.251 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-21T02:04:05.655Z,1618970645.655 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 21.729446 m in 10.606897 degC water with 7.702400 ug/l chlorophyll fluorescence.
2021-04-21T02:04:06.033Z,1618970646.033 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-21T02:04:06.440Z,1618970646.440 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-21T02:04:08.461Z,1618970648.461 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-21T02:04:42.809Z,1618970682.809 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PRIMING (@10,archive_bac)
2021-04-21T02:04:44.415Z,1618970684.415 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PRIMING (@100.0)
2021-04-21T02:04:45.226Z,1618970685.226 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-21T02:07:17.129Z,1618970837.129 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_FILTERING (@0.0)
2021-04-21T02:07:17.527Z,1618970837.527 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-21T03:07:49.127Z,1618974469.127 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PAUSED (@1000.0)
2021-04-21T03:07:49.549Z,1618974469.549 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_PROCESSING (@1000.0)
2021-04-21T03:07:51.964Z,1618974471.964 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_WAITING_FOR_PROCESSED (@10,archive_bac)
2021-04-21T03:19:55.531Z,1618975195.531 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPING (@10,archive_bac)
2021-04-21T03:20:08.071Z,1618975208.071 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-21T03:20:08.492Z,1618975208.492 [ESPComponent](IMPORTANT): [sample #3] ESP sampling state: S_STOPPED
2021-04-21T03:25:08.228Z,1618975508.228 [ESPComponent](IMPORTANT): [sample #3] ESP sampling sequence completed normally.
2021-04-21T03:25:08.631Z,1618975508.631 [ESPComponent](IMPORTANT): [sample #3] ESP log summary report (2 messages):
@19:04:18.42 Selecting Cartridge 10
@20:07:47.29 Sampled  1000.0ml

2021-04-21T03:31:53.776Z,1618975913.776 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap
2021-04-21T03:31:54.246Z,1618975914.246 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 
2021-04-21T03:31:54.587Z,1618975914.587 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 
2021-04-21T03:31:55.003Z,1618975915.003 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 
2021-04-21T03:31:55.433Z,1618975915.433 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 
2021-04-21T03:31:59.858Z,1618975919.858 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 
2021-04-21T05:02:09.308Z,1618981329.308 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000
2021-04-21T05:08:30.103Z,1618981710.103 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-21T05:08:41.830Z,1618981721.830 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T05:08:41.943Z,1618981721.943 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T05:09:29.849Z,1618981769.849 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-21T05:09:30.667Z,1618981770.667 [ESPComponent](IMPORTANT): [sample #4] Setting cartridge argument to -6
2021-04-21T05:09:30.667Z,1618981770.667 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-21T05:09:31.083Z,1618981771.083 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 16.407915 m in 11.268488 degC water with 12.504800 ug/l chlorophyll fluorescence.
2021-04-21T05:09:31.457Z,1618981771.457 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-21T05:09:31.855Z,1618981771.855 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-21T05:09:33.872Z,1618981773.872 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-21T05:10:08.213Z,1618981808.213 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PRIMING (@9,archive_bac)
2021-04-21T05:10:09.840Z,1618981809.840 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PRIMING (@100.0)
2021-04-21T05:10:10.637Z,1618981810.637 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-21T05:12:42.547Z,1618981962.547 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_FILTERING (@0.0)
2021-04-21T05:12:42.977Z,1618981962.977 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-21T05:55:14.318Z,1618984514.318 [CBIT](IMPORTANT): Beginning ground fault scan
2021-04-21T05:55:25.221Z,1618984525.221 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): 0.000942
CHAN A1 (24V): -0.001084
CHAN A2 (12V): -0.000460
CHAN A3 (5V): -0.006620
CHAN B0 (3.3V): -0.000002
CHAN B1 (3.15aV): -0.000028
CHAN B2 (3.15bV): 0.001725
CHAN B3 (GND): -0.001120
OPEN: 0.007683
Full Scale Calc: 4.765 mA, -1.589 mA
2021-04-21T06:02:31.776Z,1618984951.776 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PAUSED (@800.9)
2021-04-21T06:02:32.179Z,1618984952.179 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_PROCESSING (@800.9)
2021-04-21T06:02:34.603Z,1618984954.603 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_WAITING_FOR_PROCESSED (@9,archive_bac)
2021-04-21T06:14:38.241Z,1618985678.241 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPING (@9,archive_bac)
2021-04-21T06:14:50.694Z,1618985690.694 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-21T06:14:51.097Z,1618985691.097 [ESPComponent](IMPORTANT): [sample #4] ESP sampling state: S_STOPPED
2021-04-21T06:19:50.881Z,1618985990.881 [ESPComponent](IMPORTANT): [sample #4] ESP sampling sequence completed normally.
2021-04-21T06:19:51.268Z,1618985991.268 [ESPComponent](IMPORTANT): [sample #4] ESP log summary report (3 messages):
@22:09:43.67 Selecting Cartridge 9
@23:02:28.95 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged
@23:02:29.14 Sampled  800.9ml

2021-04-21T06:25:56.925Z,1618986356.925 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 
2021-04-21T07:44:46.659Z,1618991086.659 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600
2021-04-21T07:50:22.094Z,1618991422.094 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-21T07:50:33.890Z,1618991433.890 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T07:50:34.010Z,1618991434.010 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T07:51:21.456Z,1618991481.456 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-21T07:51:22.272Z,1618991482.272 [ESPComponent](IMPORTANT): [sample #5] Setting cartridge argument to -6
2021-04-21T07:51:22.273Z,1618991482.273 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-21T07:51:22.765Z,1618991482.765 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 21.141191 m in 10.856989 degC water with 7.690800 ug/l chlorophyll fluorescence.
2021-04-21T07:51:23.081Z,1618991483.081 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-21T07:51:23.481Z,1618991483.481 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-21T07:51:25.509Z,1618991485.509 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-21T07:51:59.842Z,1618991519.842 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PRIMING (@8,archive_bac)
2021-04-21T07:52:01.457Z,1618991521.457 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PRIMING (@100.0)
2021-04-21T07:52:02.261Z,1618991522.261 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-21T07:54:34.595Z,1618991674.595 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_FILTERING (@0.0)
2021-04-21T07:54:34.983Z,1618991674.983 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-21T08:46:31.085Z,1618994791.085 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PAUSED (@1000.0)
2021-04-21T08:46:31.495Z,1618994791.495 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_PROCESSING (@1000.0)
2021-04-21T08:46:33.909Z,1618994793.909 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_WAITING_FOR_PROCESSED (@8,archive_bac)
2021-04-21T08:58:37.485Z,1618995517.485 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPING (@8,archive_bac)
2021-04-21T08:58:50.004Z,1618995530.004 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-21T08:58:50.401Z,1618995530.401 [ESPComponent](IMPORTANT): [sample #5] ESP sampling state: S_STOPPED
2021-04-21T09:03:50.175Z,1618995830.175 [ESPComponent](IMPORTANT): [sample #5] ESP sampling sequence completed normally.
2021-04-21T09:03:50.570Z,1618995830.570 [ESPComponent](IMPORTANT): [sample #5] ESP log summary report (2 messages):
@00:51:35.41 Selecting Cartridge 8
@01:46:29.16 Sampled  1000.0ml

2021-04-21T09:09:38.857Z,1618996178.857 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 
2021-04-21T11:55:25.568Z,1619006125.568 [CBIT](IMPORTANT): Beginning ground fault scan
2021-04-21T11:55:36.476Z,1619006136.476 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.009864
CHAN A1 (24V): -0.001185
CHAN A2 (12V): -0.000520
CHAN A3 (5V): -0.000195
CHAN B0 (3.3V): -0.001474
CHAN B1 (3.15aV): 0.000083
CHAN B2 (3.15bV): 0.000019
CHAN B3 (GND): 0.000638
OPEN: 0.007928
Full Scale Calc: 4.765 mA, -1.589 mA
2021-04-21T12:00:40.630Z,1619006440.630 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900
2021-04-21T12:04:06.359Z,1619006646.359 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-21T12:04:18.159Z,1619006658.159 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T12:04:18.270Z,1619006658.270 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T12:06:48.361Z,1619006808.361 [ESPComponent](FAULT): Timeout while waiting for ESP to connect (elapsed=150.0910)
2021-04-21T12:07:00.182Z,1619006820.182 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T12:07:00.312Z,1619006820.312 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T12:09:30.362Z,1619006970.362 [ESPComponent](FAULT): Timeout while waiting for ESP to connect (elapsed=150.0510)
2021-04-21T12:09:42.195Z,1619006982.195 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T12:09:42.314Z,1619006982.314 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T12:10:29.389Z,1619007029.389 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-21T12:10:30.199Z,1619007030.199 [ESPComponent](IMPORTANT): [sample #6] Setting cartridge argument to -6
2021-04-21T12:10:30.199Z,1619007030.199 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-21T12:10:30.631Z,1619007030.631 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 14.173707 m in 10.284296 degC water with 5.243200 ug/l chlorophyll fluorescence.
2021-04-21T12:10:31.019Z,1619007031.019 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-21T12:10:31.410Z,1619007031.410 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-21T12:10:33.444Z,1619007033.444 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-21T12:11:07.817Z,1619007067.817 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PRIMING (@7,archive_bac)
2021-04-21T12:11:09.383Z,1619007069.383 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PRIMING (@100.0)
2021-04-21T12:11:10.190Z,1619007070.190 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-21T12:13:43.307Z,1619007223.307 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_FILTERING (@0.0)
2021-04-21T12:13:43.711Z,1619007223.711 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-21T13:06:05.649Z,1619010365.649 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PAUSED (@1000.0)
2021-04-21T13:06:06.069Z,1619010366.069 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_PROCESSING (@1000.0)
2021-04-21T13:06:08.496Z,1619010368.496 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_WAITING_FOR_PROCESSED (@7,archive_bac)
2021-04-21T13:15:22.763Z,1619010922.763 [DVL_micro](FAULT): Failed to initialize within timeout.
2021-04-21T13:18:12.068Z,1619011092.068 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPING (@7,archive_bac)
2021-04-21T13:18:24.575Z,1619011104.575 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-21T13:18:24.970Z,1619011104.970 [ESPComponent](IMPORTANT): [sample #6] ESP sampling state: S_STOPPED
2021-04-21T13:23:24.741Z,1619011404.741 [ESPComponent](IMPORTANT): [sample #6] ESP sampling sequence completed normally.
2021-04-21T13:23:25.156Z,1619011405.156 [ESPComponent](IMPORTANT): [sample #6] ESP log summary report (2 messages):
@05:10:42.98 Selecting Cartridge 7
@06:06:03.66 Sampled  1000.0ml

2021-04-21T13:27:47.596Z,1619011667.596 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap
2021-04-21T13:27:48.001Z,1619011668.001 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 
2021-04-21T13:27:48.418Z,1619011668.418 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 
2021-04-21T13:27:48.801Z,1619011668.801 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 
2021-04-21T13:27:49.205Z,1619011669.205 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 
2021-04-21T13:27:53.655Z,1619011673.655 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 
2021-04-21T14:11:07.864Z,1619014267.864 [ThrusterServo](FAULT): Invalid EZ Servo response:"/0@5242"
2021-04-21T14:40:29.983Z,1619016029.983 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000
2021-04-21T14:45:44.157Z,1619016344.157 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-21T14:45:55.978Z,1619016355.978 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T14:45:56.176Z,1619016356.176 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T14:48:26.397Z,1619016506.397 [ESPComponent](FAULT): Timeout while waiting for ESP to connect (elapsed=150.2209)
2021-04-21T14:48:38.262Z,1619016518.262 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T14:48:38.378Z,1619016518.378 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T14:49:25.447Z,1619016565.447 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-21T14:49:26.266Z,1619016566.266 [ESPComponent](IMPORTANT): [sample #7] Setting cartridge argument to -6
2021-04-21T14:49:26.267Z,1619016566.267 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-21T14:49:26.675Z,1619016566.675 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 44.228939 m in 9.212214 degC water with 0.406000 ug/l chlorophyll fluorescence.
2021-04-21T14:49:27.065Z,1619016567.065 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-21T14:49:27.463Z,1619016567.463 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-21T14:49:29.488Z,1619016569.488 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-21T14:50:03.829Z,1619016603.829 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PRIMING (@6,archive_bac)
2021-04-21T14:50:05.449Z,1619016605.449 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PRIMING (@100.0)
2021-04-21T14:50:06.257Z,1619016606.257 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-21T14:52:51.902Z,1619016771.902 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_FILTERING (@0.0)
2021-04-21T14:52:52.297Z,1619016772.297 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-21T15:02:21.606Z,1619017341.606 [ThrusterServo](FAULT): Invalid EZ Servo response:"/0@5242"
2021-04-21T15:50:20.495Z,1619020220.495 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PAUSED (@927.7)
2021-04-21T15:50:20.898Z,1619020220.898 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_PROCESSING (@927.7)
2021-04-21T15:50:23.317Z,1619020223.317 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_WAITING_FOR_PROCESSED (@6,archive_bac)
2021-04-21T16:02:26.882Z,1619020946.882 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPING (@6,archive_bac)
2021-04-21T16:02:39.830Z,1619020959.830 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-21T16:02:40.214Z,1619020960.214 [ESPComponent](IMPORTANT): [sample #7] ESP sampling state: S_STOPPED
2021-04-21T16:07:39.979Z,1619021259.979 [ESPComponent](IMPORTANT): [sample #7] ESP sampling sequence completed normally.
2021-04-21T16:07:40.386Z,1619021260.386 [ESPComponent](IMPORTANT): [sample #7] ESP log summary report (3 messages):
@07:49:39.13 Selecting Cartridge 6
@08:50:17.49 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged
@08:50:17.68 Sampled  927.7ml

2021-04-21T16:13:22.575Z,1619021602.575 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 
2021-04-21T16:58:11.134Z,1619024291.134 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout
2021-04-21T17:35:07.381Z,1619026507.381 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600
2021-04-21T17:41:21.117Z,1619026881.117 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-21T17:41:32.919Z,1619026892.919 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T17:41:33.020Z,1619026893.020 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T17:42:20.485Z,1619026940.485 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-21T17:42:21.308Z,1619026941.308 [ESPComponent](IMPORTANT): [sample #8] Setting cartridge argument to -6
2021-04-21T17:42:21.309Z,1619026941.309 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-21T17:42:21.718Z,1619026941.718 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 21.718533 m in 10.402399 degC water with 6.635200 ug/l chlorophyll fluorescence.
2021-04-21T17:42:22.094Z,1619026942.094 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-21T17:42:22.517Z,1619026942.517 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-21T17:42:24.524Z,1619026944.524 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-21T17:42:58.876Z,1619026978.876 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PRIMING (@5,archive_bac)
2021-04-21T17:43:00.486Z,1619026980.486 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PRIMING (@100.0)
2021-04-21T17:43:01.291Z,1619026981.291 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-21T17:45:33.200Z,1619027133.200 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_FILTERING (@0.0)
2021-04-21T17:45:33.605Z,1619027133.605 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-21T17:55:36.862Z,1619027736.862 [CBIT](IMPORTANT): Beginning ground fault scan
2021-04-21T17:55:48.217Z,1619027748.217 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.004178
CHAN A1 (24V): -0.000779
CHAN A2 (12V): -0.001537
CHAN A3 (5V): -0.000449
CHAN B0 (3.3V): 0.000434
CHAN B1 (3.15aV): 0.000160
CHAN B2 (3.15bV): 0.000831
CHAN B3 (GND): -0.000055
OPEN: 0.007565
Full Scale Calc: 4.765 mA, -1.589 mA
2021-04-21T18:06:13.587Z,1619028373.587 [ElevatorServo](FAULT): Reading outside of valid range:0.000000

2021-04-21T18:06:14.226Z,1619028374.226 [ElevatorServo](FAULT): Elevator uart error - getPosition.serial timeout
2021-04-21T18:31:34.977Z,1619029894.977 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PAUSED (@725.6)
2021-04-21T18:31:35.397Z,1619029895.397 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_PROCESSING (@725.6)
2021-04-21T18:31:37.803Z,1619029897.803 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_WAITING_FOR_PROCESSED (@5,archive_bac)
2021-04-21T18:43:41.370Z,1619030621.370 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPING (@5,archive_bac)
2021-04-21T18:43:53.105Z,1619030633.105 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-21T18:43:53.489Z,1619030633.489 [ESPComponent](IMPORTANT): [sample #8] ESP sampling state: S_STOPPED
2021-04-21T18:48:53.262Z,1619030933.262 [ESPComponent](IMPORTANT): [sample #8] ESP sampling sequence completed normally.
2021-04-21T18:48:53.677Z,1619030933.677 [ESPComponent](IMPORTANT): [sample #8] ESP log summary report (4 messages):
@10:42:34.22 Selecting Cartridge 5
@11:31:32.33 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged
@11:31:32.52 Sampled  725.6ml
@11:43:43.34 Slide::Error in LRAUV -- Archive Syringe (ArS) positionErr at 212uL (sensed 189uL)

2021-04-21T19:01:33.112Z,1619031693.112 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 
2021-04-21T20:47:16.507Z,1619038036.507 [DataOverHttps](FAULT): Server acknowledged file Logs/20210421T012532/Express0067.lzma, not
2021-04-21T21:50:08.004Z,1619041808.004 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900
2021-04-21T21:56:52.091Z,1619042212.091 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-21T21:57:03.882Z,1619042223.882 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-21T21:57:04.009Z,1619042224.009 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-21T21:57:51.508Z,1619042271.508 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-21T21:57:52.321Z,1619042272.321 [ESPComponent](IMPORTANT): [sample #9] Setting cartridge argument to -6
2021-04-21T21:57:52.322Z,1619042272.322 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-21T21:57:52.764Z,1619042272.764 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 18.506062 m in 10.054590 degC water with 3.793200 ug/l chlorophyll fluorescence.
2021-04-21T21:57:53.123Z,1619042273.123 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-21T21:57:53.534Z,1619042273.534 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-21T21:57:55.548Z,1619042275.548 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-21T21:58:29.887Z,1619042309.887 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PRIMING (@4,archive_bac)
2021-04-21T21:58:31.505Z,1619042311.505 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PRIMING (@100.0)
2021-04-21T21:58:32.332Z,1619042312.332 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-21T22:01:04.635Z,1619042464.635 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_FILTERING (@0.0)
2021-04-21T22:01:05.049Z,1619042465.049 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-21T22:59:18.876Z,1619045958.876 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PAUSED (@1000.0)
2021-04-21T22:59:19.260Z,1619045959.260 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_PROCESSING (@1000.0)
2021-04-21T22:59:21.685Z,1619045961.685 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_WAITING_FOR_PROCESSED (@4,archive_bac)
2021-04-21T23:11:25.273Z,1619046685.273 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPING (@4,archive_bac)
2021-04-21T23:11:37.798Z,1619046697.798 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-21T23:11:38.213Z,1619046698.213 [ESPComponent](IMPORTANT): [sample #9] ESP sampling state: S_STOPPED
2021-04-21T23:16:37.986Z,1619046997.986 [ESPComponent](IMPORTANT): [sample #9] ESP sampling sequence completed normally.
2021-04-21T23:16:38.361Z,1619046998.361 [ESPComponent](IMPORTANT): [sample #9] ESP log summary report (2 messages):
@14:58:07.74 Selecting Cartridge 4
@15:59:16.98 Sampled  1000.0ml

2021-04-21T23:22:43.926Z,1619047363.926 [esp_sample_station:Lap:A](IMPORTANT): Starting Lap
2021-04-21T23:22:44.329Z,1619047364.329 [esp_sample_station:Lap:B](IMPORTANT): Lat[1] = 36.797000 arcdeg Lon[1] = -121.847000 arcdeg 
2021-04-21T23:22:44.763Z,1619047364.763 [esp_sample_station:Lap:C](IMPORTANT): Lat[2] = 36.821900 arcdeg Lon[2] = -121.885600 arcdeg 
2021-04-21T23:22:45.150Z,1619047365.150 [esp_sample_station:Lap:D](IMPORTANT): Lat[3] = 36.766000 arcdeg Lon[3] = -121.864900 arcdeg 
2021-04-21T23:22:45.564Z,1619047365.564 [esp_sample_station:Lap:E](IMPORTANT): Lat[4] = nan arcdeg Lon[4] = nan arcdeg 
2021-04-21T23:22:50.001Z,1619047370.001 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.797000 arcdeg , -121.847000 arcdeg 
2021-04-21T23:55:48.534Z,1619049348.534 [CBIT](IMPORTANT): Beginning ground fault scan
2021-04-21T23:55:59.517Z,1619049359.517 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002907
CHAN A1 (24V): 0.013023
CHAN A2 (12V): -0.000018
CHAN A3 (5V): -0.001688
CHAN B0 (3.3V): 0.005540
CHAN B1 (3.15aV): 0.000154
CHAN B2 (3.15bV): -0.000411
CHAN B3 (GND): 0.000187
OPEN: 0.007648
Full Scale Calc: 4.765 mA, -1.589 mA
2021-04-22T00:32:37.313Z,1619051557.313 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.797000,-121.847000
2021-04-22T00:37:46.074Z,1619051866.074 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-22T00:37:57.854Z,1619051877.854 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-22T00:37:57.990Z,1619051877.990 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-22T00:38:45.429Z,1619051925.429 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-22T00:38:46.262Z,1619051926.262 [ESPComponent](IMPORTANT): [sample #10] Setting cartridge argument to -6
2021-04-22T00:38:46.262Z,1619051926.262 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-22T00:38:46.648Z,1619051926.648 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 19.579899 m in 9.740808 degC water with 1.241200 ug/l chlorophyll fluorescence.
2021-04-22T00:38:47.045Z,1619051927.045 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-22T00:38:47.445Z,1619051927.445 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-22T00:38:49.465Z,1619051929.465 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-22T00:39:23.799Z,1619051963.799 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PRIMING (@3,archive_bac)
2021-04-22T00:39:25.415Z,1619051965.415 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PRIMING (@100.0)
2021-04-22T00:39:26.225Z,1619051966.225 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-22T00:41:58.972Z,1619052118.972 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_FILTERING (@0.0)
2021-04-22T00:41:59.342Z,1619052119.342 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-22T01:06:43.667Z,1619053603.667 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PAUSED (@476.6)
2021-04-22T01:06:44.068Z,1619053604.068 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_PROCESSING (@476.6)
2021-04-22T01:06:46.505Z,1619053606.505 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_WAITING_FOR_PROCESSED (@3,archive_bac)
2021-04-22T01:18:50.078Z,1619054330.078 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPING (@3,archive_bac)
2021-04-22T01:19:02.594Z,1619054342.594 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-22T01:19:03.022Z,1619054343.022 [ESPComponent](IMPORTANT): [sample #10] ESP sampling state: S_STOPPED
2021-04-22T01:24:02.778Z,1619054642.778 [ESPComponent](IMPORTANT): [sample #10] ESP sampling sequence completed normally.
2021-04-22T01:24:03.176Z,1619054643.176 [ESPComponent](IMPORTANT): [sample #10] ESP log summary report (3 messages):
@17:38:59.27 Selecting Cartridge 3
@18:06:40.82 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged
@18:06:41.01 Sampled  476.6ml

2021-04-22T01:26:29.024Z,1619054789.024 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=2917, MT Status=1, MTMSN=200
2021-04-22T01:26:29.618Z,1619054789.618 [CommandLine](IMPORTANT): got command ! onESPclient preston -- Cartridge.release 1
2021-04-22T01:26:30.127Z,1619054790.127 [CommandLine](IMPORTANT): /mnt/mmc/started/2021-04-22UTC012630_onESPclient
  
2021-04-22T01:26:40.947Z,1619054800.947 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=2918, MT Status=1, MTMSN=201
2021-04-22T01:26:41.454Z,1619054801.454 [CommandLine](IMPORTANT): got command ! check
2021-04-22T01:26:41.721Z,1619054801.721 [CommandLine](IMPORTANT): Waiting for ESP to boot... 
2021-04-22T01:29:27.763Z,1619054967.763 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.821900 arcdeg , -121.885600 arcdeg 
2021-04-22T03:00:57.571Z,1619060457.571 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.821900,-121.885600
2021-04-22T03:04:01.783Z,1619060641.783 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=2932, MT Status=1, MTMSN=202
2021-04-22T03:04:02.375Z,1619060642.375 [CommandLine](IMPORTANT): got command ! check
2021-04-22T03:04:02.531Z,1619060642.531 [CommandLine](IMPORTANT): Waiting for ESP to boot... Connected.
  Thu Apr 22 01:27:14 UTC 2021 -- ssh esp@10.89.11.3 espclient preston
  1
  shutting ESP off
  
2021-04-22T03:07:48.355Z,1619060868.355 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-22T03:08:00.106Z,1619060880.106 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-22T03:08:00.211Z,1619060880.211 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-22T03:08:47.691Z,1619060927.691 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-22T03:08:48.509Z,1619060928.509 [ESPComponent](IMPORTANT): [sample #11] Setting cartridge argument to -6
2021-04-22T03:08:48.509Z,1619060928.509 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-22T03:08:48.936Z,1619060928.936 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 19.133879 m in 10.552912 degC water with 5.498400 ug/l chlorophyll fluorescence.
2021-04-22T03:08:49.316Z,1619060929.316 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-22T03:08:49.724Z,1619060929.724 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-22T03:08:51.746Z,1619060931.746 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-22T03:09:26.087Z,1619060966.087 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PRIMING (@2,archive_bac)
2021-04-22T03:09:27.695Z,1619060967.695 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PRIMING (@100.0)
2021-04-22T03:09:28.492Z,1619060968.492 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-22T03:12:04.041Z,1619061124.041 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_FILTERING (@0.0)
2021-04-22T03:12:04.465Z,1619061124.465 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-22T03:25:47.492Z,1619061947.492 [ThrusterServo](FAULT): Invalid EZ Servo response:"/0@5242"
2021-04-22T03:53:21.811Z,1619063601.811 [DVL_micro](FAULT): Failed to initialize within timeout.
2021-04-22T04:13:15.648Z,1619064795.648 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PAUSED (@1000.0)
2021-04-22T04:13:16.057Z,1619064796.057 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_PROCESSING (@1000.0)
2021-04-22T04:13:18.481Z,1619064798.481 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_WAITING_FOR_PROCESSED (@2,archive_bac)
2021-04-22T04:25:22.048Z,1619065522.048 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPING (@2,archive_bac)
2021-04-22T04:25:34.571Z,1619065534.571 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-22T04:25:34.999Z,1619065534.999 [ESPComponent](IMPORTANT): [sample #11] ESP sampling state: S_STOPPED
2021-04-22T04:30:34.753Z,1619065834.753 [ESPComponent](IMPORTANT): [sample #11] ESP sampling sequence completed normally.
2021-04-22T04:30:35.162Z,1619065835.162 [ESPComponent](IMPORTANT): [sample #11] ESP log summary report (2 messages):
@20:09:01.53 Selecting Cartridge 2
@21:13:13.76 Sampled  1000.0ml

2021-04-22T04:33:00.189Z,1619065980.189 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=2943, MT Status=1, MTMSN=203
2021-04-22T04:33:00.782Z,1619065980.782 [CommandLine](IMPORTANT): got command schedule clear
2021-04-22T04:33:00.782Z,1619065980.782 [CommandLine](IMPORTANT): #Cleared scheduled commands:0
2021-04-22T04:33:00.782Z,1619065980.782 [CommandLine](IMPORTANT): got command schedule resume
2021-04-22T04:33:00.782Z,1619065980.782 [CommandLine](IMPORTANT): Scheduling is resumed
2021-04-22T04:35:03.096Z,1619066103.096 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:C](IMPORTANT): Navigating to WP 36.766000 arcdeg , -121.864900 arcdeg 
2021-04-22T05:25:58.622Z,1619069158.622 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=2944, MT Status=1, MTMSN=204
2021-04-22T05:25:59.207Z,1619069159.207 [CommandLine](IMPORTANT): got command schedule resume
2021-04-22T05:26:09.985Z,1619069169.985 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=2945, MT Status=1, MTMSN=205
2021-04-22T05:26:10.694Z,1619069170.694 [CommandLine](IMPORTANT): got command schedule "load Science/profile_station.xml;set profile_station.MissionTimeout 12 hour;set profile_station.Lat 36.797 degree;set profile_station.Lon -121.847 degree;set profile_station.YoYoMaxDepth 50 meter;run"
2021-04-22T05:26:10.700Z,1619069170.700 [CommandLine](IMPORTANT): Scheduled #1: "load Science/profile_station.xml;set profile_station.MissionTimeout 12 hour;set profile_station.Lat 36.797 degree;set profile_station.Lon -121.847 degree;set profile_station.YoYoMaxDepth 50 meter;run"
2021-04-22T05:55:59.778Z,1619070959.778 [CBIT](IMPORTANT): Beginning ground fault scan
2021-04-22T05:56:10.685Z,1619070970.685 [CBIT](IMPORTANT): No ground fault detected
mA:
CHAN A0 (Batt): -0.002851
CHAN A1 (24V): -0.000997
CHAN A2 (12V): 0.000231
CHAN A3 (5V): -0.000345
CHAN B0 (3.3V): 0.000221
CHAN B1 (3.15aV): 0.000683
CHAN B2 (3.15bV): -0.000700
CHAN B3 (GND): 0.000255
OPEN: 0.007766
Full Scale Calc: 4.765 mA, -1.589 mA
2021-04-22T05:59:08.730Z,1619071148.730 [RudderServo](FAULT): Rudder uart error - getPosition..serial timeout
2021-04-22T06:21:29.804Z,1619072489.804 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2021-04-22T07:03:54.327Z,1619075034.327 [esp_sample_station:Lap:WaypointTrajectory:Station:GoToStation:D.Waypoint](IMPORTANT): Reached Waypoint: 36.766000,-121.864900
2021-04-22T07:08:49.081Z,1619075329.081 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:SelectESPCartridge:A](IMPORTANT): Requesting ESP cartridge type: -6.000000 count 
2021-04-22T07:09:00.890Z,1619075340.890 [ESPComponent](IMPORTANT): Starting PPPD with command:/sbin/pppd xonxoff asyncmap A0000 /dev/ttyTX1 linkname esp noauth local lock 115200 10.89.5.2:10.89.5.3 persist maxfail 0 holdoff 10 lcp-echo-interval 60 lcp-echo-failure 3 proxyarp ktune deflate 12 ms-dns 8.8.8.8
2021-04-22T07:09:01.006Z,1619075341.006 [ESPComponent](IMPORTANT): Waiting for ESP to connect (timeout=150.0000)
2021-04-22T07:09:48.524Z,1619075388.524 [ESPComponent](IMPORTANT): ESP has connected as client
2021-04-22T07:09:49.344Z,1619075389.344 [ESPComponent](IMPORTANT): [sample #12] Setting cartridge argument to -6
2021-04-22T07:09:49.344Z,1619075389.344 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence starting. Sampling state: S_WAITING_INITIAL_PROMPT
2021-04-22T07:09:49.778Z,1619075389.778 [esp_sample_station:Sample:SampleWrapper:ESP:AcquireESP:C](IMPORTANT): ESP sampling at 23.731989 m in 10.656610 degC water with 7.331200 ug/l chlorophyll fluorescence.
2021-04-22T07:09:50.141Z,1619075390.141 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_LOG
2021-04-22T07:09:50.549Z,1619075390.549 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PREPARING_SHOW_STATUS
2021-04-22T07:09:52.584Z,1619075392.584 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_LOADING_CARTRIDGE
2021-04-22T07:10:26.909Z,1619075426.909 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PRIMING (@1,archive_bac)
2021-04-22T07:10:28.525Z,1619075428.525 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PRIMING (@100.0)
2021-04-22T07:10:29.347Z,1619075429.347 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_FILTERING (@100.0)
2021-04-22T07:13:02.053Z,1619075582.053 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_FILTERING (@0.0)
2021-04-22T07:13:02.454Z,1619075582.454 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PAUSED (@0.0)
2021-04-22T08:10:53.668Z,1619079053.668 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PAUSED (@931.7)
2021-04-22T08:10:54.095Z,1619079054.095 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_PROCESSING (@931.7)
2021-04-22T08:10:56.506Z,1619079056.506 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_WAITING_FOR_PROCESSED (@1,archive_bac)
2021-04-22T08:20:55.628Z,1619079655.628 [DVL_micro](FAULT): Failed to initialize within timeout.
2021-04-22T08:23:00.055Z,1619079780.055 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPING (@1,archive_bac)
2021-04-22T08:23:12.590Z,1619079792.590 [ESPComponent](IMPORTANT): Syncing ESP logs with command: rsync -azPq --timeout=60 esp@esp:'/var/log/esp/real /var/log/esp/real.out /var/log/esp/real.slot /var/log/esp/*.spr' /LRAUV/ESPlogs &
2021-04-22T08:23:13.012Z,1619079793.012 [ESPComponent](IMPORTANT): [sample #12] ESP sampling state: S_STOPPED
2021-04-22T08:28:12.771Z,1619080092.771 [ESPComponent](IMPORTANT): [sample #12] ESP sampling sequence completed normally.
2021-04-22T08:28:13.173Z,1619080093.173 [ESPComponent](IMPORTANT): [sample #12] ESP log summary report (3 messages):
@00:10:02.42 Selecting Cartridge 1
@01:10:50.83 Cartridge::Sampler::Clogged in FILTERING -- Cartridge clogged
@01:10:51.02 Sampled  931.7ml

2021-04-22T08:33:37.394Z,1619080417.394 [CommandLine](IMPORTANT): got command load ./Missions/Science/profile_station.xml
2021-04-22T08:33:39.271Z,1619080419.271 [CommandLine](IMPORTANT): Loaded ./Missions/Science/profile_station.xml
2021-04-22T08:33:39.662Z,1619080419.662 [MissionManager](IMPORTANT): Started mission Default
2021-04-22T08:33:50.743Z,1619080430.743 [CommandLine](IMPORTANT): got command set profile_station.MissionTimeout 12.000000 hour
2021-04-22T08:33:50.744Z,1619080430.744 [CommandLine](IMPORTANT): got command set profile_station.Lat 36.797001 degree
2021-04-22T08:33:50.744Z,1619080430.744 [CommandLine](IMPORTANT): got command set profile_station.Lon -121.847000 degree
2021-04-22T08:33:50.758Z,1619080430.758 [CommandLine](IMPORTANT): got command set profile_station.YoYoMaxDepth 50.000000 meter
2021-04-22T08:33:50.758Z,1619080430.758 [CommandLine](IMPORTANT): got command run
2021-04-22T08:33:50.761Z,1619080430.761 [CommandLine](IMPORTANT): Running 
2021-04-22T08:33:50.990Z,1619080430.990 [MissionManager](IMPORTANT): Started mission profile_station
2021-04-22T08:39:34.878Z,1619080774.878 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2021-04-22T11:56:10.826Z,1619092570.826 [CBIT](IMPORTANT): Beginning ground fault scan
2021-04-22T11:56:21.805Z,1619092581.805 [CBIT](IMPORTANT): Ground fault detected
mA:
CHAN A0 (Batt): -0.132695
CHAN A1 (24V): -0.001020
CHAN A2 (12V): 0.000132
CHAN A3 (5V): -0.000990
CHAN B0 (3.3V): -0.000512
CHAN B1 (3.15aV): 0.000698
CHAN B2 (3.15bV): -0.000606
CHAN B3 (GND): 0.000291
OPEN: 0.008031
Full Scale Calc: 4.765 mA, -1.589 mA
2021-04-22T15:14:02.602Z,1619104442.602 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=2994, MT Status=1, MTMSN=206
2021-04-22T15:14:03.211Z,1619104443.211 [CommandLine](IMPORTANT): got command stop
2021-04-22T15:14:03.222Z,1619104443.222 [CommandLine](IMPORTANT): Scheduling is paused
2021-04-22T15:14:03.860Z,1619104443.860 [MissionManager](IMPORTANT): Started mission Default
2021-04-22T15:14:04.282Z,1619104444.282 [WetLabsBB2FL](FAULT): LCB fault:  Software Overcurrent.
2021-04-22T15:14:39.411Z,1619104479.411 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=2995, MT Status=1, MTMSN=207
2021-04-22T15:14:40.203Z,1619104480.203 [CommandLine](IMPORTANT): got command schedule asap "load Science/profile_station.xml;set profile_station.MissionTimeout 12 hour;set profile_station.NeedCommsTime 30 minute;set profile_station.Lat 36.797 degree;set profile_station.Lon -121.847 degree" 21dxz 1 2.000000
2021-04-22T15:14:40.204Z,1619104480.204 [CommandLine](IMPORTANT): Scheduling command #1 of 2 with id=21dxz
2021-04-22T15:14:40.209Z,1619104480.209 [CommandLine](IMPORTANT): Scheduled #2 (#1 of 2 with id='21dxz'): "load Science/profile_station.xml;set profile_station.MissionTimeout 12 hour;set profile_station.NeedCommsTime 30 minute;set profile_station.Lat 36.797 degree;set profile_station.Lon -121.847 degree" ASAP
2021-04-22T15:15:27.685Z,1619104527.685 [NAL9602](IMPORTANT): SBD MO Status=1, MOMSN=2996, MT Status=1, MTMSN=208
2021-04-22T15:15:28.303Z,1619104528.303 [CommandLine](IMPORTANT): got command schedule asap "set profile_station.YoYoMaxDepth 50 meter;run" 21dxz 2 2.000000
2021-04-22T15:15:28.303Z,1619104528.303 [CommandLine](IMPORTANT): Scheduling command #2 of 2 with id=21dxz
2021-04-22T15:15:28.304Z,1619104528.304 [CommandLine](IMPORTANT): Scheduled #3 (#2 of 2 with id='21dxz'): "set profile_station.YoYoMaxDepth 50 meter;run" ASAP
2021-04-22T15:15:28.545Z,1619104528.545 [CommandLine](IMPORTANT): got command load ./Missions/Science/profile_station.xml
2021-04-22T15:15:30.610Z,1619104530.610 [CommandLine](IMPORTANT): Loaded ./Missions/Science/profile_station.xml
2021-04-22T15:15:41.102Z,1619104541.102 [CommandLine](IMPORTANT): got command set profile_station.MissionTimeout 12.000000 hour
2021-04-22T15:15:41.102Z,1619104541.102 [CommandLine](IMPORTANT): got command set profile_station.NeedCommsTime 30.000000 minute
2021-04-22T15:15:41.103Z,1619104541.103 [CommandLine](IMPORTANT): got command set profile_station.Lat 36.797001 degree
2021-04-22T15:15:41.104Z,1619104541.104 [CommandLine](IMPORTANT): got command set profile_station.Lon -121.847000 degree
2021-04-22T15:15:44.088Z,1619104544.088 [CommandLine](IMPORTANT): got command set profile_station.YoYoMaxDepth 50.000000 meter
2021-04-22T15:15:44.089Z,1619104544.089 [CommandLine](IMPORTANT): got command run
2021-04-22T15:15:44.091Z,1619104544.091 [CommandLine](IMPORTANT): Running 
2021-04-22T15:15:44.278Z,1619104544.278 [MissionManager](IMPORTANT): Started mission profile_station
2021-04-22T15:21:24.995Z,1619104884.995 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2021-04-22T15:38:46.931Z,1619105926.931 [DVL_micro](FAULT): Failed to initialize within timeout.
2021-04-22T15:51:19.471Z,1619106679.471 [DataOverHttps](IMPORTANT): SBD MTMSN=20210422T155119
2021-04-22T15:51:27.339Z,1619106687.339 [CommandLine](IMPORTANT): got command stop
2021-04-22T15:51:28.094Z,1619106688.094 [MissionManager](IMPORTANT): Started mission Default
2021-04-22T15:53:13.924Z,1619106793.924 [NAL9602](FAULT): MT Queue status failed to be acquired within timeout. Will not retry this session.
2021-04-22T15:59:12.009Z,1619107152.009 [Default:CheckIn:E](IMPORTANT): Default mission has been running for 7.725250 min 
2021-04-22T16:00:36.442Z,1619107236.442 [NAL9602](IMPORTANT): SBD MO Status=0, MOMSN=2997, MT Status=1, MTMSN=209
2021-04-22T16:00:37.002Z,1619107237.002 [CommandLine](IMPORTANT): got command stop
2021-04-22T16:00:59.743Z,1619107259.743 [DataOverHttps](IMPORTANT): SBD MTMSN=20210422T160059