INFO 2025-06-26 05:47:21,606 process.py process_mission():429 [1] commandline = src/data/process_dorado.py --start_year 2003 --end_year 2014 --num_cores 1 -v --noinput --clobber INFO 2025-06-26 05:47:21,606 process.py process_mission():434 [1] dorado_info[mission]["comment"] = Something whaky with lat lon calculations start at (237.77744, 36.94086) and end at (255.1847, 53.14181) - ctdToUse = ctd1 INFO 2025-06-26 05:47:21,606 process.py process_mission():449 [1] Processing BIOLUME mission 2004.094.04 by user docker_user on host 671049448ece INFO 2025-06-26 05:47:21,606 process.py download_process():165 [1] Download and processing steps for 2004.094.04 INFO 2025-06-26 05:47:21,607 logs2netcdfs.py download_process_logs():739 [1] src_dir = /Volumes/AUVCTD/missionlogs/2004/2004094/2004.094.04 INFO 2025-06-26 05:47:21,607 logs2netcdfs.py download_process_logs():768 [1] Rsyncing /Volumes/AUVCTD/missionlogs/2004/2004094/2004.094.04 to /data/auv_data/dorado/missionlogs/2004.094.04 INFO 2025-06-26 05:47:23,512 logs2netcdfs.py download_process_logs():779 [1] Processing mission: dorado 2004.094.04 INFO 2025-06-26 05:47:23,512 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2004.094.04/ctdDriver.log (4898123 bytes) INFO 2025-06-26 05:47:23,881 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2004.094.04/ctdDriver2.log (4898117 bytes) INFO 2025-06-26 05:47:24,242 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2004.094.04/gps.log (38444 bytes) INFO 2025-06-26 05:47:24,256 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2004.094.04/hydroscatlog.log (3342314 bytes) INFO 2025-06-26 05:47:24,664 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2004.094.04/navigation.log (9739285 bytes) INFO 2025-06-26 05:47:25,398 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2004.094.04/isuslog.log (269397 bytes) INFO 2025-06-26 05:47:25,428 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2004.094.04/parosci.log (3344150 bytes) INFO 2025-06-26 05:47:25,715 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2004.094.04/tailCone.log (3343819 bytes) INFO 2025-06-26 05:47:26,128 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2004.094.04/biolume.log (3793724 bytes) INFO 2025-06-26 05:47:26,495 logs2netcdfs.py write_variables():575 [1] Expanding original timeTag to time60Hz variable for raw data INFO 2025-06-26 05:47:26,677 logs2netcdfs.py _process_log_file():687 [1] Non-monotonic increasing time indices in /data/auv_data/dorado/missionlogs/2004.094.04/biolume.log: [1] INFO 2025-06-26 05:47:26,690 logs2netcdfs.py download_process_logs():807 [1] Time to process: 3.18 seconds WARNING 2025-06-26 05:47:26,690 process.py download_process():196 [1] No lopc.bin file for 2004.094.04 INFO 2025-06-26 05:47:26,690 process.py calibrate():227 [1] Calibration steps for 2004.094.04 INFO 2025-06-26 05:47:26,822 calibrate.py _navigation_process():1524 [1] Converting from Easting/Northing to lat/lon for mission 2004.094.04 INFO 2025-06-26 05:47:26,839 calibrate.py _nudge_pos():1680 [1] seg# end_sec_diff end_lon_diff end_lat_diff len(segi) seg_min u_drift (cm/s) v_drift (cm/s) start datetime of segment INFO 2025-06-26 05:47:26,841 calibrate.py _nudge_pos():1711 [1] - - - 0 0.00 - - - INFO 2025-06-26 05:47:28,447 calibrate.py _nudge_pos():1806 [1] 185: 0.200 -0.0014902 -0.0001377 12175 40.58 -5.442 -0.629 2004-04-03T07:19:54.403469056 INFO 2025-06-26 05:47:28,959 calibrate.py _nudge_pos():1806 [1] 238: 0.200 0.0000029 0.0000078 44 0.14 3.014 10.062 2004-04-03T07:20:55.409289856 INFO 2025-06-26 05:47:29,008 calibrate.py _nudge_pos():1806 [1] 243: 0.200 -0.0014387 -0.0001089 12424 41.41 -5.151 -0.487 2004-04-03T08:02:24.408781824 INFO 2025-06-26 05:47:29,681 calibrate.py _nudge_pos():1765 [1] 314: 0.200 17.1738176 16.2979490 12849 0.01 44589.294 70516.935 2004-04-03T08:46:24.407929472 ERROR 2025-06-26 05:47:29,682 calibrate.py _nudge_pos():1770 [1] End of underwater segment dead reckoned position is too different from GPS fix: abs(end_lon_diff) (17.173817614522093) > 1 or abs(end_lat_diff) (16.297949012653774) > 1 INFO 2025-06-26 05:47:29,682 calibrate.py _nudge_pos():1776 [1] Fix this error by calling _range_qc_combined_nc() in _navigation_process() and/or _gps_process() for dorado 2004.094.04 ERROR 2025-06-26 05:47:29,682 calibrate.py process_logs():3546 [1] Error processing gps Traceback (most recent call last): File "/src/data/calibrate.py", line 3535, in process_logs self._process(sensor, logs_dir, netcdfs_dir) File "/src/data/calibrate.py", line 3480, in _process self._gps_process(sensor) File "/src/data/calibrate.py", line 2085, in _gps_process nudged_longitude, nudged_latitude = self._nudge_pos() ^^^^^^^^^^^^^^^^^ File "/src/data/calibrate.py", line 1786, in _nudge_pos raise ValueError(error_message) ValueError: abs(end_lon_diff) (17.173817614522093) > 1 or abs(end_lat_diff) (16.297949012653774) > 1 WARNING 2025-06-26 05:47:29,738 calibrate.py _depth_process():2193 [1] Expected sample_rate to be 10 Hz, instead it's 5.00 Hz INFO 2025-06-26 05:47:30,242 calibrate.py _geometric_depth_correction():3423 [1] hs2: d_beg_time_diff: 3 seconds, d_end_time_diff: -1 seconds INFO 2025-06-26 05:47:30,764 calibrate.py _geometric_depth_correction():3423 [1] ctd1: d_beg_time_diff: 19 seconds, d_end_time_diff: -1 seconds INFO 2025-06-26 05:47:31,177 calibrate.py _geometric_depth_correction():3423 [1] ctd1: d_beg_time_diff: 19 seconds, d_end_time_diff: -1 seconds INFO 2025-06-26 05:47:31,179 calibrate.py _ctd_process():2919 [1] Performing range checking of ['ctd1_salinity'] in 2004.094.04/ctd1.nc INFO 2025-06-26 05:47:31,180 calibrate.py _range_qc_combined_nc():898 [1] Setting 0 ctd1_salinity values to NaN INFO 2025-06-26 05:47:31,180 calibrate.py _range_qc_combined_nc():910 [1] Checked for data outside of these variables and ranges: [('ctd1_salinity', Range(min=30, max=40))] INFO 2025-06-26 05:47:31,180 calibrate.py _range_qc_combined_nc():936 [1] Done range checking ctd1 INFO 2025-06-26 05:47:31,662 calibrate.py _geometric_depth_correction():3423 [1] ctd2: d_beg_time_diff: 19 seconds, d_end_time_diff: -1 seconds INFO 2025-06-26 05:47:32,080 calibrate.py _geometric_depth_correction():3423 [1] ctd2: d_beg_time_diff: 19 seconds, d_end_time_diff: -1 seconds INFO 2025-06-26 05:47:32,082 calibrate.py _ctd_process():2919 [1] Performing range checking of ['ctd2_salinity'] in 2004.094.04/ctd2.nc INFO 2025-06-26 05:47:32,083 calibrate.py _range_qc_combined_nc():898 [1] Setting 0 ctd2_salinity values to NaN INFO 2025-06-26 05:47:32,084 calibrate.py _range_qc_combined_nc():910 [1] Checked for data outside of these variables and ranges: [('ctd2_salinity', Range(min=30, max=40))] INFO 2025-06-26 05:47:32,084 calibrate.py _range_qc_combined_nc():936 [1] Done range checking ctd2 INFO 2025-06-26 05:47:32,107 calibrate.py _biolume_process():3128 [1] Checking for non-monotonic increasing time60hz INFO 2025-06-26 05:47:32,776 calibrate.py _biolume_process():3131 [1] Removing non-monotonic increasing time60hz at indices: [ 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119] INFO 2025-06-26 05:47:32,895 calibrate.py _geometric_depth_correction():3423 [1] biolume: d_beg_time_diff: -2 seconds, d_end_time_diff: -1 seconds ERROR 2025-06-26 05:47:33,152 calibrate.py process_logs():3548 [1] Error processing biolume: missing variable "No variable named 'avg_biolume'. Variables on the dataset include ['time', 'rrr', 'cal_striing', 'pump', 'flow', 'humidity', 'temperature', 'time60hz', 'raw']" ERROR 2025-06-26 05:47:33,152 calibrate.py process_logs():3542 [1] Error processing lopc: lopc has no orig_data - likely a missing or zero-sized .log file in missionlogs/2004.094.04 ERROR 2025-06-26 05:47:33,152 calibrate.py process_logs():3542 [1] Error processing ecopuck: ecopuck has no orig_data - likely a missing or zero-sized .log file in missionlogs/2004.094.04 INFO 2025-06-26 05:47:33,209 calibrate.py write_netcdf():3508 [1] Writing calibrated instrument data to /data/auv_data/dorado/missionnetcdfs/2004.094.04/dorado_2004.094.04_cal.nc INFO 2025-06-26 05:47:33,278 calibrate.py write_netcdf():3512 [1] Data variables written: biolume_depth, biolume_flow, biolume_time, ctd1_conductivity, ctd1_conductivity_onboard, ctd1_depth, ctd1_dissolvedO2, ctd1_flow1, ctd1_oxygen_mll, ctd1_oxygen_umolkg, ctd1_salinity, ctd1_temperature, ctd1_temperature_onboard, ctd1_time, ctd2_conductivity, ctd2_conductivity_onboard, ctd2_depth, ctd2_flow2, ctd2_salinity, ctd2_temperature, ctd2_temperature_onboard, ctd2_time, depth_filtdepth, depth_filtpres, depth_time, gps_latitude, gps_longitude, gps_time, hs2_bbp470, hs2_bbp676, hs2_depth, hs2_fl676, hs2_time, isus_nitrate, isus_quality, isus_temp, isus_time, navigation_depth, navigation_latitude, navigation_longitude, navigation_mWaterSpeed, navigation_pitch, navigation_posx, navigation_posy, navigation_roll, navigation_time, navigation_yaw, tailcone_propRpm, tailcone_time INFO 2025-06-26 05:47:33,282 process.py align():251 [1] Alignment steps for 2004.094.04 INFO 2025-06-26 05:47:33,307 align.py process_cal():139 [1] Processing dorado_2004.094.04_cal.nc from /data/auv_data/dorado/missionnetcdfs/2004.094.04 ERROR 2025-06-26 05:47:33,308 process.py process_mission_exception_wrapper():537 [1] An exception occurred Traceback (most recent call last): File "/src/data/process.py", line 529, in process_mission_exception_wrapper self.process_mission(mission, src_dir=src_dir) File "/src/data/process.py", line 485, in process_mission self.align(mission) File "/src/data/process.py", line 263, in align netcdf_dir = align_netcdf.process_cal() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/src/data/align.py", line 182, in process_cal raise InvalidCalFile(error_message) from None align.InvalidCalFile: No nudged_latitude data in dorado_2004.094.04_cal.nc ERROR 2025-06-26 05:47:33,309 process.py process_mission_exception_wrapper():538 [1] Failed to process to completion: 2004.094.04 INFO 2025-06-26 05:47:33,310 process.py archive():337 [1] Archiving steps for 2004.094.04 INFO 2025-06-26 05:47:33,311 process.py archive():347 [1] nc_file_base = /data/auv_data/dorado/missionnetcdfs/2004.094.04/dorado_2004.094.04, BASE_PATH = /data/auv_data INFO 2025-06-26 05:47:33,312 archive.py copy_to_AUVTCD():60 [1] Archiving /data/auv_data/dorado/missionnetcdfs/2004.094.04/dorado_2004.094.04 files to /Volumes/AUVCTD/surveys/2004/netcdf INFO 2025-06-26 05:47:33,313 archive.py copy_to_AUVTCD():78 [1] Removing /Volumes/AUVCTD/surveys/2004/netcdf/dorado_2004.094.04_cal.nc INFO 2025-06-26 05:47:33,368 archive.py copy_to_AUVTCD():82 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/dorado_2004.094.04_cal.nc /Volumes/AUVCTD/surveys/2004/netcdf done. INFO 2025-06-26 05:47:33,390 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/ctdDriver.nc /Volumes/AUVCTD/missionnetcdfs/2004/2004094/2004.094.04 done. INFO 2025-06-26 05:47:33,408 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/ctdDriver2.nc /Volumes/AUVCTD/missionnetcdfs/2004/2004094/2004.094.04 done. INFO 2025-06-26 05:47:33,412 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/gps.nc /Volumes/AUVCTD/missionnetcdfs/2004/2004094/2004.094.04 done. INFO 2025-06-26 05:47:33,426 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/hydroscatlog.nc /Volumes/AUVCTD/missionnetcdfs/2004/2004094/2004.094.04 done. INFO 2025-06-26 05:47:33,458 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/navigation.nc /Volumes/AUVCTD/missionnetcdfs/2004/2004094/2004.094.04 done. INFO 2025-06-26 05:47:33,465 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/isuslog.nc /Volumes/AUVCTD/missionnetcdfs/2004/2004094/2004.094.04 done. INFO 2025-06-26 05:47:33,481 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/parosci.nc /Volumes/AUVCTD/missionnetcdfs/2004/2004094/2004.094.04 done. INFO 2025-06-26 05:47:33,497 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/tailCone.nc /Volumes/AUVCTD/missionnetcdfs/2004/2004094/2004.094.04 done. INFO 2025-06-26 05:47:33,541 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/biolume.nc /Volumes/AUVCTD/missionnetcdfs/2004/2004094/2004.094.04 done. INFO 2025-06-26 05:47:33,542 archive.py copy_to_AUVTCD():116 [1] Archiving product files INFO 2025-06-26 05:47:33,542 archive.py copy_to_AUVTCD():161 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2004.094.04/dorado_2004.094.04_processing.log /Volumes/AUVCTD/surveys/2004/netcdf