INFO 2025-06-26 05:32:57,790 process.py process_mission():429 [1] commandline = src/data/process_dorado.py --start_year 2003 --end_year 2014 --num_cores 1 -v --noinput --clobber
INFO 2025-06-26 05:32:57,791 process.py process_mission():434 [1] dorado_info[mission]["comment"] = Portaled files exist in archive, but not processed by reprocess_surveys.m - ctdToUse = ctd1 
INFO 2025-06-26 05:32:57,792 process.py process_mission():449 [1] Processing AUVCTD mission 2003.199.03 by user docker_user on host 671049448ece
INFO 2025-06-26 05:32:57,792 process.py download_process():165 [1] Download and processing steps for 2003.199.03
INFO 2025-06-26 05:32:57,793 logs2netcdfs.py download_process_logs():739 [1] src_dir = /Volumes/AUVCTD/missionlogs/2003/2003199/2003.199.03
INFO 2025-06-26 05:32:57,793 logs2netcdfs.py download_process_logs():768 [1] Rsyncing /Volumes/AUVCTD/missionlogs/2003/2003199/2003.199.03 to /data/auv_data/dorado/missionlogs/2003.199.03
INFO 2025-06-26 05:32:58,213 logs2netcdfs.py download_process_logs():779 [1] Processing mission: dorado 2003.199.03
INFO 2025-06-26 05:32:58,213 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2003.199.03/ctdDriver.log (46507 bytes)
INFO 2025-06-26 05:32:58,223 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2003.199.03/ctdDriver2.log (46061 bytes)
INFO 2025-06-26 05:32:58,232 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2003.199.03/gps.log (0 bytes)
WARNING 2025-06-26 05:32:58,232 logs2netcdfs.py download_process_logs():790 [1] /data/auv_data/dorado/missionlogs/2003.199.03/gps.log is empty
INFO 2025-06-26 05:32:58,232 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2003.199.03/hydroscatlog.log (38354 bytes)
INFO 2025-06-26 05:32:58,242 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2003.199.03/navigation.log (135598 bytes)
INFO 2025-06-26 05:32:58,263 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2003.199.03/isuslog.log (1749 bytes)
INFO 2025-06-26 05:32:58,267 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2003.199.03/parosci.log (38198 bytes)
INFO 2025-06-26 05:32:58,274 logs2netcdfs.py download_process_logs():788 [1] Processing file /data/auv_data/dorado/missionlogs/2003.199.03/tailCone.log (37915 bytes)
WARNING 2025-06-26 05:32:58,274 logs2netcdfs.py download_process_logs():795 [1] Failed to parse long_name & units from header in file /data/auv_data/dorado/missionlogs/2003.199.03/tailCone.log
INFO 2025-06-26 05:32:58,274 logs2netcdfs.py download_process_logs():807 [1] Time to process: 0.06 seconds
WARNING 2025-06-26 05:32:58,274 process.py download_process():196 [1] No lopc.bin file for 2003.199.03
INFO 2025-06-26 05:32:58,274 process.py calibrate():227 [1] Calibration steps for 2003.199.03
INFO 2025-06-26 05:32:58,309 calibrate.py _navigation_process():1524 [1] Converting from Easting/Northing to lat/lon for mission 2003.199.03
WARNING 2025-06-26 05:32:58,311 calibrate.py _navigation_process():1588 [1] No positive depths found in 2003.199.03/navigation.nc
ERROR 2025-06-26 05:32:58,311 calibrate.py process_logs():3542 [1] Error processing gps: gps has no orig_data - likely a missing or zero-sized .log file in missionlogs/2003.199.03
WARNING 2025-06-26 05:32:58,313 calibrate.py _depth_process():2193 [1] Expected sample_rate to be 10 Hz, instead it's 5.00 Hz
WARNING 2025-06-26 05:32:58,317 calibrate.py _hs2_process():2289 [1] Cannot process HS2 data: [Errno 2] No such file or directory: '/data/auv_data/dorado/missionlogs/2003.199.03/hs2Calibration.dat'
ERROR 2025-06-26 05:32:58,331 calibrate.py process_logs():3548 [1] Error processing ctd1: missing variable 'ctd'
INFO 2025-06-26 05:32:58,347 calibrate.py _geometric_depth_correction():3423 [1] ctd2: d_beg_time_diff: 28 seconds, d_end_time_diff: -1 seconds
INFO 2025-06-26 05:32:58,352 calibrate.py _geometric_depth_correction():3423 [1] ctd2: d_beg_time_diff: 28 seconds, d_end_time_diff: -1 seconds
INFO 2025-06-26 05:32:58,353 calibrate.py _ctd_process():2919 [1] Performing range checking of ['ctd2_salinity'] in 2003.199.03/ctd2.nc
WARNING 2025-06-26 05:32:58,354 calibrate.py _range_qc_combined_nc():890 [1] More than 500 (516) ctd2_salinity values found outside of range. This may indicate a problem with the ctd2 data.
INFO 2025-06-26 05:32:58,354 calibrate.py _range_qc_combined_nc():898 [1] Setting 516 ctd2_salinity values to NaN
INFO 2025-06-26 05:32:58,354 calibrate.py _range_qc_combined_nc():910 [1] Checked for data outside of these variables and ranges: [('ctd2_salinity', Range(min=30, max=40))]
INFO 2025-06-26 05:32:58,355 calibrate.py _range_qc_combined_nc():936 [1] Done range checking ctd2
ERROR 2025-06-26 05:32:58,359 calibrate.py process_logs():3542 [1] Error processing biolume: biolume has no orig_data - likely a missing or zero-sized .log file in missionlogs/2003.199.03
ERROR 2025-06-26 05:32:58,359 calibrate.py process_logs():3542 [1] Error processing lopc: lopc has no orig_data - likely a missing or zero-sized .log file in missionlogs/2003.199.03
ERROR 2025-06-26 05:32:58,359 calibrate.py process_logs():3542 [1] Error processing ecopuck: ecopuck has no orig_data - likely a missing or zero-sized .log file in missionlogs/2003.199.03
ERROR 2025-06-26 05:32:58,360 calibrate.py process_logs():3542 [1] Error processing tailcone: tailcone has no orig_data - likely a missing or zero-sized .log file in missionlogs/2003.199.03
INFO 2025-06-26 05:32:58,360 calibrate.py write_netcdf():3508 [1] Writing calibrated instrument data to /data/auv_data/dorado/missionnetcdfs/2003.199.03/dorado_2003.199.03_cal.nc
INFO 2025-06-26 05:32:58,383 calibrate.py write_netcdf():3512 [1] Data variables written: ctd1_conductivity, ctd1_conductivity_onboard, ctd1_dissolvedO2, ctd1_salinity, ctd1_temperature, ctd1_temperature_onboard, ctd1_time, ctd2_conductivity, ctd2_conductivity_onboard, ctd2_depth, ctd2_flow2, ctd2_salinity, ctd2_temperature, ctd2_temperature_onboard, ctd2_time, depth_filtdepth, depth_filtpres, depth_time, isus_nitrate, isus_quality, isus_temp, isus_time, navigation_depth, navigation_latitude, navigation_longitude, navigation_mWaterSpeed, navigation_pitch, navigation_posx, navigation_posy, navigation_roll, navigation_time, navigation_yaw
INFO 2025-06-26 05:32:58,386 process.py align():251 [1] Alignment steps for 2003.199.03
INFO 2025-06-26 05:32:58,397 align.py process_cal():139 [1] Processing dorado_2003.199.03_cal.nc from /data/auv_data/dorado/missionnetcdfs/2003.199.03
ERROR 2025-06-26 05:32:58,398 process.py process_mission_exception_wrapper():537 [1] An exception occurred
Traceback (most recent call last):
  File "/src/data/process.py", line 529, in process_mission_exception_wrapper
    self.process_mission(mission, src_dir=src_dir)
  File "/src/data/process.py", line 485, in process_mission
    self.align(mission)
  File "/src/data/process.py", line 263, in align
    netcdf_dir = align_netcdf.process_cal()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/data/align.py", line 182, in process_cal
    raise InvalidCalFile(error_message) from None
align.InvalidCalFile: No nudged_latitude data in dorado_2003.199.03_cal.nc
ERROR 2025-06-26 05:32:58,399 process.py process_mission_exception_wrapper():538 [1] Failed to process to completion: 2003.199.03
INFO 2025-06-26 05:32:58,400 process.py archive():337 [1] Archiving steps for 2003.199.03
INFO 2025-06-26 05:32:58,400 process.py archive():347 [1] nc_file_base = /data/auv_data/dorado/missionnetcdfs/2003.199.03/dorado_2003.199.03, BASE_PATH = /data/auv_data
INFO 2025-06-26 05:32:58,400 archive.py copy_to_AUVTCD():60 [1] Archiving /data/auv_data/dorado/missionnetcdfs/2003.199.03/dorado_2003.199.03 files to /Volumes/AUVCTD/surveys/2003/netcdf
INFO 2025-06-26 05:32:58,401 archive.py copy_to_AUVTCD():78 [1] Removing /Volumes/AUVCTD/surveys/2003/netcdf/dorado_2003.199.03_cal.nc
INFO 2025-06-26 05:32:58,408 archive.py copy_to_AUVTCD():82 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2003.199.03/dorado_2003.199.03_cal.nc /Volumes/AUVCTD/surveys/2003/netcdf done.
INFO 2025-06-26 05:32:58,415 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2003.199.03/ctdDriver.nc /Volumes/AUVCTD/missionnetcdfs/2003/2003199/2003.199.03 done.
INFO 2025-06-26 05:32:58,420 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2003.199.03/ctdDriver2.nc /Volumes/AUVCTD/missionnetcdfs/2003/2003199/2003.199.03 done.
INFO 2025-06-26 05:32:58,424 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2003.199.03/hydroscatlog.nc /Volumes/AUVCTD/missionnetcdfs/2003/2003199/2003.199.03 done.
INFO 2025-06-26 05:32:58,429 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2003.199.03/navigation.nc /Volumes/AUVCTD/missionnetcdfs/2003/2003199/2003.199.03 done.
INFO 2025-06-26 05:32:58,433 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2003.199.03/isuslog.nc /Volumes/AUVCTD/missionnetcdfs/2003/2003199/2003.199.03 done.
INFO 2025-06-26 05:32:58,439 archive.py copy_to_AUVTCD():108 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2003.199.03/parosci.nc /Volumes/AUVCTD/missionnetcdfs/2003/2003199/2003.199.03 done.
INFO 2025-06-26 05:32:58,439 archive.py copy_to_AUVTCD():116 [1] Archiving product files
INFO 2025-06-26 05:32:58,439 archive.py copy_to_AUVTCD():161 [1] copyfile /data/auv_data/dorado/missionnetcdfs/2003.199.03/dorado_2003.199.03_processing.log /Volumes/AUVCTD/surveys/2003/netcdf