More robust rejection of ETC fibers with stellar contamination

[dkirkby]

Exposures 106015-8 of tile 3072 (RA 293.37 DEC 66.43) on 20211025 have sudden jumps in the sky level that appear to correlate with downward spikes in FFRAC:

This issue is to document what happened, as a starting point for possibly updating the algorithm to avoid situations like this (in case we start to see this more often).

[dkirkby]

Use the following code to examine the frames just before and after the first jump around 2:20:

import pathlib
import numpy as np
import fitsio
import desietc.sky

SKY_CALIB = pathlib.Path('/global/cfs/cdirs/desi/cmx/sky/calib/SKY_calib.fits')
SKY = desietc.sky.SkyCamera(calib_name=SKY_CALIB)

def dump_sky(night, expid, camera='SKYCAM0', frame=0):
    exptag = str(expid).zfill(8)
    exp_path = DATA / str(night) / exptag
    sky_path = exp_path / f'sky-{exptag}.fits.fz'
    
    meta = fitsio.read(str(sky_path), ext=camera+'T')[frame]
    print({name:meta[name] for name in meta.dtype.names})
    
    raw = fitsio.read(str(sky_path), ext=camera)[frame]
    flux, dflux = SKY.setraw(raw, camera)
    
    print('flux =', np.round(SKY.flux, 0))
    print('fluxerr =', np.round(SKY.fluxerr,1))
    print('bgfit =', np.round(SKY.bgfit,4))
    print('ndrop =', SKY.ndrop, 'chisq =', SKY.chisq)
    
    exptime = meta['EXPTIME']
    print(f'combined flux / sec = {flux/exptime:.3f} +/- {dflux/exptime:.3f}')

The results for SKYCAM0 are:

# dump_sky(20211025, 106015, 'SKYCAM0', frame=0)
{'EXPTIME': 60.0, 'NIGHT': 20211025, 'DATE-OBS': '2021-10-26T02:16:53.575293', 'TIME-OBS': '02:16:53.575293', 'MJD-OBS': 59513.09506453, 'OPENSHUT': '2021-10-26T02:16:53.575293', 'ST': '21:10:26.670', 'HEXPOS': 'None', 'GAMBNTT': 'None', 'GFPGAT': 'None', 'GFILTERT': 'None', 'GCOLDTEC': 'None', 'GHOTTEC': 'None', 'GCCDTEMP': 'None', 'GCAMTEMP': 'None', 'GHUMID2': 'None', 'GHUMID3': 'None'}
flux = [2680.  140. 2530. 2313. 2482. 3186. 1893. 1885. 2391.  487.]
fluxerr = [240.9 240.8 224.5 217.6 209.9 164.6 156.7 272.5 212.3 304.1]
bgfit = [0.1722 0.4732 0.3031 0.3543 0.2955 0.3124 0.5525 0.3056 0.2157 0.4473]
ndrop = 0 chisq = 0.7619612890188725
combined flux / sec = 1.752 +/- 0.047

# dump_sky(20211025, 106015, 'SKYCAM0', frame=1)
{'EXPTIME': 60.0, 'NIGHT': 20211025, 'DATE-OBS': '2021-10-26T02:18:07.761765', 'TIME-OBS': '02:18:07.761765', 'MJD-OBS': 59513.09592317, 'OPENSHUT': '2021-10-26T02:18:07.761765', 'ST': '21:11:41.820', 'HEXPOS': 'None', 'GAMBNTT': 'None', 'GFPGAT': 'None', 'GFILTERT': 'None', 'GCOLDTEC': 'None', 'GHOTTEC': 'None', 'GCCDTEMP': 'None', 'GCAMTEMP': 'None', 'GHUMID2': 'None', 'GHUMID3': 'None'}
flux = [2457.   37. 2158. 3002. 2415. 3186. 2006. 1620. 2315.  531.]
fluxerr = [237.2 240.2 222.7 221.2 212.3 164.9 158.6 274.9 206.2 299.6]
bgfit = [0.307  0.4663 0.4022 0.1605 0.1977 0.1956 0.3429 0.3028 0.4076 0.566 ]
ndrop = 0 chisq = 1.0575703387332398
combined flux / sec = 1.742 +/- 0.047

and SKYCAM1:

# dump_sky(20211025, 106015, 'SKYCAM1', frame=0)
{'EXPTIME': 60.0, 'NIGHT': 20211025, 'DATE-OBS': '2021-10-26T02:16:53.575326', 'TIME-OBS': '02:16:53.575326', 'MJD-OBS': 59513.09506453, 'OPENSHUT': '2021-10-26T02:16:53.575326', 'ST': '21:10:26.620', 'HEXPOS': 'None', 'GAMBNTT': 'None', 'GFPGAT': 'None', 'GFILTERT': 'None', 'GCOLDTEC': 'None', 'GHOTTEC': 'None', 'GCCDTEMP': 'None', 'GCAMTEMP': 'None', 'GHUMID2': 'None', 'GHUMID3': 'None'}
flux = [16638.  9078.  3635.  2755.  2634.   878.  2557.  2160.  2561.  1199.]
fluxerr = [269.5 170.9 186.3 164.5 134.6 178.8 187.9 272.5 211.3 306. ]
bgfit = [0.081  0.3569 0.3792 0.3754 0.3477 0.3745 0.2951 0.3226 0.3259 0.3008]
ndrop = 1 chisq = 2.833889486509751
combined flux / sec = 1.764 +/- 0.048

#dump_sky(20211025, 106015, 'SKYCAM1', frame=1)
{'EXPTIME': 60.0, 'NIGHT': 20211025, 'DATE-OBS': '2021-10-26T02:18:07.763783', 'TIME-OBS': '02:18:07.763783', 'MJD-OBS': 59513.09592319, 'OPENSHUT': '2021-10-26T02:18:07.763783', 'ST': '21:11:40.970', 'HEXPOS': 'None', 'GAMBNTT': 'None', 'GFPGAT': 'None', 'GFILTERT': 'None', 'GCOLDTEC': 'None', 'GHOTTEC': 'None', 'GCCDTEMP': 'None', 'GCAMTEMP': 'None', 'GHUMID2': 'None', 'GHUMID3': 'None'}
flux = [16437. 12675.  2989.  2835.  2849.  1204.  2588.  2160.  2561.  1199.]
fluxerr = [269.  180.  185.1 164.6 135.5 180.8 184.5 272.5 211.3 306. ]
bgfit = [0.0465 0.3676 0.3873 0.274  0.3298 0.2386 0.3867 0.3226 0.3259 0.3008]
ndrop = 2 chisq = 1058.131972253187
combined flux / sec = 3.719 +/- 0.046

This reveals that the jump is due to SKYCAM1, most likely from its fiber[1] which has a >10 sigma increase in flux.

The algorithm is supposed to drop up to 2 fibers until chisq < 5. It did drop the max of 2 fibers, but with chisq still > 1000.

It looks like a cut on the final chisq would be an easy fix, at least when the other camera has a reasonable chisq. However, I am suspicious that the fiber dropping code is not doing the right thing (after seeing this example) so that needs to be checked first.

[djschlegel]

Another suspicious case reported by the observers on 2021-11-06 for exposures 107643-107644, where the sky level was 3. Adjacent exposures had sky levels of 2, and were only a few degrees away on the sky. This may be due to clouds, but it's suspicious that the discontinuities coincide with starting new tiles.

[dkirkby]

This was a similar incident:

# dump_sky(20211106, 107643, 'SKYCAM0', frame=0)
{'EXPTIME': 60.0, 'NIGHT': 20211106, 'DATE-OBS': '2021-11-07T02:37:55.019894', 'TIME-OBS': '02:37:55.019894', 'MJD-OBS': 59525.10966458, 'OPENSHUT': '2021-11-07T02:37:55.019894', 'ST': '22:18:50.350', 'HEXPOS': 'None', 'GAMBNTT': 'None', 'GFPGAT': 'None', 'GFILTERT': 'None', 'GCOLDTEC': 'None', 'GHOTTEC': 'None', 'GCCDTEMP': 'None', 'GCAMTEMP': 'None', 'GHUMID2': 'None', 'GHUMID3': 'None'}
flux = [2673.  246. 2094. 2214. 2636. 2445. 1787. 1663. 2362. 1366.]
fluxerr = [271.8 276.6 253.9 248.4 239.3 182.5 177.9 305.  239.5 350.4]
bgfit = [0.3733 0.6369 0.609  0.415  0.3645 0.4223 0.642  0.5188 0.4331 0.3941]
ndrop = 0 chisq = 1.3977179175549126
combined flux / sec = 1.570 +/- 0.053

# dump_sky(20211106, 107643, 'SKYCAM1', frame=0)
{'EXPTIME': 60.0, 'NIGHT': 20211106, 'DATE-OBS': '2021-11-07T02:37:55.019775', 'TIME-OBS': '02:37:55.019775', 'MJD-OBS': 59525.10966458, 'OPENSHUT': '2021-11-07T02:37:55.019775', 'ST': '22:18:50.190', 'HEXPOS': 'None', 'GAMBNTT': 'None', 'GFPGAT': 'None', 'GFILTERT': 'None', 'GCOLDTEC': 'None', 'GHOTTEC': 'None', 'GCCDTEMP': 'None', 'GCAMTEMP': 'None', 'GHUMID2': 'None', 'GHUMID3': 'None'}
flux = [16002. 19120.  3300.  2588.  2671.   602.  2618.  1663.  2362.  1366.]
fluxerr = [292.5 213.  206.  178.2 145.  193.3 203.6 305.  239.5 350.4]
bgfit = [0.1891 0.4103 0.307  0.5498 0.5634 0.583  0.4716 0.5188 0.4331 0.3941]
ndrop = 2 chisq = 2244.315232930171
combined flux / sec = 4.488 +/- 0.051

Since that's twice in 2 weeks, I will start working on a fix...

[djschlegel]

The spectro redux confirm that for this last example (exposures 107643-107644), the sky is fainter and the reported EFFTIME_SPEC is approx twice the EFFTIME_ETC.

[djschlegel]

Another likely example tile 2484 observed 2021-11-10 expid 108309.

[dkirkby]

For reference, FITS files for the SKYCAM data in the examples above are at:

• https://data.desi.lbl.gov/desi/spectro/data/20211025/00106015/sky-00106015.fits.fz
• https://data.desi.lbl.gov/desi/spectro/data/20211106/00107643/sky-00107643.fits.fz

[dkirkby]

I have tested that increasing the max allowed number of dropped fibers (due to being chisq outliers) from 1 to 2 fixes both of the examples above. To be safe, I am increasing the max to 3.

[djschlegel]

Re-opening issue since it looks like it may still be with us. In particular, for exposure 111452 on TILEID 3254 on 2021-11-29, the sky level was alternating between ~1.5 and ~2.5. Earlier in the evening, exposure 111440 on TILEID 11621 has somewhat suspiciously high sky as well.

We could always revisit the option of moving the Sky Monitor fibers to known blank sky positions instead if leaving them stuck where they are. Or, easier yet, pre-compute which fibers are known to be near sources (the same as we do for the stuck skies). We would need to measure the physical locations of those fibers by backlighting them for FVC images, something we maybe haven't done since early in commissioning.

.

[dkirkby]

Thanks for reporting this. I will take a look.

[schlafly]

Another one of these on 20220104/116678.

[djschlegel]

This might be worth a more exhaustive search of such cases since the last change, but I'll note two more from last night (2022-03-07): Exposure 125318 on tile 24546 and exposure 125324 on tile 26228.

[schlafly]

Adding some discussion from a past survey-ops call here... the fiberassign files do include a SKY_MONITOR extension which looks to intend to have the information for the 20 sky fibers. This does have FIBERASSIGN_{X/Y} and TARGET_{RA/DEC}, which we presumably transform in the usual way for the stuck fibers. I don't see something actually saying whether we think those locations are good sky, unfortunately, but I may be missing something.

So I think we'd have the action item of propagating that bit into this extension in the usual way for the stuck skies. Then we'd also have to make sure that FIBERASSIGN_X and FIBERASSIGN_Y are correct for these fibers, which would involve backlighting them. And then the ETC would have to use the bit. But that may be everything?

[djschlegel]

For the exercise of determining the mapping of the ETC fibers on the Sky Monitor images: On the night of 2022-04-16, we observed five BACKUP tiles at the start of last at low Galactic latitude, where a good fraction of the ETC fibers should be landing on or near stars. The exposures are 130554-130557 and 130563.

[dkirkby]

I looked at this data, but only 1 tile shows any sign of stellar contamination (in just one fiber) so this will not help us with the mapping:

[djschlegel]

Another likely example of contaminated sky fibers is tile 9759 exposure 209314 observed on 2023-12-12, based on the sky level jumping up for only that one exposure.