Official algorithms for creating the bad pixel mask¶

The definition of the official algorithm for the creation of the bad pixel mask reference file currently describes methods for finding DEAD, LOW QE, OPEN, and ADJACENT TO OPEN pixels.

Using this definition, the bad_pixel_mask.py module has been written and added to the JWST_reffiles repository. This module is currently undergoing testing before being fully integrated into the JWST_reffiles framework.

If you encounter any problems or have any questions about the code or its use, feel free to open an issue on the jwst_reffiles github page.

There are two separate modules that are called when running bad_pixel_mask.py. These are badpix_from_flats.py and badpix_from_darks.py. The former takes a collection of internal flat field exposures in order to search for DEAD, LOW QE, OPEN, and ADJACENT TO OPEN pixels. The latter takes a collection of dark current exposures in order to search for NOISY, HOT, RC, TELEGRAPH, and LOW_PEDESTAL pixels. Both of these modules expect input data in at least 2 calibration states, as detailed in the table below.

Module	Calibration States	Typical filename suffixes
badpix_from_flats	slope images, uncalibrated ramps	rate, uncal
badpix_from_darks	slope images, “fitopt” files, CR-flagged ramps, (uncalibrated ramps: MIRI-only)	rate, fitopt, jump, uncal

When running bad_pixel_mask.py as a standalone package, the input file lists described in the table above must be provided manually. In the future, when running via mkrefs.py, raw or partially-calibrated files will be allowed as inputs. mkrefs.py will then call the calibration pipeline and produce the needed files.

To use the bad pixel mask generator function as a standalone package, use the code shown below. Keywords in the call are linked to detailed descriptions below the code snippet. Note that all of the keywords in the call below have defaults defined in the code (and in fact the call below is using all default values), so you do not have to specify any keywords where you wish to use the default.

from glob import glob
from jwst_reffiles.bad_pixel_mask.bad_pixel_mask import bad_pixels

flat_rate_files = sorted(glob('/path/to/flats/*_rate.fits'))
flat_fluxcheck_files = sorted(glob('/path/to/flats/*uncal.fits'))

dark_rate_files = sorted(glob('/path/to/darks/*_rate.fits'))
dark_jump_files = sorted(glob('/path/to/darks/*_jump.fits'))
dark_fitopt_files = sorted(glob('/path/to/darks/*_fitopt.fits'))
dark_uncal_files = sorted(glob('/path/to/darks/*_uncal.fits'))

bad_pixels(flat_slope_files=flat_rate_files,
           dead_search =True,
           low_qe_and_open_search =True,
           dead_search_type ='sigma_rate',
           flat_mean_sigma_threshold =3,
           flat_mean_normalization_method ='smoothed',
           smoothing_box_width =15,
           smoothing_type ='Box2D',
           dead_sigma_threshold =5.,
           max_dead_norm_signal =None,
           run_dead_flux_check =False,
           dead_flux_check_files =None,
           flux_check =45000,
           max_low_qe_norm_signal =0.5,
           max_open_adj_norm_signal =1.05,
           manual_flag_file ='default',
           flat_do_not_use =[],
           dark_slope_files=dark_files,
           dark_uncal_files=None,
           dark_jump_files=dark_jump_files,
           dark_fitopt_files=dark_fitopt_files,
           dark_stdev_clipping_sigma =5.,
           dark_max_clipping_iters =5,
           dark_noisy_threshold =5,
           max_saturated_fraction =0.5,
           max_jump_limit =10,
           jump_ratio_threshold =5,
           early_cutoff_fraction =0.25,
           pedestal_sigma_threshold =5,
           rc_fraction_threshold =0.8,
           low_pedestal_fraction =0.8,
           high_cr_fraction =0.8,
           flag_values ={'hot': ['HOT'], 'rc': ['RC'], 'low_pedestal': ['OTHER_BAD_PIXEL'], 'high_cr': ["TELEGRAPH"]},
           dark_do_not_use =['hot', 'rc', 'low_pedestal', 'high_cr'],
           plot =False,
           output_file ='./test_bpm.fits',
           author ='jwst_reffiles',
           description ='A bad pix mask',
           pedigree ='GROUND',
           useafter ='2019-04-01 00:00:00',
           history ='',
           quality_check =False)

Dead Search¶

Boolean. If True, a dead pixel search (of type dead_search_type) is performed.

Low QE and Open Search¶

Boolean. If True, a search for Low QE, Open, and Adjacent to Open pixels is performed.

Dead Search Type¶

Use this string parameter to specify which type of dead pixel search to perform. Options are:

'sigma_rate': Using a normalized signal rate image, dead pixels
              are defined as those with a rate smaller than
              dead_sigma_threshold standard deviations below
              the mean.
'absolute_rate': Using a normalized signal rate image, dead pixels
                 are defined as those with a rate less than
                 max_dead_norm_signal.

Flat Mean Sigma Threshold¶

Number of standard deviations to use when sigma-clipping to calculate the mean slope image or the mean across the detector when working with flat field images.

Flat Mean Normalization Method¶

Specify how the mean flat field image is normalized prior to searching for bad pixels. Options are:

'smoothed': Mean image will be smoothed using a smoothing_box_width x smoothing_box_width box kernel. The mean
image is then normalized by this smoothed image.

'none': No normalization is done. Mean slope image is used as is

'mean': Mean image is normalized by its sigma-clipped mean

Smoothing Box Width¶

Width in pixels of the 2D box kernel to use to compute the smoothed mean flat field image

Smoothing Type¶

Type of smoothing to do when creating a smoothed mean flat field image. Box2D or Median. Box2D uses an astropy Box2DKernel, while Median uses a scipy median_filter.

Dead Sigma Threshold¶

Number of standard deviations below the mean at which a pixel is considered dead when using the sigma_rate dead search type.

Maximum Dead Normalized Signal¶

Maximum normalized signal rate of a pixel that is considered dead when using the absolute_rate dead search type.

Run Dead Flux Check¶

Boolean controlling whether or not to search flagged dead pixels for flux. This search potentially removes false positives, as pixels that are saturated in all groups of an integration will have a value of zero in the slope image and therefore appear dead.

Dead Flux Check Files¶

Files to use for the dead flux check. These should be raw (i.e. unal) files.

Flux Check¶

Signal level threshold to use during the dead flux check test. Pixels flagged as dead and with signals less than this signal level are considered dead.

Maximum Low QE Normalized Signal¶

The maximum normalized signal a pixel can have and be considered low QE.

Maximum Normalized Signal in Adjacent to Open Pixels¶

The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN

Manual Flag File¶

Ascii file containing a list of pixel coordinates and bad pixel types to be added to those found in badpix_from_flats.py and placed in the output bad pixel file. If left as ‘default’, the bad pixel file in the jwst_reffiles repository will be used.

Flat Do Not Use¶

List of bad pixel types (from the flat field files) where the DO_NOT_USE flag should also be applied (e.g. [‘DEAD’, ‘LOW_QE’, ‘OPEN’, ‘ADJ_OPEN’])

Dark Stdev Clipping Sigma¶

Number of sigma to use when sigma-clipping the 2D array of standard deviation values from the dark current slope files. The sigma-clipped mean and standard deviation are used to locate noisy pixels.

Dark Max Clipping Iterations¶

Maximum number of iterations to use when sigma clipping to find the mean and standard deviation values that are used when locating noisy pixels.

Dark Noisy Threshold¶

Number of sigma above the mean noise (associated with the slope) to use as a threshold for identifying noisy pixels in the dark current data.

Maximum Saturated Fraction¶

When identifying pixels that are fully saturated (in all groups of an integration), this is the fraction of integrations within which a pixel must be fully saturated before flagging it as HOT.

Maximum Jump Limit¶

The maximum number of jumps a pixel can have in an integration before it is flagged as a high jump pixel (which may be flagged as noisy later).

Jump Ratio Threshold¶

Cutoff for the ratio of jumps early in the ramp to jumps later in the ramp. Pixels with a ratio greater than this value (and which also have a high total number of jumps) will be flagged as potential (I)RC pixels.

Early Cutoff Fraction¶

Fraction of the integration to use when comparing the jump rate early in the integration to that across the entire integration. Must be <= 0.5

Pedestal Sigma Threshold¶

Used when searching for RC pixels via the pedestal image. Pixels with pedestal values more than pedestal_sigma_threshold above the mean are flagged as potential RC pixels

RC Fraction Threshold¶

Used when searching for RC pixels. This is the fraction of input files within which the pixel must be identified as an RC pixel before it will be flagged as a permanent RC pixel

Low Pedestal Fraction¶

This is the fraction of input files within which a pixel must be identified as a low pedestal pixel before it will be flagged as a permanent low pedestal pixel

High CR Fraction¶

This is the fraction of input files within which a pixel must be flagged as having a high number of jumps before it will be flagged as permanently noisy

Flag Values¶

This dictionary maps the types of bad pixels searched for to the flag mnemonics to use when creating the bad pixel file. Keys are the types of bad pixels searched for, and values are lists that include mnemonics recognized by the jwst calibration pipeline.

e.g. {‘hot’: [‘HOT’], ‘rc’: [‘RC’], ‘low_pedestal’: [‘OTHER_BAD_PIXEL’], ‘high_cr’: [“TELEGRAPH”]}

Dark Do Not Use¶

List of bad pixel types from the dark current data to be flagged as DO_NOT_USE.

e.g. [‘hot’, ‘rc’, ‘low_pedestal’, ‘high_cr’]

Plot¶

If True, produce plots of intermediate results.

Output File¶

Name of the CRDS-formatted bad pixel reference file to save the final bad pixel map into

Author¶

CRDS-required name of the reference file author, to be placed in the referece file header

Description¶

CRDS-required description of the reference file, to be placed in the reference file header

Pedigree¶

CRDS-required pedigree of the data used to create the reference file

Useafter¶

CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01 00:00:00’)

History¶

String containing any text you wish to place in the HISTORY keyword of the output bad pixel mask reference file. Note that all input filenames will automatically be placed in the HISTORY keyword independent of the string entered here.

Quality Check¶

Boolean. If True, the pipeline is run using the output reference file to be sure the pipeline doens’t crash

Current Limitations¶

Currently, only one type of dead pixel search can be performed for a given call.