API Documentation

create reference files for JWST instruments A. Rest

Bad Pixel Mask

Wrapper script that calls badpix_from_flats.py and badpix_from_darks.py. The results are combined to create a single bad pixel reference file.

jwst_reffiles.bad_pixel_mask.bad_pixel_mask.bad_pixels(flat_slope_files=None, dead_search=True, low_qe_and_open_search=True, dead_search_type='sigma_rate', flat_mean_sigma_threshold=3, flat_mean_normalization_method='smoothed', smoothing_box_width=15, smoothing_type='Box2D', dead_sigma_threshold=5.0, max_dead_norm_signal=None, run_dead_flux_check=False, dead_flux_check_files=None, flux_check=45000, max_low_qe_norm_signal=0.5, max_open_adj_norm_signal=1.05, manual_flag_file='default', flat_do_not_use=[], dark_slope_files=None, dark_uncal_files=None, dark_jump_files=None, dark_fitopt_files=None, dark_stdev_clipping_sigma=5.0, dark_max_clipping_iters=5, dark_noisy_threshold=5, max_saturated_fraction=0.5, max_jump_limit=10, jump_ratio_threshold=5, early_cutoff_fraction=0.25, pedestal_sigma_threshold=5, rc_fraction_threshold=0.8, low_pedestal_fraction=0.8, high_cr_fraction=0.8, flag_values={'high_cr': ['TELEGRAPH'], 'hot': ['HOT'], 'low_pedestal': ['OTHER_BAD_PIXEL'], 'rc': ['RC']}, dark_do_not_use=['hot', 'rc', 'low_pedestal', 'high_cr'], plot=False, output_file=None, author='jwst_reffiles', description='A bad pix mask', pedigree='GROUND', useafter='2019-04-01 00:00:00', history='', quality_check=False)

Wrapper that calls the two modules for finding bad pixels from input flat field files, and bad pixels from dark current files.

Parameters:
  • flat_slope_files (list) – List of flat field slope files to be used for the dead pixel search. If None, the search is skipped.
  • dead_search (bool) – Whether or not to search for DEAD pixels using the flat field files
  • low_qe_and_open_search (bool) – Whether or not to search for LOW_QE, OPEN, and ADJ_OPEN pixels using the flat field files
  • dead_search_type (str) –

    Type of search to use when looking for dead pixels. Options are: sigma_rate: Using a normalized signal rate image, dead pixels

    are defined as those with a rate smaller than dead_sigma_threshold standard deviations below the mean.
    absolute_rate: Using a normalized signal rate image, dead pixels
    are defined as those with a rate less than max_dead_norm_signal.
  • flat_mean_sigma_threshold (float) – Number of standard deviations to use when sigma-clipping to calculate the mean slope image or the mean across the detector
  • flat_mean_normalization_method (str) –

    Specify how the mean image is normalized prior to searching for bad pixels. Options are: ‘smoothed’: Mean image will be smoothed using a

    smoothing_box_width x smoothing_box_width box kernel. The mean image is then normalized by this smoothed image.

    ’none’: No normalization is done. Mean slope image is used as is ‘mean’: Mean image is normalized by its sigma-clipped mean ‘fit2d’: Mean image will be normalized by a fit of 2-D surface to

    mean image. The degree of the fit is controlled by the fit_degree parameters
  • smoothing_box_width (float) – Width in pixels of the box kernel to use to compute the smoothed mean image
  • smoothing_typ (string) – Type of smoothing to do Box2D `` or ``median filtering
  • smoothing_sigma (float) – Number of standard deviations to use when smoothing in a box defined by smoothing_box_width.
  • dead_sigma_threshold (float) – Number of standard deviations below the mean at which a pixel is considered dead.
  • max_dead_norm_signal (float) – Maximum normalized signal rate of a pixel that is considered dead
  • run_dead_flux_check (bool) – Whether or not to check for dead pixels using an absolute flux value
  • dead_flux_check_files (list) – List of ramp (uncalibrated) files to use to check the flux of average of last 4 groups. If None then the uncalibration files are not read in and no flux_check is done.
  • flux_check (float) – Tolerance on average signal in last 4 groups. If dead_flux_check_files is a list of uncalibrated files, then the average of the last four groups for all the integrations is determined. If this average > flux_check then this pixel is not a dead pixel.
  • max_low_qe_norm_signal (float) – The maximum normalized signal a pixel can have and be considered low QE.
  • max_open_adj_norm_signal (float) – The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN
  • manual_flag_file (str) – Name of the ascii file containing a list of pixels to be added manually to the output bad pixel mask file. Default is ‘default’, in which case the file contained in the bad_pixel_mask directory of the repo will be used.
  • flat_do_not_use (list) – List of bad pixel types where the DO_NOT_USE flag should also be applied (e.g. [‘DEAD’, ‘LOW_QE’])
  • dark_slope_files (list) – List of dark current slope files to be used for the noisy pixel search. If None, the search is skipped.
  • dark_uncal_files (list) – List of uncalibrated dark current ramp files. These should correspond 1-to-1 with the files listed in dark_slope_files. If None, the code assumes the files are in the same location as the slope files and have names ending in uncal.fits
  • dark_jump_files (list) – List of dark current ramp files output from the jump step of the pipeline. These should correspond 1-to-1 with the files listed in dark_slope_files. If None, the code assumes the files are in the same location as the slope files and have names ending in jump.fits
  • dark_fitopt_files (list) – List of optional output files produced by the ramp-fitting step of the pipeline. These should correspond 1-to-1 with the files listed in dark_slope_files. If None, the code assumes the files are in the same location as the slope files and have names ending in fitopt.fits
  • dark_stdev_clipping_sigma (int) – Number of sigma to use when sigma-clipping the 2D array of standard deviation values. The sigma-clipped mean and standard deviation are used to locate noisy pixels.
  • dark_max_clipping_iters (int) – Maximum number of iterations to use when sigma clipping to find the mean and standard deviation values that are used when locating noisy pixels.
  • dark_noisy_threshold (int) – Number of sigma above the mean noise (associated with the slope) to use as a threshold for identifying noisy pixels.
  • max_saturated_fraction (float) – When identifying pixels that are fully saturated (in all groups of an integration), this is the fraction of integrations within which a pixel must be fully saturated before flagging it as HOT
  • max_jump_limit (int) – The maximum number of jumps a pixel can have in an integration before it is flagged as a high jump pixel (which may be flagged as noisy later)
  • jump_ratio_threshold (int) – Cutoff for the ratio of jumps early in the ramp to jumps later in the ramp. Pixels with a ratio greater than this value (and which also have a high total number of jumps) will be flagged as potential (I)RC pixels.
  • early_cutoff_fraction (float) – Fraction of the integration to use when comparing the jump rate early in the integration to that across the entire integration. Must be <= 0.5
  • pedestal_sigma_threshold (int) – Used when searching for RC pixels via the pedestal image. Pixels with pedestal values more than pedestal_sigma_threshold above the mean are flagged as potential RC pixels
  • rc_fraction_threshold (float) – Used when searching for RC pixels. This is the fraction of input files within which the pixel must be identified as an RC pixel before it will be flagged as a permanent RC pixel
  • low_pedestal_fraction (float) – This is the fraction of input files within which a pixel must be identified as a low pedestal pixel before it will be flagged as a permanent low pedestal pixel
  • high_cr_fraction (float) – This is the fraction of input files within which a pixel must be flagged as having a high number of jumps before it will be flagged as permanently noisy
  • flag_values (dict) – This dictionary maps the types of bad pixels searched for to the flag mnemonics to use when creating the bad pixel file. Keys are the types of bad pixels searched for, and values are lists that include mnemonics recognized by the jwst calibration pipeline e.g. {‘hot’: [‘HOT’], ‘rc’: [‘RC’], ‘low_pedestal’: [‘OTHER_BAD_PIXEL’], ‘high_cr’: [“TELEGRAPH”]}
  • dark_do_not_use (list) – List of bad pixel types to be flagged as DO_NOT_USE e.g. [‘hot’, ‘rc’, ‘low_pedestal’, ‘high_cr’]
  • plot (bool) – If True, produce and save intermediate results from noisy pixel search
  • output_file (str) – Name of the CRDS-formatted bad pixel reference file to save the final bad pixel map into
  • author (str) – CRDS-required name of the reference file author, to be placed in the referece file header
  • description (str) – CRDS-required description of the reference file, to be placed in the reference file header
  • pedigree (str) – CRDS-required pedigree of the data used to create the reference file
  • useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01 00:00:00’)
  • history (str) – Text to be added to the HISOTRY section of the output bad pixel file
  • quality_check (bool) – If True, the pipeline is run using the output reference file to be sure the pipeline doens’t crash
jwst_reffiles.bad_pixel_mask.bad_pixel_mask.create_output_filename(inst_name, det_name)

Create a default output filename for the bad pixel mask given instrument and detector names

Parameters:
  • inst_name (str) – Instrument name
  • det_name (str) – Detector name
Returns:

outfile – Default bad pixel mask filename
Return type:

str
jwst_reffiles.bad_pixel_mask.bad_pixel_mask.instrument_info(filename)

Get the instrument and detector name from the header of the input file

Parameters:filename (str) – Name of fits file
Returns:
  • inst (str) – Instrument name
  • det (str) – Detector name
jwst_reffiles.bad_pixel_mask.bad_pixel_mask.save_final_map(bad_pix_map, instrument, detector, hdulist, files, author, description, pedigree, useafter, history_text, outfile)

Save a bad pixel map into a CRDS-formatted reference file

Parameters:
  • bad_pix_map (numpy.ndarray) – 2D bad pixel array
  • instrument (str) – Name of instrument associated with the bad pixel array
  • detector (str) – Name of detector associated with the bad pixel array
  • hdulist (astropy.fits.HDUList) – HDUList containing “extra” fits keywords
  • files (list) – List of files used to create bad_pix_map
  • author (str) – Author of the bad pixel mask reference file
  • description (str) – CRDS description to use in the final bad pixel file
  • pedigree (str) – CRDS pedigree to use in the final bad pixel file
  • useafter (str) – CRDS useafter string for the bad pixel file
  • history_text (list) – List of strings to add as HISTORY entries to the bad pixel file
  • outfile (str) – Name of the output bad pixel file

Badpix From Flats

This module can be used to create JWST-format bad pixel mask reference files for use in the dq_init step of the JWST calibration pipeline.

Author

  • Bryan Hilbert

Use

This module can be imported and used as such:

from jwst_reffiles.bad_pixel_mask import badpix_from_flats
badpix_from_flats.find_bad_pix(arguments)

Notes

This algorithm is used to identify types of bad pixels that are flagged in the bad pixel mask. This includes the following types: DEAD, LOW_QE, OPEN, ADJ_OPEN

This is based on discussions within the JWST Reference File Generation Working Group in February 2019. The goal is to produce an algorithm that can be used by all JWST instruments to produce bad pixel mask reference files.

Overview: Inputs: A set of flatfield countrate images (NIRCam, NIRISS, NIRSpec)

A set of dark current exposures i.e. Ramps (MIRI)
Algorithm:

  1. For MIRI, extract and use only the 10th group from each exposure

  2. Create average image from the set of input images, using sigma-clipping

  3. Create smoothed version of the average image (15x15 smoothing)

  4. Divide average image by smoothed image

  5. Calculate sigma-clipped mean and stdev in normalized image

  6. Find bad pix:

    NIRCam, NIRISS: DEAD+DO_NOT_USE if signal < (mean-N*stdev) NIRSPec, MIRI: DEAD+DO_NOT_USE if signal < 0.05

    LOW_QE if 0.05 < signal < 0.5 and signal in 4 adjacent pixels are below ADJ_OPEN threshold (1.05)

References

Working group discussion and algorithm details are presented at: https://outerspace.stsci.edu/display/JWSTCC/Algorithm+details%3A+DQ+Init

Dependencies

  • jwst
  • astropy
  • numpy
jwst_reffiles.bad_pixel_mask.badpix_from_flats.average_last4groups(hdu_list)

Find the median of all the group from each integration

Parameters:data (numpy.ndarray) – 3D or 4D array of data
Returns:data – 3D array (10th group only)
Return type:numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.combine_individual_maps(bad_maps, do_not_use_flags)

Given multiple separate bad pixel maps, combine using bit values matching those defined in the JWST calibraiton pipeline

Parameters:bad_maps (dict) – Dictionary containing the various individual bad pixel maps to be combined. Format is (e.g.) {‘DEAD’: dead_map, ‘LOW_QE’: low_qe_map}
Returns:final_map – 2D array containing the combined bad pixel map
Return type:numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.crds_input_checks(author_name, descrip, pedigree, use_after)

Perform some basic checks on the input values to be placed in the header of the reference file. These check against CRDS requirements

Parameters:
  • Results –
  • ------- –
jwst_reffiles.bad_pixel_mask.badpix_from_flats.create_dqdef()

Create the DQ definition data needed to populate the final bad pixel reference file

Returns:definitions – Bad pixel bit definitions
Return type:list
jwst_reffiles.bad_pixel_mask.badpix_from_flats.create_dummy_hdu_list(sigma_threshold, smoothing_width, dead_sigma_threshold, dead_max_rate, low_qe_max_rate, open_adj_max_rate)

Create a fits HDU List to contain the “extra” fits header keywords that specific to this module and not included in the datamodels. This will be used to initialize the datamodel when creating the reference file

Parameters:
  • sigma_threshold (float) – Number of standard deviations to use when sigma-clipping to calculate the mean slope image or the mean across the detector
  • smoothing_width (float) – Width in pixels of the box kernel to use to compute the smoothed mean image
  • dead_sigma_threshold (float) – Number of standard deviations below the mean at which a pixel is considered dead.
  • dead_max_rate (float) – Maximum normalized signal rate of a pixel that is considered dead
  • low_qe_max_rate (float) – The maximum normalized signal a pixel can have and be considered low QE.
  • open_adj_max_rate (float) – The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN
Returns:

hdu_list – HDU List containing a header with just the extra keywords specific to this module
Return type:

astropy.io.fits.HDUList
jwst_reffiles.bad_pixel_mask.badpix_from_flats.dead_pixels_absolute_rate(rate_image, max_dead_signal=0.05)

Create a map of dead pixels given a normalized rate image. In this case pixels are flagged as dead if their normalized signal rate is less than max_dead_signal.

Parameters:
  • rate_image (numpy.ndarray) – 2D normalized rate image
  • max_dead_signal (float) – Maximum normalized signal rate of a pixel that is considered dead
Returns:

dead_pix_map – 2D map showing DEAD pixels. Good pixels have a value of 0.
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.dead_pixels_flux_check(dead_pix_map, ave_group, flux_check)

Check the dead_pix_map with the average_rate of the last 4 groups. We want to remove cases where a pixel was flagged as dead because it is a hot pixel that saturates on the first group causing the rate for this pixel to be close to 0.

Parameters:
  • dead_pix_map (numpy.ndarray) – 2D map showing DEAD pixels. Good pixels have a value of 0.
  • ave_group (numpy.ndaarray) – 2D array of the average signal in the last 4 groups from an average of all integrations and exposures.
  • flux_check (float) – if pixel have a ave_group > flux_check this is not a dead pixel
Returns:

clean_dead_pix_map – 2D map showing DEAD pixels. Good pixels have a value of 0. Clean up dead pixel map removing saturated pixels from dead flag.
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.dead_pixels_sigma_rate(rate_image, mean_rate, stdev_rate, sigma=5.0)

Create a map of dead pixels given a normalized rate image. In this case pixels are flagged as dead if their normalized signal rate is less than sigma standard deviations below the mean

Parameters:
  • rate_image (numpy.ndarray) – 2D normalized rate image
  • mean_rate (float) – Sigma-clipped mean value of the rate_image
  • stdev_rate (float) – Sigma-clipped standard deviation of the rate_image
  • sigma (float) – Number of standard deviations below the mean at which a pixel is considered dead.
Returns:

dead_pix_map – 2D map showing DEAD pixels. Good pixels have a value of 0.
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.determine_manual_file(inst_name, detector_name)

Find the default manual bad pixel file for the given instrument and detector

Parameters:
  • inst_name (str) – Name of JWST instrument
  • detector_name (str) – Name of detector
Returns:

filename – Full path and filename of the bad pixel file
Return type:

str
jwst_reffiles.bad_pixel_mask.badpix_from_flats.extract_10th_group(hdu_list)

Keep only the 10th group from each integration

Parameters:data (numpy.ndarray) – 3D or 4D array of data
Returns:data – 3D array (10th group only)
Return type:numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.find_bad_pix(input_files, dead_search=True, low_qe_and_open_search=True, dead_search_type='sigma_rate', sigma_threshold=3, normalization_method='smoothed', smoothing_box_width=15, smoothing_type='Box2D', dead_sigma_threshold=5.0, max_dead_norm_signal=None, run_dead_flux_check=False, dead_flux_check_files=None, flux_check=45000, max_low_qe_norm_signal=0.5, max_open_adj_norm_signal=1.05, manual_flag_file='default', do_not_use=[], output_file=None, author='jwst_reffiles', description='A bad pix mask', pedigree='GROUND', useafter='2019-04-01 00:00:00', history='', quality_check=True)

MAIN SCRIPT: Given a set of input files, create dead, low QE, and open pixel maps

Parameters:
  • input_files (list) – Fits files to be used in the search for dead/open pixels
  • dead_search (bool) – Whether or not to search for DEAD pixels
  • low_qe_and_open_search (bool) – Whether or not to search for LOW_QE, OPEN, and ADJ_OPEN pixels
  • dead_search_type (str) –

    Type of search to use when looking for dead pixels. Options are: sigma_rate: Using a normalized signal rate image, dead pixels

    are defined as those with a rate smaller than dead_sigma_threshold standard deviations below the mean.
    absolute_rate: Using a normalized signal rate image, dead pixels
    are defined as those with a rate less than max_dead_norm_signal.
  • sigma_threshold (float) – Number of standard deviations to use when sigma-clipping to calculate the mean slope image or the mean across the detector
  • normalization_method (str) –

    Specify how the mean image is normalized prior to searching for bad pixels. Options are: ‘smoothed’: Mean image will be smoothed using a

    smoothing_box_width x smoothing_box_width box kernel. The mean image is then normalized by this smoothed image.

    ’none’: No normalization is done. Mean slope image is used as is ‘mean’: Mean image is normalized by its sigma-clipped mean

  • smoothing_box_width (float) – Width in pixels of the box kernel to use to compute the smoothed mean image
  • smoothing_typ (string) – Type of smoothing to do Box2D `` or ``median filtering
  • smoothing_sigma (float) – Number of standard deviations to use when smoothing in a box defined by smoothing_box_width.
  • dead_sigma_threshold (float) – Number of standard deviations below the mean at which a pixel is considered dead.
  • max_dead_norm_signal (float) – Maximum normalized signal rate of a pixel that is considered dead
  • run_dead_flux_check (bool) – Whether or not to check for dead pixels using an absolute flux value
  • dead_flux_check_files (list) – List of ramp (uncalibrated) files to use to check the flux of average of last 4 groups. If None then the uncalibration files are not read in and no flux_check is done.
  • flux_check (float) – Tolerance on average signal in last 4 groups. If dead_flux_check_files is a list of uncalibrated files, then the average of the last four groups for all the integrations is determined. If this average > flux_check then this pixel is not a dead pixel.
  • max_low_qe_norm_signal (float) – The maximum normalized signal a pixel can have and be considered low QE.
  • max_open_adj_norm_signal (float) – The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN
  • manual_flag_file (str) – Name of the ascii file containing a list of pixels to be added manually to the output bad pixel mask file. Default is ‘default’, in which case the file contained in the bad_pixel_mask directory of the repo will be used.
  • do_not_use (list) – List of bad pixel types where the DO_NOT_USE flag should also be applied (e.g. [‘DEAD’, ‘LOW_QE’])
  • output_file (str) – Name of the CRDS-formatted bad pixel reference file to save the final bad pixel map into
  • author (str) – CRDS-required name of the reference file author, to be placed in the referece file header
  • description (str) – CRDS-required description of the reference file, to be placed in the reference file header
  • pedigree (str) – CRDS-required pedigree of the data used to create the reference file
  • useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01 00:00:00’)
  • history (str) – Text to be added to the HISOTRY section of the output bad pixel file
  • quality_check (bool) – If True, the pipeline is run using the output reference file to be sure the pipeline doens’t crash
Returns:

final_map – 2D array containing the combined bad pixel map
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.find_open_and_low_qe_pixels(rate_image, max_dead_signal=0.05, max_low_qe=0.5, max_adj_open=1.05)

Create maps of open (and adjacent to open) and low QE pixels given a normalized rate image

Parameters:
  • rate_image (numpy.ndarray) – 2D normalized rate image
  • mean_rate (float) – Sigma-clipped mean value of the rate_image
  • stdev_rate (float) – Sigma-clipped standard deviation of the rate_image
  • max_dead_signal (float) – The maximum normalized signal a pixel can have and be considered dead
  • max_low_qe (float) – The maximum normalized signal a pixel can have and be considered low QE.
  • max_adj_open (float) – The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN
Returns:

  • low_qe_map (numpy.ndarray) – 2D map showing LOW QE pixels. Good pixels have a value of 0.
  • open_pix_map (numpy.ndarray) – 2D map showing OPEN pixels. Good pixels have a value of 0.
  • adj_pix_map (numpy.ndarray) – 2D map showing ADJ_OPEN pixels. Good pixels have a value of 0.
jwst_reffiles.bad_pixel_mask.badpix_from_flats.fit_surface(data, deg)

Fit the rate image with a 2-D surface plot

Parameters:
  • data (numpy.ndarray) – 2D image
  • deg (degree of polynomial fit) –
Returns:

surface_image – 2D image of polynomial fit
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.flag_type_check(flags)

Check that the flags in the input list are valid JWST bad pixel types

Parameters:flag (list) – List of flag types (e.g. [‘DEAD’, ‘LOW_QE’])
Returns:flag – Modified list (all flags made uppercase)
Return type:list
jwst_reffiles.bad_pixel_mask.badpix_from_flats.get_adjacent_pixels(x_val, y_val, x_dim, y_dim)

Return coordinates of the pixels immediately adjacent to the input pixel coordinate

Parameters:
  • x_val (int) – x coordinate of central pixel
  • y_val (int) – y coordinate of central pixel
  • x_dim (int) – Size of the array in the x dimension
  • y_dim (int) – Size of the array in the y dimension
Returns:

  • adj_x (numpy.ndarray) – x coordinates of adjacent pixel coordinates
  • adj_y (numpy.ndarray) – y coordinates of adjacent pixel coordinates
jwst_reffiles.bad_pixel_mask.badpix_from_flats.get_fastaxis(filename)

Get the fastaxis and slowaxis values from the input file

Parameters:filename (str) – Name of fits file to get values from
Returns:
  • fastaxis (int) – Value of FASTAXIOS keyword
  • slowaxis (int) – Value of SLOWAXIS keyword
jwst_reffiles.bad_pixel_mask.badpix_from_flats.get_max_dead_norm_signal_default(instrument, detector, normalization_method)

Finds the default max_dead_norm_signal value to use for the absolute_rate dead pixel search type.

Parameters:
  • instrument (str) – The name of the instrument
  • detector (str) – The name of the detector
  • normalization_method (str) –

    Specify how the mean image is normalized prior to searching for bad pixels. Options are: ‘smoothed’: Mean image will be smoothed using a

    smoothing_box_width x smoothing_box_width box kernel. The mean image is then normalized by this smoothed image.

    ’none’: No normalization is done. Mean slope image is used as is ‘mean’: Mean image is normalized by its sigma-clipped mean
Returns:

default_value – The default max_dead_norm_signal value
Return type:

float
jwst_reffiles.bad_pixel_mask.badpix_from_flats.image_stats(image, sigma=3.0)

Calculate the sigma-clipped mean and standard deviation across the given image

Parameters:
  • image (numpy.ndarray) – 2D image
  • sigma (float) – Sigma threshold to use for sigma-clipping
Returns:

  • mean_val (float) – Sigma-clipped mean value
  • stdev_val (float) – Sigma-clipped standard deviation
jwst_reffiles.bad_pixel_mask.badpix_from_flats.manual_flags(filename, map_shape, no_use)

Create a bad pixel mask from an input ascii file containing pixel indicies and flag types

Parameters:
  • filename (str) – Name of ascii file containing flags
  • map_shape (tup) – Shape of the bad pixel mask to create (y, x)
  • no_use (list) – List of mnemonics to flag as DO_NOT_USE
Returns:

mask – 2D array of bad pixels, with bit values equal to the standard JWST values
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.mean_stdev_images(data, sigma=3.0)

Calculate the sigma-clipped mean and stdev images from a stack of input images

Parameters:
  • data (numpy.ndarray) – Stack of 2D images (i.e 3D array)
  • sigma (float) – Threshold value to use for sigma clipping.
Returns:

  • mean_image (numpy.ndarray) – 2D array of the mean image
  • stdev_image (numpy.ndarray) – 2D array of the stdev image
jwst_reffiles.bad_pixel_mask.badpix_from_flats.miri_bad_columns(dimensions)

Create a map that flags the shorted columns on the MIRI detector

Parameters:dimensions (tup) – (y, x) dimensions, in pixels, of the map to create
Returns:shorted_map – 2D map showing the locations of the shorted columns (1)
Return type:numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.none_check(value)

If value is a string containing ‘none’, then change it to a NoneType object

Parameters:value (str or NoneType) –
Returns:new_value
Return type:NoneType
jwst_reffiles.bad_pixel_mask.badpix_from_flats.pad_with_refpix(data, instrument_name)

Pad the given image with an outer 4 rows and columns of (zeroed out) reference pixels.

Parameters:
  • data (numpy.ndarray) – 2D image
  • instrument_name (str) – Name of the JWST instrument associated with the data
Returns:

padded – 2D image with 4 rows and columns of reference pixels added
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.pipeline_check(reference_filename, filename)

Run the dq_init step using the provided file, to check that the pipeline runs successfully

Parameters:
  • reference_filename (str) – Name of the bad pixel mask reference filename
  • filename (str) – Name of the fit exposure on which to test the reference file
jwst_reffiles.bad_pixel_mask.badpix_from_flats.range_format(table_cell, total_dimension)

Turn a string containing a possible shorthand list of indexes into an array of the specified indexes. Support the notation that if no beginning or ending index is given that the range will be to the start or end of total_dimension. Note that this function assumes that the input strings do not follow the python convention of using a maximum index value of one beyond the index you really want. (e.g. it assumes that ‘0:3’ means columns 0 through 3 inclusive are what you are interested in (meaning indexing in python would need to use [0:4] to get all the values.))

Parameters:
  • table_cell (str or int) – Value to process (e.g. ‘34:36’ or 93)
  • total_dimension (int) – The maximum index number to be returned
jwst_reffiles.bad_pixel_mask.badpix_from_flats.read_files(filenames, dead_search_type)

Read the data in the input files. Perform basic sanity checks to be sure data are consistent (same instrument, aperture, etc)

Parameters:
  • filenames (list) – List of fits filenames to be opened
  • dead_search_type (str) –

    Type of search to use when looking for dead pixels. Options are: sigma_rate: Using a normalized signal rate image, dead pixels

    are defined as those with a rate smaller than dead_sigma_threshold standard deviations below the mean.
    absolute_rate: Using a normalized signal rate image, dead pixels
    are defined as those with a rate less than max_dead_norm_signal.
    saturation_check: this option is set by the program if dead_flux_check_files
    is a list of ramp files. The average of the last four groups is determined to test if pixel is close to saturation.
Returns:

data – 3D stack of data
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.read_manual_flag_file(filename)

Read in the text file containing the list of pixels to be flagged manually.

Parameters:filename (str) – Name of ascii file to be read in
Returns:bad_table – Table of bad pixels
Return type:astropy.table.Table
jwst_reffiles.bad_pixel_mask.badpix_from_flats.reference_pixel_map(dimensions, instrument_name)

Create a map that flags all reference pixels as such

Parameters:
  • dimensions (tup) – (y, x) dimensions, in pixels, of the map to create
  • instrument_name (str) – Name of JWST instrument associated with the data
Returns:

ref_map – 2D map showing the locations of reference pixels (1)
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.save_final_map(bad_pix_map, instrument, detector, files, author, description, pedigree, useafter, history_text, sigma_thresh, smooth_width, dead_sigma_thresh, max_dead_rate, max_low_qe_rate, max_open_adj_rate, do_not_use_list, outfile)

Save a bad pixel map into a CRDS-formatted reference file

jwst_reffiles.bad_pixel_mask.badpix_from_flats.science_pixels(data, instrument_name)

Given a full frame image, strip off the reference pixels and return only the science pixels. At the moment, assume 4 rows and columns of reference pixels

Parameters:
  • data (numpy.ndarray) – 2D image
  • instrument_name (str) – Name of JWST instrument associated wit the data
Returns:

data – 2D image with outer 4 rows and columns removed
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.smooth(data, box_width=15, type='Box2D')

Create a smoothed version of the 2D input data

Parameters:
  • data (numpy.ndarray) – 2D array of data to be smoothed
  • box_width (int) – Width of the smoothing box, in pixels
  • sigma (float) – value to use for sigma clipping
Returns:

smoothed – A smoothed version of data
Return type:

numpy.ndarray
jwst_reffiles.bad_pixel_mask.badpix_from_flats.split_row(line, colname)

Convert the indicies in a given row of the manual bad pixel ascii file into integers

Parameters:
  • line (astropy.table.row.Row) – Row containing x, y, and flag values
  • colname (str) – Column name to examine
Returns:

values – If the input is an integer, the list will simply be [value]. If a range is given (e.g. 200:210), then values will be [200, 210].
Return type:

list

Badpix From Darks

Find bad pixels from dark current files

Start with a stack of dark ramps and slope images? Or maybe a stack of ramps that have been processed through the jump step and then ramp-fitting is performed here?

Input for the dark current reference file step is going to be a stack of ramps. So maybe use that and ramp-fit here.

inputs: 1. list of dark current ramps that have been run through jump step 2. list of same exposures after ramp-fitting has been done

Plot summary:

  1. Check to see if IPC correction has been run
  2. Look through stack of slopes images, get mean and rms per pixel (do we sigma-clip the inputs or not?)
  3. Potential bad pixels are those with noise values above some threshold

NOTE: when producing slope images of these data, make sure to save the optional output parameters into the *fitopt.fits files. https://jwst-pipeline.readthedocs.io/en/stable/jwst/ramp_fitting/main.html?highlight=intercept “A third, optional output product is also available and is produced only when the step parameter ‘save_opt’ is True”

jwst_reffiles.dark_current.badpix_from_darks.add_refpix(array, to_add)

Place map within a larger array that contains the reference pixels.

Parameters:
  • array (numpy.ndarray) – 2D array of bad pixels that does not contain reference pixels
  • to_add (tup) – 4-element tuple containing the number of rows and columns to add around the outside of the science pixels. (left cols, right cols, bottom rows, top rows)
Returns:

array – 2D array with rows and columns added
Return type:

numpy.ndarray
jwst_reffiles.dark_current.badpix_from_darks.apply_flags(pixmap, flag_list)

Beginning with a map indicating locations of a particular type of bad pixel, apply the bits specified in flag_list to come up with the jwst bad pixel value.

Parameters:
  • pixmap (numpy.ndarray) – 2D array indicating bad pixels. 1 for a bad pixel and 0 for a good pixel
  • flag_list (list) – List of bad pixel mnemonics to be applied. These mnemonics must be in the dictionary of bad pixel types recognized by the jwst calibration pipeline
Returns:

pixmap – Map updated with proper bad pixel values
Return type:

numpy.ndarray
jwst_reffiles.dark_current.badpix_from_darks.check_metadata(hdr, comp)

Basic metadata check for consistency from one file to another

Parameters:
  • hdr (astropy.fits.header) – Header read in from fits file
  • comp (astropy.fits.header) – Baseline header to which the comparison is done
jwst_reffiles.dark_current.badpix_from_darks.combine_bad_pixel_types(sat_map, rc_map, low_pedestal_map, high_cr_map)

Copmbine individual maps of bad pixel types into a final bad pixel map, using flag values defined in dq_flags.

Parameters:
  • sat_map (numpy.ndarray) – 2D array giving the location of pixels saturated all the way up the ramp
  • rc_map (numpy.ndarray) – 2D array giving the location of RC pixels
  • low_pededtal_map (numpy.ndarray) – 2D array giving the location of pixels with abnormally low pedestal values
  • high_cr_map (numpy.ndarray) – 2D array giving the location of pixels with abnormally high numbers of jumps
Returns:

final_map – 2D array containing the bitwise combined bad pixel maps
Return type:

numpy.ndarray
jwst_reffiles.dark_current.badpix_from_darks.combine_clean_slopes(slope_stack, islope_stack)

Combine the stack of slopes and form the stanard deviation of the pixel slopes

Parameters:
  • slope_stack (list) – A list of slopes for full array stacked for each integration
  • islope_stack (list) – A list of of 1 or 0 for each integration. A 1 is a good slope and 0 is slope with cosmic ray
jwst_reffiles.dark_current.badpix_from_darks.extract_group2(filename, refpix)

Read in the PEDESTAL values from a *fitopt.fits file

Parameters:
  • filename (str) – Name of uncalibrated file. This should be a *uncal.fits file.
  • refpix (tup) – 4-element tuple listing the number of outer rows and columns that are reference pixels
Returns:

group2 – 3D array of group 2
Return type:

numpy.ndarray
jwst_reffiles.dark_current.badpix_from_darks.find_bad_pix(filenames, uncal_filenames=None, jump_filenames=None, fitopt_filenames=None, clipping_sigma=5.0, max_clipping_iters=5, noisy_threshold=5, max_saturated_fraction=0.5, max_jump_limit=10, jump_ratio_threshold=5, early_cutoff_fraction=0.25, pedestal_sigma_threshold=5, rc_fraction_threshold=0.8, low_pedestal_fraction=0.8, high_cr_fraction=0.8, flag_values={'high_cr': ['TELEGRAPH'], 'hot': ['HOT'], 'low_pedestal': ['OTHER_BAD_PIXEL'], 'rc': ['RC']}, do_not_use=['hot', 'rc', 'low_pedestal', 'high_cr'], outfile=None, plot=False)

MAIN FUNCTION

Parameters:
  • filenames (list) – List of dark current slope files. These should be slope images.
  • uncal_filenames (list) – List of uncal files. Should have a 1-to-1 correspondence to the files in filenames. If None, the scipt will look in the same directory containing filenames, and assume that the only difference in filename is that rate.fits is replaced with uncal.fits. Uncal files are only used when working with MIRI data.
  • jump_filenames (list) – List of exposures output from the jump step of the pipeline. Should have a 1-to-1 correspondence to the files in filenames. If None, the scipt will look in the same directory containing filenames, and assume that the only difference in filename is that rate.fits is replaced with jump.fits
  • fitopt_filenames (list) – List of exposures from the optional output from the ramp_fitting step of the pipeline. Should have a 1-to-1 correspondence to the files in filenames. If None, the scipt will look in the same directory containing filenames, and assume that the only difference in filename is that rate.fits is replaced with fitopt.fits
  • clipping_sigma (int) – Number of sigma to use when sigma-clipping the 2D array of standard deviation values from the dark current slope files. The sigma-clipped mean and standard deviation are used to locate noisy pixels.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma clipping to find the mean and standard deviation values that are used when locating noisy pixels.
  • noisy_threshold (int) – Number of sigma above the mean noise (associated with the slope) to use as a threshold for identifying noisy pixels.
  • max_saturated_fraction (float) – When identifying pixels that are fully saturated (in all groups of an integration), this is the fraction of integrations within which a pixel must be fully saturated before flagging it as HOT
  • max_jump_limit (int) – The maximum number of jumps a pixel can have in an integration before it is flagged as a high jump pixel (which may be flagged as noisy later)
  • jump_ratio_threshold (int) – Cutoff for the ratio of jumps early in the ramp to jumps later in the ramp. Pixels with a ratio greater than this value (and which also have a high total number of jumps) will be flagged as potential (I)RC pixels.
  • early_cutoff_fraction (float) – Fraction of the integration to use when comparing the jump rate early in the integration to that across the entire integration. Must be <= 0.5
  • pedestal_sigma_threshold (int) – Used when searching for RC pixels via the pedestal image. Pixels with pedestal values more than pedestal_sigma_threshold above the mean are flagged as potential RC pixels
  • rc_fraction_threshold (float) – Used when searching for RC pixels. This is the fraction of input files within which the pixel must be identified as an RC pixel before it will be flagged as a permanent RC pixel
  • low_pedestal_fraction (float) – This is the fraction of input files within which a pixel must be identified as a low pedestal pixel before it will be flagged as a permanent low pedestal pixel
  • high_cr_fraction (float) – This is the fraction of input files within which a pixel must be flagged as having a high number of jumps before it will be flagged as permanently noisy
  • flag_values (dict) – This dictionary maps the types of bad pixels searched for to the flag mnemonics to use when creating the bad pixel file. Keys are the types of bad pixels searched for, and values are lists that include mnemonics recognized by the jwst calibration pipeline e.g. {‘hot’: [‘HOT’], ‘rc’: [‘RC’], ‘low_pedestal’: [‘OTHER_BAD_PIXEL’], ‘high_cr’: [“TELEGRAPH”]}
  • do_not_use (list) – List of bad pixel types to be flagged as DO_NOT_USE e.g. [‘hot’, ‘rc’, ‘low_pedestal’, ‘high_cr’]
  • plot (bool) – If True, produce plots of intermediate results.
  • outfile (str) – Name of fits file to save the resulting bad pixel mask to
jwst_reffiles.dark_current.badpix_from_darks.find_pix_with_many_jumps(jump_map, max_jump_limit=10, jump_ratio_threshold=5, early_cutoff_fraction=0.25)

Identify pixels that have an abnormal number of flagged jumps. Do this by finding the jump rate early in the ramp versus that later in the ramp.

Parameters:
  • jump_map (numpy.ndarray) – Map of jump flags for all pixels (e.g. output from dqflags.flag_map) Assume we are working one integration at a time.
  • max_jump_limit (int) – The maximum number of jumps a pixel can have in an integration before it is flagged as a high jump pixel (which may be flagged as noisy later)
  • jump_ratio_threshold (int) – Cutoff for the ratio of jumps early in the ramp to jumps later in the ramp. Pixels with a ratio greater than this value (and which also have a high total number of jumps) will be flagged as potential (I)RC pixels.
  • early_cutoff_fraction (float) – Fraction of the integration to use when comparing the jump rate early in the integration to that across the entire integration. Must be <= 0.5
Returns:

  • high_jumps (numpy.ndarray) – Map of pixels that have more than max_jump_limit jumps.
  • potential_rc (numpy.ndarray) – Map of pixels which have: 1) a large number of jumps, and 2) a higher rate of jumps early in the ramp than later in the ramp.
jwst_reffiles.dark_current.badpix_from_darks.pedestal_stats(pedestal_array, threshold=5)

Get statsitics on the pedestal array corresponding to an integration

Parameters:
  • pedestal_array (numpy.ndarray) – Array of pedestal values
  • threshold (int) – Number of sigma above or below the mean at which a pedestal value is considered bad
Returns:

suspicioius_pedestal – Map of bad pedestal values
Return type:

numpy.ndarray
jwst_reffiles.dark_current.badpix_from_darks.plot_histogram_stats(data_array, cut_limit, nbins, outdir, titleplot, fileout, xaxis_log=False)

Plot a histogram of stats and over the upper limit cut off

Parameters:
  • data_array (numpy.ndarray) – 2D data to make a histogram from
  • sigma_threshold (float) – used to plotting sigma clip line on plot
  • nbins (integer) – number of bins in creating histogram
  • titleplot (string) – title of the plot
  • fileout (string) – output file of the plot
Returns:
Return type:

prints the plot to disk
jwst_reffiles.dark_current.badpix_from_darks.plot_image(image, image_max, outdir, titleplot, fileout)

Plot an Image

Parameters:
  • image (numpy.ndarray) – 2D image to plot
  • image_max (float) – maximum of image to use for scaling the image
  • titleplot (string) – title of the plot
  • fileout (string) – output file of the plot
Returns:
Return type:

prints the plot to disk
jwst_reffiles.dark_current.badpix_from_darks.read_pedestal_data(filename, refpix)

Read in the PEDESTAL values from a *fitopt.fits file

Parameters:
  • filename (str) – Name of output file to check. This should be a *fitopt.fits file.
  • refpix (tup) – 4-element tuple listing the number of outer rows and columns that are reference pixels
Returns:

pedestal – 3D array of pedestal values (signal extrapolated to time=0)
Return type:

numpy.ndarray
jwst_reffiles.dark_current.badpix_from_darks.read_slope_data(filename, refpix)

Read in the science extension from a group of slope images

Parameters:
  • filenames (list) – List of fits files containing slope values
  • refpix (tup) – 4-element tuple listing the number of outer rows and columns that are reference pixels
Returns:

slope – 2D or 3D array of slope values (signal extrapolated to time=0) Reference pixels have been stripped off.
Return type:

numpy.ndarray
jwst_reffiles.dark_current.badpix_from_darks.read_slope_files(filenames)

Read in the science extension from a group of slope images

Parameters:filenames (list) – List of fits files containing slope values
Returns:
  • slope_data (numpy.ndarray) – 3D array containing slope values for science pixels only. Reference pixels have been stripped off.
  • left_cols (int) – Number of columns of reference pixels on the left side of the array
  • right_cols (int) – Number of columns of reference pixels on the right side of the array
  • bottom_rows (int) – Number of rows of reference pixels on the bottom of the array
  • top_rows (int) – Number of rows of reference pixels on the top of the array
jwst_reffiles.dark_current.badpix_from_darks.read_slope_integrations(filenames)

Read in the science extension from a group of slope images

Parameters:filenames (list) – List of fits files containing slope values
Returns:
  • instrument (str) – Name of instrument associated with the data
  • slope_data (numpy.ndarray) – 3D array containing slope values for science pixels only. Reference pixels have been stripped off.
  • starting_indexes (list) – List of numbers corresponding to the index numbers within slope_data where each file’s data begins. For example, if slope_data is an array of size (10, 2048, 2048), and starting_indexes = [0, 5, 7, 10] then we can pull apart slope_data into its constituent exposures using slope_data[starting_indexes[0]: starting_indexes[1]], etc.
  • left_cols (int) – Number of columns of reference pixels on the left side of the array
  • right_cols (int) – Number of columns of reference pixels on the right side of the array
  • bottom_rows (int) – Number of rows of reference pixels on the bottom of the array
  • top_rows (int) – Number of rows of reference pixels on the top of the array
jwst_reffiles.dark_current.badpix_from_darks.saturated_in_all_groups(pedestal_array, first_group_sat)

Generate a list of pixels that are saturated in all groups

Parameters:
  • pedestal_array (numpy.ndarray) – 2D array of pedestal values (signal extrapolated to time=0)
  • first_group_sat (numpy.ndarray) – 2D array of the first group DQ containing either 0 = not saturated or 2 = saturated.
Returns:

full_saturation – Boolean array describing which pixels are saturated in all reads
Return type:

numpy.ndarray
jwst_reffiles.dark_current.badpix_from_darks.slopes_not_cr(slope, number_of_jumps)

Create an array of pixel slopes which are clean and have no detected cosmic rays

Parameters:
  • slope (numpy.ndarray) – Array of pixel slopes for integration
  • number_of_jumps (numpy.ndarray) – Array holding the number of jumps detected for each pixel ramp
Returns:

  • clean_slope (numpy.ndarray) –

    array of slopes for an integration containing no cosmic rays

    slopes for a pixel ramp that have a cosmic ray detected are set to nan

  • iclean_slope (numpy.ndarray) – an array of that holds if a good slope was detected. A value of 1 is assigned to array elements where there are no CRs

Readnoise

This module creates a readnoise reference file that can be used in the JWST calibration pipeline.

Author

  • Ben Sunnquist

Use

This module can be imported and used as such:

from jwst_reffiles.readnoise import readnoise
readnoise.make_readnoise(arguments)

Notes

Overview: Inputs: A list of dark current ramps

Algorithm:

Method 1 (method=’stack’): Create CDS images for each input ramp. Stack all of these images from each input ramp together. The readnoise is the sigma-clipped standard deviation through this master CDS image stack.

Method 2 (method=’ramp’): Calculate the readnoise in each ramp individually, and then average these readnoise images together to produce a final readnoise map.

jwst_reffiles.readnoise.readnoise.calculate_mean(stack, clipping_sigma=3.0, max_clipping_iters=3)

Calculates the sigma-clipped mean through a stack of images.

Parameters:
  • stack (numpy.ndarray) – A 3D stack of images.
  • clipping_sigma (float) – Number of sigmas to use when sigma-clipping the input stack.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the input stack.
Returns:

mean_image – 2D image of the sigma-clipped mean through the input stack.
Return type:

numpy.ndarray
jwst_reffiles.readnoise.readnoise.calculate_stddev(stack, clipping_sigma=3.0, max_clipping_iters=3)

Calculates the sigma-clipped standard deviation through a stack of images.

Parameters:
  • stack (numpy.ndarray) – A 3D stack of images.
  • clipping_sigma (float) – Number of sigmas to use when sigma-clipping the input stack.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the input stack.
Returns:

stddev_image – 2D image of the sigma-clipped standard deviation through the input stack.
Return type:

numpy.ndarray
jwst_reffiles.readnoise.readnoise.get_crds_info(filename, subarray, readpatt)

Get CRDS-required header information to put into the final readnoise reference file.

Parameters:
  • filename (str) – Path to one of the files being used to generate the readnoise reference file. Will be used to find default values.
  • subarray (str) – CRDS-required subarray for which to use this reference file for.
  • readpatt (str) – CRDS-required read pattern for which to use this reference file for.
Returns:

  • instrument (str) – CRDS-required instrument for which to use this reference file for.
  • detector (str) – CRDS-required detector for which to use this reference file for.
  • subarray (str) – CRDS-required subarray for which to use this reference file for.
  • readpatt (str) – CRDS-required read pattern for which to use this reference file for.
  • fastaxis (int) – CRDS-required fastaxis of the reference file.
  • slowaxis (int) – CRDS-required slowaxis of the reference file.
  • substrt1 (int) – CRDS-required starting pixel in axis 1 direction.
  • substrt2 (int) – CRDS-required starting pixel in axis 2 direction.
jwst_reffiles.readnoise.readnoise.make_cds_stack(data, group_diff_type='independent')

Creates a stack of CDS images (group difference images) using the input ramp data, combining multiple integrations if necessary.

Parameters:
  • data (numpy.ndarray) – The input ramp data. The data shape is assumed to be a 4D array in DMS format (integration, group, y, x).
  • group_diff_type (str) –

    The method for calculating group differences. Options are: independent: Each group is only differenced once (e.g. 6-5, 4-3,

    2-1)
    consecutive: Each group is differenced to its neighbors (e.g.
    4-3, 3-2, 2-1)
Returns:

cds_stack – A 3D stack of the group difference images.
Return type:

numpy.ndarray
jwst_reffiles.readnoise.readnoise.make_readnoise(filenames, method='stack', group_diff_type='independent', clipping_sigma=3.0, max_clipping_iters=3, nproc=1, slice_width=50, single_value=False, outfile='readnoise_jwst_reffiles.fits', author='jwst_reffiles', description='CDS Noise Image', pedigree='GROUND', useafter='2015-10-01T00:00:00', history='', subarray=None, readpatt=None, save_tmp=False)

The main function. Creates a readnoise reference file using the input dark current ramps. See module docstring for more details.

Parameters:
  • filenames (list) – List of dark current files. These should be calibrated ramp images. The data shape in these images is assumed to be a 4D array in DMS format (integration, group, y, x).
  • method (str) –

    The method to use when calculating the readnoise. Options are: stack: Creates a master stack of all CDS images by combining the

    CDS images for each ramp and integration together. The final readnoise map is calculated by taking the stddev through this master stack.
    ramp: Calculates the readnoise in each ramp individually. The
    final readnoise map is calculated by averaging these individual readnoise maps together.
  • group_diff_type (str) –

    The method for calculating group differences. Options are: independent: Each group is only differenced once (e.g. 6-5, 4-3,

    2-1).
    consecutive: Each group is differenced to its neighbors (e.g.
    4-3, 3-2, 2-1).
  • clipping_sigma (float) – Number of sigma to use when sigma-clipping.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping.
  • nproc (int) – The number of processes to use during multiprocessing.
  • slice_width (int) – The width (in pixels) of the image slice to use during multiprocessing. The readnoise of each slice is calculated separately during multiprocessing and combined together at the end of processing. Only relevant if method==stack.
  • single_value (bool) – Option to use a single readnoise value (the average of all readnoise values) for all pixels.
  • outfile (str) – Name of the CRDS-formatted readnoise reference file to save the final readnoise map to.
  • author (str) – CRDS-required name of the reference file author, to be placed in the referece file header.
  • description (str) – CRDS-required description of the reference file, to be placed in the reference file header.
  • pedigree (str) – CRDS-required pedigree of the data used to create the reference file.
  • useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01T00:00:00’).
  • history (str) – CRDS-required history section to place in the reference file header.
  • subarray (str) – CRDS-required subarray for which to use this reference file for.
  • readpatt (str) – CRDS-required read pattern for which to use this reference file for.
  • save_tmp (bool) – Option to save the final readnoise map before turning it into CRDS format. This is useful if the CRDS transformation fails; in this scenario, you won’t lose all of the final readnoise results so all of the previous processing wasn’t for nothing.
jwst_reffiles.readnoise.readnoise.none_check(value)

If value is a string containing ‘none’, then change it to a NoneType object.

Parameters:value (str or NoneType) –
Returns:new_value
Return type:NoneType
jwst_reffiles.readnoise.readnoise.readnoise_by_ramp(filename, group_diff_type='independent', clipping_sigma=3.0, max_clipping_iters=3)

Calculates the readnoise for the given input dark current ramp.

Parameters:
  • filename (str) – The dark current ramp file. The data shape in this image is assumed to be a 4D array in DMS format (integration, group, y, x).
  • group_diff_type (str) –

    The method for calculating group differences. Options are: independent: Each group is only differenced once (e.g. 6-5, 4-3,

    2-1)
    consecutive: Each group is differenced to its neighbors (e.g.
    4-3, 3-2, 2-1)
  • clipping_sigma (float) – Number of sigma to use when sigma-clipping.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping.
jwst_reffiles.readnoise.readnoise.readnoise_by_slice(filenames, group_diff_type='independent', clipping_sigma=3.0, max_clipping_iters=3, column=0, slice_width=50)

Calculates the readnoise for a given slice in the input dark file ramps. Useful for multiprocessing and avoiding memory issues for large image stacks.

Parameters:
  • filenames (list) – List of dark current files. The data shape in these images is assumed to be a 4D array in DMS format (integration, group, y, x).
  • group_diff_type (str) –

    The method for calculating group differences. Options are: independent: Each group is only differenced once (e.g. 6-5, 4-3,

    2-1)
    consecutive: Each group is differenced to its neighbors (e.g.
    4-3, 3-2, 2-1)
  • clipping_sigma (float) – Number of sigma to use when sigma-clipping.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping.
  • column (int) – The index of the starting image column for each slice.
  • slice_width (int) – The width of the slice in pixels.
Returns:

readnoise – 2D image of the calculated readnoise.
Return type:

numpy.ndarray
jwst_reffiles.readnoise.readnoise.save_readnoise(readnoise, instrument='', detector='', subarray='GENERIC', readpatt='ANY', outfile='readnoise_jwst_reffiles.fits', author='jwst_reffiles', description='CDS Noise Image', pedigree='GROUND', useafter='2015-10-01T00:00:00', history='', fastaxis=-1, slowaxis=2, substrt1=1, substrt2=1, filenames=[])

Saves a CRDS-formatted readnoise reference file.

Parameters:
  • readnoise (numpy.ndarray) – The 2D readnoise image.
  • instrument (str) – CRDS-required instrument for which to use this reference file for.
  • detector (str) – CRDS-required detector for which to use this reference file for.
  • subarray (str) – CRDS-required subarray for which to use this reference file for.
  • readpatt (str) – CRDS-required read pattern for which to use this reference file for.
  • outfile (str) – Name of the CRDS-formatted readnoise reference file to save the final readnoise map to.
  • author (str) – CRDS-required name of the reference file author, to be placed in the referece file header.
  • description (str) – CRDS-required description of the reference file, to be placed in the reference file header.
  • pedigree (str) – CRDS-required pedigree of the data used to create the reference file.
  • useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01T00:00:00’).
  • history (str) – CRDS-required history section to place in the reference file header.
  • fastaxis (int) – CRDS-required fastaxis of the reference file.
  • slowaxis (int) – CRDS-required slowaxis of the reference file.
  • substrt1 (int) – CRDS-required starting pixel in axis 1 direction.
  • substrt2 (int) – CRDS-required starting pixel in axis 2 direction.
  • filenames (list) – List of dark current files that were used to generate the reference file.
jwst_reffiles.readnoise.readnoise.use_single_value(readnoise, instrument, detector, subarray, clipping_sigma=3.0, max_clipping_iters=3)

Replaces all readnoise image values with the average of all readnoise values.

Parameters:
  • readnoise (numpy.ndarray) – The 2D readnoise image.
  • instrument (str) – CRDS-required instrument for which to use this reference file for.
  • detector (str) – CRDS-required detector for which to use this reference file for.
  • subarray (str) – CRDS-required subarray for which to use this reference file for.
  • clipping_sigma (float) – Number of sigma to use when sigma-clipping.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping.
Returns:

readnoise_single_value – 2D readnoise image where all of the readnoise values are the same (the average of all of the original readnoise values).
Return type:

numpy.ndarray
jwst_reffiles.readnoise.readnoise.wrapper_readnoise_by_ramp(args)

A wrapper around the readnoise_by_ramp function to allow for multiprocessing.

Parameters:args (tuple) – A tuple containing the input arguments for the readnoise_by_ramp function. See readnoise_by_ramp docstring for more details.
Returns:readnoise – 2D image of the calculated readnoise (i.e. the output from the readnoise_by_ramp function).
Return type:numpy.ndarray
jwst_reffiles.readnoise.readnoise.wrapper_readnoise_by_slice(args)

A wrapper around the readnoise_by_slice function to allow for multiprocessing.

Parameters:args (tuple) – A tuple containing the input arguments for the readnoise_by_slice function. See readnoise_by_slice docstring for more details.
Returns:readnoise – 2D image of the calculated readnoise (i.e. the output from the readnoise_by_slice function).
Return type:numpy.ndarray

Dark Current

This module contains code for creating a dark current reference file.

Input:

List of exposures at a calibration state from which you wish to create dark current reference files. For NIR detectors, this should be after:

  1. group_scale
  2. dq_init
  3. saturation
  4. ipc* - not currently run
  5. superbias
  6. refpix
  7. linearity
  8. persistence

For MIRI, this should be after:

  1. group_scale
  2. dq_init
  3. saturation
  4. linearity
  5. rscd

Also, versions of these files that have been run up through the JUMP step must also be present. This is because we need to know where the CR hits are in the data.

******************POSSIBLE INPUT CHANGE************************** (An alternate plan: Run the pipeline once on the input files. Run up through the JUMP step. For NIR data, skip the dark subtraction step. For MIRI, skip refpix and dark subtraction steps. Then the CR flags will be present, and the data will be in the proper state to create dark current reference files.) ******************POSSIBLE INPUT CHANGE**************************

All files should be from the same instrument/detector/aperture/readpattern

Ouputs:

A dark current reference file in the proper format A series of images showing for each pixel in each group, how many

measurements of the signal were used to calculate the mean dark value.

Function: Get file headers from each file. Check for consistency. Extract CR flags from the JUMP version of the files.

From these, create arrays giving the group number of the first cosmic ray hit in each pixel for each integration
Create images of how many dark current measurements are considered good
for each pixel and each group. All groups after the first CR hit in a pixel are considered bad and not used.
Read in the data. To avoid memory problems only read in N groups at a time,
where N is determined from user inputs.
Flag all groups that occur between the first CR hit and the end of the integration
as bad.
From the remaining good data, calculate the sigma-clipped mean dark signal
in each pixel for each group (i.e. take the mean across integrations). Also calculate the sigma-clipped stdard deviation.
Save the sigma-clipped mean as the dark data, and the sigma-clipped stdev
as the uncertainty in a DarkModel or MIRIDarkModel instance. The DQ array is currently all zeros, anticipating bad pixels being placed in the bad pixel reference file.

Superbias

This module creates a superbias reference file that can be used in the JWST calibration pipeline.

Author

  • Ben Sunnquist

Use

This module can be imported and used as such:

from jwst_reffiles.superbias import superbias
superbias.make_superbias(arguments)

Notes

Overview: Inputs: A list of dark current ramps

Algorithm: The superbias is the sigma-clipped mean of all 0th frames.
The superbias error is the sigma-clipped stddev of all 0th frames. Pixels with high superbias error values are flagged as UNRELIABLE_BIAS and DO_NOT_USE in the superbias DQ array.
jwst_reffiles.superbias.superbias.calculate_mean(stack, clipping_sigma=3.0, max_clipping_iters=3)

Calculates the sigma-clipped mean through a stack of images.

Parameters:
  • stack (numpy.ndarray) – A 3D stack of images.
  • clipping_sigma (float) – Number of sigmas to use when sigma-clipping the input stack.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the input stack.
Returns:

mean_image – 2D image of the sigma-clipped mean through the input stack.
Return type:

numpy.ndarray
jwst_reffiles.superbias.superbias.calculate_stddev(stack, clipping_sigma=3.0, max_clipping_iters=3)

Calculates the sigma-clipped standard deviation through a stack of images.

Parameters:
  • stack (numpy.ndarray) – A 3D stack of images.
  • clipping_sigma (float) – Number of sigmas to use when sigma-clipping the input stack.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the input stack.
Returns:

stddev_image – 2D image of the sigma-clipped standard deviation through the input stack.
Return type:

numpy.ndarray
jwst_reffiles.superbias.superbias.find_unreliable_bias_pixels(error, bad_clipping_sigma=3.0, bad_max_clipping_iters=3, bad_sigma_threshold=5.0, plot=False)

Identifies pixels with high superbias error values (i.e. UNRELIABLE_BIAS pixels).

Parameters:
  • error (numpy.ndarray) – A 2D superbias error image.
  • bad_clipping_sigma (float) – Number of sigma to use when sigma-clipping the superbias error image to find UNRELIABLE_BIAS pixels.
  • bad_max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the superbias error image to find UNRELIABLE_BIAS pixels.
  • bad_sigma_threshold (float) – Number of standard deviations above the mean of the superbias error image at which a pixel is flagged as UNRELIABLE_BIAS.
  • plot (bool) – Option to save a histogram of the superbias error image values, with the threshold used for flagging UNRELIABLE_BIAS pixels also shown.
Returns:

dq – A 2D DQ image that flags pixels with high superbias error values as UNRELIABLE_BIAS (2) and DO_NOT_USE (1).
Return type:

numpy.ndarray
jwst_reffiles.superbias.superbias.get_crds_info(filename, subarray, readpatt)

Get CRDS-required header information to put into the final superbias reference file.

Parameters:
  • filename (str) – Path to one of the files being used to generate the superbias reference file. Will be used to find default values.
  • subarray (str) – CRDS-required subarray for which to use this reference file for.
  • readpatt (str) – CRDS-required read pattern for which to use this reference file for.
Returns:

  • instrument (str) – CRDS-required instrument for which to use this reference file for.
  • detector (str) – CRDS-required detector for which to use this reference file for.
  • subarray (str) – CRDS-required subarray for which to use this reference file for.
  • readpatt (str) – CRDS-required read pattern for which to use this reference file for.
  • fastaxis (int) – CRDS-required fastaxis of the reference file.
  • slowaxis (int) – CRDS-required slowaxis of the reference file.
  • substrt1 (int) – CRDS-required starting pixel in axis 1 direction.
  • substrt2 (int) – CRDS-required starting pixel in axis 2 direction.
jwst_reffiles.superbias.superbias.make_filled_superbias(superbias, refpix_map, bad_clipping_sigma=3.0, bad_max_clipping_iters=3, bad_sigma_threshold=5.0, box_width=3, outfile='superbias_jwst_reffiles.fits')

Create a version of the superbias image where the values of high superbias pixels are replaced by their smoothed values.

Parameters:
  • superbias (numpy.ndarray) – A 2D superbias image.
  • refpix_map (numpy.ndarray) – A 2D image where refpix have a value of 1 and all other pixels have a value of 0.
  • bad_clipping_sigma (float) – Number of sigma to use when sigma-clipping to find UNRELIABLE_BIAS pixels.
  • bad_max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping to find UNRELIABLE_BIAS pixels.
  • bad_sigma_threshold (float) – Number of standard deviations above the mean at which a pixel is flagged as UNRELIABLE_BIAS.
  • box_width (int) – Width in pixels of the box kernel to use to compute the smoothed superbias image.
  • outfile (str) – Name of the CRDS-formatted superbias reference file to save the final superbias map to. This is only used in this function to name the filled output file.
jwst_reffiles.superbias.superbias.make_refpix_map(filename)

Creates a map of the reference pixels based on the input fits file.

Parameters:filename (str) – The fits image file to create the refpix map from. The PIXELDQ extension is used to find the refpix.
Returns:refpix_map – A map of the reference pixels (same dimensions as the input fits file image extensions), where refpix are 1 and all other pixels are 0. An array of all zeros is returned if no refpix are found or if no PIXELDQ extension exists.
Return type:numpy.ndarray
jwst_reffiles.superbias.superbias.make_superbias(filenames, clipping_sigma=3.0, max_clipping_iters=3, bad_clipping_sigma=3.0, bad_max_clipping_iters=3, bad_sigma_threshold=5.0, plot=False, save_filled=False, box_width=3, save_reffile=True, outfile='superbias_jwst_reffiles.fits', author='jwst_reffiles', description='Super Bias Image', pedigree='GROUND', useafter='2000-01-01T00:00:00', history='', subarray=None, readpatt=None)

The main function. Creates a superbias reference file using the input dark current ramps. See module docstring for more details.

Parameters:
  • filenames (list) – List of dark current ramp files. The data shape in these images is assumed to be a 4D array in DMS format (integration, group, y, x).
  • clipping_sigma (float) – Number of sigma to use when sigma-clipping through the 0th frame stack to create the superbias image.
  • max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping through the 0th frame stack to create the superbias error image.
  • bad_clipping_sigma (float) – Number of sigma to use when sigma-clipping the superbias error image to find UNRELIABLE_BIAS pixels.
  • bad_max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the superbias error image to find UNRELIABLE_BIAS pixels.
  • bad_sigma_threshold (float) – Number of standard deviations above the mean of the superbias error image at which a pixel is flagged as UNRELIABLE_BIAS.
  • plot (bool) – Option to save a histogram of the superbias error image values, with the threshold used for flagging UNRELIABLE_BIAS pixels also shown.
  • save_filled (bool) – Option to save an image where high superbias pixels are replaced by their smoothed values.
  • box_width (int) – Width in pixels of the box kernel to use to compute the smoothed superbias image. Only relevant if save_filled=True.
  • save_reffile (bool) – Option to save the generated superbias map into a CRDS-formatted reference file.
  • outfile (str) – Name of the CRDS-formatted superbias reference file to save the final superbias map to.
  • author (str) – CRDS-required name of the reference file author, to be placed in the referece file header.
  • description (str) – CRDS-required description of the reference file, to be placed in the reference file header.
  • pedigree (str) – CRDS-required pedigree of the data used to create the reference file.
  • useafter (str) – CRDS-required date of earliest data with which this referece file should be used (e.g. ‘2019-04-01T00:00:00’).
  • history (str) – CRDS-required history section to place in the reference file header.
  • subarray (str) – CRDS-required subarray for which to use this reference file for.
  • readpatt (str) – CRDS-required read pattern for which to use this reference file for.
jwst_reffiles.superbias.superbias.none_check(value)

If value is a string containing ‘none’, then change it to a NoneType object.

Parameters:value (str or NoneType) –
Returns:new_value
Return type:NoneType
jwst_reffiles.superbias.superbias.save_superbias(superbias, error, dq, instrument='', detector='', subarray='GENERIC', readpatt='ANY', outfile='superbias_jwst_reffiles.fits', author='jwst_reffiles', description='Super Bias Image', pedigree='GROUND', useafter='2000-01-01T00:00:00', history='', fastaxis=-1, slowaxis=2, substrt1=1, substrt2=1, filenames=[])

Saves a CRDS-formatted superbias reference file.

Parameters:
  • superbias (numpy.ndarray) – The 2D superbias image.
  • error (numpy.ndarray) – The 2D superbias error image.
  • dq (numpy.ndarray) – The 2D superbias data quality image.
  • instrument (str) – CRDS-required instrument for which to use this reference file for.
  • detector (str) – CRDS-required detector for which to use this reference file for.
  • subarray (str) – CRDS-required subarray for which to use this reference file for.
  • readpatt (str) – CRDS-required read pattern for which to use this reference file for.
  • outfile (str) – Name of the CRDS-formatted superbias reference file to save the final superbias map to.
  • author (str) – CRDS-required name of the reference file author, to be placed in the referece file header.
  • description (str) – CRDS-required description of the reference file, to be placed in the reference file header.
  • pedigree (str) – CRDS-required pedigree of the data used to create the reference file.
  • useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01T00:00:00’).
  • history (str) – CRDS-required history section to place in the reference file header.
  • fastaxis (int) – CRDS-required fastaxis of the reference file.
  • slowaxis (int) – CRDS-required slowaxis of the reference file.
  • substrt1 (int) – CRDS-required starting pixel in axis 1 direction.
  • substrt2 (int) – CRDS-required starting pixel in axis 2 direction.
  • filenames (list) – List of dark current files that were used to generate the reference file.