JWST_REFFILES: Reference File Generator Tool¶

Installing JWST_REFFILES¶

There are two main methods for installing jwst_reffiles. Note that for both of these installation methods, the main script, mkrefs.py, will be made available from the command line (i.e. it can be called from anywhere).

Installation for Users¶

If you are not planning on doing any development work for jwst_reffiles but only to use it, then installation can be done by (optionally) creating a new environment, activating it, and then using pip to install jwst_reffiles.

(optional) conda create -n jwst-reffiles python
conda activate jwst-reffiles
pip install git+https://github.com/spacetelescope/jwst_reffiles.git

Installation for Developers¶

For those who will be working on development of jwst_reffiles, installation should be done by cloning the repository from GitHub and then installing the local copy. It’s recommended that you install jwst_reffiles in a new environment. Also, when installing, it’s recommended that you install using the “-e” flag, which allows you to any changes to the code without reinstalling each time.

conda create -n jwst-reffiles python
conda activate jwst-reffiles
git clone https://github.com/spacetelescope/jwst_reffiles.git
cd jwst_reffiles
pip install -e .

Calling JWST_REFFILES¶

Currently, the top-level script in jwst_reffiles, which is called mkrefs.py, is designed such that it can only be called from the command line. Given a configuration file, as well as paths to collections of the necessary input files, mkrefs.py can be called in the following way from any directory (mkrefs.py need not be in the working directory):

mkrefs.py -vvv --cfgfile main_config_file.cfg example_readnoise_module /path/to/dark/files/*DARK*_uncal.fits /path/to/flatfield/files/NRCN815A-LIN-53650[5678]*uncal.fits

The -vvv flag sets the verbosity. More v’s leads to more details printed to the screen.

example_readnoise_module is the reference file creation module that you wish to run. You can create more than one reference file type by providing a comma-separated list of reference file creation modules.

You can provide as many input file lists/paths as you need. jwst_reffiles combines the input file lists into a single list of files. It then examines each of the input files and determines what type of file it is (dark or flat. jwst_reffiles currently only supports these two file types.).

Identification and Orgnization of Input Files¶

The primary function performed by jwst_reffiles is to perform the organization and bookkeeping of input files to be used in the creation of reference files. The package is designed such that users can provide lists of input files (e.g. dark current files, flat field files), along with information on which types of files should be used to create a given type of reference file, as well as some simple rules on which input files can be used in combination with others. jwst_reffiles then performs several tasks to prepare these input files for use in creating reference files.

Group input files¶

Within a configuration file, the user can specify which combinations of inputs are needed to create a given type of reference file. For example, the user can specify that an individual gain reference file should be created using a pair of flat field files and a pair of dark current files.

In addition, the user can define some simple rules for creating these groups of input files. Currently this includes a maximum time between files in the group. For example, to minimize any systematic variation in dark current, the user may specify that all dark current files in a single input group must have been observed within a 2 day time period. Are there any other rules?

Using the grouping definitions and input file rules, jwst_reffiles will then create groups of input files, where each group can be used to create an individual reference file.

Determine which pipeline calibration steps must be run¶

Also within the main configuration file, the user can specify which calibration pipeline steps are necessary to be complete on the input files prior to creating individual reference files. jwst_reffiles will then examine the input files, determine which pipeline steps have already been completed, and which steps remain to be done. It then generates, for each file, the appropriate call to the JWST calibration pipeline such that the output will have all necessary steps completed. jwst_reffiles will run the pipeline commands, save the outputs, and use these in the input file groups.

Create individal reference files¶

Once the input files have been organized and calibrated to the requested level, jwst_reffiles will create an individual reference file from each group of input files. The software package used to create each type of reference file is specified by the user in the main configuration file, towards the bottom. An example, called example_readnoise_module, is shown in the linked configuration file. This package must be plugged in to jwst_reffiles in the manner described in the plug-in page.

Values for package-specific variables are provided by the user in this section of the configuration file as well. There are several inputs that are required for all reference file generation scripts. These are the reftype, imtypes, and ssbsteps entries. In addition to these, any variables that are unique to the package can be listed here. In the example_readnoise_module example, these include verbose, boxsize, and sigma_threshold. As described in the Plug-in page, values entered here will be passed to the example_readnoise_module when it is called, unless they are overridden on the command line.

For each reference file generation module specified in the configuration file’s reflabels section, the appropriate software will be called once for each of the previously created group of pipeline-processed input files. Output files will be placed in the directory specified by the reference file output directory in the configuration file.

Note that each reference file creation module must take care of creating and saving the CRDS-formatted reference file itself. This means that the plugged-in module should use the appropriate jwst data model, correctly populate all data and metadata, and save the file in the location provided by jwst_reffiles.

Create final reference files¶

The creation of final reference files is similar to that of the individual reference files, where the user can plug in a module that takes as input a list of individual reference files and produces (and saves) a final reference file.

Several basic and widely-uised combination methods (simple pixel-by-pixel mean, median, etc) will be included in the jwst_reffiles package for convenience.

How to Plug In A User-Generated Module¶

Jwst_reffiles is simply a framework upon which reference file generation modules can be built. In order to add functionality for creating and averaging/combining reference files, users will need to plug-in their code to the jwst_reffiles framework. This page describes how to plug an existing module into jwst_reffiles. An example is provided in jwst_reffiles/example, where we have plugged in example_readnoise_module.py by creating mkref_example_readnoise_module.py following the steps below.

Assume you have a python script that creates a gain reference file. It takes as input several fits files containing dark current and flat field exposures, as well as optional arguments, and produces a CRDS-formatted gain reference file which is saved as a fits file. For this example, we will call this script my_gain_module.py.

Plug-in Steps¶

The steps needed to plug my_gain_module.py into jwst_reffiles are:

Make a copy of jwst_reffiles/templates/plugin_template.py and rename it to the name of your module with mkref_ prepended. (e.g. mkref_my_gain_module.py). This is your plug-in module.
Make your plug-in module executable. (e.g. chmod ugo+x mkref_my_gain_module.py). This is necessary so that the plug-in module can be called via the command line without prepending python to the call. This is helpful for cases where commands are being sent to Condor.
Add the appropriate parameters and a call to your script (e.g. my_gain_module.py). Details on edits are given in the edits list below.
Add the appropriate parameters for your script to a new section towards the bottom (see for example the block for example_readnoise_module) of the main config file. An example of what to add is shown in the Config File Update section below.

Plug-in Wrapper Edits¶

Within your plug-in module, make the following edits:

Add an import statement for your script. e.g. from my_package.gain import my_gain_module
In __init__, set the reflabel and reftype. The reflabel is simply the name of your script. (e.g. my_gain_module) The reftype is the jwst_reffiles shorthand for the type of reference file that your script produces. (e.g. gain) The list of allowed reftypes is in ????
In the extra_optional_arguments function, add any parameters for your script that you may want to call via the command line. These parameters will be available during command line calls to mkrefs.py. Parameters set via the command line will override the values set in the config file.
Within the callalgorithm function, create a call to your script (e.g. my_gain_module.py) When creating the call to your module, you should have access to all necessary information. The names of input fits files for a single run of your script will be in a list in self.inputimages['output_name']. Similarly, the output name of the reference file to be created will be in self.args.outputreffilename. Any additional parameters (as defined in the config file and/or on the command line) will be available in the self.parameters dictionary.

Config File Update¶

You may add any parameters for your module to the mkrefs config file. This is done in a block under the name of your module.

In this example, let’s assume that mkref_my_gain_module.py takes two arguments:

num_boxes: The detector will be divided up into a num_boxes*x*num_boxes grid of squares
max_signal: The maximum signal value to use in the calculation of the gain

Add a section to the config file containing the arguments and corresponding values.

my_gain_module:
    num_boxes: 16
    max_signal: 20000

The values for these parameters will then be read in from the config file when mkrefs.py is called. Keep in mind that if you have added these parameters to extra_optional_arguments function, then you can override the values in the config file via the command line.

Configuration Files¶

Jwst_reffiles makes use of multiple configuration files to control the behavior of the main mkrefs.py script, as well as describe the reference files to be created and define the parameters needed for each.

An example of the main configuration file:

instrument: NIRCam
reflabels: [example_readnoise_module]
reftypes: [bpm,rdnoise,gain,linearity]

default_reflabels:
   bpm: bpm
   rdnoise: example_readnoise_module
   gain: gain_armin

inputfiles:
   dark_pattern: Bias|Dark|BIAS|DARK|ZERO
   flat_pattern: GAIN|LIN

   basenamepattern: (.*)_uncal.fits$

   # specificy the fits keywords to determine MJD time of
   # observation. They are automatically added to the list of required
   # fits keywords. if MJD-OBS does not exist, then DATE-OBS (and
   # TIME-OBS if specified) are used to determine the MJD.
   dateobs_fitskey: DATE-END
   timeobs_fitskey: TIME-END
   mjdobs_fitskey:

   requiredfitskeys: [INSTRUME,DETECTOR,NGROUPS,NINTS,TGROUP,SUBARRAY]
   optionalfitskeys: [FILTER,PUPIL]

DD:
   max_Delta_MJD: 2.0

FF:
   max_Delta_MJD: 2.0
DDFF:
   max_Delta_MJD: 10.0

# reference file output basename: outrootdir[/outsubdir][/runID]/reflabel/reflabel[_outsubdir][_runID][.addsuffix].cmdID.reftype.fits
# if runID=AUTO, then the latest runID in the output dir outrootdir[/outsubdir] is taken
output:
   outrootdir: $JWST_MKREFS_OUTDIR
   outsubdir: test1
   runID: AUTO
   # how many leading zeros in runID
   runIDNdigits: 3
   addsuffix: bla

   # directory in which ssb products are saved.
   # if not specified, default is outrootdir[/outsubdir][/runID]/ssb
   # if skip_runID_ssbdir, then the runID is skipped
   ssbdir:
   skip_runID_ssbdir: False
   # If ssblogFlag=True, the stdout from the strun command is saved in a log
   # file, same name as reduced filename, but with suffix log.txt
   ssblogFlag: False
   # If ssberrorlogFlag=True, the stderr from the strun command is
   # saved in a log file, same name as reduced filename, but with
   # suffix err.txt
   ssberrorlogFlag: False

   # directories in which ssb products are looked for. comma-separated list!
   pipeline_prod_search_dir:

batch:
   batch_enabled: False
   batchmode: Condor


test1: 4

# values allowed: CRDS, SELF, filepattern
reffile4ssb:
   gain: CRDS
   bpm: CRDS
   rdnoise: CRDS

# values allowed: CRDS, filename
validation:
   gainreffile: CRDS
   bpmreffile: CRDS

example_readnoise_module:
    reftype: rdnoise
    imtypes: D
    ssbsteps: dq_init
    verbose: 3
    boxsize: 128
    sigma_threshold: 4

bpm:
    reftype: bpm
    imtypes: D
    ssbsteps: refpix-

rdnoise_nircam:
    reftype: rdnoise
    imtypes: DD
    ssbsteps: dq_init
    test1: 6

gain_armin:
    reftype: gain
    imtypes: DDFF
    ssbsteps: superbias-
    reffile4ssb:
        bpmreffile: /bla/bpm.fits

gain_bryan:
    reftype: gain
    imtypes: FF
    ssbsteps: rate-
    reffile4ssb:
        bpmreffile: /bla/bpm.fits

Instrument¶

The name of the instrument whose reference files are being created. Must be one of the JWST instruments. Capitalization is not important.

Reference Labels¶

The list of allowed reference labels. These are simply the names of the reference file creation packages that can be run. In the near future, this list will be generated automatically by searching for the wrapper scripts around plugged-in modules. At that time, reflabels will be removed from the config file.

Reference Types¶

List of the allowed reference file types that can be created. At a minimum, this list must contain the reference file types corresponding to all of the entries in the reflabels list.

Default Reference Labels¶

(NOT YET IMPLEMENTED) In this section, the user can define which of the reference file creation packages (reflabels) is the default for each reference type. This is a convenience feature for the user. If you define my_nircam_gain_script as the default package to use for gain files, then you can call mkrefs.py from the command line and simply request gain rather than my_nircam_gain_script.

Dark Pattern¶

List of strings that will be used to identify dark current files within the list of input files. mkrefs will look for these strings in the filenames. Note that the values shown in the example configuration file above are designed around ground testing files. JWST data obtained in flight will not have unique features in the filenames that identify them as darks, flats, etc. Future improvements to jwst_reffiles will address this.

Flat Pattern¶

List of strings that will be used to identify flat field files within the list of input files. jwst_reffiles will look for these strings in the filenames. Note that the values shown in the example configuration file above are designed around ground testing files. JWST data obtained in flight will not have unique features in the filenames that identify them as darks, flats, etc. Future improvements to jwst_reffiles will address this.

Basename Pattern¶

String to use when looking for input files. It may be that there are different versions (e.g. outputs from different points within the pipeline) of a file specified by the user in the input directory. If the basename pattern is set, jwst_reffiles will look only for the files matching that pattern. In the example above, we limit inputs to uncalibrated files.

Date-Obs Fits Header Keyword¶

The header keyword in the input files that contains the date of the observation. Dates are needed to enforce rules when pairing darks and flats. For JWST, set this to DATE-END.

Time-Obs Fits Header Keyword¶

The header keyword in the input files that contains the time of the observation. Times are needed to enforce rules when pairing darks and flats. For JWST, set this to TIME-END.

Obervation MJD Fits Header Keyword¶

The header keyword in the input files that contains the time of the observation in MJD. If specified, the value associated with this keyword is used to determinte the modified julian date (MJD) of the observation.

Required Fits File Header Keywords¶

List of header keywords which must be present in the input files. Values from these keywords will be copied into the master table created by jwst_reffiles. If any input files do not contain all of these keywords, an error will be raised.

Optional FitsFile Header Keywords¶

List of optional header keywords in the input files. Values from these keywords will be copied into the master table created by jwst_reffiles. If any of the keywords are missing in any of the input files, they are simply not copied to the master table, and jwst_reffiles proceeds without raising an error.

DD¶

Stands for dark dark, and indicates a situation where a pair of dark current ramps are needed to produce an individual reference file.

FF¶

Stands for flat flat, and indicates a situation where a pair of flat field ramps are needed to produce an individual reference file.

DDFF¶

Stands for dark dark flat flat, and indicates a situation where a pair of dark current ramps and a pair of flat field ramps are needed to produce an individual reference file.

Max_Delta_MJD¶

The maximum time allowed, in days, between input observations when creating pairs/groups of files. For example, to minimize the chances of dark current varying enough to impact reference file creation, you can set max_Delta_MJD to (e.g.) 2. In this case, when pairing dark current files (i.e. DD) jwst_reffiles will not pair observations taken more than 2 days apart.

Reference File Output Basename¶

Format of the output names for individual reference files. Output names will be automatically generated by jwst_reffiles to ensure accurate bookkeeping. The overall format of the reference file output names follows the convention shown below. cmdID is an ID assigned by jwst_reffiles for a particular run of the package. This helps to ensure unique file and directory names for the outputs.

outrootdir_[/outsubdir][/runID]/reflabel/reflabel[_outsubdir_][_runID_][.addsuffix_].cmdID.reftype_.fits

Output Root Directory¶

Path to the top level output directory for jwst_reffiles. Default is to define this within the JWST_MKREFS_OUTDIR environment variable, but any valid path is acceptable.

Output Subdirectory¶

Subdirectory name to add to the Output Root Directory when creating outputs.

Run ID¶

An integer that will be used to create a unique subdirectory for jwst_reffiles outputs for a given “run” of the software. With this parameter, it is easy to organize the outputs and prevent conflicts for multiple runs using the same input files. Leaving this set to the default value of AUTO will cause jwst_reffiles to search for the most recent/highest existing run ID, and add 1 for the next run.

Number of Digits in the Run ID¶

The total number of digits in the run ID. Leading zeros are added as necessary. The default is 3.

Add Suffix¶

Optional suffix that can be added to the output files from jwst_reffiles.

SSB Directory¶

Directory where JWST calibration pipeline output files, which are often created in the process of running jwst_reffiles, will be saved. If this entry is left blank, the default value of outrootdir_[/outsubdir][/runID]/ssb/ will be used.

Skip RunID for SSB Directory¶

Boolean. If True, the output directory (ssbdir) for the pipeline output files will follow the convention outrootdir_[/outsubdir]/ssb/.

SSB Log Flag¶

If ssblogFlag=True, the stdout from the JWST calibration pipeline call is saved in a log file, same name as pipeline output filename, but with suffix log.txt.

SSB Error Log Flag¶

If ssberrorlogFlag=True, the stderr from the JWST calibration pipeline call is saved in a log file, same name as reduced filename, but with suffix err.txt.

Pipeline Products Search Directories¶

This is comma-separated list of directories. Prior to calling the calibration pipeline for a given input fits file, jwst_reffiles will search the directories listed here to see if the proper pipeline-processed file already exists. If blank, no searching will be done.

Batch Enabled¶

Boolean entry. If True, jwst_reffiles will be run in batch mode. (BATCH MODE NOT YET IMPLEMENTED)

Batch Mode¶

The batch system to use when running in batch mode. Default is Condor.

Reference Files for SSB¶

NOT YET IMPLEMENTED. In this section, list the types of reference files to use in the calls to the calibration pipeline. Options are CRDS, SELF, or a file pattern. If CRDS is used, then the appropriate reference files will be selected from the Calibration Reference Data System (CRDS). This system contains only officially delivered reference files.

If SELF is used, then calls to the calibration pipeline will use reference files generated from the current run of jwst_reffiles. Note that in this case, running jwst_reffiles becomes an iterative process. For example, run once to produce a superbias reference file. Then run again to use this superbias reference file to calibrate inputs when creating dark current reference files.

Finally, you can enter a file pattern (e.g. /my/files/reffiles/gain/gain.fits), in which case *jwst_reffiles will use that file.

Validation¶

NOT YET IMPLEMENTED. Can be either CRDS or a filename. When comparing an output reference file to a previous version, this controls where the comparison file comes from. If set to CRDS, the most recent matching file in CRDS is used for comparison. If set to a filename, that file is used.

Example Readnoise Module (example of a reference file creation module)¶

This point in the config file is where you define the options that are specific to each reference file creation module. Any modules that are going to be run must be listed in the reflabels list. Any modules not in the reflabels list will be ignored.

Reference Type¶

Define the type of reference file created by this module. (e.g. gain, rdnoise)

Image Types¶

The type of inputs required by this package. (e.g. DD for a pair of dark ramps. FF for a pair of flat field ramps. DDFF for a pair of each.)

SSB Steps¶

Comma-separated list of calibration pipeline steps which must be complete on the input files prior to creating the reference file. Note that if the input files have been partially processed by the pipeline, the full list of completed steps must still be given here. For convenience, there is also a “-” shorthand that can be used. If the input files require all pipeline steps up to and including dark current subtraction, then you can enter “dark-”. The pipeline steps currently recognized by jwst_reffiles includes all of those in the calwebb_detetor1 pipeline, and are called using the values from the list below. Other values will not be recognized.

::: group_scale, dq_init, saturation, ipc, firstframe, lastframe, superbias, refpix, linearity, persistence, rscd, dark_current, jump, rate.

Example entries:

dq_init, saturation, refpix refpix-

Official algorithms for creating refrence files¶

The JWST Calibration Reference File Generation Tools group is tasked with defining a set of officially endorsed algorithms for creating JWST reference files. The goals of the group, and the defined algorithms, is to reduce duplication of effort and identify algorithms that may be used across instruments. Once defined, the Referene File Implementation subgroup will implement these algorithms in python modules which will be added to the JWST_reffiles repository so that they are available for all instrument teams to use.

The current list of complete or in-progress officially-defined algorithms include:

Bad Pixel Mask (used in DQ_Init pipeline step)

Official algorithms for creating the bad pixel mask¶

The definition of the official algorithm for the creation of the bad pixel mask reference file currently describes methods for finding DEAD, LOW QE, OPEN, and ADJACENT TO OPEN pixels.

Using this definition, the bad_pixel_mask.py module has been written and added to the JWST_reffiles repository. This module is currently undergoing testing before being fully integrated into the JWST_reffiles framework.

If you encounter any problems or have any questions about the code or its use, feel free to open an issue on the jwst_reffiles github page.

There are two separate modules that are called when running bad_pixel_mask.py. These are badpix_from_flats.py and badpix_from_darks.py. The former takes a collection of internal flat field exposures in order to search for DEAD, LOW QE, OPEN, and ADJACENT TO OPEN pixels. The latter takes a collection of dark current exposures in order to search for NOISY, HOT, RC, TELEGRAPH, and LOW_PEDESTAL pixels. Both of these modules expect input data in at least 2 calibration states, as detailed in the table below.

Module	Calibration States	Typical filename suffixes
badpix_from_flats	slope images, uncalibrated ramps	rate, uncal
badpix_from_darks	slope images, “fitopt” files, CR-flagged ramps, (uncalibrated ramps: MIRI-only)	rate, fitopt, jump, uncal

When running bad_pixel_mask.py as a standalone package, the input file lists described in the table above must be provided manually. In the future, when running via mkrefs.py, raw or partially-calibrated files will be allowed as inputs. mkrefs.py will then call the calibration pipeline and produce the needed files.

To use the bad pixel mask generator function as a standalone package, use the code shown below. Keywords in the call are linked to detailed descriptions below the code snippet. Note that all of the keywords in the call below have defaults defined in the code (and in fact the call below is using all default values), so you do not have to specify any keywords where you wish to use the default.

from glob import glob
from jwst_reffiles.bad_pixel_mask.bad_pixel_mask import bad_pixels

flat_rate_files = sorted(glob('/path/to/flats/*_rate.fits'))
flat_fluxcheck_files = sorted(glob('/path/to/flats/*uncal.fits'))

dark_rate_files = sorted(glob('/path/to/darks/*_rate.fits'))
dark_jump_files = sorted(glob('/path/to/darks/*_jump.fits'))
dark_fitopt_files = sorted(glob('/path/to/darks/*_fitopt.fits'))
dark_uncal_files = sorted(glob('/path/to/darks/*_uncal.fits'))

bad_pixels(flat_slope_files=flat_rate_files,
           dead_search =True,
           low_qe_and_open_search =True,
           dead_search_type ='sigma_rate',
           flat_mean_sigma_threshold =3,
           flat_mean_normalization_method ='smoothed',
           smoothing_box_width =15,
           smoothing_type ='Box2D',
           dead_sigma_threshold =5.,
           max_dead_norm_signal =None,
           run_dead_flux_check =False,
           dead_flux_check_files =None,
           flux_check =45000,
           max_low_qe_norm_signal =0.5,
           max_open_adj_norm_signal =1.05,
           manual_flag_file ='default',
           flat_do_not_use =[],
           dark_slope_files=dark_files,
           dark_uncal_files=None,
           dark_jump_files=dark_jump_files,
           dark_fitopt_files=dark_fitopt_files,
           dark_stdev_clipping_sigma =5.,
           dark_max_clipping_iters =5,
           dark_noisy_threshold =5,
           max_saturated_fraction =0.5,
           max_jump_limit =10,
           jump_ratio_threshold =5,
           early_cutoff_fraction =0.25,
           pedestal_sigma_threshold =5,
           rc_fraction_threshold =0.8,
           low_pedestal_fraction =0.8,
           high_cr_fraction =0.8,
           flag_values ={'hot': ['HOT'], 'rc': ['RC'], 'low_pedestal': ['OTHER_BAD_PIXEL'], 'high_cr': ["TELEGRAPH"]},
           dark_do_not_use =['hot', 'rc', 'low_pedestal', 'high_cr'],
           plot =False,
           output_file ='./test_bpm.fits',
           author ='jwst_reffiles',
           description ='A bad pix mask',
           pedigree ='GROUND',
           useafter ='2019-04-01 00:00:00',
           history ='',
           quality_check =False)

Dead Search¶

Boolean. If True, a dead pixel search (of type dead_search_type) is performed.

Low QE and Open Search¶

Boolean. If True, a search for Low QE, Open, and Adjacent to Open pixels is performed.

Dead Search Type¶

Use this string parameter to specify which type of dead pixel search to perform. Options are:

'sigma_rate': Using a normalized signal rate image, dead pixels
              are defined as those with a rate smaller than
              dead_sigma_threshold standard deviations below
              the mean.
'absolute_rate': Using a normalized signal rate image, dead pixels
                 are defined as those with a rate less than
                 max_dead_norm_signal.

Flat Mean Sigma Threshold¶

Number of standard deviations to use when sigma-clipping to calculate the mean slope image or the mean across the detector when working with flat field images.

Flat Mean Normalization Method¶

Specify how the mean flat field image is normalized prior to searching for bad pixels. Options are:

'smoothed': Mean image will be smoothed using a smoothing_box_width x smoothing_box_width box kernel. The mean
image is then normalized by this smoothed image.

'none': No normalization is done. Mean slope image is used as is

'mean': Mean image is normalized by its sigma-clipped mean

Smoothing Box Width¶

Width in pixels of the 2D box kernel to use to compute the smoothed mean flat field image

Smoothing Type¶

Type of smoothing to do when creating a smoothed mean flat field image. Box2D or Median. Box2D uses an astropy Box2DKernel, while Median uses a scipy median_filter.

Dead Sigma Threshold¶

Number of standard deviations below the mean at which a pixel is considered dead when using the sigma_rate dead search type.

Maximum Dead Normalized Signal¶

Maximum normalized signal rate of a pixel that is considered dead when using the absolute_rate dead search type.

Run Dead Flux Check¶

Boolean controlling whether or not to search flagged dead pixels for flux. This search potentially removes false positives, as pixels that are saturated in all groups of an integration will have a value of zero in the slope image and therefore appear dead.

Dead Flux Check Files¶

Files to use for the dead flux check. These should be raw (i.e. unal) files.

Flux Check¶

Signal level threshold to use during the dead flux check test. Pixels flagged as dead and with signals less than this signal level are considered dead.

Maximum Low QE Normalized Signal¶

The maximum normalized signal a pixel can have and be considered low QE.

Maximum Normalized Signal in Adjacent to Open Pixels¶

The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN

Manual Flag File¶

Ascii file containing a list of pixel coordinates and bad pixel types to be added to those found in badpix_from_flats.py and placed in the output bad pixel file. If left as ‘default’, the bad pixel file in the jwst_reffiles repository will be used.

Flat Do Not Use¶

List of bad pixel types (from the flat field files) where the DO_NOT_USE flag should also be applied (e.g. [‘DEAD’, ‘LOW_QE’, ‘OPEN’, ‘ADJ_OPEN’])

Dark Stdev Clipping Sigma¶

Number of sigma to use when sigma-clipping the 2D array of standard deviation values from the dark current slope files. The sigma-clipped mean and standard deviation are used to locate noisy pixels.

Dark Max Clipping Iterations¶

Maximum number of iterations to use when sigma clipping to find the mean and standard deviation values that are used when locating noisy pixels.

Dark Noisy Threshold¶

Number of sigma above the mean noise (associated with the slope) to use as a threshold for identifying noisy pixels in the dark current data.

Maximum Saturated Fraction¶

When identifying pixels that are fully saturated (in all groups of an integration), this is the fraction of integrations within which a pixel must be fully saturated before flagging it as HOT.

Maximum Jump Limit¶

The maximum number of jumps a pixel can have in an integration before it is flagged as a high jump pixel (which may be flagged as noisy later).

Jump Ratio Threshold¶

Cutoff for the ratio of jumps early in the ramp to jumps later in the ramp. Pixels with a ratio greater than this value (and which also have a high total number of jumps) will be flagged as potential (I)RC pixels.

Early Cutoff Fraction¶

Fraction of the integration to use when comparing the jump rate early in the integration to that across the entire integration. Must be <= 0.5

Pedestal Sigma Threshold¶

Used when searching for RC pixels via the pedestal image. Pixels with pedestal values more than pedestal_sigma_threshold above the mean are flagged as potential RC pixels

RC Fraction Threshold¶

Used when searching for RC pixels. This is the fraction of input files within which the pixel must be identified as an RC pixel before it will be flagged as a permanent RC pixel

Low Pedestal Fraction¶

This is the fraction of input files within which a pixel must be identified as a low pedestal pixel before it will be flagged as a permanent low pedestal pixel

High CR Fraction¶

This is the fraction of input files within which a pixel must be flagged as having a high number of jumps before it will be flagged as permanently noisy

Flag Values¶

This dictionary maps the types of bad pixels searched for to the flag mnemonics to use when creating the bad pixel file. Keys are the types of bad pixels searched for, and values are lists that include mnemonics recognized by the jwst calibration pipeline.

e.g. {‘hot’: [‘HOT’], ‘rc’: [‘RC’], ‘low_pedestal’: [‘OTHER_BAD_PIXEL’], ‘high_cr’: [“TELEGRAPH”]}

Dark Do Not Use¶

List of bad pixel types from the dark current data to be flagged as DO_NOT_USE.

e.g. [‘hot’, ‘rc’, ‘low_pedestal’, ‘high_cr’]

Plot¶

If True, produce plots of intermediate results.

Output File¶

Name of the CRDS-formatted bad pixel reference file to save the final bad pixel map into

Author¶

CRDS-required name of the reference file author, to be placed in the referece file header

Description¶

CRDS-required description of the reference file, to be placed in the reference file header

Pedigree¶

CRDS-required pedigree of the data used to create the reference file

Useafter¶

CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01 00:00:00’)

History¶

String containing any text you wish to place in the HISTORY keyword of the output bad pixel mask reference file. Note that all input filenames will automatically be placed in the HISTORY keyword independent of the string entered here.

Quality Check¶

Boolean. If True, the pipeline is run using the output reference file to be sure the pipeline doens’t crash

Current Limitations¶

Currently, only one type of dead pixel search can be performed for a given call.

API Documentation¶

create reference files for JWST instruments A. Rest

Bad Pixel Mask¶

Wrapper script that calls badpix_from_flats.py and badpix_from_darks.py. The results are combined to create a single bad pixel reference file.

jwst_reffiles.bad_pixel_mask.bad_pixel_mask.bad_pixels(flat_slope_files=None, dead_search=True, low_qe_and_open_search=True, dead_search_type='sigma_rate', flat_mean_sigma_threshold=3, flat_mean_normalization_method='smoothed', smoothing_box_width=15, smoothing_type='Box2D', dead_sigma_threshold=5.0, max_dead_norm_signal=None, run_dead_flux_check=False, dead_flux_check_files=None, flux_check=45000, max_low_qe_norm_signal=0.5, max_open_adj_norm_signal=1.05, manual_flag_file='default', flat_do_not_use=[], dark_slope_files=None, dark_uncal_files=None, dark_jump_files=None, dark_fitopt_files=None, dark_stdev_clipping_sigma=5.0, dark_max_clipping_iters=5, dark_noisy_threshold=5, max_saturated_fraction=0.5, max_jump_limit=10, jump_ratio_threshold=5, early_cutoff_fraction=0.25, pedestal_sigma_threshold=5, rc_fraction_threshold=0.8, low_pedestal_fraction=0.8, high_cr_fraction=0.8, flag_values={'high_cr': ['TELEGRAPH'], 'hot': ['HOT'], 'low_pedestal': ['OTHER_BAD_PIXEL'], 'rc': ['RC']}, dark_do_not_use=['hot', 'rc', 'low_pedestal', 'high_cr'], plot=False, output_file=None, author='jwst_reffiles', description='A bad pix mask', pedigree='GROUND', useafter='2019-04-01 00:00:00', history='', quality_check=False)¶

Wrapper that calls the two modules for finding bad pixels from input flat field files, and bad pixels from dark current files.

Parameters:

flat_slope_files (list) – List of flat field slope files to be used for the dead pixel search. If None, the search is skipped.
dead_search (bool) – Whether or not to search for DEAD pixels using the flat field files
low_qe_and_open_search (bool) – Whether or not to search for LOW_QE, OPEN, and ADJ_OPEN pixels using the flat field files
dead_search_type (str) –
Type of search to use when looking for dead pixels. Options are: sigma_rate: Using a normalized signal rate image, dead pixels

are defined as those with a rate smaller than dead_sigma_threshold standard deviations below the mean.

absolute_rate: Using a normalized signal rate image, dead pixels

are defined as those with a rate less than max_dead_norm_signal.
flat_mean_sigma_threshold (float) – Number of standard deviations to use when sigma-clipping to calculate the mean slope image or the mean across the detector
flat_mean_normalization_method (str) –
Specify how the mean image is normalized prior to searching for bad pixels. Options are: ‘smoothed’: Mean image will be smoothed using a

smoothing_box_width x smoothing_box_width box kernel. The mean image is then normalized by this smoothed image.

’none’: No normalization is done. Mean slope image is used as is ‘mean’: Mean image is normalized by its sigma-clipped mean ‘fit2d’: Mean image will be normalized by a fit of 2-D surface to

mean image. The degree of the fit is controlled by the fit_degree parameters
smoothing_box_width (float) – Width in pixels of the box kernel to use to compute the smoothed mean image
smoothing_typ (string) – Type of smoothing to do Box2D `` or ``median filtering
smoothing_sigma (float) – Number of standard deviations to use when smoothing in a box defined by smoothing_box_width.
dead_sigma_threshold (float) – Number of standard deviations below the mean at which a pixel is considered dead.
max_dead_norm_signal (float) – Maximum normalized signal rate of a pixel that is considered dead
run_dead_flux_check (bool) – Whether or not to check for dead pixels using an absolute flux value
dead_flux_check_files (list) – List of ramp (uncalibrated) files to use to check the flux of average of last 4 groups. If None then the uncalibration files are not read in and no flux_check is done.
flux_check (float) – Tolerance on average signal in last 4 groups. If dead_flux_check_files is a list of uncalibrated files, then the average of the last four groups for all the integrations is determined. If this average > flux_check then this pixel is not a dead pixel.
max_low_qe_norm_signal (float) – The maximum normalized signal a pixel can have and be considered low QE.
max_open_adj_norm_signal (float) – The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN
manual_flag_file (str) – Name of the ascii file containing a list of pixels to be added manually to the output bad pixel mask file. Default is ‘default’, in which case the file contained in the bad_pixel_mask directory of the repo will be used.
flat_do_not_use (list) – List of bad pixel types where the DO_NOT_USE flag should also be applied (e.g. [‘DEAD’, ‘LOW_QE’])
dark_slope_files (list) – List of dark current slope files to be used for the noisy pixel search. If None, the search is skipped.
dark_uncal_files (list) – List of uncalibrated dark current ramp files. These should correspond 1-to-1 with the files listed in dark_slope_files. If None, the code assumes the files are in the same location as the slope files and have names ending in uncal.fits
dark_jump_files (list) – List of dark current ramp files output from the jump step of the pipeline. These should correspond 1-to-1 with the files listed in dark_slope_files. If None, the code assumes the files are in the same location as the slope files and have names ending in jump.fits
dark_fitopt_files (list) – List of optional output files produced by the ramp-fitting step of the pipeline. These should correspond 1-to-1 with the files listed in dark_slope_files. If None, the code assumes the files are in the same location as the slope files and have names ending in fitopt.fits
dark_stdev_clipping_sigma (int) – Number of sigma to use when sigma-clipping the 2D array of standard deviation values. The sigma-clipped mean and standard deviation are used to locate noisy pixels.
dark_max_clipping_iters (int) – Maximum number of iterations to use when sigma clipping to find the mean and standard deviation values that are used when locating noisy pixels.
dark_noisy_threshold (int) – Number of sigma above the mean noise (associated with the slope) to use as a threshold for identifying noisy pixels.
max_saturated_fraction (float) – When identifying pixels that are fully saturated (in all groups of an integration), this is the fraction of integrations within which a pixel must be fully saturated before flagging it as HOT
max_jump_limit (int) – The maximum number of jumps a pixel can have in an integration before it is flagged as a high jump pixel (which may be flagged as noisy later)
jump_ratio_threshold (int) – Cutoff for the ratio of jumps early in the ramp to jumps later in the ramp. Pixels with a ratio greater than this value (and which also have a high total number of jumps) will be flagged as potential (I)RC pixels.
early_cutoff_fraction (float) – Fraction of the integration to use when comparing the jump rate early in the integration to that across the entire integration. Must be <= 0.5
pedestal_sigma_threshold (int) – Used when searching for RC pixels via the pedestal image. Pixels with pedestal values more than pedestal_sigma_threshold above the mean are flagged as potential RC pixels
rc_fraction_threshold (float) – Used when searching for RC pixels. This is the fraction of input files within which the pixel must be identified as an RC pixel before it will be flagged as a permanent RC pixel
low_pedestal_fraction (float) – This is the fraction of input files within which a pixel must be identified as a low pedestal pixel before it will be flagged as a permanent low pedestal pixel
high_cr_fraction (float) – This is the fraction of input files within which a pixel must be flagged as having a high number of jumps before it will be flagged as permanently noisy
flag_values (dict) – This dictionary maps the types of bad pixels searched for to the flag mnemonics to use when creating the bad pixel file. Keys are the types of bad pixels searched for, and values are lists that include mnemonics recognized by the jwst calibration pipeline e.g. {‘hot’: [‘HOT’], ‘rc’: [‘RC’], ‘low_pedestal’: [‘OTHER_BAD_PIXEL’], ‘high_cr’: [“TELEGRAPH”]}
dark_do_not_use (list) – List of bad pixel types to be flagged as DO_NOT_USE e.g. [‘hot’, ‘rc’, ‘low_pedestal’, ‘high_cr’]
plot (bool) – If True, produce and save intermediate results from noisy pixel search
output_file (str) – Name of the CRDS-formatted bad pixel reference file to save the final bad pixel map into
author (str) – CRDS-required name of the reference file author, to be placed in the referece file header
description (str) – CRDS-required description of the reference file, to be placed in the reference file header
pedigree (str) – CRDS-required pedigree of the data used to create the reference file
useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01 00:00:00’)
history (str) – Text to be added to the HISOTRY section of the output bad pixel file
quality_check (bool) – If True, the pipeline is run using the output reference file to be sure the pipeline doens’t crash

jwst_reffiles.bad_pixel_mask.bad_pixel_mask.create_output_filename(inst_name, det_name)¶

Create a default output filename for the bad pixel mask given instrument and detector names

Parameters:	inst_name (str) – Instrument name det_name (str) – Detector name
Returns:	outfile – Default bad pixel mask filename
Return type:	str

jwst_reffiles.bad_pixel_mask.bad_pixel_mask.instrument_info(filename)¶

Get the instrument and detector name from the header of the input file

Parameters:	filename (str) – Name of fits file
Returns:	inst (str) – Instrument name det (str) – Detector name

jwst_reffiles.bad_pixel_mask.bad_pixel_mask.save_final_map(bad_pix_map, instrument, detector, hdulist, files, author, description, pedigree, useafter, history_text, outfile)¶

Save a bad pixel map into a CRDS-formatted reference file

Parameters:

bad_pix_map (numpy.ndarray) – 2D bad pixel array
instrument (str) – Name of instrument associated with the bad pixel array
detector (str) – Name of detector associated with the bad pixel array
hdulist (astropy.fits.HDUList) – HDUList containing “extra” fits keywords
files (list) – List of files used to create bad_pix_map
author (str) – Author of the bad pixel mask reference file
description (str) – CRDS description to use in the final bad pixel file
pedigree (str) – CRDS pedigree to use in the final bad pixel file
useafter (str) – CRDS useafter string for the bad pixel file
history_text (list) – List of strings to add as HISTORY entries to the bad pixel file
outfile (str) – Name of the output bad pixel file

Badpix From Flats¶

This module can be used to create JWST-format bad pixel mask reference files for use in the dq_init step of the JWST calibration pipeline.

Author¶

Bryan Hilbert

Use¶

This module can be imported and used as such:

from jwst_reffiles.bad_pixel_mask import badpix_from_flats
badpix_from_flats.find_bad_pix(arguments)

Notes

This algorithm is used to identify types of bad pixels that are flagged in the bad pixel mask. This includes the following types: DEAD, LOW_QE, OPEN, ADJ_OPEN

This is based on discussions within the JWST Reference File Generation Working Group in February 2019. The goal is to produce an algorithm that can be used by all JWST instruments to produce bad pixel mask reference files.

Overview: Inputs: A set of flatfield countrate images (NIRCam, NIRISS, NIRSpec)

A set of dark current exposures i.e. Ramps (MIRI)

Algorithm:

For MIRI, extract and use only the 10th group from each exposure
Create average image from the set of input images, using sigma-clipping
Create smoothed version of the average image (15x15 smoothing)
Divide average image by smoothed image
Calculate sigma-clipped mean and stdev in normalized image
Find bad pix:

NIRCam, NIRISS: DEAD+DO_NOT_USE if signal < (mean-N*stdev) NIRSPec, MIRI: DEAD+DO_NOT_USE if signal < 0.05

LOW_QE if 0.05 < signal < 0.5 and signal in 4 adjacent pixels are below ADJ_OPEN threshold (1.05)

References

Working group discussion and algorithm details are presented at: https://outerspace.stsci.edu/display/JWSTCC/Algorithm+details%3A+DQ+Init

Dependencies¶

jwst

astropy

numpy

jwst_reffiles.bad_pixel_mask.badpix_from_flats.average_last4groups(hdu_list)¶

Find the median of all the group from each integration

Parameters:	data (numpy.ndarray) – 3D or 4D array of data
Returns:	data – 3D array (10th group only)
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.combine_individual_maps(bad_maps, do_not_use_flags)¶

Given multiple separate bad pixel maps, combine using bit values matching those defined in the JWST calibraiton pipeline

Parameters:	bad_maps (dict) – Dictionary containing the various individual bad pixel maps to be combined. Format is (e.g.) {‘DEAD’: dead_map, ‘LOW_QE’: low_qe_map}
Returns:	final_map – 2D array containing the combined bad pixel map
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.crds_input_checks(author_name, descrip, pedigree, use_after)¶

Perform some basic checks on the input values to be placed in the header of the reference file. These check against CRDS requirements

Parameters:	Results – ------- –

jwst_reffiles.bad_pixel_mask.badpix_from_flats.create_dqdef()¶

Create the DQ definition data needed to populate the final bad pixel reference file

Returns:	definitions – Bad pixel bit definitions
Return type:	list

jwst_reffiles.bad_pixel_mask.badpix_from_flats.create_dummy_hdu_list(sigma_threshold, smoothing_width, dead_sigma_threshold, dead_max_rate, low_qe_max_rate, open_adj_max_rate)¶

Create a fits HDU List to contain the “extra” fits header keywords that specific to this module and not included in the datamodels. This will be used to initialize the datamodel when creating the reference file

Parameters:	sigma_threshold (float) – Number of standard deviations to use when sigma-clipping to calculate the mean slope image or the mean across the detector smoothing_width (float) – Width in pixels of the box kernel to use to compute the smoothed mean image dead_sigma_threshold (float) – Number of standard deviations below the mean at which a pixel is considered dead. dead_max_rate (float) – Maximum normalized signal rate of a pixel that is considered dead low_qe_max_rate (float) – The maximum normalized signal a pixel can have and be considered low QE. open_adj_max_rate (float) – The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN
Returns:	hdu_list – HDU List containing a header with just the extra keywords specific to this module
Return type:	astropy.io.fits.HDUList

jwst_reffiles.bad_pixel_mask.badpix_from_flats.dead_pixels_absolute_rate(rate_image, max_dead_signal=0.05)¶

Create a map of dead pixels given a normalized rate image. In this case pixels are flagged as dead if their normalized signal rate is less than max_dead_signal.

Parameters:	rate_image (numpy.ndarray) – 2D normalized rate image max_dead_signal (float) – Maximum normalized signal rate of a pixel that is considered dead
Returns:	dead_pix_map – 2D map showing DEAD pixels. Good pixels have a value of 0.
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.dead_pixels_flux_check(dead_pix_map, ave_group, flux_check)¶

Check the dead_pix_map with the average_rate of the last 4 groups. We want to remove cases where a pixel was flagged as dead because it is a hot pixel that saturates on the first group causing the rate for this pixel to be close to 0.

Parameters:	dead_pix_map (numpy.ndarray) – 2D map showing DEAD pixels. Good pixels have a value of 0. ave_group (numpy.ndaarray) – 2D array of the average signal in the last 4 groups from an average of all integrations and exposures. flux_check (float) – if pixel have a ave_group > flux_check this is not a dead pixel
Returns:	clean_dead_pix_map – 2D map showing DEAD pixels. Good pixels have a value of 0. Clean up dead pixel map removing saturated pixels from dead flag.
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.dead_pixels_sigma_rate(rate_image, mean_rate, stdev_rate, sigma=5.0)¶

Create a map of dead pixels given a normalized rate image. In this case pixels are flagged as dead if their normalized signal rate is less than sigma standard deviations below the mean

Parameters:	rate_image (numpy.ndarray) – 2D normalized rate image mean_rate (float) – Sigma-clipped mean value of the `rate_image` stdev_rate (float) – Sigma-clipped standard deviation of the `rate_image` sigma (float) – Number of standard deviations below the mean at which a pixel is considered dead.
Returns:	dead_pix_map – 2D map showing DEAD pixels. Good pixels have a value of 0.
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.determine_manual_file(inst_name, detector_name)¶

Find the default manual bad pixel file for the given instrument and detector

Parameters:	inst_name (str) – Name of JWST instrument detector_name (str) – Name of detector
Returns:	filename – Full path and filename of the bad pixel file
Return type:	str

jwst_reffiles.bad_pixel_mask.badpix_from_flats.extract_10th_group(hdu_list)¶

Keep only the 10th group from each integration

Parameters:	data (numpy.ndarray) – 3D or 4D array of data
Returns:	data – 3D array (10th group only)
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.find_bad_pix(input_files, dead_search=True, low_qe_and_open_search=True, dead_search_type='sigma_rate', sigma_threshold=3, normalization_method='smoothed', smoothing_box_width=15, smoothing_type='Box2D', dead_sigma_threshold=5.0, max_dead_norm_signal=None, run_dead_flux_check=False, dead_flux_check_files=None, flux_check=45000, max_low_qe_norm_signal=0.5, max_open_adj_norm_signal=1.05, manual_flag_file='default', do_not_use=[], output_file=None, author='jwst_reffiles', description='A bad pix mask', pedigree='GROUND', useafter='2019-04-01 00:00:00', history='', quality_check=True)¶

MAIN SCRIPT: Given a set of input files, create dead, low QE, and open pixel maps

Parameters:	input_files (list) – Fits files to be used in the search for dead/open pixels dead_search (bool) – Whether or not to search for DEAD pixels low_qe_and_open_search (bool) – Whether or not to search for LOW_QE, OPEN, and ADJ_OPEN pixels dead_search_type (str) – Type of search to use when looking for dead pixels. Options are: `sigma_rate`: Using a normalized signal rate image, dead pixels are defined as those with a rate smaller than `dead_sigma_threshold` standard deviations below the mean. `absolute_rate`: Using a normalized signal rate image, dead pixels are defined as those with a rate less than `max_dead_norm_signal`. sigma_threshold (float) – Number of standard deviations to use when sigma-clipping to calculate the mean slope image or the mean across the detector normalization_method (str) – Specify how the mean image is normalized prior to searching for bad pixels. Options are: ‘smoothed’: Mean image will be smoothed using a `smoothing_box_width` x `smoothing_box_width` box kernel. The mean image is then normalized by this smoothed image. ’none’: No normalization is done. Mean slope image is used as is ‘mean’: Mean image is normalized by its sigma-clipped mean smoothing_box_width (float) – Width in pixels of the box kernel to use to compute the smoothed mean image smoothing_typ (string) – Type of smoothing to do Box2D `` or ``median filtering smoothing_sigma (float) – Number of standard deviations to use when smoothing in a box defined by smoothing_box_width. dead_sigma_threshold (float) – Number of standard deviations below the mean at which a pixel is considered dead. max_dead_norm_signal (float) – Maximum normalized signal rate of a pixel that is considered dead run_dead_flux_check (bool) – Whether or not to check for dead pixels using an absolute flux value dead_flux_check_files (list) – List of ramp (uncalibrated) files to use to check the flux of average of last 4 groups. If None then the uncalibration files are not read in and no flux_check is done. flux_check (float) – Tolerance on average signal in last 4 groups. If dead_flux_check_files is a list of uncalibrated files, then the average of the last four groups for all the integrations is determined. If this average > flux_check then this pixel is not a dead pixel. max_low_qe_norm_signal (float) – The maximum normalized signal a pixel can have and be considered low QE. max_open_adj_norm_signal (float) – The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN manual_flag_file (str) – Name of the ascii file containing a list of pixels to be added manually to the output bad pixel mask file. Default is ‘default’, in which case the file contained in the `bad_pixel_mask` directory of the repo will be used. do_not_use (list) – List of bad pixel types where the DO_NOT_USE flag should also be applied (e.g. [‘DEAD’, ‘LOW_QE’]) output_file (str) – Name of the CRDS-formatted bad pixel reference file to save the final bad pixel map into author (str) – CRDS-required name of the reference file author, to be placed in the referece file header description (str) – CRDS-required description of the reference file, to be placed in the reference file header pedigree (str) – CRDS-required pedigree of the data used to create the reference file useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01 00:00:00’) history (str) – Text to be added to the HISOTRY section of the output bad pixel file quality_check (bool) – If True, the pipeline is run using the output reference file to be sure the pipeline doens’t crash
Returns:	final_map – 2D array containing the combined bad pixel map
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.find_open_and_low_qe_pixels(rate_image, max_dead_signal=0.05, max_low_qe=0.5, max_adj_open=1.05)¶

Create maps of open (and adjacent to open) and low QE pixels given a normalized rate image

Parameters:

rate_image (numpy.ndarray) – 2D normalized rate image
mean_rate (float) – Sigma-clipped mean value of the rate_image
stdev_rate (float) – Sigma-clipped standard deviation of the rate_image
max_dead_signal (float) – The maximum normalized signal a pixel can have and be considered dead
max_low_qe (float) – The maximum normalized signal a pixel can have and be considered low QE.
max_adj_open (float) – The maximum normalized signal a pixel adjacent to a low QE pixel can have in order for the low QE pixel to be reclassified as OPEN

Returns:

low_qe_map (numpy.ndarray) – 2D map showing LOW QE pixels. Good pixels have a value of 0.
open_pix_map (numpy.ndarray) – 2D map showing OPEN pixels. Good pixels have a value of 0.
adj_pix_map (numpy.ndarray) – 2D map showing ADJ_OPEN pixels. Good pixels have a value of 0.

jwst_reffiles.bad_pixel_mask.badpix_from_flats.fit_surface(data, deg)¶

Fit the rate image with a 2-D surface plot

Parameters:	data (numpy.ndarray) – 2D image deg (degree of polynomial fit) –
Returns:	surface_image – 2D image of polynomial fit
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.flag_type_check(flags)¶

Check that the flags in the input list are valid JWST bad pixel types

Parameters:	flag (list) – List of flag types (e.g. [‘DEAD’, ‘LOW_QE’])
Returns:	flag – Modified list (all flags made uppercase)
Return type:	list

jwst_reffiles.bad_pixel_mask.badpix_from_flats.get_adjacent_pixels(x_val, y_val, x_dim, y_dim)¶

Return coordinates of the pixels immediately adjacent to the input pixel coordinate

Parameters:

x_val (int) – x coordinate of central pixel
y_val (int) – y coordinate of central pixel
x_dim (int) – Size of the array in the x dimension
y_dim (int) – Size of the array in the y dimension

Returns:

adj_x (numpy.ndarray) – x coordinates of adjacent pixel coordinates
adj_y (numpy.ndarray) – y coordinates of adjacent pixel coordinates

jwst_reffiles.bad_pixel_mask.badpix_from_flats.get_fastaxis(filename)¶

Get the fastaxis and slowaxis values from the input file

Parameters:	filename (str) – Name of fits file to get values from
Returns:	fastaxis (int) – Value of FASTAXIOS keyword slowaxis (int) – Value of SLOWAXIS keyword

jwst_reffiles.bad_pixel_mask.badpix_from_flats.get_max_dead_norm_signal_default(instrument, detector, normalization_method)¶

Finds the default max_dead_norm_signal value to use for the absolute_rate dead pixel search type.

Parameters:	instrument (str) – The name of the instrument detector (str) – The name of the detector normalization_method (str) – Specify how the mean image is normalized prior to searching for bad pixels. Options are: ‘smoothed’: Mean image will be smoothed using a `smoothing_box_width` x `smoothing_box_width` box kernel. The mean image is then normalized by this smoothed image. ’none’: No normalization is done. Mean slope image is used as is ‘mean’: Mean image is normalized by its sigma-clipped mean
Returns:	default_value – The default max_dead_norm_signal value
Return type:	float

jwst_reffiles.bad_pixel_mask.badpix_from_flats.image_stats(image, sigma=3.0)¶

Calculate the sigma-clipped mean and standard deviation across the given image

Parameters:

image (numpy.ndarray) – 2D image
sigma (float) – Sigma threshold to use for sigma-clipping

Returns:

mean_val (float) – Sigma-clipped mean value
stdev_val (float) – Sigma-clipped standard deviation

jwst_reffiles.bad_pixel_mask.badpix_from_flats.manual_flags(filename, map_shape, no_use)¶

Create a bad pixel mask from an input ascii file containing pixel indicies and flag types

Parameters:	filename (str) – Name of ascii file containing flags map_shape (tup) – Shape of the bad pixel mask to create (y, x) no_use (list) – List of mnemonics to flag as DO_NOT_USE
Returns:	mask – 2D array of bad pixels, with bit values equal to the standard JWST values
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.mean_stdev_images(data, sigma=3.0)¶

Calculate the sigma-clipped mean and stdev images from a stack of input images

Parameters:

data (numpy.ndarray) – Stack of 2D images (i.e 3D array)
sigma (float) – Threshold value to use for sigma clipping.

Returns:

mean_image (numpy.ndarray) – 2D array of the mean image
stdev_image (numpy.ndarray) – 2D array of the stdev image

jwst_reffiles.bad_pixel_mask.badpix_from_flats.miri_bad_columns(dimensions)¶

Create a map that flags the shorted columns on the MIRI detector

Parameters:	dimensions (tup) – (y, x) dimensions, in pixels, of the map to create
Returns:	shorted_map – 2D map showing the locations of the shorted columns (1)
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.none_check(value)¶

If value is a string containing ‘none’, then change it to a NoneType object

Parameters:	value (str or NoneType) –
Returns:	new_value
Return type:	NoneType

jwst_reffiles.bad_pixel_mask.badpix_from_flats.pad_with_refpix(data, instrument_name)¶

Pad the given image with an outer 4 rows and columns of (zeroed out) reference pixels.

Parameters:	data (numpy.ndarray) – 2D image instrument_name (str) – Name of the JWST instrument associated with the data
Returns:	padded – 2D image with 4 rows and columns of reference pixels added
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.pipeline_check(reference_filename, filename)¶

Run the dq_init step using the provided file, to check that the pipeline runs successfully

Parameters:	reference_filename (str) – Name of the bad pixel mask reference filename filename (str) – Name of the fit exposure on which to test the reference file

jwst_reffiles.bad_pixel_mask.badpix_from_flats.range_format(table_cell, total_dimension)¶

Turn a string containing a possible shorthand list of indexes into an array of the specified indexes. Support the notation that if no beginning or ending index is given that the range will be to the start or end of total_dimension. Note that this function assumes that the input strings do not follow the python convention of using a maximum index value of one beyond the index you really want. (e.g. it assumes that ‘0:3’ means columns 0 through 3 inclusive are what you are interested in (meaning indexing in python would need to use [0:4] to get all the values.))

Parameters:	table_cell (str or int) – Value to process (e.g. ‘34:36’ or 93) total_dimension (int) – The maximum index number to be returned

jwst_reffiles.bad_pixel_mask.badpix_from_flats.read_files(filenames, dead_search_type)¶

Read the data in the input files. Perform basic sanity checks to be sure data are consistent (same instrument, aperture, etc)

Parameters:	filenames (list) – List of fits filenames to be opened dead_search_type (str) – Type of search to use when looking for dead pixels. Options are: `sigma_rate`: Using a normalized signal rate image, dead pixels are defined as those with a rate smaller than `dead_sigma_threshold` standard deviations below the mean. `absolute_rate`: Using a normalized signal rate image, dead pixels are defined as those with a rate less than `max_dead_norm_signal`. `saturation_check`: this option is set by the program if `dead_flux_check_files` is a list of ramp files. The average of the last four groups is determined to test if pixel is close to saturation.
Returns:	data – 3D stack of data
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.read_manual_flag_file(filename)¶

Read in the text file containing the list of pixels to be flagged manually.

Parameters:	filename (str) – Name of ascii file to be read in
Returns:	bad_table – Table of bad pixels
Return type:	astropy.table.Table

jwst_reffiles.bad_pixel_mask.badpix_from_flats.reference_pixel_map(dimensions, instrument_name)¶

Create a map that flags all reference pixels as such

Parameters:	dimensions (tup) – (y, x) dimensions, in pixels, of the map to create instrument_name (str) – Name of JWST instrument associated with the data
Returns:	ref_map – 2D map showing the locations of reference pixels (1)
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.save_final_map(bad_pix_map, instrument, detector, files, author, description, pedigree, useafter, history_text, sigma_thresh, smooth_width, dead_sigma_thresh, max_dead_rate, max_low_qe_rate, max_open_adj_rate, do_not_use_list, outfile)¶: Save a bad pixel map into a CRDS-formatted reference file

jwst_reffiles.bad_pixel_mask.badpix_from_flats.science_pixels(data, instrument_name)¶

Given a full frame image, strip off the reference pixels and return only the science pixels. At the moment, assume 4 rows and columns of reference pixels

Parameters:	data (numpy.ndarray) – 2D image instrument_name (str) – Name of JWST instrument associated wit the data
Returns:	data – 2D image with outer 4 rows and columns removed
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.smooth(data, box_width=15, type='Box2D')¶

Create a smoothed version of the 2D input data

Parameters:	data (numpy.ndarray) – 2D array of data to be smoothed box_width (int) – Width of the smoothing box, in pixels sigma (float) – value to use for sigma clipping
Returns:	smoothed – A smoothed version of `data`
Return type:	numpy.ndarray

jwst_reffiles.bad_pixel_mask.badpix_from_flats.split_row(line, colname)¶

Convert the indicies in a given row of the manual bad pixel ascii file into integers

Parameters:	line (astropy.table.row.Row) – Row containing x, y, and flag values colname (str) – Column name to examine
Returns:	values – If the input is an integer, the list will simply be [value]. If a range is given (e.g. 200:210), then values will be [200, 210].
Return type:	list

Badpix From Darks¶

Find bad pixels from dark current files

Start with a stack of dark ramps and slope images? Or maybe a stack of ramps that have been processed through the jump step and then ramp-fitting is performed here?

Input for the dark current reference file step is going to be a stack of ramps. So maybe use that and ramp-fit here.

inputs: 1. list of dark current ramps that have been run through jump step 2. list of same exposures after ramp-fitting has been done

Plot summary:

Check to see if IPC correction has been run
Look through stack of slopes images, get mean and rms per pixel (do we sigma-clip the inputs or not?)
Potential bad pixels are those with noise values above some threshold

NOTE: when producing slope images of these data, make sure to save the optional output parameters into the *fitopt.fits files. https://jwst-pipeline.readthedocs.io/en/stable/jwst/ramp_fitting/main.html?highlight=intercept “A third, optional output product is also available and is produced only when the step parameter ‘save_opt’ is True”

jwst_reffiles.dark_current.badpix_from_darks.add_refpix(array, to_add)¶

Place map within a larger array that contains the reference pixels.

Parameters:	array (numpy.ndarray) – 2D array of bad pixels that does not contain reference pixels to_add (tup) – 4-element tuple containing the number of rows and columns to add around the outside of the science pixels. (left cols, right cols, bottom rows, top rows)
Returns:	array – 2D array with rows and columns added
Return type:	numpy.ndarray

jwst_reffiles.dark_current.badpix_from_darks.apply_flags(pixmap, flag_list)¶

Beginning with a map indicating locations of a particular type of bad pixel, apply the bits specified in flag_list to come up with the jwst bad pixel value.

Parameters:	pixmap (numpy.ndarray) – 2D array indicating bad pixels. 1 for a bad pixel and 0 for a good pixel flag_list (list) – List of bad pixel mnemonics to be applied. These mnemonics must be in the dictionary of bad pixel types recognized by the jwst calibration pipeline
Returns:	pixmap – Map updated with proper bad pixel values
Return type:	numpy.ndarray

jwst_reffiles.dark_current.badpix_from_darks.check_metadata(hdr, comp)¶

Basic metadata check for consistency from one file to another

Parameters:	hdr (astropy.fits.header) – Header read in from fits file comp (astropy.fits.header) – Baseline header to which the comparison is done

jwst_reffiles.dark_current.badpix_from_darks.combine_bad_pixel_types(sat_map, rc_map, low_pedestal_map, high_cr_map)¶

Copmbine individual maps of bad pixel types into a final bad pixel map, using flag values defined in dq_flags.

Parameters:	sat_map (numpy.ndarray) – 2D array giving the location of pixels saturated all the way up the ramp rc_map (numpy.ndarray) – 2D array giving the location of RC pixels low_pededtal_map (numpy.ndarray) – 2D array giving the location of pixels with abnormally low pedestal values high_cr_map (numpy.ndarray) – 2D array giving the location of pixels with abnormally high numbers of jumps
Returns:	final_map – 2D array containing the bitwise combined bad pixel maps
Return type:	numpy.ndarray

jwst_reffiles.dark_current.badpix_from_darks.combine_clean_slopes(slope_stack, islope_stack)¶

Combine the stack of slopes and form the stanard deviation of the pixel slopes

Parameters:	slope_stack (list) – A list of slopes for full array stacked for each integration islope_stack (list) – A list of of 1 or 0 for each integration. A 1 is a good slope and 0 is slope with cosmic ray

jwst_reffiles.dark_current.badpix_from_darks.extract_group2(filename, refpix)¶

Read in the PEDESTAL values from a *fitopt.fits file

Parameters:	filename (str) – Name of uncalibrated file. This should be a uncal.fits file. refpix* (tup) – 4-element tuple listing the number of outer rows and columns that are reference pixels
Returns:	group2 – 3D array of group 2
Return type:	numpy.ndarray

jwst_reffiles.dark_current.badpix_from_darks.find_bad_pix(filenames, uncal_filenames=None, jump_filenames=None, fitopt_filenames=None, clipping_sigma=5.0, max_clipping_iters=5, noisy_threshold=5, max_saturated_fraction=0.5, max_jump_limit=10, jump_ratio_threshold=5, early_cutoff_fraction=0.25, pedestal_sigma_threshold=5, rc_fraction_threshold=0.8, low_pedestal_fraction=0.8, high_cr_fraction=0.8, flag_values={'high_cr': ['TELEGRAPH'], 'hot': ['HOT'], 'low_pedestal': ['OTHER_BAD_PIXEL'], 'rc': ['RC']}, do_not_use=['hot', 'rc', 'low_pedestal', 'high_cr'], outfile=None, plot=False)¶

MAIN FUNCTION

Parameters:

filenames (list) – List of dark current slope files. These should be slope images.
uncal_filenames (list) – List of uncal files. Should have a 1-to-1 correspondence to the files in filenames. If None, the scipt will look in the same directory containing filenames, and assume that the only difference in filename is that rate.fits is replaced with uncal.fits. Uncal files are only used when working with MIRI data.
jump_filenames (list) – List of exposures output from the jump step of the pipeline. Should have a 1-to-1 correspondence to the files in filenames. If None, the scipt will look in the same directory containing filenames, and assume that the only difference in filename is that rate.fits is replaced with jump.fits
fitopt_filenames (list) – List of exposures from the optional output from the ramp_fitting step of the pipeline. Should have a 1-to-1 correspondence to the files in filenames. If None, the scipt will look in the same directory containing filenames, and assume that the only difference in filename is that rate.fits is replaced with fitopt.fits
clipping_sigma (int) – Number of sigma to use when sigma-clipping the 2D array of standard deviation values from the dark current slope files. The sigma-clipped mean and standard deviation are used to locate noisy pixels.
max_clipping_iters (int) – Maximum number of iterations to use when sigma clipping to find the mean and standard deviation values that are used when locating noisy pixels.
noisy_threshold (int) – Number of sigma above the mean noise (associated with the slope) to use as a threshold for identifying noisy pixels.
max_saturated_fraction (float) – When identifying pixels that are fully saturated (in all groups of an integration), this is the fraction of integrations within which a pixel must be fully saturated before flagging it as HOT
max_jump_limit (int) – The maximum number of jumps a pixel can have in an integration before it is flagged as a high jump pixel (which may be flagged as noisy later)
jump_ratio_threshold (int) – Cutoff for the ratio of jumps early in the ramp to jumps later in the ramp. Pixels with a ratio greater than this value (and which also have a high total number of jumps) will be flagged as potential (I)RC pixels.
early_cutoff_fraction (float) – Fraction of the integration to use when comparing the jump rate early in the integration to that across the entire integration. Must be <= 0.5
pedestal_sigma_threshold (int) – Used when searching for RC pixels via the pedestal image. Pixels with pedestal values more than pedestal_sigma_threshold above the mean are flagged as potential RC pixels
rc_fraction_threshold (float) – Used when searching for RC pixels. This is the fraction of input files within which the pixel must be identified as an RC pixel before it will be flagged as a permanent RC pixel
low_pedestal_fraction (float) – This is the fraction of input files within which a pixel must be identified as a low pedestal pixel before it will be flagged as a permanent low pedestal pixel
high_cr_fraction (float) – This is the fraction of input files within which a pixel must be flagged as having a high number of jumps before it will be flagged as permanently noisy
flag_values (dict) – This dictionary maps the types of bad pixels searched for to the flag mnemonics to use when creating the bad pixel file. Keys are the types of bad pixels searched for, and values are lists that include mnemonics recognized by the jwst calibration pipeline e.g. {‘hot’: [‘HOT’], ‘rc’: [‘RC’], ‘low_pedestal’: [‘OTHER_BAD_PIXEL’], ‘high_cr’: [“TELEGRAPH”]}
do_not_use (list) – List of bad pixel types to be flagged as DO_NOT_USE e.g. [‘hot’, ‘rc’, ‘low_pedestal’, ‘high_cr’]
plot (bool) – If True, produce plots of intermediate results.
outfile (str) – Name of fits file to save the resulting bad pixel mask to

jwst_reffiles.dark_current.badpix_from_darks.find_pix_with_many_jumps(jump_map, max_jump_limit=10, jump_ratio_threshold=5, early_cutoff_fraction=0.25)¶

Identify pixels that have an abnormal number of flagged jumps. Do this by finding the jump rate early in the ramp versus that later in the ramp.

Parameters:

jump_map (numpy.ndarray) – Map of jump flags for all pixels (e.g. output from dqflags.flag_map) Assume we are working one integration at a time.
max_jump_limit (int) – The maximum number of jumps a pixel can have in an integration before it is flagged as a high jump pixel (which may be flagged as noisy later)
jump_ratio_threshold (int) – Cutoff for the ratio of jumps early in the ramp to jumps later in the ramp. Pixels with a ratio greater than this value (and which also have a high total number of jumps) will be flagged as potential (I)RC pixels.
early_cutoff_fraction (float) – Fraction of the integration to use when comparing the jump rate early in the integration to that across the entire integration. Must be <= 0.5

Returns:

high_jumps (numpy.ndarray) – Map of pixels that have more than max_jump_limit jumps.
potential_rc (numpy.ndarray) – Map of pixels which have: 1) a large number of jumps, and 2) a higher rate of jumps early in the ramp than later in the ramp.

jwst_reffiles.dark_current.badpix_from_darks.pedestal_stats(pedestal_array, threshold=5)¶

Get statsitics on the pedestal array corresponding to an integration

Parameters:	pedestal_array (numpy.ndarray) – Array of pedestal values threshold (int) – Number of sigma above or below the mean at which a pedestal value is considered bad
Returns:	suspicioius_pedestal – Map of bad pedestal values
Return type:	numpy.ndarray

jwst_reffiles.dark_current.badpix_from_darks.plot_histogram_stats(data_array, cut_limit, nbins, outdir, titleplot, fileout, xaxis_log=False)¶

Plot a histogram of stats and over the upper limit cut off

Parameters:	data_array (numpy.ndarray) – 2D data to make a histogram from sigma_threshold (float) – used to plotting sigma clip line on plot nbins (integer) – number of bins in creating histogram titleplot (string) – title of the plot fileout (string) – output file of the plot
Returns:
Return type:	prints the plot to disk

jwst_reffiles.dark_current.badpix_from_darks.plot_image(image, image_max, outdir, titleplot, fileout)¶

Plot an Image

Parameters:	image (numpy.ndarray) – 2D image to plot image_max (float) – maximum of image to use for scaling the image titleplot (string) – title of the plot fileout (string) – output file of the plot
Returns:
Return type:	prints the plot to disk

jwst_reffiles.dark_current.badpix_from_darks.read_pedestal_data(filename, refpix)¶

Read in the PEDESTAL values from a *fitopt.fits file

Parameters:	filename (str) – Name of output file to check. This should be a fitopt.fits file. refpix* (tup) – 4-element tuple listing the number of outer rows and columns that are reference pixels
Returns:	pedestal – 3D array of pedestal values (signal extrapolated to time=0)
Return type:	numpy.ndarray

jwst_reffiles.dark_current.badpix_from_darks.read_slope_data(filename, refpix)¶

Read in the science extension from a group of slope images

Parameters:	filenames (list) – List of fits files containing slope values refpix (tup) – 4-element tuple listing the number of outer rows and columns that are reference pixels
Returns:	slope – 2D or 3D array of slope values (signal extrapolated to time=0) Reference pixels have been stripped off.
Return type:	numpy.ndarray

jwst_reffiles.dark_current.badpix_from_darks.read_slope_files(filenames)¶

Read in the science extension from a group of slope images

Parameters: filenames (list) – List of fits files containing slope values

Returns:

slope_data (numpy.ndarray) – 3D array containing slope values for science pixels only. Reference pixels have been stripped off.
left_cols (int) – Number of columns of reference pixels on the left side of the array
right_cols (int) – Number of columns of reference pixels on the right side of the array
bottom_rows (int) – Number of rows of reference pixels on the bottom of the array
top_rows (int) – Number of rows of reference pixels on the top of the array

jwst_reffiles.dark_current.badpix_from_darks.read_slope_integrations(filenames)¶

Read in the science extension from a group of slope images

Parameters: filenames (list) – List of fits files containing slope values

Returns:

instrument (str) – Name of instrument associated with the data
slope_data (numpy.ndarray) – 3D array containing slope values for science pixels only. Reference pixels have been stripped off.
starting_indexes (list) – List of numbers corresponding to the index numbers within slope_data where each file’s data begins. For example, if slope_data is an array of size (10, 2048, 2048), and starting_indexes = [0, 5, 7, 10] then we can pull apart slope_data into its constituent exposures using slope_data[starting_indexes[0]: starting_indexes[1]], etc.
left_cols (int) – Number of columns of reference pixels on the left side of the array
right_cols (int) – Number of columns of reference pixels on the right side of the array
bottom_rows (int) – Number of rows of reference pixels on the bottom of the array
top_rows (int) – Number of rows of reference pixels on the top of the array

jwst_reffiles.dark_current.badpix_from_darks.saturated_in_all_groups(pedestal_array, first_group_sat)¶

Generate a list of pixels that are saturated in all groups

Parameters:	pedestal_array (numpy.ndarray) – 2D array of pedestal values (signal extrapolated to time=0) first_group_sat (numpy.ndarray) – 2D array of the first group DQ containing either 0 = not saturated or 2 = saturated.
Returns:	full_saturation – Boolean array describing which pixels are saturated in all reads
Return type:	numpy.ndarray

jwst_reffiles.dark_current.badpix_from_darks.slopes_not_cr(slope, number_of_jumps)¶

Create an array of pixel slopes which are clean and have no detected cosmic rays

Parameters:

slope (numpy.ndarray) – Array of pixel slopes for integration
number_of_jumps (numpy.ndarray) – Array holding the number of jumps detected for each pixel ramp

Returns:

clean_slope (numpy.ndarray) –

array of slopes for an integration containing no cosmic rays

slopes for a pixel ramp that have a cosmic ray detected are set to nan
iclean_slope (numpy.ndarray) – an array of that holds if a good slope was detected. A value of 1 is assigned to array elements where there are no CRs

Readnoise¶

This module creates a readnoise reference file that can be used in the JWST calibration pipeline.

Author¶

Ben Sunnquist

Use¶

This module can be imported and used as such:
from jwst_reffiles.readnoise import readnoise
readnoise.make_readnoise(arguments)

Notes

Overview: Inputs: A list of dark current ramps

Algorithm:

Method 1 (method=’stack’): Create CDS images for each input ramp. Stack all of these images from each input ramp together. The readnoise is the sigma-clipped standard deviation through this master CDS image stack.

Method 2 (method=’ramp’): Calculate the readnoise in each ramp individually, and then average these readnoise images together to produce a final readnoise map.

jwst_reffiles.readnoise.readnoise.calculate_mean(stack, clipping_sigma=3.0, max_clipping_iters=3)¶

Calculates the sigma-clipped mean through a stack of images.

Parameters:	stack (numpy.ndarray) – A 3D stack of images. clipping_sigma (float) – Number of sigmas to use when sigma-clipping the input stack. max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the input stack.
Returns:	mean_image – 2D image of the sigma-clipped mean through the input stack.
Return type:	numpy.ndarray

jwst_reffiles.readnoise.readnoise.calculate_stddev(stack, clipping_sigma=3.0, max_clipping_iters=3)¶

Calculates the sigma-clipped standard deviation through a stack of images.

Parameters:	stack (numpy.ndarray) – A 3D stack of images. clipping_sigma (float) – Number of sigmas to use when sigma-clipping the input stack. max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the input stack.
Returns:	stddev_image – 2D image of the sigma-clipped standard deviation through the input stack.
Return type:	numpy.ndarray

jwst_reffiles.readnoise.readnoise.get_crds_info(filename, subarray, readpatt)¶

Get CRDS-required header information to put into the final readnoise reference file.

Parameters:

filename (str) – Path to one of the files being used to generate the readnoise reference file. Will be used to find default values.
subarray (str) – CRDS-required subarray for which to use this reference file for.
readpatt (str) – CRDS-required read pattern for which to use this reference file for.

Returns:

instrument (str) – CRDS-required instrument for which to use this reference file for.
detector (str) – CRDS-required detector for which to use this reference file for.
subarray (str) – CRDS-required subarray for which to use this reference file for.
readpatt (str) – CRDS-required read pattern for which to use this reference file for.
fastaxis (int) – CRDS-required fastaxis of the reference file.
slowaxis (int) – CRDS-required slowaxis of the reference file.
substrt1 (int) – CRDS-required starting pixel in axis 1 direction.
substrt2 (int) – CRDS-required starting pixel in axis 2 direction.

jwst_reffiles.readnoise.readnoise.make_cds_stack(data, group_diff_type='independent')¶

Creates a stack of CDS images (group difference images) using the input ramp data, combining multiple integrations if necessary.

Parameters:	data (numpy.ndarray) – The input ramp data. The data shape is assumed to be a 4D array in DMS format (integration, group, y, x). group_diff_type (str) – The method for calculating group differences. Options are: `independent`: Each group is only differenced once (e.g. 6-5, 4-3, 2-1) `consecutive`: Each group is differenced to its neighbors (e.g. 4-3, 3-2, 2-1)
Returns:	cds_stack – A 3D stack of the group difference images.
Return type:	numpy.ndarray

jwst_reffiles.readnoise.readnoise.make_readnoise(filenames, method='stack', group_diff_type='independent', clipping_sigma=3.0, max_clipping_iters=3, nproc=1, slice_width=50, single_value=False, outfile='readnoise_jwst_reffiles.fits', author='jwst_reffiles', description='CDS Noise Image', pedigree='GROUND', useafter='2015-10-01T00:00:00', history='', subarray=None, readpatt=None, save_tmp=False)¶

The main function. Creates a readnoise reference file using the input dark current ramps. See module docstring for more details.

Parameters:

filenames (list) – List of dark current files. These should be calibrated ramp images. The data shape in these images is assumed to be a 4D array in DMS format (integration, group, y, x).
method (str) –
The method to use when calculating the readnoise. Options are: stack: Creates a master stack of all CDS images by combining the

CDS images for each ramp and integration together. The final readnoise map is calculated by taking the stddev through this master stack.

ramp: Calculates the readnoise in each ramp individually. The

final readnoise map is calculated by averaging these individual readnoise maps together.
group_diff_type (str) –
The method for calculating group differences. Options are: independent: Each group is only differenced once (e.g. 6-5, 4-3,

2-1).

consecutive: Each group is differenced to its neighbors (e.g.

4-3, 3-2, 2-1).
clipping_sigma (float) – Number of sigma to use when sigma-clipping.
max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping.
nproc (int) – The number of processes to use during multiprocessing.
slice_width (int) – The width (in pixels) of the image slice to use during multiprocessing. The readnoise of each slice is calculated separately during multiprocessing and combined together at the end of processing. Only relevant if method==stack.
single_value (bool) – Option to use a single readnoise value (the average of all readnoise values) for all pixels.
outfile (str) – Name of the CRDS-formatted readnoise reference file to save the final readnoise map to.
author (str) – CRDS-required name of the reference file author, to be placed in the referece file header.
description (str) – CRDS-required description of the reference file, to be placed in the reference file header.
pedigree (str) – CRDS-required pedigree of the data used to create the reference file.
useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01T00:00:00’).
history (str) – CRDS-required history section to place in the reference file header.
subarray (str) – CRDS-required subarray for which to use this reference file for.
readpatt (str) – CRDS-required read pattern for which to use this reference file for.
save_tmp (bool) – Option to save the final readnoise map before turning it into CRDS format. This is useful if the CRDS transformation fails; in this scenario, you won’t lose all of the final readnoise results so all of the previous processing wasn’t for nothing.

jwst_reffiles.readnoise.readnoise.none_check(value)¶

If value is a string containing ‘none’, then change it to a NoneType object.

Parameters:	value (str or NoneType) –
Returns:	new_value
Return type:	NoneType

jwst_reffiles.readnoise.readnoise.readnoise_by_ramp(filename, group_diff_type='independent', clipping_sigma=3.0, max_clipping_iters=3)¶

Calculates the readnoise for the given input dark current ramp.

Parameters:

filename (str) – The dark current ramp file. The data shape in this image is assumed to be a 4D array in DMS format (integration, group, y, x).
group_diff_type (str) –
The method for calculating group differences. Options are: independent: Each group is only differenced once (e.g. 6-5, 4-3,

2-1)

consecutive: Each group is differenced to its neighbors (e.g.

4-3, 3-2, 2-1)
clipping_sigma (float) – Number of sigma to use when sigma-clipping.
max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping.

jwst_reffiles.readnoise.readnoise.readnoise_by_slice(filenames, group_diff_type='independent', clipping_sigma=3.0, max_clipping_iters=3, column=0, slice_width=50)¶

Calculates the readnoise for a given slice in the input dark file ramps. Useful for multiprocessing and avoiding memory issues for large image stacks.

Parameters:	filenames (list) – List of dark current files. The data shape in these images is assumed to be a 4D array in DMS format (integration, group, y, x). group_diff_type (str) – The method for calculating group differences. Options are: `independent`: Each group is only differenced once (e.g. 6-5, 4-3, 2-1) `consecutive`: Each group is differenced to its neighbors (e.g. 4-3, 3-2, 2-1) clipping_sigma (float) – Number of sigma to use when sigma-clipping. max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping. column (int) – The index of the starting image column for each slice. slice_width (int) – The width of the slice in pixels.
Returns:	readnoise – 2D image of the calculated readnoise.
Return type:	numpy.ndarray

jwst_reffiles.readnoise.readnoise.save_readnoise(readnoise, instrument='', detector='', subarray='GENERIC', readpatt='ANY', outfile='readnoise_jwst_reffiles.fits', author='jwst_reffiles', description='CDS Noise Image', pedigree='GROUND', useafter='2015-10-01T00:00:00', history='', fastaxis=-1, slowaxis=2, substrt1=1, substrt2=1, filenames=[])¶

Saves a CRDS-formatted readnoise reference file.

Parameters:

readnoise (numpy.ndarray) – The 2D readnoise image.
instrument (str) – CRDS-required instrument for which to use this reference file for.
detector (str) – CRDS-required detector for which to use this reference file for.
subarray (str) – CRDS-required subarray for which to use this reference file for.
readpatt (str) – CRDS-required read pattern for which to use this reference file for.
outfile (str) – Name of the CRDS-formatted readnoise reference file to save the final readnoise map to.
author (str) – CRDS-required name of the reference file author, to be placed in the referece file header.
description (str) – CRDS-required description of the reference file, to be placed in the reference file header.
pedigree (str) – CRDS-required pedigree of the data used to create the reference file.
useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01T00:00:00’).
history (str) – CRDS-required history section to place in the reference file header.
fastaxis (int) – CRDS-required fastaxis of the reference file.
slowaxis (int) – CRDS-required slowaxis of the reference file.
substrt1 (int) – CRDS-required starting pixel in axis 1 direction.
substrt2 (int) – CRDS-required starting pixel in axis 2 direction.
filenames (list) – List of dark current files that were used to generate the reference file.

jwst_reffiles.readnoise.readnoise.use_single_value(readnoise, instrument, detector, subarray, clipping_sigma=3.0, max_clipping_iters=3)¶

Replaces all readnoise image values with the average of all readnoise values.

Parameters:	readnoise (numpy.ndarray) – The 2D readnoise image. instrument (str) – CRDS-required instrument for which to use this reference file for. detector (str) – CRDS-required detector for which to use this reference file for. subarray (str) – CRDS-required subarray for which to use this reference file for. clipping_sigma (float) – Number of sigma to use when sigma-clipping. max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping.
Returns:	readnoise_single_value – 2D readnoise image where all of the readnoise values are the same (the average of all of the original readnoise values).
Return type:	numpy.ndarray

jwst_reffiles.readnoise.readnoise.wrapper_readnoise_by_ramp(args)¶

A wrapper around the readnoise_by_ramp function to allow for multiprocessing.

Parameters:	args (tuple) – A tuple containing the input arguments for the readnoise_by_ramp function. See readnoise_by_ramp docstring for more details.
Returns:	readnoise – 2D image of the calculated readnoise (i.e. the output from the readnoise_by_ramp function).
Return type:	numpy.ndarray

jwst_reffiles.readnoise.readnoise.wrapper_readnoise_by_slice(args)¶

A wrapper around the readnoise_by_slice function to allow for multiprocessing.

Parameters:	args (tuple) – A tuple containing the input arguments for the readnoise_by_slice function. See readnoise_by_slice docstring for more details.
Returns:	readnoise – 2D image of the calculated readnoise (i.e. the output from the readnoise_by_slice function).
Return type:	numpy.ndarray

Dark Current¶

This module contains code for creating a dark current reference file.

Input:

List of exposures at a calibration state from which you wish to create dark current reference files. For NIR detectors, this should be after:

group_scale
dq_init
saturation
ipc* - not currently run
superbias
refpix
linearity
persistence

For MIRI, this should be after:

group_scale
dq_init
saturation
linearity
rscd

Also, versions of these files that have been run up through the JUMP step must also be present. This is because we need to know where the CR hits are in the data.

******************POSSIBLE INPUT CHANGE************************** (An alternate plan: Run the pipeline once on the input files. Run up through the JUMP step. For NIR data, skip the dark subtraction step. For MIRI, skip refpix and dark subtraction steps. Then the CR flags will be present, and the data will be in the proper state to create dark current reference files.) ******************POSSIBLE INPUT CHANGE**************************

All files should be from the same instrument/detector/aperture/readpattern

Ouputs:

A dark current reference file in the proper format A series of images showing for each pixel in each group, how many

measurements of the signal were used to calculate the mean dark value.

Function: Get file headers from each file. Check for consistency. Extract CR flags from the JUMP version of the files.

From these, create arrays giving the group number of the first cosmic ray hit in each pixel for each integration

Create images of how many dark current measurements are considered good: for each pixel and each group. All groups after the first CR hit in a pixel are considered bad and not used.
Read in the data. To avoid memory problems only read in N groups at a time,: where N is determined from user inputs.
Flag all groups that occur between the first CR hit and the end of the integration: as bad.
From the remaining good data, calculate the sigma-clipped mean dark signal: in each pixel for each group (i.e. take the mean across integrations). Also calculate the sigma-clipped stdard deviation.
Save the sigma-clipped mean as the dark data, and the sigma-clipped stdev: as the uncertainty in a DarkModel or MIRIDarkModel instance. The DQ array is currently all zeros, anticipating bad pixels being placed in the bad pixel reference file.

Superbias¶

This module creates a superbias reference file that can be used in the JWST calibration pipeline.

Author¶

Ben Sunnquist

Use¶

This module can be imported and used as such:
from jwst_reffiles.superbias import superbias
superbias.make_superbias(arguments)

Notes

Overview: Inputs: A list of dark current ramps

Algorithm: The superbias is the sigma-clipped mean of all 0th frames.: The superbias error is the sigma-clipped stddev of all 0th frames. Pixels with high superbias error values are flagged as UNRELIABLE_BIAS and DO_NOT_USE in the superbias DQ array.

jwst_reffiles.superbias.superbias.calculate_mean(stack, clipping_sigma=3.0, max_clipping_iters=3)¶

Calculates the sigma-clipped mean through a stack of images.

Parameters:	stack (numpy.ndarray) – A 3D stack of images. clipping_sigma (float) – Number of sigmas to use when sigma-clipping the input stack. max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the input stack.
Returns:	mean_image – 2D image of the sigma-clipped mean through the input stack.
Return type:	numpy.ndarray

jwst_reffiles.superbias.superbias.calculate_stddev(stack, clipping_sigma=3.0, max_clipping_iters=3)¶

Calculates the sigma-clipped standard deviation through a stack of images.

Parameters:	stack (numpy.ndarray) – A 3D stack of images. clipping_sigma (float) – Number of sigmas to use when sigma-clipping the input stack. max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the input stack.
Returns:	stddev_image – 2D image of the sigma-clipped standard deviation through the input stack.
Return type:	numpy.ndarray

jwst_reffiles.superbias.superbias.find_unreliable_bias_pixels(error, bad_clipping_sigma=3.0, bad_max_clipping_iters=3, bad_sigma_threshold=5.0, plot=False)¶

Identifies pixels with high superbias error values (i.e. UNRELIABLE_BIAS pixels).

Parameters:	error (numpy.ndarray) – A 2D superbias error image. bad_clipping_sigma (float) – Number of sigma to use when sigma-clipping the superbias error image to find UNRELIABLE_BIAS pixels. bad_max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the superbias error image to find UNRELIABLE_BIAS pixels. bad_sigma_threshold (float) – Number of standard deviations above the mean of the superbias error image at which a pixel is flagged as UNRELIABLE_BIAS. plot (bool) – Option to save a histogram of the superbias error image values, with the threshold used for flagging UNRELIABLE_BIAS pixels also shown.
Returns:	dq – A 2D DQ image that flags pixels with high superbias error values as UNRELIABLE_BIAS (2) and DO_NOT_USE (1).
Return type:	numpy.ndarray

jwst_reffiles.superbias.superbias.get_crds_info(filename, subarray, readpatt)¶

Get CRDS-required header information to put into the final superbias reference file.

Parameters:

filename (str) – Path to one of the files being used to generate the superbias reference file. Will be used to find default values.
subarray (str) – CRDS-required subarray for which to use this reference file for.
readpatt (str) – CRDS-required read pattern for which to use this reference file for.

Returns:

instrument (str) – CRDS-required instrument for which to use this reference file for.
detector (str) – CRDS-required detector for which to use this reference file for.
subarray (str) – CRDS-required subarray for which to use this reference file for.
readpatt (str) – CRDS-required read pattern for which to use this reference file for.
fastaxis (int) – CRDS-required fastaxis of the reference file.
slowaxis (int) – CRDS-required slowaxis of the reference file.
substrt1 (int) – CRDS-required starting pixel in axis 1 direction.
substrt2 (int) – CRDS-required starting pixel in axis 2 direction.

jwst_reffiles.superbias.superbias.make_filled_superbias(superbias, refpix_map, bad_clipping_sigma=3.0, bad_max_clipping_iters=3, bad_sigma_threshold=5.0, box_width=3, outfile='superbias_jwst_reffiles.fits')¶

Create a version of the superbias image where the values of high superbias pixels are replaced by their smoothed values.

Parameters:

superbias (numpy.ndarray) – A 2D superbias image.
refpix_map (numpy.ndarray) – A 2D image where refpix have a value of 1 and all other pixels have a value of 0.
bad_clipping_sigma (float) – Number of sigma to use when sigma-clipping to find UNRELIABLE_BIAS pixels.
bad_max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping to find UNRELIABLE_BIAS pixels.
bad_sigma_threshold (float) – Number of standard deviations above the mean at which a pixel is flagged as UNRELIABLE_BIAS.
box_width (int) – Width in pixels of the box kernel to use to compute the smoothed superbias image.
outfile (str) – Name of the CRDS-formatted superbias reference file to save the final superbias map to. This is only used in this function to name the filled output file.

jwst_reffiles.superbias.superbias.make_refpix_map(filename)¶

Creates a map of the reference pixels based on the input fits file.

Parameters:	filename (str) – The fits image file to create the refpix map from. The PIXELDQ extension is used to find the refpix.
Returns:	refpix_map – A map of the reference pixels (same dimensions as the input fits file image extensions), where refpix are 1 and all other pixels are 0. An array of all zeros is returned if no refpix are found or if no PIXELDQ extension exists.
Return type:	numpy.ndarray

jwst_reffiles.superbias.superbias.make_superbias(filenames, clipping_sigma=3.0, max_clipping_iters=3, bad_clipping_sigma=3.0, bad_max_clipping_iters=3, bad_sigma_threshold=5.0, plot=False, save_filled=False, box_width=3, save_reffile=True, outfile='superbias_jwst_reffiles.fits', author='jwst_reffiles', description='Super Bias Image', pedigree='GROUND', useafter='2000-01-01T00:00:00', history='', subarray=None, readpatt=None)¶

The main function. Creates a superbias reference file using the input dark current ramps. See module docstring for more details.

Parameters:

filenames (list) – List of dark current ramp files. The data shape in these images is assumed to be a 4D array in DMS format (integration, group, y, x).
clipping_sigma (float) – Number of sigma to use when sigma-clipping through the 0th frame stack to create the superbias image.
max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping through the 0th frame stack to create the superbias error image.
bad_clipping_sigma (float) – Number of sigma to use when sigma-clipping the superbias error image to find UNRELIABLE_BIAS pixels.
bad_max_clipping_iters (int) – Maximum number of iterations to use when sigma-clipping the superbias error image to find UNRELIABLE_BIAS pixels.
bad_sigma_threshold (float) – Number of standard deviations above the mean of the superbias error image at which a pixel is flagged as UNRELIABLE_BIAS.
plot (bool) – Option to save a histogram of the superbias error image values, with the threshold used for flagging UNRELIABLE_BIAS pixels also shown.
save_filled (bool) – Option to save an image where high superbias pixels are replaced by their smoothed values.
box_width (int) – Width in pixels of the box kernel to use to compute the smoothed superbias image. Only relevant if save_filled=True.
save_reffile (bool) – Option to save the generated superbias map into a CRDS-formatted reference file.
outfile (str) – Name of the CRDS-formatted superbias reference file to save the final superbias map to.
author (str) – CRDS-required name of the reference file author, to be placed in the referece file header.
description (str) – CRDS-required description of the reference file, to be placed in the reference file header.
pedigree (str) – CRDS-required pedigree of the data used to create the reference file.
useafter (str) – CRDS-required date of earliest data with which this referece file should be used (e.g. ‘2019-04-01T00:00:00’).
history (str) – CRDS-required history section to place in the reference file header.
subarray (str) – CRDS-required subarray for which to use this reference file for.
readpatt (str) – CRDS-required read pattern for which to use this reference file for.

jwst_reffiles.superbias.superbias.none_check(value)¶

If value is a string containing ‘none’, then change it to a NoneType object.

Parameters:	value (str or NoneType) –
Returns:	new_value
Return type:	NoneType

jwst_reffiles.superbias.superbias.save_superbias(superbias, error, dq, instrument='', detector='', subarray='GENERIC', readpatt='ANY', outfile='superbias_jwst_reffiles.fits', author='jwst_reffiles', description='Super Bias Image', pedigree='GROUND', useafter='2000-01-01T00:00:00', history='', fastaxis=-1, slowaxis=2, substrt1=1, substrt2=1, filenames=[])¶

Saves a CRDS-formatted superbias reference file.

Parameters:

superbias (numpy.ndarray) – The 2D superbias image.
error (numpy.ndarray) – The 2D superbias error image.
dq (numpy.ndarray) – The 2D superbias data quality image.
instrument (str) – CRDS-required instrument for which to use this reference file for.
detector (str) – CRDS-required detector for which to use this reference file for.
subarray (str) – CRDS-required subarray for which to use this reference file for.
readpatt (str) – CRDS-required read pattern for which to use this reference file for.
outfile (str) – Name of the CRDS-formatted superbias reference file to save the final superbias map to.
author (str) – CRDS-required name of the reference file author, to be placed in the referece file header.
description (str) – CRDS-required description of the reference file, to be placed in the reference file header.
pedigree (str) – CRDS-required pedigree of the data used to create the reference file.
useafter (str) – CRDS-required date of earliest data with which this referece file should be used. (e.g. ‘2019-04-01T00:00:00’).
history (str) – CRDS-required history section to place in the reference file header.
fastaxis (int) – CRDS-required fastaxis of the reference file.
slowaxis (int) – CRDS-required slowaxis of the reference file.
substrt1 (int) – CRDS-required starting pixel in axis 1 direction.
substrt2 (int) – CRDS-required starting pixel in axis 2 direction.
filenames (list) – List of dark current files that were used to generate the reference file.

The jwst_reffiles package contains a framework to organize and automate the creation of calibration reference files for JWST instruments. These reference files are used by the JWST calibration pipeline for data calibration.

There are three general stages within jwst_reffiles for reference file creation.

In this documentation, we use the term individual reference file to refer to a reference file created from a single input file or group of files. For example, if a single dark current exposure can be used to create a readnoise image, then this readnoise image, onced saved in the appropriate format, is an individual reference file. Conversely, we refer to the mean readnoise image created from a stack of individual readnoise images as the final reference file.