Zeustools API documentation

zeustools.rooneystools module

class zeustools.rooneystools.ArrayMapper(path=None)

Bases: object

This class makes it easy to address mce data.

MCE data is stored in “Row,Col” format, where the rows and columns correspond to how the pixels are wired to the electronics. This format is useful when debugging electronics issues but not so much for viewing data. So, to index raw MCE data you usually call

mcefile = mce_data.SmallMCEFile(filename)
mcedata = mcefile.Read(row_col=True).data

Then, mcedata will be a 3-dimensional datacube. You index this cube with mcedata[row, col, time]

The physical layout of the detector has a spatial direction and a spectral direction. To index a pixel using spatial position and spectral position, you either need to reformat the mcedata object or convert the spatial position and spectral position to a row and column. This does the latter.

The phys_to_mce function in this class will return an index you can use immediately to index an mce_data datacube in order to address a pixel by its physical location.

Parameters: path – (optional) path to a folder containing the three arrayX_map.dat files. The default array map is the one most recently updated in 2021 with input from Bo, and is now included in the package by default!

grid_map()

Return a grid of spectral positions and spatial positions for each mce_row and mce_column

Note that for ease of use the 400 array has 10 added to all its spatial positions, and the 200 array has 20 added to its spectral positions. If the entries are masked, there is nothing wired to that MCE location (usually indicates broken wires that have been routed around).

Returns: Numpy array of shape (33,24,2). The first two axes match the MCE; axis 0 selects an MCE row, while axis 1 selects an MCE column. This way if you take np.grid_map()[mce_row,mce_col] you get back [spatial_position,spectral_position]

phys_to_mce(spec, spat, array)

Given the physical position of a pixel (spectral position, spatial position, array name) returns the mce_row and mce_col of that pixel. That can be used to directly address the pixel from the mce datacube.

Parameters

spec – The spectral position of the pixel to index.
spat – The spatial position of the pixel to index.
array – The array the pixel is on. Should be a number or string, like “400” for the 400 micron array.

Returns

Tuple of (mce_row, mce_col) of the pixel.

Example

To load a data file and get the time series for spectral pixel 7 on spatial position 1 of the 400 micron array:

chop, ts, datacube = load_data_raw(...)
am = ArrayMapper()
idx = am.phys_to_mce(7,1,400)
time_series = datacube[idx]

It is recommended that you initialize am = ArrayMapper() once and reuse the am object over the lifetime of your code.

TODO: better way to index multiple pixels.

zeustools.rooneystools.array_name(name)

“A rose by any other name would smell as sweet”

Converts various names for the 3 different physical arrays into the internal scheme used by this class (which is the same as the letter used by arrayX_map.dat, much to Thomas’s dismay. At least this is an easy way to convert to that scheme.)

That scheme is follows:

array “A” is the 400 um array (350/450)
array “B” is the 200 um array
array “C” is the 600 um array.

simply pass any name (e.g. "400", 400, 350, "200") and this will return the correct letter to use in arrayX_map.dat

Parameters: name – Human readable name for the array. Should be a string or number.
Returns: one of ‘a’, ‘b’, or ‘c’.

zeustools.rooneystools.big_signal_bad_px_finder(chop, cube, corrthresh=0.5)

class zeustools.rooneystools.chi_sq_solver(bins, ys, function, guesses, bounds=None)

Bases: object

A chi squared analysis tool I built. I don’t know whether it’s still useful. Kept around just in case. It might also assume your data is Poisson… Or I’ve just forgotten how chi squared works?

centers(bins)

chi_sq(args)

zeustools.rooneystools.cube_nan_interpolator(cube)

zeustools.rooneystools.gaussian(x, sigma, mu, a): I never understood why numpy or scipy don’t have their own gaussian function.

zeustools.rooneystools.gaussian_integral(sigma, mu, a): Same parameter order as my “gaussian” function, but no x input. Returns integral from -infinity to infinity of gaussian(x,sigma,mu,a) dx

zeustools.rooneystools.get_ts_time_offset(filename)

Turns out that it’s not straightforward what’s going on between the gps card (and therefore the info in the .ts file) and the time on the pc (ie, what you get when you call datetime.now() or CTIME).

I thought it could be like the TAI/GPS/leap second stuff, but that isn’t really it either. (unless the MCE PC clock is set to TAI somehow.)

Anyway, this method returns gps_card_time - pc_time by looking at the run file and the ts file. The run file contains the pc_time that mce_run was launched, and the ts file contains the gps_card_time of the first data point.

Note, I do not do this here, but you could also check date modified for the data file and the run file—the runfile is written before data collection is commenced, and obviously the data file has to be modified after the last data point is collected. or the last entry in the ts file

returns t_GPS - t_PC = t_offset so t_GPS = t_offset + t_PC (i.e. add return value to PC time to convert so it matches GPS time)

zeustools.rooneystools.histGaussian(something, nbins=50)

zeustools.rooneystools.load_data_handle_outliers(filename, mode='labchop')

zeustools.rooneystools.load_data_raw(filename)

Loads an MCE data file, chop file and TS file, returning the raw data for each of them.

Note that if this turns out to be a “stare” dataset, there won’t be chop or TS files. So we make some surrogate chop and ts data sets. The chop will be all 0s and the ts will be a monotonicall increasing array.

Parameters

filename – the file name that you want to load.

Returns

A tuple containing 3 numpy arrays:

Chop phase (whether we’re looking at the source or the sky)
Time stamp
Time series data cube as usual from mce_data (use ArrayMapper to index it)

Return type

3-Tuple of numpy.array

zeustools.rooneystools.loadts(filename)

Really simple wrapper function to load a .ts file.

Does literally the following:

times=np.loadtxt(f"{filename}.ts")[:,1]

Parameters: filename – the location of the data file to read the time stamps for. Don’t include the .ts extension
Returns: 1-D numpy array of timestamps

zeustools.rooneystools.makeFileName(date, name, number)

Simple routine to convert a date, filename, and file index into a real data file.

Example

Say you have a data file in /data/cryo/20191130/saturn_191130_0023. Calling:

makeFileName('20191130','saturn',23)

will return:

'20191130/saturn_191130_0023`

which you can easily append to the parent folder name.

Parameters

date – String date in yyyymmdd format
name – source name
number – integer of file number

Returns

folder and file path relative to the data folder (e.g. /data/cryo)

TODO: it would be nice if this didn’t depend so much on data types and such

zeustools.rooneystools.nan_helper(y)

Helper to handle indices and logical indices of NaNs.

Input:

y, 1d numpy array with possible NaNs

Output:

nans, logical indices of NaNs
index, a function, with signature indices= index(logical_indices), to convert logical indices of NaNs to ‘equivalent’ indices

Example:

>>> # linear interpolation of NaNs
>>> nans, x= nan_helper(y)
>>> y[nans]= np.interp(x(nans), x(~nans), y[~nans])

zeustools.rooneystools.nan_interp(y)

zeustools.rooneystools.nd_mad(nparray, axis, extrainfo=False)

This is the backend for nd_mads_from_median(). Given a numpy ndarray, we calculate the median absolute deviation

NOTE! in scipy version 1.3 they added a median absolute deviation method! you might consider using that instead in some cases.

However this function gives you the option of returning the distance from the median for every point on the array.

zeustools.rooneystools.nd_mads_from_median(nparray, axis, zero_padding_factor=0.0001): Returns the number of mads from the median for each data point, with median and mad computed over the specified axis

zeustools.rooneystools.nd_nan_outliers(nparray, MAD_chop=5, axis=2)

Returns an array with nans instead of outliers on a certain axis

The input array is examined using the median absolute deviation method and outliers greater than the MAD_chop number are changed to np.nan.

zeustools.rooneystools.nd_reject_outliers(nparray, MAD_chop=5, axis=2)

Returns a masked array masking outliers on a certain axis.

The input array is examined using the median absolute deviation method and outliers greater than the MAD_chop number are masked.

zeustools.rooneystools.parseCmdArgs(argumentList, helpList, typeList)

Parses command-line arguments.

arguments: 3 lists of argument data. First is the list of arguments. Each argument is a list that will be passed to the add_argument function of argparse Each element in the second list is the help string for that argument, The last list contains the type.

Usage:

common.parseCmdArgs([['settings'],
                     ['-v','--verbose']],
                    ['Path to settings file','flag to print extra info'],
                    [str,'bool'])

zeustools.rooneystools.processChop(filename)

Loads in MCE data file, chop file, and TS file.

Returns

a tuple containing:

data points where chop is on
ts for chop on
data for chop off
ts for chop off

zeustools.rooneystools.processChopBetter(fname)

Loads in MCE data file and chop file.

returns an array with the medians of each chop cycle for all pixels.

TODO: needs a better name

zeustools.rooneystools.readPf(filename)

Loads in a .pf file. Those contain the exact data that was sent to the APEX servers for pointing and/or focus calibration. We send lots of 1s to APEX because they do on-off/off and we think we know better than them so we do all kinds of acrobatics.

This script undoes the acrobatics so you see exactly what APEX would see.

Parameters: filename – The filename you want to load (INCLUDE the .pf extension)
Returns: 1-D array of data points (usually a time series)
Return type: numpy.array

zeustools.rooneystools.twoD_Gaussian(pos, amplitude, xo, yo, sigma_x, offset): Good for doing pointing. Doesn’t have too many parameters, for example we set sigma_x = sigma_y and we don’t have a theta.

zeustools.codystools module

zeustools.codystools.createModelSnakeEntireArray(tseries, goodSnakeCorr=0.85, minSnakeNumber=10, bestPixel=(6, 14), fastFreq=0.25, quiet=False)

zeustools.codystools.gaussianSimple(vec, a, v0, sigma)

zeustools.codystools.readChopFile(fname): Reads in the .chop file so you know when the chopper was on or off

zeustools.plotting module

class zeustools.plotting.ClickType(value)

Bases: Enum

An enumeration.

TS_ADD = 2

TS_FLAT_ADD = 4

TS_FLAT_ONLY = 3

TS_ONLY = 1

class zeustools.plotting.FakeEvent(key, xdata, ydata)

Bases: object

Used internally in case you want to manually trigger one of the event handlers

class zeustools.plotting.MultiInteractivePlotter(multi_data, multi_cube, ts=None, flat=None, chop=None)

Bases: ZeusInteractivePlotter

onkey(event)

set_index(i)

class zeustools.plotting.ZeusInteractivePlotter(data, cube, ts=None, flat=None, chop=None)

Bases: object

This is an awesome object that gives you some of the features of the zalpha script!

Tested in ipython notebooks, you first need to initialize such that you are plotting interactively. This is accomplished by running something like

%matplotlib notebook

Then, you can pass in a data array in mce/flux format (like you would get out of the super naive data reduction np.median(-cube[:,:,chop==1],axis=2)+np.median(cube[:,:,chop==0],axis=2) or use in plot_array() above) and a data cube (raw mce format), call interactive_plot() and TA DAH! You can now click on any pixel on the array and immediately view its time series. You have a few additional options too:

Click on a pixel to view its time series

Right click to view a pixel’s time series and flat time series (if the flat cube is provided)

Middle click (or if that doesn’t work press ‘a’ while hovering) to add a time series without clearing the previous one

Press ‘f’ while hovering to add a time series+flat series without clearing

Press ‘c’ while hovering to mask that pixel, removing it from the colorbar calculation and array plot.

This is a work in progress, and is not very robust. If you suspect something is wrong, try accessing error member variable. If it exists it contains the most recent error that occurred. You can also:

import traceback
traceback.print_tb(z2ip.error.__traceback__)
z2ip.error

that blurb will print the error type and also its traceback. Usually exceptions are thrown into Ginnungagap by Trickster God Loki on behalf of the matplotlib event system, so they don’t appear immediately.

bottom_flat()

bottom_plot()

check_for_errors(): Untested. Prints out the most recent exception thrown in the interactive plot routine, since those get lost in the void otherwise.

interactive_plot()

onclick(event)

onkey(event)

redraw_top_plot()

run_signal_correlator()

static_plot(px_for_btm_plot=None, btm_plot_type=None)

zeustools.plotting.get_physical_location(xdata, ydata): Used internally. The center of each pixel is at integer values, so the pixel extends from value - 0.5 to value+0.5. Here we correct for that and also tell you which array your pixel is on. Usable with the plot_array and ZeusInteractivePlotter.

zeustools.plotting.plot_array(somedata, ax=None, s=60, bad_px=False)

Plots mce data nicely but in a less advanced way than Bo.

Parameters

somedata – this should be an array of size (33,24), i.e., mce data format. Each element is the ‘flux’ value of that pixel. This routine will colorize based on flux.
ax – a pyplot axis object in case you want to use yours instead of generating a new one.
s – size value for the squares representing pixels. Tweak this to your liking.

Returns

A tuple containing 3 mpl objects:

scatter plot object
color bar object
axis that the plots were added to

Return type

3-Tuple

zeustools.plotting.spectrum_atm_plotter(velocity, spectrum, errbars, title, atm_helper, pwv, y_scaling=1e-18, bounds=None): Makes a 2-panel plot with atmospheric transmission on the bottom and the observed spectrum on top

zeustools.pointing module

class zeustools.pointing.CachingRaster

Bases: object

Reduces latency on the raster_plot function by only loading in each mce datafile once

go(filestem, start_filename, num_beams, pixels)

Load in pointing raster scan. NOW WITH CACHING! FOR ALL YOUR LOW-LATENCY PLOTTING NEEDS. This is a wrapper around raster_load, and has the same paramters except existing_data.

Parameters

filestem – The folder that contains date folders. e.g.: /data/cryo/current_data/
start_filename – The first number that was part of the raster
num_beams – The number of raster pointings on a side e.g.: For a 3x3 raster would be 3.
pixels – array of (mce_row,mce_col) for pixels to sum to make the raster plot tip: use ArrayMapper.phys_to_mce(spec,spat) to get this number easily
do_plot – Set this to False if you want to just load in the data and not make a plot
existing_data – Used by the Caching Raster to speed up the loading process

Returns

amplitudes of signal at each pointing position

Return type

numpy.array

zeustools.pointing.make_added_pointing_plots(date, fstem, fnums, specs, spats)

zeustools.pointing.make_spectral_pointing_plots(date, fstem, fnums, specs, spats, array='a')

zeustools.pointing.pfRaster(filestem, firstnum, sidesize)

Uses the .pf files generated by the acquisition software to make a raster

Parameters

filestem – The folder that contains date folders. e.g.: /data/cryo/current_data/
firstnum – The first number that was part of the raster
sidesize – The number of raster pointings on a side e.g.: For a 3x3 raster would be 3.

Returns

amplitudes of signal at each pointing position

Return type

numpy.array

zeustools.pointing.raster_calc(arr, stepsize, initial_guess=(1000, 0, 0, 8, 0), do_plot=True, bounds=None)

Calculate the pointing offset to apply to center the source. Prints the pointing correction command that you should send to APECS. This does a nicer job plotting the raster than the other functions.

Parameters

arr – a numpy array containing the values of each position
stepsize – arcseconds offset between raster positions
initial_guess – the initial guess for the 2d gaussian fit. It’s a tuple of (amplitude, x position, y position, sigma, baseline)
do_plot – whether or not to make a plot of the raster and gaussian fit. (default: True)
bounds – same as initial guess, except 2 tuples for max and min bounds for each parameters

Returns

best-fit paramteters

Return type

numpy.array

zeustools.pointing.raster_load(filestem, firstnum, sidesize, pixels, existing_data=[])

Load in and plot a pointing raster scan. Modifies the existing_data array to contain newly-loaded information.

Parameters

filestem – All of the filename except the number. e.g.: /data/cryo/current_data/saturn_191202_
firstnum – The first number that was part of the raster
sidesize – The number of raster pointings on a side e.g.: For a 3x3 raster would be 3.
pixels – array of (mce_row,mce_col) for pixels to sum to make the raster plot tip: use ArrayMapper.phys_to_mce(spec,spat) to get this number easily
existing_data – Used by the Caching Raster to speed up the loading process

Returns

amplitudes of signal at each pointing position

Return type

numpy.array