p3d Changelog

What's new in p3d 2.2.3 Revision 2792

Jun 17, 2013

All tools that combine raw-data images:
Major bug fix - When I optimized the C routine of the image combination routine I introduced a serious bug. This bug is present in revisions 2753-2779 of p3d_cimcom.c, and is fixed with revision >=2780
C routines:
Optimization - Optimized the C routines p3d_cpixex.c and p3d_cmpd.c

New in p3d 2.1.1 Revision 1615 (Feb 6, 2012)

Combining multiple files with extracted data (p3d_cexposure):
New functionality.
Added a new tool that combines mutliple extracted exposures that are
taken illuminating the exact same field on the sky. The supplied data
are at first checked for consistency in image size and equal values on
the wavelength calibration parameters CRVAL and CDELT. Thereafter the
files are either averaged, weighting with the exposure time of the
individual exposures. Alternatively, setting the /SATURATED keyword it
is expected that all files have a different exposure time. In this case
the longest exposure is used as reference. Portions of individual
spectra, where a mask indicates that a pixel was saturated, are then
replaced with scaled data of the same spectrum and portion of a file
using a shorter exposure time. The routine accepts any number of files.
The routine produces one output file with merged data, and a file with
corresponding errors if all the error files of the input images are
available.
o Handling of saturated pixels in the raw data:
New functionality.
From now on a new mask file is written along with extracted data
whenever the keyword SATMASK is set in the routines p3d_cdmask,
p3d_cflatf, and p3d_cobjex. This mask file indicates, for each element
in the spectra, if any of the used raw-data pixels were saturated. Only
the center part of the cross-dispersion profile is considered for
optimally extracted spectra. This functionality will be used to combine
extracted spectra that use different exposure times.
The science-object extraction procedure (p3d_cobjex):
Minor bugs affecting the functionality.
A cosmic-ray mask file that was provided as an input keyword (CRMASK)
was never identified.
If a run would contain several groups, where some would require the
combination of images, and some others not, while /crclean was set, then
the combined images would not be properly reduced.
o The dispersion-mask creation procedure (p3d_cdmask):
Major bug affecting PMAS 4kx4k reductions.
A new method to handle arc-line entries in vignetted regions of the
PMAS 4kx4k CCD, both Larr and PPAK, was introduced in version 2.1.
However, the initial version is completely filled with bugs. This has
been corrected with revision 1557, also the log-file output has been
improved. Additionally, the vignetted arc-line entries (wavelengths) are
indicated in the "Check the final fits" tab, for the affected spectra.
Note! To continue to use version 2.0, when creating a dispersion-mask,
it is necessary to comment the "vignettedfile_dm" entries in the
instrument-parameter files larr4k.prm and ppak4k.prm.
Minor issue affecting any data with calibration spectra.
Corrected the display of the affected spectrum (number) in the
"Check the final fit" tab. The number is now showing the correct
spectrum also for data that contain calibration spectra.
Compatibility with PINGSoft:
Added functionality
Added PINGSoft-formatted fiber-position table files (to
{p3d_path}/data/tables/pingsoft_postables/), which can be used with the
reduced outcome of p3d to analyze the data using the IFS analysis tool
PINGSoft: http://www.ast.cam.ac.uk/ioa/research/pings/html/index.html.
These new tables were introduced with revision 1602.
Reduction work with GMOS:
Minor issue affecting cosmic-ray cleaning with GMOS data
Fixed the cosmic-ray cleaning routine so that it works with the
multi-extension data of GMOS. It did not until now. This was fixed with
revision 1577.
Reduction work with MPFS:
Major issue affecting any accurate spectrum extraction
Updated the instrument parameters so that it is possible to calculate
accurate cross-dispersion line profiles for optimal extraction by
default. This was made with revision 1584.
Reduction work with VIRUS-P:
Major bug affecting VIRUS-P reductions.
Fixed a bug that concerned how the parsing of the CCD binning parameter
CCDSUM. The bug caused all VIRUS-P reductions to not work, since quite a
while. This was fixed with revision 1572.
The (automatic) feature to compile the C-language source files...
Minor bug affecting those with a src/ directory unavailable for writing
Corrected the minor bug (a forgotten path separator) so that it works
again.
Various minor bug fixes and adjustments to comments throughout the code.

New in p3d 2.1 Revision 1529 (Jan 17, 2012)

Correction for differential atmospheric refraction (p3d_darc):
Added a new tool that corrects extracted science images for differential
atmospheric refraction. The new tool works out-of-the-box with all the
IFUs that use square-shaped spatial elements (VIMOS, FLAMES, PMAS/Larr,
MPFS, and SPIRAL). In some cases the required physical properties are
unavailable in the data header. In that case they must be entered by
hand.
The routine allows the user to choose between three implemented
approaches to calculate the offsets. The results of all three approaches
are collected in a plot, which allows the user to choose the most
accurate method as he or she believes is more correct.
The offsets are calculated using the wavelength-dependent
refractive-index equation of Ciddor (1996; as well as Edlén 1966).
A second approach uses a height-integrating approach to calculate the
refractive-index that comes from the Starlink project. The routine
integrates the refractive index through the trophosphere and the
stratosphere.
A third method uses an empirical fit of a two-dimensional Gaussian
profile to the data itself to calculate the DAR-caused offsets on both
the X and the Y axis.
Most of the features of the DAR-correction routine of J.Walsh (ESO) have
been included with his permission.
Please note that the calculated errors are off as correlated errors are
completely ignored (at the moment).
Removing of cosmic-ray hits in single raw-data images (p3d_cr):
Added a new tool that cleans single raw-data images of cosmic-ray hits.
The new tool works with all raw data that are used with p3d.
The tool uses an IFS-adapted version of the L.A. Cosmic IDL routine of
J.Bloom (Berkeley); this adaptation has been made with the permission of
both P.van Dokkum (Yale) and J.Bloom. The modifications have been
implemented partly according to the suggestions of B.Husemann (AIP).
By default the routine only produces a cleaned image and a cosmic-ray
hit mask. However, if the keyword WRITEALL is set then intermediate
products are also written to the disk at each iteration; these products
allow a close inspection of the quality of the cleaning procedure, and
may also require huge amounts of disk space.
The cosmic-ray mask is used as a bad-pixel mask during (optimal)
spectrum extraction. Thereby the, so-called, weight of masked pixels are
set to zero - i.e. they are excluded from the calculations.
The routine can be called separately, but is also available from
p3d_ctrace, p3d_cdmask, p3d_cflatf, and p3d_cobjex, when the new keyword
/CRCLEAN is set.
The spectrum extraction:
The resampling of the extracted spectra from pixel to wavelength units
can now be done using either a one-dimensional drizzling algorithm, or
the original linear-interpolation approach. The error calculation of the
original linear-interpolation method has been improved. The older less
accurate errors are still available for comparison; through the use of
the ORIGINALERRORS keyword.
The multi-profile deconvolution method that uses a band-diagonal matrix
has been fully ported to C for fast and parallelized execution. The new
routine, which has been written by F. Tabataba-Vakili at AIP, compiles
and executes automatically; together with the other C routines.
The RSS to cube converter (p3d_rss2cube):
Added a new tool that converts the extracted images that p3d writes into
cubes, which may be easier to work with. This routine is only available
as a stand-alone tool, and also only works with the IFUs that use
square-shaped elements, i.e. PMAS/Larr, SPIRAL, VIMOS, FLAMES, and MPFS.
The scattered-light subtraction:
The scattered-light subtraction can now be switched on and off using a
keyword to the call of p3d_cflatf and p3d_cobjex.
Regarding the instrument support for VIMOS and FLAMES:
Due to an extremely serious bug in the routine that reads the gain and
the readout noise values from the data header, all errors calculated for
these two instruments so far have been wrong - they have been too small.
Regarding the instrument support for VIMOS:
Corrected a bug that would prevent a proper extraction of data that were
sampled using the medium resolution setup (the DETSEC parameter was
improperly read from the instrument-parameter file).
Regarding the instrument support for PMAS/4kx4k CCD (both PPAK and Larr):
The new PMAS 4kx4k CCD has a region in all four corners of the CCD that
are vignetted. When creating a dispersion-mask image, any arc line
inside of these vignetted regions are now automatically excluded from
the chosen selection of line-list entries before the arc-line positions
are fitted with a polynomial. This way it is thought that the wavelength
calibration can be made more accurate in affected spectra. Note that a
exact definition of the vignetted region is difficult to make. If you
think that you have a better solution on how to define the extent of
the vignetted regions, then please, have a look at the two files
ppak4k_vignet.dat and larr4k_vignet.dat.
o The spectrum viewer (p3d_rss):
The spatial maps are now saved as two- or three-dimensional image arrays
instead as of an unordered one-dimensional array (which is how has been
until now). In the new file each pixel corresponds to a spatial element.
This feature is now only available on instruments with square-shaped
spatial elements, i.e. PMAS/Larr, SPIRAL, VIMOS, FLAMES, and MPFS.
o The instrument- and user-parameter file parser (p3d_misc_read_params):
The routine did not return the correct flag if a parameter was only
found in the user-parameter file and not in the instrument-parameter
file. For example, if crval is only set in the user-parameter file then
it would always be reported as not found. This has been corrected with
revision 1414.
o Added the (automatic) feature to compile the C-language source files in a
temporary directory on the local computer. Up to now the compilation was
always done in ${p3d_path}/src/, which would not work on a central
installation.
o Various minor bug fixes and adjustments to comments throughout the code.

New in p3d 2.0.3 Revision 1181 (Oct 13, 2011)

New in p3d 2.0.2 Revision 1150 (Sep 29, 2011)

Regarding the instrument support for VIMOS:
Added a new feature to the combination routine (p3d_cvimos_combine).
Flux-calibrated data are now re-normalized using a telluric line. This final normalization corrects for smaller inaccuracies in the data that are left after the flux calibration (which is performed using, e.g., IRAF). This new re-normalization can be switched off, and the telluric line that is used can be chosen freely (or automatically). The re-normalization can be done both per spectrum and per quadrant.
The telluric-line fits are now illustrated in a plot that includes the raw data and the fit. One plot is created for each spectrum and each detector. The file is compressed automatically to save space.
The science-object extraction (p3d_cobjex):
The telluric-line fits of the sky alignment procedure, which is applied with the application of the dispersion solution, are now illustrated in a plot that includes the raw data and the respective fit. One plot is created for each spectrum and each detector. The file is compressed automatically to save space.
The spectrum viewer (p3d_rss):
Added a colorbar for the spectrum image and the spatial image.
Corrected a format mix-up when saving a mean or total spectrum. Also
corrected a serious bug affecting the error calculations of the "sky"-subtracted spectrum.
Corrected the annotation of the "viewed" wavelength in the spectrum plot.
The wavelength-range zoom value and the wavelength-range shift value are now reset to the initial values when the 'Reset' button is pressed.
Corrected a bug in the object-extraction routine p3d_cobjex, which could
appear if the GROUP keyword was used together with flat fielding. The result could be seen as (kind of) randomly rearranged spatial maps (fixed with revision 1054).
Corrected the World Coordinate System header entries for the
cross-dispersion axis. Now the data are properly labeled in ds9. /PW
Corrected the e3d output; due to an (old) -1 image-row offset the fiber position table inthe e3d file was always wrong (corrected with
revision 1048).
Cleaned up all routines of the trace/disp.mask/object shift reference files. These were never used, and are not required anymore
(fixed with revision 1056).
Various minor bug fixes and adjustments to comments throughout the code.

New in p3d 2.0 Revision 1011 (Jun 17, 2011)

Regarding the instrument support:
VLT/UT2/FLAMES/(ARGUS, IFU1, IFU2) (/FLAMES)
Corrected the tracing procedure for the mini-IFUs. Francesco D'Eugenio
informed us of this inconsistency. Fibers 192, 229, and 278 (IFU1) and
192 and 303 (IFU2) are dead, as well as the last (off-chip) fibers. This
was corrected with revision 877 (tracings using all previous revisions
are consequently wrong).
o Recentering the calculated cross-dispersion line profiles on object data:
Due to flexure data of most instruments are shifted on the CCD as the
time progresses (as a function on where on the sky the telescope is
pointing). Starting with Revision 897 p3d now attempts to correct for
such offsets in the cross-dispersion direction, whenever optimal
extraction is used and the parameter recenterprofiles is set to yes.
These offsets are normally very small, but even an offset of 0.10 pixel
can have a strong influence on the extracted intensities; the influence
becomes stronger the closer the spectra are arranged on the CCD. There
are three methods available to calculate these offsets: either the data
are smoothed (median-filtered) on the dispersion axis before the line
profiles are fitted anew (for one or a few pixels on the dispersion
axis), or the routine first searches for the brightest telluric line in
the data, before that line is fitted (this approach is only available if
a dispersion-mask image is provided as input to p3d_cobjex), or an
offset is provided explicitly. The recentering can be configured either
through the user parameter file (all settings), or through the RECENTER
keyword to p3d_cobjex (only the basic setup - which normally is enough).
o The fiber-flat field normalization (p3d_flatfield):
Starting with revision 966 any dead, unused, or low-transmission fibers
are ignored when calculating a mean spectrum, which is used to
normalize the flat-field data using /NDISP.
The dispersion-mask creation (p3d_cdmask and p3d_wavecal_dispmask_gui):
Up to now it was always necessary to create a dispersion-mask image from
scratch interactively, which could be a quite time-consuming process if
there are many dispersion masks to create. Starting with revision 892 it
is now possible to specify a starting dispersion-mask image as an input.
Using such a starting image, which -must- use the same observing
conditions as the new dispersion-mask image, the creation process is
very fast!
An arc-line list file can now be specified in the p3d main GUI, before
it launches p3d_cdmask.
Starting with revision 1004 the used arc-line list is stored in the
header of the dispersion-mask image. Because of this it is not
necessary, anymore, to (first store and then remember to) provide
p3d_cdmask with the same arc-line list file that was used to create the
DISPMASKIN dispersion-mask image - the data is loaded automatically from
the header.
The vimos extracted spectra combination program:
The previously very preliminary version of the VIMOS extracted-objects
image combination program is now debugged, fully documented, it writes
log messages to both the status line and the log file. The routine,
p3d_cvimos_combine, can be called from both the IDL command line, the
shell prompt (using p3d_cvimos_combine_vm.sh), and from the GUI (using
the lowermost righthand-side tab in the VIMOS configurations).
The spectrum viewer:
Added error handling to all (mean) spectrum-writing commands. The mean
spectrum file now also contains the extended header keyword, to make the
file fully consistent with the FITS format.
The log files:
Starting with revision 909, added the file revision number to the
routine-specific identification string "rname". This addition makes it
possible to see which routine version was used to produced the outcome.
The scattered-light subtraction:
Starting with revision 963 the calculation of a median-filtered image,
using all parts of the image where there are no spectra, is now carried
out by a C routine. The C routine is orders of magnitude faster than the
IDL routine, and also makes use of multi-processor platforms (through
OpenMP). The routine is compiled automatically by p3d.
Corrected numerous smaller bugs.

New in p3d 2.0b (Jun 16, 2011)

Modified the implementation:
VLT/UT3/VIMOS (/VIMOS, /NVIMOS, /MVIMOS, /MNVIMOS):
Data using the high (medium) resolution grism (HR [MR]) are now selected
using the /VIMOS and /NVIMOS (/MVIMOS and /MNVIMOS) keywords. Data from
before (after) the instrument reburbish in May-August 2010 must now use
the /VIMOS and /MVIMOS (/NVIMOS and /MNVIMOS) keywords.
The IDL VM scripts:
Made some minor corrections to how the code is executed using the IDL VM.
The routine scripts (p3d_c*_vm.sh) would have returned an error regarding
a missing environmental variable "path", which is never set, or used, in
these routines. Removed the statement "cd $path". The script "p3d_vm.sh"
has prevented a correct use of the data output directory. The script no
more changes directory to ${p3d_path} before launching the p3d GUI; this
allows a correct use of the /CWD keyword.
These scripts contain a large number of additional improvements.
The spectrum extraction:
Added the option to subtract a scattered light component, which could just
as well be a dark current component. This mode is activated by setting the
darkscattsubtract parameter to yes in the user parameter file. The
subtraction works by fitting a two-dimensional image to the
bias-subtracted image where the spectra should be extracted from. The fit
is only made on the unmasked pixels; all pixels nearby to spectra are
masked. All properties of the fit are controlled with the options in the
(master) user parameter file.
The cross-dispersion profile fitting routine:
Has now been ported to C. Instead of the IDL version of MPFIT the new
routine uses the C version, which is named cmpfit. The necessary source
code is retrieved automatically with the script p3d_acquire_xtools.sh.
The C source code is compiled when it is used, completely transparently
to the user (using the IDL routine MAKE_DLL). If you see that all your
cores, on your multi-core computer, are not used, then check if your
compiler supports OpenMP. The routine has been found to compile
correctly on linux and macosx, but should also work as it is on solaris-
based platforms. We still need to set up the proper compiler options for
windows-based platforms.
If the compilation fails, for whatever reason, then the original IDL
routine is used instead.
The dispersion-mask creation GUI:
Dead or unused fibers are no more fitted in the final step, when the
individual arc-line pixel positions are fitted to the raw data. This way
the probability that any arc-line in a spectrum after, or before, the
current spectrum will be incorrectly fitted is minimized.
Corrected the behavior when reading a arc-line file using the File menu
option. Using the same approach as when the routine first starts, the
read line list is now keeping arc-line entries outside the current
wavelength range.
Various smaller improvements:
The default plot colors are now black lines on a white background. It
used to be the opposite.
Also the spectrum-gap files (gapfile) can be put in the directory of the
user parameter file now.
Corrected various bugs:
Exporting the extracted science spectra in Euro3D format is now working.
Corrected a bug in the spectrum viewer, where the summed spectrum was
saved instead of the average spectrum, even though an average spectrum
was selected.

New in p3d 2.0a (Jun 16, 2011)

Documentation:
All main data-reduction routines are now documented; this includes the six
files: p3d.pro, p3d_cmbias.pro, p3d_ctrace.pro, p3d_cdmask.pro,
p3d_cflatf.pro, and p3d_cobjex.pro. Also the spectrum viewer, p3d_rss.pro,
has been documented.
The documentation includes the following information:
A short description of what the routine does.
The main GUI routine (p3d.pro) and the dispersion-mask routine
(p3d_cdmask) both contain a complete description of how they are used.
A section with information on where additional (detailed) information
can be found on how the routine works, and where the parameter files can
be found, which are required to run the routine.
A section about the format of the input and the output files.
Additional sections with more detailed information on specific tasks.
A description of all input and output parameters/keywords.
An example section with information on how to use the routine; both by
itself from the IDL command prompt, or using shell scripts together with
the IDL Virtual Machine.
Added support for six additional instruments:
WHT/WYFFOS/INTEGRAL (/INTEGRAL, /OLDINTEGRAL):
p3d has been configured to work with data of all four IFUs of INTEGRAL,
i.e.: SB1, SB2, SB3, and the Equalized IFU. In the standard setup the
data must come from the newer 2-detector configuration. If you have data
of the older one-detector configuration you must use the old detector
configuration (/OLDINTEGRAL).
As is the case with the IFU of MPFS all spectra of INTEGRAL are placed
on the CCD without any gaps. Consequently the spectrum profile fitting
becomes very slow, as all spectra are fitted simultaneously.
VLT/UT2/FLAMES/(ARGUS, IFU1, IFU2) (/FLAMES):
p3d has been configured to work with all available configurations of
FLAMES/ARGUS and the mini IFUs FLAMES/IFU1 and FLAMES/IFU2.
VLT/UT3/VIMOS (/VIMOS):
Specifically, p3d is now capable of working with data using the medium-
and high-resolution grisms (i.e. HR Blue, HR Orange, HR Red, and MR).
No support for the low-resolution grisms is planned, since these modes
produce data that is very difficult to reduce accurately.
Thanks to an improved tracing procedure it should not be necessary to
modify any parameters when tracing the spectra. But as always, please
check the outcome to see if p3d could indeed identify all spectra
correctly. VIMOS is a complex IFU to trace since there are so many
unused and dead fibers; actually, several spectra simply fall off the
CCDs. Creating a dispersion solution p3d starts off with a linear
dispersion solution, which has been calibrated by hand for the HR Blue
and HR Orange grisms. The remaining two grisms, i.e. HR Red and MR, have
not yet been calibrated, with respect to the wavelength ranges. So it
will be more tricky to create a dispersion solution for these grims
if you are uncertain on which arc lines you can see.
VIMOS data is provided for four separate detectors, referred to as
quadrants. The data must be reduced for each quadrant separately,
before they are combined into one data set. p3d checks all raw data
files for the header keyword that indicates which detector was used, it
is not possible to combine images originating in different detectors.
Starting October 2010 VIMOS has been upgraded with aligned CCDs. The
current configuration will unlikely work directly for data of this
upgraded IFU. However, the required changes should be limited to
providing new dead fiber files.
Gemini/GMOS (GMOS-S and GMOS-N) (/GMOSS,/GMOSN):
Currently p3d supports the two one-slit modes; i.e. the red and the blue
slits. The two-slit mode is more complex to reduce, and is not (yet)
supported. Since we do not have data for GMOS-N the data files are not
completely configured for this IFU yet (in principle it should only be
necessary to modify the dead fibers file). Additionally, note that the
wavelength axis of data of these IFUs cover more than 6000 pixels. The
initial guess of the linear dispersion solution works poorly across all
the detectors, which is why it is necessary to make a manual dispersion
solution, shifting lines at the opposite end by as much as several
hundred pixels.
MPFS (/MPFS3K):
p3d supports the newer 3kx2k CCD. The spectrum separation is nearly the
same as the FWHM; hence, the spectrum-to-spectrum cross talk is most
significant. The strong overlap currently prevents good fits of the
profiles. Because the spectra are so tightly packed it seems necessary
to limit the permitted FWHM range during the fitting process. The same
profile width does not seem to work for both the blue and the red ends.
A more complete solution will likely have to account for a varying
FWHM across the dispersion axis. Note that there is no gap between the
fibers of this IFU (O.K. a gap due to a dead fiber is used as a gap to
have two spectrum groups). The spectrum profile fitting procedure
becomes very slow since all spectra are fitted simultaneously.
VIRUS-P (/VIRUS):
Both the setup of fiber bundles 1 and 2 are now available using the
/VIRUS keyword; the keyword /OLDVIRUS for fiber bundle 1 has been
removed from the routine p3d.pro.
and PMAS:
Through a minor change in the gratings file the new V500 grating is now
fully supported (the only modification was to replace U300 with V500).
o Cross-dispersion line profile fitting procedure:
Before the signal of different spectra can be deconvolved during the
spectrum extraction it is necessary to calculate a line profile for each
spectrum. p3d now allows the background linear function to be controlled:
By default the constant value and the slope are fitted together with
all the other profile parameters of each group of spectra.
The slope can now be set to zero explicitly by setting the parameter
"lprofnobgslope" to "yes".
The constant value can be set to any positive value (including 0.0),
by setting the parameter "lprofbgzlevel" to that value. If the parameter
is instead set to -1 the default fitting is achieved, while a parameter
value of -2 results in using the lowest value outside the fiber regions.
This last option is useful whenever scattered light seems to be present.
The tracing procedure:
When fitting profiles it is now also possible to fit the width of each
spectrum individually; by setting the 'lprofwidth' parameter to yes.
Please note, however, that this procedure currently seems to work well
only with Gaussian profiles. The other profile functions seem to need to
be more constrained when also the spectrum width is fitted. Since this
mode makes the profile fitting even slower the default is lprofwidth=no.
It is now possible to use a list of spectrum separations for the second
tracing step. Such a list must have SPNUM-1 entries, where SPNUM is the
number of spectra. It is important for this tracing mode that the first
spectrum is actually found in the data. If it is not then it is
absolutely compulsory that the first spectrum/spectra are indicated in
the dead fibers file as being dead, unused, or possibly unused.
Currently this mode is used with the FLAMES/ARGUS IFU, where the
calibration fiber positions are difficult to fit with a constant
separation.
Dead, unused, and possibly unused fibers are not used anymore in the
third tracing step; this is where the spectrum pattern is matched with
the spectra found in the second tracing step. This improvement was
necessary to trace spectra of IFUs with many unused or dead fibers,
such as VIMOS.
Dead, unused, and possibly unused fibers are now indicated in the trace
inspector GUI with dark green symbols and lines.
The flat fielding procedure:
Has been modified to include a better fiber-to-fiber transmission
normalization, as well as a division by the average spectrum. The new
modes are available through options in the user parameter file, as is the
old approach (where all spectra where divided with the average spectrum).
The fiber-to-fiber transmission function can now either be taken from the
current or a separate dataset. This approach is useful if you want to use
twilight flats to perform the fiber-to-fiber transmission correction, while
you at the same want to use a continuum flat to first smooth and replace
the dataset with a polynomial fit, before the full spectra are normalized.
Using p3d with the IDL VM - i.e., without an IDL license:
This was previously a bit tricky as all configuration had to be done using
the GUI. This has now all become much easier.
All commands that are required to start p3d (i.e. those of p3d.pro
and p3d_gui.pro - actually, nearly all commands) can now be entered at
the shell prompt when starting p3d using the VM script p3d_vm.sh.
For example, to start p3d for the VIMOS IFS instrument, with a log file
dred.log, and writing all output data to the current directory, you
would give the following command (this, currently, works on all the
UN*X platforms):
{p3d_path}/vm/p3d_vm.sh /vimos logfile=dred.log /cwd
It is now also possible to call the individual routines using the IDL
VM. Five new shell scripts are provided in the vm/ directory, one for
each data-reduction task: p3d_cmbias_vm.sh, p3d_ctrace_vm.sh,
p3d_cdmask_vm.sh, p3d_cflatf_vm.sh, and p3d_cobjex_vm.sh. These scripts
are most easily called by writing shell scripts that define all the
required input parameters of the respective task. Each IDL routine
(i.e., p3d_cmbias.pro, p3d_ctrace.pro, p3d_cdmask.pro, p3d_cflatf.pro,
and p3d_cobjex.pro) has been modified to read command line parameters.
How the parameters are passed to the shell script is described in the
respective IDL routine. Here is an example on how to call p3d_ctrace.pro
from the terminal (outside IDL):
parfile=${p3d_path}/data/instruments/pmas/larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
userparfile=${opath}/user_p3d.prm
p3d_ctrace_vm.sh tracefile_1.fits.gz,tracefilefile_2.fits.gz \
parfile masterbias=$masterbias userparfile=$userparfile \
opath=$opath logfile=$opath/dred.log /verbose
Note that it is necessary to click on the splash screen that pops up
each time the IDL VM is started. A trick on how to automatically click
on this splash screen (to actually start IDL) is given in README.
The spectrum viewer can also be called using the IDL VM. A new shell
script is provided in the vm/ directory "p3d_rss_vm.sh". In comparison
to when this routine is called from inside IDL the first argument is
the filename of the spectrum fits-file. Inside IDL the first argument
would be the already loaded image variable.
The fiber position tables for PMAS/PPAK/4kx4kCCD and PMAS/PPAK/2kx4k CCD:
now also contain the proper positions of the sky fibers. These sky fibers
are consequently shown properly when viewing them in the spatial map of the
spectrum viewer.
o Corrected numerous bugs related to configurations where the dispersion axis
is in the second dimension (i.e. on the Y axis).
1.1.1 (2010.04.21; Revision 182)
Corrected a problem that has affected all optimally extracted spectra of
all revisions and releases so far. The problem has been that it has been
necessary to apply a 0.5pixel offset to the profile spectrum positions
before the spectrum extraction. Previously this offset was applied in an
improper way which has not solved the problem. From now on all profile
positions are offset by 0.5 pixel before the data is saved to the *_cdlp*
file. In order to achieve a better accuracy in you extracted spectra it is
necessary to either recalculate the profiles or apply the 0.5 pixel offset
by hand, before extracting the spectra anew. If you want to know more about
how important it is with accurate profile center positions for the accuracy
of your extracted spectra please have a look in Sections 4.1 and 4.4 in
the p3d paper.
Note! Most instruments are affected by flexure, which has the potential to
cause an additional offset between the profiles calculated using a
continuum or sky flat exposure and the object exposure. p3d does not -yet-
correct for such offsets (although support might be added soon).
o The image viewer p3d_imv now also looks for raw data in the directory of
the main input file (FILENAME) and in the parent directory of FILENAME.
This makes it easier to inspect the data if the reductions are carried out
on one machine, and are thereafter transferred to another computer (where
the path is different).

New in p3d 1.1 (Jun 16, 2011)

Much improved support for use with the PMAS 4kx4k CCD.
Implemented a much more strict checking of the available parameters in the
instrument keywords file(s).
Included support for flipping the raw data on the dispersion axis. The
flipping is performed when the DFLIP parameter is set (either in the
instrument parameter file or in the user parameter ditto).
File suffixes, and the string that is used with combined images, can now be
set in the user parameter file.
All parameters in the master user parameter file
(data/master_userparfile.dat) are now fully described.
Added transparent support for gzip-compressed input data files.
Z-compressed files (which were compressed using the UN*X-command compress)
and bzip2-compressed files are also supported on UN*X platforms - see the
READFITS routine of astro-lib for more information. Output data files are
compressed using gzip (all platforms!) if the user parameter compress is
set to yes (default==no).
All references to PASP have been changed to A&A, which is where the paper
on p3d is published (yes, we changed our mind).
Updated PPAK tracing parameters to work more generally.
Document headers have been adjusted to work better with the IDL->html
translation tools of O. Streicher.
Fixed numerous smaller bugs.
GUI:
Added several different filters in the file dialogs, supporting
compressed files as well as non-compressed ones.
Only the first-block files of multi-block instruments are now shown by
default in the file dialogs.
The default width of file dialogs is now set to 500px when using the VM
(see the Xdefaults file in vm/).
Image combination:
All image-combination-related operations have been moved to a separate
routine (routines/p3d_misc_imcombine_wrapper.pro) to comply with the
"Don't Repeat Yourself" coding principle.
Added a slow combination method since the fast MIN() function of IDL
gives strange results (with PMAS at least). This slow method is now also
the default.
Spectrum extraction:
The optimal extraction routines MOX and MPD have been moved to their own
routines.
The default extraction method for MPD is now 'banddiagonal' and not 'svd'
as previously ('svd' is painfully slow).
Information about what is going on is now also printed on the console
when verbose==1.
Master Bias:
It is now possible to pick only one file and then let p3d smooth it
before it is used as a master bias. This now also works with the
PMAS 4kx4k CCD.
Wavelength calibration / Dispersion Mask creation:
Now works properly with two input files for all supported instruments.
Two files (ideally of different exposure time) can be used if arc lines
on one side of the detector are much fainter/brighter than on the other
side.
After entries in the line list were matched with the data a more precise
Gaussian fitting (or weighting) was previously performed in the
"Create dispersion mask" step. In order to allow a checking of the more
precise fitting this fitting has now been given its own function in the
GUI ("Fit centroids"). This function (button) must now be used after the
"Match centroids" step, and before the "Create dispersion mask" step.
The default behavior for p3d is now to not rebin the data on the
dispersion axis before starting the GUI. This allows big data sets to be
used on very small screens (DBIN=1). If the user still wants to have the
data rebinned then she/he can select another mode in the Options menu,
or simply set the DBIN parameter in the user parameter file.
Object extraction:
Now accepts a cosmic ray mask (although this method does not work well at
all).
p3d can now also save output spectra in the E3D-format (finally; as we
have promised in the paper on p3d).