Download Spectral Line Bandpass Removal Using a Median Filter

Spectral Line Bandpass Removal Using a Median Filter
Travis McIntyre
The University of New Mexico
December 2013
Abstract
For spectral line observations, an alternative to the position switching observation strategy is to
take one observation on source and subtract the median filter of the resulting bandpass from itself. The
median filtering technique has two great advantages over position switching in that it is at least twice as
efficient with telescope time and it does not subtract extended emission from itself. However, median
filtering can create a reduced spectrum with a flat baseline only if the general frequency of the signal is
known (i.e. within tens of widths of the signal) and narrow compared to any structure in the bandpass
(e.g. standing waves caused by a blocked aperture). Median filtering an entire bandpass is not an
effective strategy because the signal must be in a local maximum in order to recover its flux and
profile. Therefore, at least a first order fit of the bandpass around the signal must be removed before
applying a median filter. A radio recombination line from an observation with the Arecibo Radio
Telescope is used to test this method. Its flux and line width are recovered compared to position
switching.
1. Introduction
Removing a median filter from a bandpass is a desirable method of processing a spectral line
signal if source emission is too extended for position switching to work properly. It is also useful in that
it takes less than half as much telescope time to reach the same sensitivity. The following sections will
describe the median filtering technique and analyze its major properties, benefits, and limitations.
2. Process
The first step is to make at least a linear fit of the bandpass around the suspected location of the
spectral line signal. The next step is to create a running median of the fitted bandpass with a window
that is more than twice the width of the signal and less than twice the width of any bandpass structure
present. The final step is to subtract this median bandpass from the fitted bandpass in order to create a
reduced spectrum with a flat baseline. Below is a more detailed analysis of the effects of these
processes on the quality of the data. The H164α radio recombination line shown in Figure 1 is taken
from an observation of the HII region, S257, made with the 305m Arecibo Radio Telescope and is used
as an example throughout this paper.
3. Analysis
A median filter has certain features that should be understood before using it as a data reduction
technique. At least a linear fit of the bandpass around the spectral line signal must be removed before
applying a median filter or it will remove a significant amount of the signal. A median filter will always
subtract some flux from a signal, but the exact amount is well understood if the baseline is flat and the
noise is Gaussian. The wider the filter window, the less flux it will remove from a signal. A median
filter will not remove any structure (e.g. standing waves) in the bandpass less than half the width of the
filter window, which puts an upper limit on the width of the window when significant bandpass
structure is present.
Figure 1. Unprocessed spectrum, in receiver units per channel, of an observation centered on the H164α radio
recombination line of the HII region S257. This data is one polarization of a 5 minute integration using the Arecibo Radio
Telescope with the Mock Spectrometer backend. Each channel is 21 kHz. To process this data we will remove a linear fit
and then a median filter.
3.1. Removing a Linear Fit
At least a linear fit of the bandpass around the signal must be removed before applying a
median filter because any slope in the baseline will significantly harm the quality of the median filter.
For a bright signal with a channel width, w, a median filter will miss mw of its flux per channel, where
m is the slope of the baseline. This will subtract (mw)w = mw2 from the signal's integrated flux in the
reduced spectrum. When the H164α line in Figure 1 is bandpass corrected with a median filter without first
removing a linear fit, its integrated flux is 1/4 of the value given by the traditional position switching (ONOFF) data reduction method as shown in Figure 2. The integrated flux of H164α is 16000 Receiver Units x
Channels using the position switched method and 4000 Receiver Units x Channels using the median filter
method when a linear fit is not removed. The median filter window is 36 channels, 4 times wider than the signal.
Figure 2. Left Panel. Spectrum of H164α using a median filter without first removing a linear fit from the bandpass. Right
Panel. H164α using the position switching data reduction method, (ON-OFF)/OFF. The peak flux of H164α is cut in half
and the integrated flux is reduced by 3/4 when a linear fit is not removed.
If a linear fit of the bandpass is removed, then the flux of the signal is recovered. Figure 3
shows the spectrum of H164α when a median filter is applied after a linear fit is removed. The width of the
filter window used is 36 channels and the width of the signal is 9 channels. The integrated flux of
H164α in Figure 3 is 14500 Receiver Units x Channels, within 10% of the integrated flux from the position
switched method.
Figure 3. Spectrum of H164α using a median filter after a linear fit is removed from the bandpass. The integrated flux of the
source is 14500 Receiver Units x Channels compared to 16000 from the position switched spectrum in Figure 2.
3.2. The Effect of Gaussian Noise on a Median Filter
Once a first order fit around the signal is removed, a median filter can be applied. In the idl
programming language, the simplest way to do this is to subtract the median array,
Median(Data,Window), from the Data array, where Window is the number of channels to take the
median of around each channel. The median window has to be more than twice the width of the signal
to prevent subtracting the signal from itself. A median filter will always subtract flux from a signal if it
is surrounded by Gaussian noise, even if the baseline of the noise is perfectly flat. However, the amount
of flux subtracted is based off of the distribution of Gaussian noise and so is well known and the
subtracted flux can be recovered statistically. The distribution of Gaussian noise is the integral,
P(𝑁) =
1
𝑁
∫ 𝑒 −𝑥
√2𝜋 −∞
2 /2
𝑑𝑥,
where P is the fractional percent of noise values, x, above the noise value, N (in standard deviations,
σ). There is not a normal solution for the indefinite integral. A numerical solution of the definite
integral is inverted and plotted in Figure 4 to show how the noise values are sorted from high to low
before the median is taken.
Figure 4. The fractional distribution of Gaussian noise sorted from high to low noise values. Any median filter will include
this distribution of values in its window due to Gaussian noise adjacent to the signal. For instance, the median of a window
3 times wider than a bright signal is the noise value above 0.25.
The larger a median window is, the less flux it will subtract from the signal. For a signal greater
than 3σ that is a step function of width, w, and is surrounded by a flat baseline of Gaussian noise, the
median of a window, c times the width of the signal, is the value of the Gaussian noise distribution at
(c-2)/(2(c-1)). In the example of the H164α line from Figure 3, the median filter window is 4 times wider
than the signal, so the median across the signal is the value above 0.33 in Figure 4, or ~ 0.4σ per channel. This
flux is subtracted from the signal when the median filter is removed from the bandpass. This subtracts
0.4 x 400 Receiver Units x 9 channels = 1440 Receiver Units x Channels of integrated flux from the
H164α line, where the spectrum’s RMS = 400 Receiver Units. Assigning this subtracted flux back to the
H164α line raises its integrated flux to 14500 + 1440 = 15990 Receiver Units x Channels, within less
than 1% of the integrated flux measured from the position switched spectrum. A small table of chosen
noise values subtracted by a median filter in this way is given in Table 1.
Flux Subtracted from
Signal
2σ
1σ
0.5σ
0.25σ
0.1σ
Filter Window Width
/ Signal Width
2.1
2.5
3.5
5
12.5
Table 1. Noise values subtracted as flux per channel from the signal due to a median filter with a window of width c times
wider than the signal. This flux is subtracted from the signal and from the spectrum w(c/2 – 1) on either side of the signal.
A median filter subtracts the same flux from every channel across the signal, as well as the
spectrum within w(c/2 – 1) on either side of the signal. For small windows, the integrated flux
subtracted from the signal can be recovered statistically by multiplying the noise subtracted by the
width of the signal. However, for windows much wider than the signal, the baseline around the signal
will be lowered by the same value and so the relative signal above the noise will not be missing any
integrated flux as illustrated in Figure 5. For instance, a median filter with a window 10 times the width
of the signal will subtract 0.125σ per channel from both the signal, and the baseline 4 signal widths on
either side of the signal.
The major assumption being made here is that the baseline of the noise is flat. If that is not the
case, then we must take into account the effect of structure in the baseline.
Figure 5. A diagram of two different. theoretical median fits around an idealized, step-function signal in a flat baseline of
Gaussian noise using a filter window c times wider than the signal width. For a small window, c = 2, a median filter will
subtract flux from the signal that has to be recovered statistically. For a larger window, c = 4, a median filter will subtract
the same flux from the baseline around the signal, making it possible to recover the signal flux without statistical arguments.
3.3. Non-Linear Features in the Bandpass
There are many causes of non-linear structure in a bandpass: sensitivity fall-off at the edges,
receiver (and backend) response per frequency, standing waves from telescopes with partially blocked
apertures. Typically, the narrowest of these structures, and thus the least responsive to a median filter,
are the standing waves. A partially blocked telescope aperture can reflect some incoming radiation
through the telescope support structure and this radiation will reach the receiver out of phase with the
original wavefront. The Fourier transform of the time delay in this phase shift results in a standing
wave in the bandpass. The structure of these waves depends on the telescope design, the time of day,
and the zenith and azimuth angle of the observation. These structures can be removed with a median
filter, which is a nice advantage as they are complicated superpositions of wave functions that cannot
be fit with a polynomial, but only if the filter window is small compared to the width of the structure. A
median filter is blind to any structure that is smaller than half the width of the filter window. This puts
an upper limit on the width of the window that can be used for the median filter.
A simple standing wave with some amplitude A and wavelength, λ, will create a structure,
Asin(2πn/λ), in the bandpass across channels, n. A median filter window of width, W, is blind to this
standing wave if W > λ. In fact, a median filter is blind to any part of the wave that is narrower than
half the width of the window, as shown in Figure 6.
Figure 6. Any median filter of width, W, will not remove any structure with width narrower than W/2. The filled in region of
the curve above will not be fit by a median filter of width W. This is true for any bandpass feature narrower than half the
width of the filter window.
A median filter will be blind to the wave inside where
𝐴 𝑠𝑖𝑛 (
2𝜋
2𝜋
𝑊
𝑛) = 𝐴 𝑠𝑖𝑛 [ (𝑛 + )] .
𝜆
𝜆
2
Solving for n gives
𝜋
𝑠𝑖𝑛 ( 𝑊)
𝜆
𝜆
𝑛 =
𝑎𝑟𝑐𝑡𝑎𝑛 *
+,
𝜋
2𝜋
1 − 𝑐𝑜𝑠 ( 𝑊)
𝜆
meaning that the amplitude of the wave function will be reduced by
𝜋
𝑠𝑖𝑛 ( 𝑊)
𝜆
𝑠𝑖𝑛 (𝑎𝑟𝑐𝑡𝑎𝑛 *
+) ,
𝜋
1 − 𝑐𝑜𝑠 ( 𝑊)
𝜆
and the fraction of the wave successfully fit by the median filter is 1 - W/λ. For instance, if the window
of the median filter is W = λ/2, applying the technique will reduce the amplitude of the standing wave
by 70% and remove 1/2 of the wave. For Arecibo, the brightest standing waves have a frequency of 1
MHz, which will not be removed a median filter with a window wider than 250 km/s in L-band.
Therefore, the median filter technique will fail when observing galaxies in HI (50 – 500 km/s wide),
but it is quite effective for observing emission line sources coming from discrete regions in the Milky
Way, which are of the order 20 – 30 km/s wide.
Median filter windows that are too wide can fail to remove standing waves in the bandpass, so
the effectiveness of the technique can be improved if standing waves can be removed all together. This
is theoretically possible by clipping the features, corresponding with the frequency of the waves, in a
Fourier transform of the bandpass (Briggs et al., 1997). A Fourier transform of the S257 observation is
shown in Figure 7. The bandpass has 8192 channels spanning 172 MHz. The bright, 1 MHz standing
wave at Arecibo should show up around u = 172, but this area is drowned out by ringing between the
Milky Way and a spike in the center of the Mock Spectrometer bandpass. Clipping these bright sources
effectively in the frequency domain before applying the Fourier transform may help in locating the
standing waves in the time domain. It may also help to remove the steep sides of the bandpass in order
to mitigate the sinc function centered on u = 0. These tactics should be looked into going forward as
removing standing waves in the bandpass will relax the upper limits on the width of the median filter
window, thus improving its effectiveness.
Figure 7. Fourier transform of the bandpass from the S257 observation covering 8192 channels and 172 MHz bandwidth.
4. Conclusion
The median filtering technique is superior to position switching in telescope efficiency and for
extended source observations. It is inferior for blind redshift observations and when the signal is wide
compared to standing waves in the bandpass. The technique requires knowing, apriori, the observed
frequency of the spectral line and that its width is small compared to standing structure in the bandpass.
A median filter will always remove flux from a signal, but this flux is a function of the Gaussian noise
distribution and is well understood. The integrated flux of a signal can be recovered using the median
filter technique, and one example, using real data, is given here. The technique is best suited for narrow
spectral line sources, but it can be applied to wider sources if standing waves in the bandpass can be
removed effectively.
References:
Briggs, F. H.; Sorar, E.; Kraan-Korteweg, R. C.; van Driel, W., 1997, PASA, 14, 37.