These details are our interpretation of the telecon on 20th May 2016:
The "F-engine wide" will always be running and we can subscribe to its output. Total bandwidth (from L band) is 856
MHz (
we need 875). Boresight delays (coarse and fine) are already corrected for. The number of channels depends on the observing mode:
probably 1024 channels for pulsars, details not decided
4k channels for imaging, 8-tap filter (later maybe 16)
32k channel mode, 4-tap filter? (Time resolution with that is 37 microsec, still sufficient. (Beware of wider PPF response in time.)
The wide-band correlator is also running all the time and we can receive its data as well. The frequency resolution is the same as the "Feng wide" output, time resolution is 0.5 sec.
Phase solution will be computed whenever a calibrator is available. Time scale is unknown, probably 2-10 sec or so. The phase solutions will also be available and can be used by us. Are they phase solutions per band or delays?
The (Feng?) voltage buffer keeps 2.5 sec of data. Upgrade to 4 sec or so may be possible, but if 10 or more are needed, it is better to buffer in our cluster.
VLBI output will not be available within the next two years, probably only later than that.
The standard beamformer (SKARAB boards) will produce 4 beams or maybe more.
SETI will use similar approach as we are thinking of: Tap Feng output and beamform from that.
We will need 64 links of 40Gbps each. Some data can be sent back to the switch (e.g. beamformed products).
Our beamformer (unsorted notes):
Is expanding the hardware of the standard Beng beamformer an option?
Is pretty straight-forward: Take Feng output, correct for atmospheric phase terms, then shift to residual offsets with phase factors. For >= 4k channels this should be fine, at least in S band. For L band and UHF we have to check this critically.
Decimation in the beamformers, then transpose by sending back to switch (from band per node to beam per node).
Need ca. 4k channels, time resolution 50 microsec. With 8 bit per sample and only Stokes I this corresponds to 80
MB/sec (640 Mb/sec). For 400 beams that is 32
GB/sec. When spread over 64 nodes or links, this is 0.5
GB/sec or 4 Gb/sec. This should be compared to the ~ 25 Gbps x 64 for the Feng data. This should still fit in the big switch so that we can use it for additional transpose operations (from beamformer to pulsar search).
4k channels have a time resolution of 4.7 microsec. We need 50, so the decimation factor is about 11. Input data are complex with two polarisations, output is real with only one Stokes I (factor of 4). In total the decimation factor is 43. We have 400 beams, which is a factor of 6.25 more than antennas. 6.25/43=0.145, the estimated ratio of data rates is 4/25=0.16, so this is consistent.
Second half of band is more difficult: Need own Feng and maybe correlator and determination of phase solutions (dispersive and non-dispersive delay?). Expand their hardware or do own Feng in GPUs? Second half should go into same switch.
We may still want advanced beamformer options that require own correlations with full time resolution!
Is output from wide-band channeliser always available?
What is the channel width? Documents seem to indicate 32k or 64k channels for wide and 4k for narrow, but is this correct?
Are the internals of the "Feng wide" documented? Are delays really corrected for the primary beam centre? For proper beamforming we have to understand our input data in detail.
Are the delays that are applied by Feng available? Is the geometric model known? We have to apply additional delays, and these should be consistent.
Is the "X-eng wide" correlator always running and can we use its output? We can also correlate ourselves, and that may even be preferable.
Sensitivity: We need some input to judge how well we can calibrate.
Calibration in general: Is any procedure planned for the B-engine beamformer (e.g. for VLBI)? Our beamformer would have to do exactly the same.
Will the B-engine beamformer deliver output for VLBI?
Can we implement something like the "Feng wide" in our "D engines" for the second half of the S band? This should be consistent with the existing "Feng wide".
Resolution in time and frequency and bits (would 16 be sufficient?)
Do we want to combine all antennas or only short(ish) baselines?
Do we want to optimise the beam shape by weighting? (probably yes, but in which way?)
Do we want additional products (beyond the 400 beams), e.g. full-primary-beam data with reduced time resolution?
I assume that we will also produce incoherent sums, they are also needed for the without-autocorrelation-beams approach.
How much voltage buffer (per antenna) do we need?
Do we want to (or have to) feed any of our intermediate or output data back to the switch?
N_a = 64 antennas
D = 13.5 m
longest baseline L = 8 km, most stations within 1 km
frequency bands: 0.58-1.015, 1-1.75, 8-14.5
GHz. S-band: 1.7-3.5
GHz N_b = 400 beams
-
channel width: wide 4k channels → 214 kHz (??), narrow: 32k channels
corresponds to 4.7 microsec (for wide)
downsampling factor for pulsar analysis: F will be of the order 10 (????)
primary beam width: full size approximately theta = lambda/D
delays: L/c, residual delays max. theta/2*L/c
for max. baseline: 27 microsec, for 1 km: 3.3 microsec
for max. baseline at theta/2 at lowest frequency: 0.51 microsec (is sufficiently smaller than the 4.7 microsec that are the reciprocal bw for for 4k channels: sufficient)
sensitivity ???? We need SEFDs for the planning.
If maximum delays are much below the reciprocal channel width, delays can be
applied as phase factors per channel. For correlations they have to be
calculated for the phase centre and applied to each antenna signal at full
data rate.
If delays are larger than this limit, the main part is applied as
integer-sample shifts before the channelisation. After channelisation, the
reminder is applied as phase shift.
Fringe-stopping (compensation for non-zero frequency, e.g. due to downmixing)
and non-integer shifts can be applied together. Because all stations share the
same frequency setup, this is easier than in the general case.
Apparently the system does already compensate for delays of the phase centre
so that we only have to deal with residuals within the field of view (primary
beam). Even for the lowest frequencies this seems to be sufficient at the
half-power point of the primary beam.
For the second half of the bandwidth that does not fit in the current system,
we will have to do the full delay compensation (including the shift to the
phase centre) ourselves. For this we should use the exact same delays that are
used within the current system.
1. Addition of voltages
Requires full delay compensation for each antenna and each beam. This is
conceptually very simple and requires of the order N_a*N_b operations per
sample.
2. Use of cross-correlations
For each sample this requires N_a^2 operations, which have to be done
anyway for the correlations (but we may not get them out of the correlator
at high time resolution!). Delay compensation and beamforming can then be
done at the downsampled rate and requires of the order N_a^2*N_b/F
operations per original sample.
3. Cross-correlations with FFT
If we had an array on a more or less dense regular grid (as LOFAR HBA
antennas), we could use an FFT to cross-correlate the voltage field and
form beams/images directly. For a dense array this would require of the
order N_a*log(N_a) operations per sample and produce of the order N_a
beams. In our sparse-array case we would have to grid the voltage data at
full data rate and work with something like N=(L/D)**2 pixels. For 1-km
baselines, this would be N=5500 for 8-km baselines N=350000. The former may
not sound too bad (compared to N_a*N_b), but the whole procedure is way
more complicated and probably much less efficient in terms of operations
per second on a GPU. But it would fill the entire field of view with beams!
If we agree that (3) is too far-fetched, I would opt for (1) for the 400 beams
and investigate whether we can in addition also run (2) with less time
resolution but with more beams. Having the visibilities at full time
resolution available also makes visibility-based search strategies possible.
We have to correct for delays produced by the atmosphere (troposhpere and
ionosphere) and for signal delays in the system. How regularly we have to
measure correction delays (or gains) is unclear (VLA at
https://science.nrao.edu/facilities/vla/docs/manuals/obsguide/modes/vlbi says
every few min to 30 min), but we must accomodate some way of doing this. To
determine delays and relative phases, we need cross-correlations. These can be
averaged in time and frequency, but we must keep sufficient resolution to
avoid time-averaging and bandwidth smearing.
In frequency we can probably go to 1k channels, in time the limit depends on
the observing frequency. At 15 GHz we may not want to go beyond 1 second, at
0.58 GHz we can use 30 sec.
Can we get these data from the correlator or do we have to form our own
visibilities? Another important question is if we will generally find suitable
calibrator sources within the primary beam.
The VLA (slightly more collecting power) recommends (
https://science.nrao.edu/facilities/vla/docs/manuals/obsguide/modes/vlbi )
calibrator sources > 100 mJy in our frequency range.
NVSS has 62000 sources > 100 mJy (over 82 % of sky, 34000 square deg), that is 1.8
sources per square degree. Field of view in square degree: 3.8 at lowest
frequency, 0.32 at 2 GHz. This means it is unclear if there will always be a
good calibrator in the field. We can use the entire field instead of just the
brightes source, but this may still not be enough. We have to make more
realistic estimates of the sensitivity.
At 8-14.5 GHz only fields around dedicated calibrators can be used, but
calibrator scans every now and then may do the trick. Generally the
observations that we want to piggyback on will also need to be calibrated, so
they may either use good fields or include calibrator scans. The calibration
plans for the surveys should be checked for this!
incl. VLBI capabilities, have they thought about calibration and the other
issues?