Main

The production of intense coherent radiation by microbunched relativistic electron (e–) beams underlies today’s X-ray free-electron lasers (XFELs)1, which are revolutionizing our understanding of biomolecules, materials and living cells2. For an e– bunch to radiate coherently, some of its electrons must possess a space–time-correlated structure. Such microbunching emerges from noise when an e– beam interacts with a magnetic undulator, inducing the beam to radiate partially coherent self-amplified spontaneous emission3. An e– beam can also pre-bunch from interacting with electromagnetic or material structures before the undulator. Such seed microbunching can profoundly influence the FEL gain dynamics and the intensity, coherence and stability of FEL radiation4. Hence, non-intercepting diagnostics that track the evolution of microbunching from the cathode through the undulator are critical to develop and operate accelerator-based coherent light sources. Indeed, single-wavelength measurements of optical transition radiation that e– beams generate on transiting thin metal films5, particularly its strong coherent component (coherent optical transition radiation (COTR))6, were pivotal to understand how microbunching evolved in early FELs7,8,9 and thus to the emergence of today’s XFELs.

Recent observations of gain in FELs driven by e– beams from centimetre-scale laser-driven10,11 and beam-driven12,13 wakefield accelerators (WFAs) now portend a new generation of tabletop XFELs14,15. However, WFA-driven bunches differ markedly from their radio-frequency-accelerated counterparts: they are typically shorter in duration (down to a few femtoseconds), correspondingly higher in peak current (tens of kiloamperes) and emerge from the accelerator with a smaller transverse size (~1 μm), properties that can promote the FEL gain. On the other hand, they typically possess larger relative energy spread (ΔEe/Ee > 10−2), larger divergence (few milliradians) and higher shot-to-shot beam-pointing fluctuations (>0.1 mrad r.m.s.) (ref. 16), factors that inhibit the FEL gain. In addition, the complex laser–plasma or beam–plasma interaction that creates plasma accelerator structures can inject plasma electrons into them that pre-bunch on micrometre17,18 to nanometre19 scales, often with intertwined longitudinal and transverse coherence. These unique properties of WFA e– beams underlie proposals for new types of coherent X-ray source driven by chirped20,21, attosecond-compressed22 and nanobunched19 WFA beams. These same properties demand new ways of diagnosing WFA e–-beam microbunching23, which will be as essential for developing WFA-driven coherent light sources as previous diagnostics7,8,9 were for developing today’s radio-frequency-accelerator-driven FELs.

Here we report a minimally intercepting, single-shot diagnostic that unfolds the three-dimensional (3D) structure of the microbunched sub-population of laser-driven wakefield accelerator (LWFA) e– bunches from the wide-bandwidth COTR that they uniquely generate. Previous work determined one-dimensional (longitudinal) profiles of such e– bunches24,25,26 from transversely averaged, multioctave COTR spectra27 analysed with an iterative reconstruction algorithm that assumed a smooth transverse structure28. Similar analysis supplements some measurements presented here. However, the critical new element of the present work is the imaging of the COTR at near-infrared to ultraviolet wavelengths—that is, wavelengths close to the dimensions of the e– bunch itself—from the COTR foil to nine high-resolution array detectors, each filtered to detect a specific wavelength λ. In this λ range, the COTR changes from nearly fully coherent at longer λ to nearly fully incoherent at shorter λ and captures the most essential information about the beam’s transverse microbunched structure. Combining λ-resolved COTR images with longitudinal profiles obtained from, for example, multioctave COTR spectra, electron spectra or particle-in-cell (PIC) simulations then elucidates the beam’s 3D microbunched structure. For simply structured beams, a full 3D reconstruction is possible with high confidence. For more complex beams, the contributing 3D structure can still be estimated with help from PIC simulations. In either case, a combined transverse–longitudinal analysis is essential because transverse (⊥) and longitudinal (∥) form factors29 contribute either as a separable product F⊥(k⊥)F∥(kz) or a correlated function F(k) to the total beam microbunching, where \({{{\bf{k}}}}={{{{\bf{k}}}}}_{\perp }+{k}_{\parallel }\hat{z}\) is a wavevector of magnitude k = ∥k∥ = 2π/λ. Thus, oversimplified assumptions about one lead to errors in the other.

Our main result is the observation of a correlation between the size, shape, intensity and λ dependence of a multispectral COTR image set on one hand and on the other, the method of injecting plasma electrons into the LWFA. Injection by propagating the laser through a preformed plasma-density down-ramp30 in a laser-ionized helium gas jet led to annular-shaped, near-λ-independent COTR images, and to simply shaped, micrometre-scale reconstructed e– bunches. Injection by the delayed ionization of the inner-shell electrons of nitrogen dopants in the helium plasma (self-truncated ionization injection (STII)31) led to larger, λ-dependent COTR images with a more complex, but stable, structure. Aided by PIC simulations, we identified these COTR features with density modulations near the beginning and end of the ~5-μm-long e– bunch. Finally, injection via uncontrolled laser–plasma dynamics in a pure, featureless He plasma (self-injection) led to the largest, most complex and least stable COTR images. Such empirical correlations lend themselves to machine learning and feedback control of LWFAs32, even when the coherently radiating e–-beam profile cannot be uniquely reconstructed. Although the current results were obtained from a COTR foil just outside the accelerator exit, the same method can track microbunching at downstream locations. Similar to other coherent imaging techniques such as ptychography33 and coherent diffraction imaging34, multispectral COTR imaging shares the problem of iteratively reconstructing a micro-/nanostructured object of interest from the intensity measurements of coherent interference patterns. Its distinguishing features are the relativistic speed of the object of interest, which generates its own coherent radiation in a broadband ultrashort burst, and the λ multiplexing of the resulting interference pattern to form the core dataset for reconstruction.

Results

Figure 1 (top) depicts the experimental setup. Laser pulses of 67 TW (~2 J, 30 fs) peak power and λ = 800 nm from the Dresden laser acceleration source (DRACO) laser at Helmholtz-Zentrum Dresden-Rossendorf (HZDR) (Methods) were focused into a gas jet, where they drove nonlinear plasma wakes that accelerated electrons injected from the surrounding plasma to energies as high as 300 MeV (ref. 35). A tilted 65 μm aluminium foil 2 mm after the jet blocked the transmitted laser pulses while only minimally perturbing the e– beams36,37. The e– beams then passed through a 25 μm Kapton foil ~1 mm further downstream, inducing radial currents in its 0.3 μm aluminized back surface that emitted forward COTR as the beams exited the foil. A polished 100 μm thick silicon wafer reflected the COTR towards a microscope objective, which imaged it from the COTR foil to nine bandpass-filtered charge-coupled-device cameras (a camera filtered at 193 nm, which detected no signal, is not shown in Fig. 1). The e– beam passed through the wafer to an electron spectrometer (Fig. 1, top right). No COTR or electrons were observed with the gas jet turned off. Methods provides further details.

Fig. 1: Multispectral COTR imaging experiments.
figure 1

Experimental setup (top) and representative electron spectra (top right) for three injection schemes. The total charge above 100 MeV was 54 pC (down-ramp), 280 pC (STII) and 865 pC (self-injection). The laser pulses are focused to a 20 μm spot in the 3-mm-long helium gas jet, creating a plasma of electron density ne = 2.3 × 1018 cm−3 (down-ramp and STII) or 4 × 1018 cm−3 (self-injection) and a nonlinear wake. For down-ramp injection, a knife edge perturbed the gas flow, creating a shock near the jet entrance; for STII, the gas was doped with 1% nitrogen. Representative single-shot, eight-panel COTR image sets (bottom) from three types of injection (rows), bandpass filtered (10 nm FWHM) at eight wavelengths (columns), acquired on the same shots as the electron spectra in the top right. The yellow dashed lines on 400 nm images exemplify the method for defining the area of each pattern. Each image uses a grey scale that optimizes the visibility of its details.

Full size image

COTR multispectral imaging phenomenology

Experimental COTR images

The three-row bottom panel in Fig. 1 shows the representative eight-panel multispectral COTR image sets acquired in one shot for each type of injection. Down-ramp injection yielded the simplest, smallest and most consistent COTR patterns: single annuli with deep central minima and similar size for all the measured wavelengths. STII bunches yielded larger, more complex images. Self-injected bunches yielded the largest, most complex images, varying from λ to λ and shot to shot without a discernible pattern. Figure 2a plots the effective radii \({r}_{{{{\rm{eff}}}}}=\sqrt{A/\uppi }\) of COTR images as a function of λ for each type of injection. Here A is an area, illustrated by the dashed yellow lines in the 400 nm column in Fig. 1, containing all of the pixels with signals over 1/4 of the maximum along with any enclosed dark minima. Down-ramp injection (blue data points) yielded not only the smallest images (reff = 9.0 ± 2.0 μm) but also the smallest variation amongst the wavelengths and from shot to shot (error bars), whereas self-injection (green) led to the corresponding largest values.

Fig. 2: Empirical trends in multispectral COTR data.
figure 2

a,b, Effective radii \(\sqrt{A/\pi }\) of the COTR radiation patterns (a) and ratio (Nc/Ne) of the number of coherently radiating electrons Nc to the total number of electrons over 100 MeV Ne assuming form factor F(k) = 1 (b), both plotted versus λ for LWFA electrons injected by three methods. Average Ne of 0.3, 1.8 and 5.4 × 109 for down-ramp, STII and self-injection, respectively. Data are presented as mean values ± standard deviation over 100 down-ramp, 65 STII and 30 self-injected shots.

Full size image

The ratio W(λ)/W1(λ) of the total energy W(λ) deposited into each image, determined from the integrated camera counts corrected for detector quantum efficiency and optical losses, to the energy W1(λ) that one electron deposits, determined from the optical transition radiation theory5,6, is a sensitive, quantitative indicator of a microbunched (that is, coherently radiating) e–-beam substructure. This is because the coherent part of \(W(\lambda )/{W}_{1}(\lambda )={N}_{{\rm{i}}}+{N}_{{\rm{c}}}^{2}\parallel F({{{\bf{k}}}}){\parallel }^{2}\) (ref. 9) is proportional to \({N}_{{\rm{c}}}^{2}\parallel F({{{\bf{k}}}}){\parallel }^{2}\), where Nc is the number of microbunched electrons contributing to COTR with 0 < ∥F(k)∥2 ≤ 1 at each λ, whereas the incoherent optical transition radiation energy scales only linearly with the number Ni of non-microbunched electrons. Here, since Ne = Nc + Ni ≈ 109 total electrons as determined from the integrated electron spectrometer38 signal, the COTR dominates even for Nc/Ne as small as ~10−4. By crudely assuming ∥F(k)∥ = 1, we set a lower limit on Nc/Ne. Figure 2b plots the resulting Nc/Ne value as a function of λ, and shows that Nc/Ne ≪ 1, although it is higher for down-ramp (blue) and self-injection (green) than for STII (orange). The COTR image analyses shown below will reveal that, in fact, ∥F(k)∥ ≪ 1, implying that Nc/Ne approaches 1.

Synthetic COTR images

As a prelude to reconstructing unknown e– bunches, the leftmost column in Fig. 3 shows synthetic (that is, simulated) e– bunches with four different internal structures: pancake shaped, with Gaussian lengths σz = 0.3 and 0.4 μm (rows 1 and 2, respectively); two beamlets (row 3); and a sine wave (row 4). The remaining columns show the forward-calculated images of COTR (Methods) that each bunch generated at each of the four wavelengths. The images strikingly vary in shape, intensity and λ dependence, illustrating the technique’s sensitivity to sub-micrometre beam morphology.

Fig. 3: Multispectral COTR images of synthetic electron distributions.
figure 3

The left column shows the four electron distributions: width σ⊥ = 3 μm, length σz = 300 nm (row 1) and 400 nm (row 2) FWHM longitudinally Gaussian distributions; two 300 nm FWHM longitudinal Gaussians separated by 400 nm longitudinally and 6 μm transversely (row 3); a distribution with sinusoidal displacements along y and 800 nm period along z (row 4). Two-dimensional projections of each distribution are shown on the x–y, x–z and y–z planes. The total charge ratio is 1.0:1.0:1.0:1.1; electron energy, 285 MeV throughout. The remaining rows show the COTR images generated from the corresponding electron distribution at 400, 500, 600 and 800 nm with a collection angle of 0.28 mrad. The colour scale is consistent in each column, and normalized to the corresponding image in the first row.

Full size image

The bunch in row 1 is shorter than all the observation wavelengths, and yields an annular COTR profile at all λ values. The central minimum results from the radial polarization of the COTR: opposed polarizations converge and destructively interfere on the z axis in the image plane. The longer similarly shaped bunch (row 2) generates markedly less COTR near 400 nm, because emissions from its front and rear now destructively interfere. As an analogy, optical thin-film interference patterns display similar sensitivity to film-thickness changes of only λ/4, which interchange destructive and constructive interferences from the front and back surfaces of the film, respectively. The interference of COTR from the two e– beamlets in row 3 creates the analogue of a two-slit interference pattern. With 400 nm filtering, oppositely directed electric fields from the two beamlets overlap and interfere destructively on the z axis to produce an axial minimum. As λ lengthens, this feature evolves towards constructive interference and an axial maximum at 800 nm. The sinusoidal electron distribution of the bunch in row 4 yields an axial COTR minimum (maximum) at λ = 400 (800) nm as in the previous example, but with very different off-axis features. The varied structure of the COTR images suggests the possibility of uniquely reconstructing the internal coherently radiating microstructure of LPA e– bunches from them. Of course, simple geometric transformations (for example, reflections about the z = 0 plane) of asymmetric electron distributions such as rows 3 and 4 in Fig. 3 can yield identical COTR image sets, making them inherently indistinguishable by COTR analysis. Apart from such unavoidable redundancy, the following analysis of experimental COTR images explores to what extent, with what assumptions and with what uncertainty this inverse problem can be uniquely solved.

Substructure of wakefield-accelerated e– bunches

Down-ramp-injected e– bunches

Further progress in unfolding a 3D coherently radiating e–-bunch substructure requires a quantitative analysis of the multispectral COTR image sets. Down-ramp-injected e– bunches are a good starting point because of the high coherence, simple structure and near-λ independence of their COTR images, hinting at a simple underlying bunched structure. However, these image sets alone do not adequately constrain the wide range of possible longitudinal structures that iterative reconstructions must sample in search of a 3D solution. Fortunately, the longitudinal profile of a down-ramp-injected bunch can be estimated from its energy spectrum if, as here, three conditions are met: (1) the injected charge is small enough (here 30 pC) to avoid beam loading the wake (which requires ~300 pC (ref. 35)), so that the bunch chirps linearly as it accelerates39,40; (2) the down-ramp—typically a shock wave30—is short and steep enough to deterministically initiate both acceleration and chirping at a precise location in the gas jet; (3) the drive pulse focus is well matched to the plasma-guiding forces such that the wake evolves quasi-statically during acceleration. If these conditions are met, then the bunch’s energy spectrum maps linearly onto its longitudinal profile (see the ‘Calculations and simulations of wakefield acceleration’ section). Figure 4a (blue curve) shows a longitudinal profile thus estimated from the spectrum in Fig. 1 (top right). Its ~1 μm full-width at half-maximum (FWHM) and 10% energy spread are consistent with our own (Supplementary Fig. 1c) and published41 simulations of similar accelerators. Moreover, its two internal peaks, which correspond to two peaks in the energy spectrum, are shorter than the detected wavelengths, leading (Fig. 3, row 1) to coherent, λ-independent annular COTR images. Other methods that do not rely on conditions (1)–(3) above (for example, multioctave COTR spectroscopy27) can also provide the initial longitudinal profile estimate.

Fig. 4: 3D reconstruction of down-ramp-injected e– bunch.
figure 4

a, Input longitudinal profile of the e– bunch derived from its electron spectrum (blue), and its average profile reconstructed from the COTR data (orange). b, Average of 47 reconstructions from one cluster of solutions for the 3D electron density of the e– bunch, with average two-dimensional projections plotted on the bounding planes. The beam is much wider than it is long, so the z axis is scaled by a factor of 10. c, The top row shows the measured COTR at eight wavelengths. The bottom row shows the forward-calculated COTR from the cluster-averaged structure in b. The colour scale is different at each λ, but the same for the measured and reconstructed COTR at each λ. d, Average NSS at each λ for 120 reconstructions that included 100% (blue), 90% (orange), 80% (green), 70% (red) and 60% (purple) of the measured total charge. e, The grey area shows the superposed NSS values for 120 individual reconstructions, using 100% of the measured charge. The inset shows a PCA plot, revealing three colour-coded clusters of reconstructions (axes: the first two principal components of the distribution of reconstructions). The coloured plots in the main panel, NSS for structures averaged over each correspondingly colour-coded cluster. Reconstruction in b belongs to the red cluster.

Full size image

We then used a differential evolution algorithm42 to iteratively reconstruct the 3D profiles. Briefly, at each step n, we forward calculated the multispectral COTR image sets from candidate electron distributions ρn(x, y, z), evaluated residuals, then updated ρn to ρn+1, guided by a cost function that favoured evolution towards the estimated longitudinal profile, the measured total charge qm and an improved match to the COTR data. The algorithm, however, made no assumptions about the symmetry or longitudinal–transverse separability of candidate ρn. Each reconstruction ‘j’ converged towards a solution ρj that minimized the cost function. We then repeated the reconstructions dozens of times with randomly initialized ρn=1. We evaluated the set {ρj} of converged solutions for each shot for determining the quality of fit to the COTR data and for uniqueness, that is, clustering around a single centroid solution, or a small group of geometrically related solutions. Methods provides further details, and Supplementary Fig. 2 shows the reconstruction algorithm.

Figure 4b shows an average of several dozen 3D reconstructions (green) based on the down-ramp-injected COTR data in Fig. 1. The two z-separated sub-bunches described above survive. However, in the course of optimizing its fit to the COTR data, the algorithm modified their transverse and longitudinal shapes from the initial estimate (Fig. 4a, orange curve). This distribution is neither symmetric nor separable. Figure 4c directly compares the measured (top) and reconstructed (bottom) COTR images, illustrating the high level of detail that the reconstruction captured across eight detected λ. Normalized sum-of-squares (NSS) errors (Methods), plotted versus λ in Fig. 4e, were similar in magnitude for individual reconstructions (grey) as for the average of each of the three near-equally populated clusters of reconstructions that principal-component analysis (PCA) identified, as displayed in the inset (Methods provides the details of PCA). The NSS equivalence of individual and cluster-averaged structures is a strong indicator of intracluster uniqueness, since all the cluster members must share common structural attributes to survive averaging. Moreover, the three clusters prove to be close geometric relatives of each other, rather than independent solutions of the inverse problem (Supplementary Fig. 4). Thus, apart from this unavoidable geometric ambiguity, all of the 120 reconstructions clustered tightly around three otherwise equivalent solutions. Supplementary Figs. 3 and 5 show examples of the analyses of other shots.

To re-evaluate F(k) and Nc/Ne, we ran reconstructions that included 90, 80, 70, 60 and 50 per cent of the total charge. NSS increased monotonically as this percentage decreased, as shown by the coloured curves in Fig. 4d. From this, we infer that nearly all of the measured charges contributed to the COTR to some extent, that is, Nc/Ne ≈ 1, much higher than numbers obtained by assuming F(k) = 1 (Fig. 2b). This, in turn, implies F(k) ≪ 1. Evidently, transverse coherence is small, since the transverse FWHM of the reconstructed down-ramp-injected bunches averages ~6 μm, much larger than the detected λ. This re-affirms the importance of analysing both transverse and longitudinal beam structures.

Supplementary Figs. 3–5 illustrate the analyses of other down-ramp-injected bunches, which yield similarly sized envelopes with widely varying internal structures. Envelope reproducibility is a hallmark of down-ramp injection from short, steep, stable shocks. But these results show that the sub-micrometre internal structure of such bunches is correspondingly stochastic. Evidently, fine-grained non-uniformities in nonlinear shocks imprint themselves on injected bunches in ways that are difficult to control, diagnose or simulate. Moreover, off-axis electrons, which down-ramps inject plentifully, evolve unpredictably during acceleration. This internal stochasticity could influence, for example, gain in LWFA-driven FELs.

Ionization-injected e– bunches

STII occurs over an extended region of the gas jet, and generates e– bunches of sufficient charge (~300 pC) to load the wake35. Thus, the method used above to estimate the longitudinal profile of down-ramp-injected e– bunches is not valid. Fortunately, other information is available. A recent multioctave COTR spectrometry study of STII bunches from DRACO yielded a bunch length of σz ≈ 3 μm (ref. 27). The PIC-simulated STII e– bunch (Fig. 5a), generated under our experimental conditions and propagated to the location of the COTR foil, corroborates this measurement. Moreover, it reveals two concentrations of optical-wavelength density displacements that are candidate COTR sources: (1) oscillations with λ ≈ 800 nm in the leading portion of the beam, from interaction of the e– bunch with the trailing edge of drive laser pulse; (2) shorter-wavelength oscillations in the trailing portion, from late off-axis ionization injection from N7+ (ref. 43). Figure 5b shows the extracted parameterized model of the first source that matches its simulated amplitude, location and frequency spectrum, but enables fine variations to fit the data (Methods). Figure 5c shows a representation of the second source that emerged from such a fit. Figure 5d shows them combined with the featureless intervening section, which generated a negligible COTR.

Fig. 5: 3D representation of ionization-injected e– bunch.
figure 5

a, PIC simulation of density distribution ρ(x, y, z) of the STII e– bunch after exiting the LWFA, projected onto the y–z plane. b,c, Parameterized representation of laser-induced modulation at the leading edge (b) and ionization-induced higher-frequency modulation at the tail of a (c). d, Combination of the data in b and c with the unmodulated middle section of the data in a. e, The first row shows the measured COTR images from a single shot at the eight corresponding central wavelengths. The bottom three rows show the best-fit COTR images generated by the optimized distributions in b–d. The colour scale in each column is normalized to the brightest image in that column.

Full size image

Figure 5e (top row) shows the COTR data from a typical STII beam. The second row shows the calculated COTR from the source in Fig. 5b. When iteratively optimized, it reproduced the salient features of longer-λ images, but failed to model those at shorter λ. Conversely, the source in Fig. 5c reproduced the main features of short-λ images, but contributed negligibly at λ > 630 nm. COTR from the optimized combined sources (Fig. 5e, last row), however, reproduced the main observed features at all λ. NSS was no larger than that for the down-ramp-injected bunches, and with repeated reconstructions of one shot or different shots, the oscillatory leading and trailing features consistently reappeared. Only the detailed internal structure of the latter varied widely among the reconstructions of even one shot and thus could not be uniquely determined.

Self-injected e– bunches

Attempted reconstructions of self-injected e– bunches led to widely divergent solutions with high NSS. Supplementary Figs. 7 and 8 show a representative example.

Discussion

The solution to the inverse problem of reconstructing ρ(x, y, z) from COTR intensity measurements is inherently uncertain because the optical-phase information that links ρ to the COTR field E⊥(x, y) is lost. Nevertheless, similarly handicapped image reconstruction methods are used with high confidence in, for example, medicine and structural engineering. Success relies on effectively compensating for the missing phase information, screening trial solutions for fit quality and uniqueness, and repeatability of the iterative procedure. Here independent measurements of electron charge and energy, knowledge that bunch charge is negative, and reliable guidance on longitudinal bunched structure from electron spectra or PIC simulations provided compensating information that constrained possible solutions, and is readily available to most LWFA researchers. A high-quality fit of a calculated COTR image set to a measured one is a necessary, but not sufficient, condition for trusting a trial solution. Here we quantified the fit quality by calculating the NSS values based on a pixel-to-pixel comparison. Visually satisfying fits such as those shown in Figs. 4 and 5 for the down-ramp-injected and STII bunches, respectively, generated NSS values comparable with those achieved with synthetic data such as those shown in Fig. 3.

A trustworthy ρ(x, y, z) must also be unique. Here, to quantify uniqueness, we repeatedly ran the reconstruction algorithm on each dataset with randomly varied initial parameters to evaluate solution clustering44. For successful reconstructions, solutions grouped into two or three geometrically related clusters, in each of which the cluster-averaged structure yielded NSS as small as—or nearly as small as—that of individual reconstructions, demonstrating uniqueness. The difficulty of reconstructing self-injected e– bunches reflected lack of guidance on their longitudinal structure, their wide energy spread and the complexity of the COTR patterns.

Several extensions of multispectral COTR imaging can probably increase its diagnostic capability in the future. First, there is enough COTR energy to distribute to almost three times as many filtered cameras as demonstrated here, enabling larger spectral range and density and correspondingly greater resolution. Extensions to shorter λ will be important for diagnosing nanobunching in LWFA-driven XFELs45,46 and nanoprebunching schemes19. Second, by conveying the e– bunch after the COTR imaging split-off mirror (Fig. 1, top) to a multioctave spectrometer, one can combine the latter’s wide spectral range with the fine lateral resolution of multispectral imaging. Third, e–-beam perturbations by laser-block and COTR foils, though carefully minimized here, can be eliminated by using apertured47,48 foils. For applications remote from the LWFA, a laser block is not needed.

In conclusion, we have demonstrated multispectral COTR imaging as a versatile, low-tech, high-resolution diagnostic of the internal coherently radiating structure of LWFA e– bunches that are quasi-monoenergetic and for which the longitudinal charge profile can be reliably estimated. Using this tool, we have identified optical and structural fingerprints of down-ramp- and ionization-injected e– beams, and reconstructed the internal fine structure with unprecedented detail. We anticipate that this approach will play a key role in diagnosing micro- and nanobunching in future tabletop LWFA-driven X-ray sources, as well as in optimizing their performance through machine learning strategies.

Methods

Laser wakefield electron acceleration

DRACO49 at HZDR drove plasma wakes with laser pulses of 30 fs duration, 800 nm centre wavelength and ~2 J energy. An f/20 off-axis parabolic mirror with 2 m focal length focused these pulses to a spot size of 20 μm (FWHM) onto the entrance plane of a 3-mm-long Mach-10 helium (He) gas jet. The leading edge of the laser pulse fully ionized the He, after which its intense peak drove an LWFA in the He plasma. For down-ramp and ionization injection experiments, the plasma in the jet’s central density plateau (length L ≈ 1.5 mm) had an average electron density in the range of 2.0 × 1018 < ne < 3.3 × 1018 cm−3. For self-injection experiments, it had an average density of ne ≈ 4 × 1018 cm−3. For down-ramp injection, a knife edge was inserted into the gas jet, creating a thin dense shock near the beginning of the plateau, pinpointed by a transverse wide-bandwidth shadowgraphy probe with a centre wavelength of 800 nm (ref. 50) (Supplementary Fig. 1b). Laser focus and shock positions were scanned to optimize the LWFA stability and performance. For STII, we removed the shock and doped the gas with 1% nitrogen. We then scanned the pulse shape with a programmable acousto-optic dispersive filter (Fastlite Dazzler) and adjusted the pulse energy to optimize the LWFA performance. For self-injection, we used a pure He jet with no shock and shifted the laser focus several millimetres before the jet. This mismatched the laser divergence and the plasma wake’s focusing force, inducing radial wake oscillations that triggered the injection of plasma electrons into the wake.

A magnetic electron spectrometer with its entrance plane at z = 30 cm downstream of the gas-jet exit determined the electron energy distribution for each shot49. A Konica Minolta OG 400 scintillating screen recorded the dispersed electron beam (Fig. 1 (top right) shows examples). We converted the luminescence intensity into charge per unit energy per pixel using methods described elsewhere38. The absolute charge calibration uncertainty was ~20%. The r.m.s. shot-to-shot charge fluctuations were only a few per cent. Uncertainties in the >200 MeV electron energy measurement primarily originated from pointing and divergence fluctuations of LWFA electrons entering the spectrometer and amounted to ~2% for electrons in the range of 200–350 MeV.

Calculations and simulations of wakefield acceleration

In the ‘Substructure of wakefield-accelerated e– bunches’ section, we estimated the longitudinal profile Ne(ξ) and length Δξ of down-ramp-injected e– bunches from their measured energy distribution ∂Ne/∂Ue and energy spread ΔUe, respectively. Here ξ = z − v0t is the longitudinal coordinate in the frame of a bubble moving with velocity v0 along the lab coordinate z. This estimate assumed a quasi-static bubble and an accelerating field Ez = (ene/2ϵ0)ξ that is linear in ξ but exceeds the field (ene/3ϵ0)ξ of a uniformly charged sphere because of the concentration of electrons at its rear39. The PIC simulations of down-ramp-injected LWFA for our conditions, such as the example shown in Supplementary Fig. 1c, validate these assumptions for our conditions. Linear Ez is expected when accelerated charge, which averaged ~50 pC for our down-ramp-injected LWFA, is well below the beam-loading limit of ~300 pC (ref. 35). The estimate also assumed that injection is localized in space and time. Transverse shadowgraphs50 such as the example shown in Supplementary Fig. 1b validated this assumption. Under these conditions, an electron injected at position ξi within the bubble and accelerated to ξf, positions that are at potentials \({\varPhi }_{x}=(e{n}_{{\rm{e}}}/4{\epsilon }_{0}){\xi }_{x}^{2}\) (where x = i, f) with respect to the bubble centre, gains energy Ue = e(Φi − Φf) in the bubble frame. Applying the chain rule dNe/dξ = (∂Ne/∂Ue)(∂Ue/∂ξ) and transforming to the lab frame, we find that ∂Ne/∂Ue maps onto the bunch’s longitudinal profile dNe/dz via the linear chirp ∂Ue/∂z, that is,

$$\frac{{\rm{d}}{N}_{{\rm{e}}}}{{\rm{d}}z}=\frac{{n}_{{\rm{e}}}{e}^{2}}{2{\epsilon }_{0}}\frac{\partial {N}_{{\rm{e}}}}{\partial {U}_{{\rm{e}}}}{L}_{{{{\rm{acc}}}}},$$
(1)

where Lacc is the acceleration length in the lab frame. This blue curve in Fig. 4a plots this function using Lacc ≈ L, ne = 2.3 × 1018 cm−3 and the measured ∂Ne/∂Ue (Fig. 1, top right). For the down-ramp-injected shot analysed in the main text, the chirp was ∂Ue/∂ξ ≈ 3 MeV μm–1 and bunch length Δξ = (2ϵ0/e2ne)(ΔUe/L) ≈ 850 nm FWHM for ~10% energy spread at 280 MeV, which is consistent with published41 and our own data (Supplementary Fig. 1c shows the PIC simulations for this injection method). Since our spectrometer has a resolution of \(\Delta {U}_{{\rm{e}}}^{\,({\rm{res}})}\) ≈ 10 MeV, this approach determines bunch length Δξ (bunch duration Δξ/c) with 300 nm (1 fs) resolution. The r.m.s. fluctuation of ~20% in neLacc was the principal uncertainty in the estimated dNe/dz and Δξ. In iterative reconstructions of the 3D profile Ne(x, y, ξ) of down-ramp-injected bunches, the estimated longitudinal profile served as a parameterized initial guess (rather than a fixed function) that was allowed to vary over a range consistent with this experimental uncertainty, yielding, for example, the orange curve in Fig. 4a.

The 3D PIC simulations of STII were performed using the code PIConGPU51 to evaluate the spatial structure of the bunch and the longitudinal phase space. For the STII simulation (Fig. 5a), we used a simulation box of 768 × 4,608 × 768 cells with a transverse resolution of 4.5 and a longitudinal resolution of 36 sampling points per laser wavelength. The laser is modelled using a Gauss–Laguerre reconstructed laser profile measured during the experiment. To avoid numerical Cherenkov radiation, the Lehe field solver52 was used together with the Boris pusher53 and the Esirkepov current deposition scheme54. The exact code version used and all the setup files are available elsewhere55.

The COTR images directly calculated from the PIC-simulated density distribution (Fig. 5a) are shown in Supplementary Fig. 6. By extracting the oscillatory leading and trailing edges and smooth middle section of this distribution, we verified that nearly all the calculated COTR data originated from the leading and trailing edges. To optimize the match of the calculated and measured COTR images, we first constructed the following parameterized representation of the leading portion of the Fig. 5a distribution:

$${\rho }_{{\rm{l}}}(x,y,z)=A{{\rm{e}}}^{-{[(x+\delta x)/2{\sigma }_{x}]}^{2}}\times {{\rm{e}}}^{-{(z/2{\sigma }_{z})}^{2}}\times {{\rm{e}}}^{{\left\{\left[-y+\delta y-B\cos \left(\frac{2\uppi }{\lambda }z+{\phi }_{z}\right)\right]/2{\sigma }_{y}\right\}}^{2}},$$
(2)

where A is the overall amplitude in units of C μm–3; and x, y and z and all the remaining parameters were normalized to 1 μm: σx,y,z are the Gaussian widths; δx and δy are x and y offsets; and λ and B are the oscillation wavelength and amplitude, respectively. The first two Gaussians in equation (2) constrain the overall distribution in the x and z directions, and the third represents the leading oscillation. The simulated trailing oscillations in Fig. 5a had less well-defined wavelength and amplitude; therefore, we specified only the spatial boundaries of this region and left the reconstruction algorithm to find the density distribution on its own. A smooth super-Gaussian function that generated negligible COTR represented the charge density in the middle section. We then ran dozens of iterative reconstructions with varied starting parameters until a dominant cluster of solutions representing the best fit emerged. Figure 5d represents the best-fit composite bunch distribution, and the bottom row in Fig. 5d shows the corresponding best-fit COTR data. The best-fit equation (2) parameters for the data in Fig. 5 were as follows: A = 0.323 ± 0.034 pC μm–3, B = 7.70 ± 1.00 μm, σx = 3.50 ± 0.40 μm, σy = 5.40 ± 0.40 μm, σz = 0.16 ± 0.10 μm, δx = −0.65 ± 0.20 μm, δy = −0.90 ± 0.06 μm, λ = 0.89 ± 0.07 μm.

COTR generation, imaging and calibration

A schematic of the 65 μm laser-block foil and 25 μm Kapton COTR foil with a 300-nm-thick aluminium-coated back surface in relation to the gas jet and LWFA drive pulse is shown in another work18. For the results presented here, the Kapton foil also included a 65 μm low-Z adhesive layer. Here 67 such foil pairs were mounted at the perimeter of a remotely controlled wheel that maintained a fixed 1 mm distance between the foils and rotated a fresh pair into place after each shot. Before the next shot, a lamp temporarily illuminated the new foil’s COTR-emitting back surface, enabling it to be finely adjusted into focus. To image the COTR from the foil to detectors, we employed a ×10 infinity-corrected long-working-distance microscope objective (Mitutoyo Model 46-144; numerical aperture, 0.28; focal length, 20 mm) for all the measurements reported here. By inserting a resolution test target in place of the COTR foil, we measured almost ×40 magnification and better than 2 μm (σ) resolution at the foil surface. To extend the spectral range to λ = 193 nm, we temporarily used a reflective microscope objective (TECHSPEC 89-723), but detected no signals at this wavelength. All the data reported here were taken with the Mitutoyo objective.

The integrated COTR energy captured at each detector was calibrated by measuring the power of linearly polarized test lasers at 405, 532, 633 and 805 nm at the position of the COTR foil with a power meter and then exposing the cameras for both s and p polarizations for a known time, yielding a counts per nanojoule calibration. These measurements were checked against the manufacturer’s published transmission/reflection curves for beamsplitters and transmission curves for bandpass filters. The COTR is radially polarized, but encountered beamsplitters with polarization- and λ-dependent transmission/reflection coefficients en route to each detector. We strategically designed each optical path to deliver an s/p intensity ratio as close to unity as possible. Actual s/p ratios ranged from 0.7 to 2.0, except for λ = 700 nm, for which the s/p value was ~7.0. Supplementary Table 1 provides a full list. For the data analysis, each image’s actual s/p ratio was taken into account.

e– bunch reconstruction procedure

Differential evolution

For reconstructing the normalized 3D charge density ρ(x, y, z) of an e– bunch from multispectral COTR data, we used a differential evolution56 algorithm, a highly parallelizable global-maximum-search algorithm that does not require computationally expensive gradient calculations. At each step ‘n’ of an iterative reconstruction, one calculates the COTR field \({{{{\bf{E}}}}}_{\perp }^{(n)}\) that an e– bunch of candidate distribution ρn(x, y, z) propagating along z generates on emerging from a foil in the x–y plane. This starts with calculating the field produced at transverse displacement r from the point at which a single electron emerges. After a lens with numerical aperture θmax images it to a detector, this field is57

$${{{{\bf{E}}}}}_{\perp }^{({{{\rm{PSF}}}})}({{{\bf{r}}}})=\frac{2ek}{c}\int\nolimits_{0}^{{\theta }_{\max}}\frac{{\theta }^{2}{\rm{d}}\theta }{{\theta }^{2}+{\gamma }^{-2}}{J}_{1}(k\theta | {{{\bf{r}}}}| )\hat{{{{\bf{r}}}}},$$
(3)

where e and γ are the electron charge and Lorentz factor, respectively; k and c are the wave number and speed of emitted transition radiation, respectively; and J1 is a Bessel function of the first kind. Equation (3) is our point-spread function (PSF). The total transverse COTR field from an e– bunch is then the convolution of ρn(x, y, z), multiplied by the phase delay e−ikz of radiation from each longitudinal slice of ρn, with the PSF as follows:

$${{{{\bf{E}}}}}_{\perp }^{(n)}(k,x,y)=Q\int\,{\rm{d}}{x}^{{\prime} }\int\,{\rm{d}}{y}^{{\prime} }{{{{\bf{E}}}}}_{\perp }^{({{{\rm{PSF}}}})}({{{\bf{r}}}}-{{{{\bf{r}}}}}^{{\prime} })\int\,{{\rm{e}}}^{-{\rm{i}}kz}{\rho }_{n}({x}^{{\prime} },{y}^{{\prime} },z){\rm{d}}z,$$
(4)

where Q is the total charge and |r′|2 = x′2 + y′2. The total COTR intensity, which is proportional to \(| {{{{\bf{E}}}}}_{\perp }^{(n)}{| }^{2}\), is then compared with the data. At the γ and θ values involved in this work, effects from the finite size of the COTR foil or milliradian-scale divergences57 are negligible.

Digital reconstruction

For automated digital reconstruction, we segmented ρ(x, y, z) into Nv voxels with Gaussian longitudinal profiles (Δzr.m.s. = 80 nm), effectively suppressing radiation at λ < 250 nm that we never observed27, and transverse widths Δx = Δy = 1 μm, just below our resolution threshold in the foil plane. The voxel spacing was ~60 nm longitudinally and 1 μm transversely. Thus, Nv ≈ 15,000 = 21 × 21 × 35. The voxel charge was then substituted for a single-electron charge e in evaluating the PSF in equation (3), and the convolution integrals in equation (4) became sums over voxels. We similarly segmented the calculated \({{{{\bf{E}}}}}_{\perp }^{(n)}(k,x,y)\) into pixels for comparison with charge-coupled-device images. For each iteration, a dimensionless NSS error \({N}_{\lambda }^{-1}{\sum }_{p}{(\Delta {N}_{p})}^{2}\) is calculated from differences \(\Delta {N}_{p}={N}_{p}^{\,({\rm{c}})}-{N}_{p}^{\,({\rm{m}})}\) between the calculated (c) and measured (m) counts in pixel ‘p’ normalized to the summed squared count \({N}_{\lambda }={\sum }_{p}({N}_{p}^{\,({\rm{m}})})^{2}\) at each λ. Procedures for aligning the calculated and measured intensity distributions and for evaluating the alignment uncertainties are discussed in ref. 58 and in Supplementary Section 2. We generated a composite NSS for each multi-λ image set.

The overall cost function that reconstructions minimized was a weighted sum of this composite NSS and four additional terms. Three of the latter ensured that the reconstructed {ρij} (where i and j are voxel and reconstruction labels, respectively, and brackets denote the set of values for Nv voxels) adhered within designated uncertainties to (1) a prescribed longitudinal profile from an independent measurement or simulation, (2) the total measured charge and (3) a positive-definite electron density ρ. The fourth ensured that it generated (4) the measured overall COTR intensity at each λ within the detector calibration uncertainty. The longitudinal profile term had the form \({\sum }_{\zeta }{({\Delta }_{\zeta }/{q}_{\zeta })}^{2}\varTheta ({\Delta }_{\zeta })\), where qζ is the prescribed charge at longitudinal position ζ, Δζ is the difference Qζ − qζ between the candidate profile’s charge Qζ at ζ and qζ and Θ is the Heaviside step function. Thus, if Qζ > qζ, Θ(Δζ) = 1 and Qζ increased the cost, whereas if Qζ < qζ, Θ(qζ) = 0 and Qζ incurred no cost. Similarly, the total charge term had the form ΔQ2Θ(ΔQ), where ΔQ is the difference Q − qm between the candidate’s Q and the measured qm total charges. Although this term encouraged Q to converge towards qm, we typically initialized Q at ~10qm to enrich genetic diversity and hasten convergence56. The positive-definite ρ constraint contributed a term \({\sum }_{i}{n}_{i}^{2}\varTheta (-{n}_{i})\), where ni is the number of electrons in voxel i. This term is zero except where the number in a voxel is negative (that is, the charge is positive). Finally, the COTR intensity constraint contributed a term \({\sum }_{\lambda }{({\Delta }_{I\lambda }/{\epsilon }_{I\lambda })}^{2}\) to the cost function, where ΔIλ is the variation in the overall COTR intensity during a reconstruction relative to the nominal measured value at each λ, and ϵIλ is the relative uncertainty of the measurement. Weights assigned to the various terms were strong, but finite and balanced. Underweighting led to unphysical solutions, overweighting to premature termination of a reconstruction in local minima and unbalanced weighting to over-prioritization of one term at the expense of others. Nevertheless, these criteria still left a wide range of flexibility in choosing workable weighting factors. Supplementary Fig. 2 diagrammatically depicts the digital reconstruction and cost analysis procedures.

Cluster analysis

Subjective judgements of the uniqueness of reconstructed {ρij} for a given COTR dataset can be simply based on the visual inspection of similarities within and between small (N ≤ 10) groups of N randomly initialized reconstructions. Supplementary Fig. 4 shows an example of such a group. As N increases, the systematic assessments of uniqueness become necessary. Here quantitative judgements are based on cluster analysis44. Specifically, we used a K-means clustering algorithm59, which partitions N reconstructions {ρij} (where 1 ≤ j ≤ N) for a COTR dataset into a small fixed number K ≪ N of clusters in which each {ρij} belongs to the cluster with the nearest mean \(\{{\bar{\rho }}_{i\kappa }\}\), where \({\bar{\rho }}_{i\kappa }={N}_{\kappa }^{-1}\mathop{\sum }\nolimits_{j = 1}^{{N}_{\kappa }}{\rho }_{ij},{N}_{\kappa }\) is the number of reconstructions in the cluster and \(\mathop{\sum }\nolimits_{\kappa = 1}^{K}{N}_{\kappa }=N\). A simple measure of the closeness of reconstruction j to \(\{\,{\bar{\rho }}_{i\kappa }\}\) is the variance \({\delta }_{j\kappa }={N}_{{\rm{v}}}^{-1}\mathop{\sum }\nolimits_{i = 1}^{{N}_{{\rm{v}}}}{\left\Vert {\rho }_{ij}-{\bar{\rho }}_{i\kappa }\right\Vert }^{2}\). The average variance within cluster κ is then \({\Delta }_{\kappa }={N}_{\kappa }^{-1}\mathop{\sum }\nolimits_{j = 1}^{{N}_{\kappa }}{\delta }_{j\kappa }\). The algorithm then reassigns reconstructions among the K clusters until global variance \(\varLambda =\mathop{\sum }\nolimits_{\kappa = 1}^{K}{\Delta }_{\kappa }\) is minimized.

To visualize clusters within a set of N reconstructions, we generated two-dimensional plots (for example, Fig. 4e) using PCA60. In PCA, each {ρij} is treated as a point in Nv-dimensional space. Principal components are unit vectors, where the mth vector is the direction of a line that best fits the N points and being orthogonal to the first m − 1 vectors. Here best fit means that the line minimizes the average squared perpendicular distance of the N points to the line. The first principal component is, thus, a line along the direction of maximum variance of the N points. The second principal component defines the direction of maximum variance in what is left once the effect of the first component is removed. Figure 4e plots the reconstruction results with respect to the first two principal components only. Such plots help us to visualize the clusters of closely related points. This identification then provides a basis for evaluating the difference between solutions within each cluster and between different clusters.

In a K-means cluster analysis59, K is an input parameter, that is, the algorithm identifies the designated number of clusters, assigning each point {ρij} in a way that minimizes the global variance Λ. Designating K = 2 usually generated two well-separated clusters on a PCA plot of reconstruction sets {ρij}. Members of these two clusters were approximately longitudinal mirror images of each other, that is, one is nearly a reflection of the other from a plane perpendicular to the z axis. Supplementary Figs. 3d and 4 show examples of this. This happens because an e– bunch and its longitudinally inverted counterpart produce identical multispectral COTR data. Thus, any set of N reconstructions divides naturally and unavoidably into these two indistinguishable clusters of solutions. In principle, phase-sensitive measurements could distinguish them. However, the intensity measurements used here can, at best, unfold the e–-beam structure to within a longitudinal mirror image of itself.

For occasional down-ramp-injected shots, a third well-separated cluster emerged and lowered Λ when we designated K = 3. Figure 4 is an example of this. In most such cases, the elements of the third cluster retained the main structural characteristics of the other clusters and was related to them through a simple geometric transformation. Supplementary Fig. 4 shows the details. Rarely, however, did further insights emerge from designating K ≥ 4. The main indicators that the maximum useful K had been designated for a given dataset were (1) designating a higher K did not yield a well-separated additional cluster on a PCA plot; (2) designating a higher K did not lower Λ; (3) cluster-averaged structures fit the COTR data as well as (or nearly as well as) individual reconstructions; and (4) visual inspection revealed greater structural differences between clusters than within a cluster.

Indicator (3) above provided our formal metric for uniqueness. Most down-ramp-injected shots yielded two or three clusters of reconstructions in each of which NSS of the cluster-averaged structure was nearly as small as that of individual reconstructions. This indicated that cluster members possessed common features that survived averaging. We relied on visual inspection to evaluate the geometric relationship between clusters. When, as in most cases, this relationship was simple, these reconstructions were deemed unique. Most self-injected shots, on the other hand, yielded clusters in which the NSS of the cluster-averaged structure substantially exceeded that of individual reconstructions, indicating that the reconstruction had not found a unique solution. Supplementary Figs. 7 and 8 show representative examples of this.