Fig. 1: Sketch of the experimental set-up.

a, An anisotropic silica nanoparticle (schematically illustrated as an ellipsoid) is trapped by an optical tweezer in an ultrahigh vacuum. The tweezer light is linearly polarized along the y axis by a polarizing beam splitter. The long axis of the nanoparticle aligns parallel to the tweezer polarization and undergoes angular harmonic oscillations, termed librations, at a frequency Ωα/(2π) in the x–y plane. This libration motion is coupled to a high-finesse optical cavity. The high-NA lens forming the optical trap is mounted on a nanopositioner (not shown) such that the particle equilibrium position can be varied across the cavity intensity profile. The x-polarized light backscattered from the nanoparticle is collected by the trapping lens and mixed with a local oscillator of frequency ωLO in a balanced heterodyne detector. This detector provides a measurement of the libration motion unaltered by the cavity transfer function. Inset: power spectral density \({S}_{VV}^{{\rm{het}}}\) acquired at 6 mbar from this backward detector. The librational mode of the particle peaks at Ωα/(2π) = 1.08 MHz. Laser phase noise in the tweezer beam is suppressed by a noise eater composed of a phase-noise detector and an electro-optic modulator. The suppression level is varied with a gain g. b, Illustration of the libration mode. The tweezer polarization is aligned to the cavity axis (y). The libration angle α denotes the deviation of the long axis of the particle from the polarization direction of the tweezer field in the x–y plane. EOM, electro-optic modulator; LO, local oscillator; PBS, polarizing beam splitter; UHV, ultrahigh vacuum.