The transient temperature evolution of ultrathin bismuth films, epitaxially grown on a silicon single crystal, upon femtosecond laser excitation is studied by time-resolved electron diffraction. The exponential decay of the film temperature is explained by phonon reflection at the interface, which results in a strongly reduced thermal conduction in the cross plane of the layered system. The thermal boundary conductance is found to be as low as 1273 W/(K cm2). Model calculations, including phonon confinement effects, explain the linear relationship between the observed film-temperature decay constant and the film thickness. Even for 2.5 nm thin films the phonon transmission probability across the interface is given by bulk properties. Our simulations show that phonon confinement effects are negligible for bismuth-film thicknesses larger than 1 nm.