A new type of optical phase-locked loop (OPLL), called the dither loop, is mathematically analyzed. The dither loop extracts a phase-error signal by applying a small phase disturbance to the local oscillator laser, and synchronously demodulating the resulting power fluctuation in the output signal of the receiver. The dither loop is superior to other OPLL designs, because it does not need the transmission of a residual carrier, it employs a 180°/3-dB hybrid, an ac-coupled front end, and it accepts a large variety of input signals. Furthermore, in a dither loop, the amount of power which is fed to the phase-locking branch can be adaptively controlled within the receiver. The analysis first focuses on an expression for the phase detector gain in a dither loop. Using a linearized model, the phase-error variance due to phase dithering, white frequency noise induced phase noise and shot noise is evaluated. A simplified expression for the power penalty generated by the phase dither signal is presented. In a more complex calculation, the overall power penalty due to phase dithering and the residual phase error is found. This allows us to synthesize a design rule for dither loops with optimum performance measures. The design rule determines all relevant system parameters, based on specified values of the system bit rate, the laser linewidth, the photodiode responsivity and the required bit-error rate.