Face cognition is considered a specific human ability, clearly differentiable from general cognitive functioning. Its specificity is primarily supported by cognitive-experimental and neuroimaging research, but recently also from an individual differences perspective. However, no comprehensive behavioral data are available, which would allow estimating lifespan changes of the covariance structure of face-cognition abilities and general cognitive functioning as well as age-differences in face cognition after accounting for interindividual variability in general cognition. The present study aimed to fill this gap. In an age-heterogeneous (18-82 years) sample of 448 adults, we found no factorial dedifferentiation between face cognition and general cognition. Age-related differences in face memory were still salient after taking into account changes in general cognitive functioning. Face cognition thus remains a specific human ability compared with general cognition, even until old age. We discuss implications for models of cognitive aging and suggest that it is necessary to include more explicitly special social abilities in those models.