This paper studies a new challenging problem in face recognition (FR) with single sample per person (SSPP), i.e., SSPP FR with a contaminated gallery (SSPP-CG FR), where the gallery is contaminated by variations. In SSPP-CG FR, the popular generic learning methods will suffer serious performance degradation because the applied prototype plus variation (P+V) model is not suitable in such scenarios. The reasons are twofold: 1) The contaminated gallery samples yield bad prototypes to represent the persons; 2) The generated variation dictionary is simply based on the subtraction of average face from generic samples of the same person and cannot well depict the intra-personal variations. To tackle SSPPCG FR, we propose a novel Iterative Dynamic Generic Learning (IDGL) method, where the labeled gallery and unlabeled query sets are fed into a dynamic label feedback network for learning. Specifically, IDGL first recovers the prototypes via a semi-supervised low-rank representation (SSLRR) framework and learns a representative variation dictionary by extracting the 'sample-specific' corruptions from an auxiliary generic set. Then, it puts them into the P+V model to estimate labels for query samples. Subsequently, the estimated labels are used as the feedbacks to modify the SSLRR, thus updating new prototypes for the next round of P+V based label estimation. With the dynamic learning network, the accuracy of the estimated labels is improved iteratively owing to the steadily enhanced prototypes. Experiments on various benchmark databases have verified the superiority of IDGL.