As polarized light propagates through optical systems, the state folds at each refraction, and folds and flips at each reflection. Carrying the polarization state through the optical system is analogous to the parallel transport of vectors on a sphere, and leads to different Berry phase accumulation along different paths. For circular polarization this rotation of the polarization state is a phase change and manifests as wavefront aberration, albet of opposite signs for left and right circularly polarized light. For linearly polarized light, there is an overall rotation of the polarization state across the pupil, yielding a mismatch between the incident and exiting polarizations. This polarization variation has been named skew aberration because the polarization rotation is zero for meridional rays. Several examples of the effect on the point spread function and optical transfer function are presented.