SCExAO at the Subaru telescope is a visible and near-infrared high-contrast imaging instrument employing extreme adaptive optics and coronagraphy. The instrument feeds the near-infrared light (JHK) to the integralfield spectrograph CHARIS. The spectropolarimetric capability of CHARIS is enabled by a Wollaston prism and is unique among high-contrast imagers. We present a detailed Mueller matrix model describing the instrumental polarization effects of the complete optical path, thus the telescope and instrument. From measurements with the internal light source, we find that the image derotator (K-mirror) produces strongly wavelength-dependent crosstalk, in the worst case converting â1/495% of the incident linear polarization to circularly polarized light that cannot be measured. Observations of an unpolarized star show that the magnitude of the instrumental polarization of the telescope varies with wavelength between 0.5% and 1%, and that its angle is exactly equal to the altitude angle of the telescope. Using physical models of the fold mirror of the telescope, the half-wave plate, and the derotator, we simultaneously fit the instrumental polarization effects in the 22 wavelength bins. Over the full wavelength range, our model currently reaches a total polarimetric accuracy between 0.08% and 0.24% in the degree of linear polarization. We propose additional calibration measurements to improve the polarimetric accuracy to <0.1% and plan to integrate the complete Mueller matrix model into the existing CHARIS post-processing pipeline. Our calibrations of CHARIS' spectropolarimetric mode will enable unique quantitative polarimetric studies of circumstellar disks and planetary and brown dwarf companions.