Common illumination systems in short wave infrared (SWIR) hyperspectral imaging (HSI) include direct or indirect tungsten halogen lights. While direct lights provide more radiation onto the samples than dome setups, thus being more energy efficient, the acquired images often suffer from specular reflections and gloss. Glare artifacts in images increase variability in the data limiting the accuracy of machine vision algorithms for defect detection and quality inspection, or even providing false positives. Although domes are known to provide a near Lambertian illumination and glare free images, glossy regions and heterogeneities may remain in the data in practice. More particularly, in the field of fruit and vegetable quality inspection, due to their waxy surface, it remains challenging to design an efficient realistic lighting system. This paper suggests a new approach to optimize the illumination of fruit and vegetables based on measurements of the bidirectional reflectance distribution function (BRDF), shape and Stokes parameters. From these measured values, a BRDF model is loaded into ray-Tracing software for realistic illumination engineering in order to determine the most suitable illumination scheme. This concept is applied to apples and a cross polarizer (CP) with freeform optics (FO) optical configuration is proposed, which allows the FO to be optimized to maximize uniformity in the field of view of the imager and removes the parallel polarized gloss on the apples. The performance of this CP illumination system was determined experimentally for a set of apples. This cross polarized (CP) illumination system provided a uniformity (U) of 92% and an efficiency (ν) of 90%, while U = 87% and ν = 14% for an ideal dome configuration when illuminating a rectangular target. The simulated imaged apples with assigned optical properties performed better with CP (U=80%) than when using a dome (U=73%) by 7%. Finally, the sensitivity of the design for the light positioning and spectral tolerance are investigated.