Odd that you should say the individual channels are not linear… The application note says it is (that is, the readings should increase linearly with respect to the intensity of the light).
It does make sense that there is “bleeding” into G and B if you shine red light at the sensor, since in the datasheet you can see the frequency responses of the R,G,B sensors overlap. This doesn’t preclude calibration … I don’t think.
First thing - I’ll try to confirm the sensor readings are linear with respect to intensity (using a PWM’d LED at fixed distance). Hopefully it is. Yikes.