To avoid rescaling, use the original resolution of the source, yes.
One cause for slightly off colors is that for increasing compression in a compressed video, the number of different colors is slightly reduced as default (color range limited). For the same reason there is also a different resolution for color than for luma (brightness): only half, so colors might look slightly different.