Interesting find. I can also confirm that it happens in 6.5.1.
For now one workaround would be to use Thor as a muting mechanism (Audio In1 100% -> Audio Out1 scaled 100% by Button1). It seems multiplying the incoming audio by zero is the only surefire way to stop Neptune from tracking it.
By the way, Neptune's pitch CV isn't a note#, but a frequency. For example, you could connect it to OSC Pitch of a Subtractor.