Some people believe the strangest things. That doesn't make those
things true.
SRC amounts to applying a low-pass filter. That's it. There is no
other magic involved. The fact that this filter is producing output
samples at a rate which is different from the input sample rate does
not change it in any fundamental way. It just means using different
coefficients, which are no better or worse than those used by a 1:1
rate filter.
The operation will add some round-off noise, but no more than a simple
EQ or even a fader. That noise will be at a level at least 40 dB below
what will be added by reducing to 16 bit for a CD. Which means it makes
a difference of less than 1/1000 of a dB, if anything at all. It does
not add distortion or degrade the signal in any way. If you're looking
for that, you'll get a zillion times as much by using even the simplest
dynamic effect.
Given that most audio interfaces will perform better at 48 kHz than at
44.1, if there is any significant difference at all you're probably better
of producing a CD at 48 kHz and converting it to 44.1 right at the end of
the chain, the same place where you reduce to 16 bit and add dithering,
for example in Ardour's export dialog.
There is no good reason to use 44.1 kHz *ever*, except for the signal
stored on an audio CD.