Fairly certain I remember sndfile-* not needing anything X11, so not sure why it would pull in any in your case.  I could be wrong, but that is my memory.

Now if someone packaged it very strange I could see that happening, but not form the program itself.

      Seablade