As a yoga practitioner I have to admit that I'm not all that much into the whole Hindu/Buddhist undertones that many practices take on, but that's because I'm just an extremely non-religious person in general. After spending most of my childhood in church and Catholic schools, I tend to run in the opposite direction of anything that is remotely religious.
Yoga is considered a science in India, and, in fact, I've been told that many of the gurus in India actually frown on their Western students "converting" to Hinduism or Buddhism. Gurus will tell you that you can practice yoga no matter what religion you are or even if you are an agnostic or atheist because yoga has nothing to do with religion.
The yoga/eastern religion connection most likely got mixed up here in the west because early western devotees to yoga adopted aspects of eastern religions or converted because their gurus were Hindu or Buddhist.
So, when I hear Christian leaders' objections to yoga, I know that the objections stem from nothing more than just pure ignorance. After all, it was just a few years ago that the Teletubbies were being accused of promoting the gay lifestyle by this same crowd.
On the other hand, yoga is not just a physical work-out (although it can be). The physical aspect of yoga, the asanas, are just one of the eight limbs of yoga. When you look at the other seven limbs, however, even those do not demand adherence to a particular religion. Meditation is not a religious practice, although you can meditate on a higher being. Chanting is also not a religious practice, nor is pranayama (breath work).
Yogis are supposed to follow a code of ethics, the yamas and niyamas, but even those have no reference to any gods.
So I think before Pat Robertson objects to yoga because it's "spooky" he should really put the research into finding out what the chanting is all about.
After all, there are many people in this country who could benefit from yoga. It would be a shame if they feared walking into a yoga studio just because they thought it would compromise their Christian faith.