So long you do not find the spiritual master is doing against the principles of sastra or guru, then if you give up the company of spiritual master, that is not good for you. That is your downfall