Conflicts of interest and auditor termination

Can we insist that our external auditors insert a clause in theri engagement letter that allows us to terminate their appointment if there is a conflict of interest between us  and another client of theirs?  They have a clause that says they will alert us to any such conflicts, but reserve the right to carry on nonetheless.

Thanks.