I think this is only possible (currently) by editing the payment modules. In effect you could choose not to display a payment option if a particular carrier has been selected (using a hard-coded carrier_id).
In payment modules there is always a hookPayment() function which displays the payment method. Within this function you would do your test for the carrier.
So, for example if someone has picked a carrier that takes payment on delivery, you don’t want to allow payment by PayPal to be an option. In paypal.php (line 139) there is:
public function hookPayment($params)
{
global $smarty;
$address = new Address(intval($params['cart']->id_address_invoice));
$customer = new Customer(intval($params['cart']->id_customer));
$business = Configuration::get('PAYPAL_BUSINESS');
$currency = $this->getCurrency();
etc.
Lets’s say your cash on delivery carrier has the id of 2, then you could change the above to:
public function hookPayment($params)
{
global $smarty;
$cart = new Cart(intval($params['cart']));
if (intval($cart->id_carrier==2)) return;
$address = new Address(intval($params['cart']->id_address_invoice));
$customer = new Customer(intval($params['cart']->id_customer));
$business = Configuration::get('PAYPAL_BUSINESS');
$currency = $this->getCurrency();
This will disable PayPal for any checkout processes where the customer has selected the carrier with the ID of 2.
A bit messy, but it would provide a quick fix.
Paul