Same issue. With or without a product tax rule, it makes no difference.
Also, I don't get why it would be set on product in the first place, should be just dependent on order's delivery address, since I have delivery address set as what it's based on. Any answers?