late filing of self assessment

Help Needed

I have worked in the construction industry for the past 12 years,but fell behind filing my self assessment tax returns from 04/05 to 11/12,the contractor i done my work for deducted tax under the cis scheme,and i got at first vouchers then statements so i paid the tax on the money i earned,i have now filed all the tax returns.but in the mean time over the years the HMRC have added determination payments to my tax bills.does this mean i have to pay my taxes twice,HMRC say yes,even though i had my vouchers and statements to prove i have paid once,,Please any help would be great,