REVENUE DETERMINATION FOR EARLIER YEARS

 I have been hit by a revenue determination  for 2003-2004 and 2004-2005 in sum of £6850 because i have not submitted tax returns on time, however i have been out of the UK during that period and did not do any work i have filed nil returns for that period in 2011 and send a letter regarding this isssue, but hmrc says that the time limit to file tax returns for 2003-2004 and 2004-2005 expired on 31/03/2010 therefore the determination remains in force and i have to pay that amount, and can't apeal because the penalty regime has changed. I wonder if anyone can help as paying this amount of money without having any income for that period is insane.

Thanks in advance for your reply.