ESC A19 - is it worth seeking a review of my rejected appeal?

Since getting early retirement in 2005 I have received a pension from UKAEA (and their successor organisations). Whilst working I had also contributed to a superannuation 'additional voluntary contribution' fund from which I started to receive an annuity in 2008 from the Prudential*.

On 7 Nov HMRC notified me that I had personal allowances on both incomes and I owed them £5004. I appealed under ESC A19 as they knew about both incomes. I received their rejection on 24 November as they hadn't received information and 'end of year pay and tax details sent by your employer is not classified as information for the purposes of ESC A19'

I appealed against this decision based on a letter from Prudential dated April 2009 referring to a change in tax coding which confirmed that HMRC knew about the annuity.

HMRC have just rejected this appeal as the law allows them 4 years to investigate P14 / P60 information and they received no other source of information.

Prudential have confirmed they have no record of sending a P46 but they had sent the P14 each year.

Since 2005 HMRC told me I no longer need to send them a return and I didn't inform them about the annuity by any other means.

Have I any basis for further appeal with a likelihood of success (bearing in mind delays have the potential to result in penalties and interest charges)?

* Under an agreement with HMRC whereby the annuity can be taken over the period between early retirement and reaching State Pension age.