Prince Harry has said he is devastated and in shock to have to quit as patron of a charity he set up in honour of his mother.