A gallery designed to display the work of Graham Sutherland is accused of turning its back on art.