Fishermen in Sea Palling are locked in a row with holiday home owners which they fear could mean the end of the village's centuries old industry.