The downside here is that, because these hit points are temporary, they're not restored by rolling hit dice or resting.