You say at the best of times when you are making a negative or critical comment to emphasize that it is true even when the circumstances are as favourable as possible.
[emphasis] A trade war would be bad at the best of times, but in the current climate it wouldbe a disaster.