Margin shortfall rejection can occur in specific situations with open positions:
Check this sample scenarios
Scenario A: When you have a pending order for an open position, and without cancelling that order you’re trying to square off the position.
Scenario B: When you have a hedge spread and attempt to close the buy orders first, without having adequate margin for the potential naked positions after the closing.
It seems that your situation falls into Scenario B, as you mentioned you were dealing with a calendar spread.