This is more about basic physics than biology. When you hold your breath, you normally take in one last long breath and keep it in as long as possible, Your lungs are therefore already full of gas (remember that the oxygen used by our lungs is only ~22% of the total volume of air you inhale). Therefore, when you release that breath and want to take in a new one you need to first empty your lungs out in order to refill them.
In addition, you need to have your lungs empty in order to be able to expand them and draw in more air. There is no biological reason for any of this, it is the same principle as trying to use a turkey baster:
If you use it once and pull up some liquid into it, you will have to let air out before you can use it again. Your lungs work in the same way. Test this by exhaling all the air from your lungs and then holding your breath without inhaling. When you want to start breathing again, you'll be able to breathe directly in without needing to exhale first.
