If you are left holding the baby, you are put in a situation where you are responsible for something, often in an unfair way because other people fail or refuse to take responsibility for it. [British, informal]
You go off round the world, and leave me to hold the baby.
regional note: in AM, use be left holding the bag