[She shrugs. Again. Isn't this a nice theme.]

I saw someone try to leave without theirs and they got stopped and told that it was too dangerous to go alone. Considering the logic of this world I have the feeling that someone will pop out of nowhere to tell you that if you try to leave without it. I don't know how it works, whether it's magic or something else, but somehow someone will know that you've broken the rules.