Joy said we could

The family lives on what we would consider the 6th floor of the apartment. They have a tiny balcony off of the kitchen that we generally hang clothes off of to dry and keep outside toys such as soccer balls. Occasionally D and Ir will take their blades outside on the balcony and go back and forth while I clean up after snack time. Today E was gone in the evening so while I made dinner they proceeded to do this. It is very rare that E isn't there for dinner, this might be the second time the entire five months that I have been here that she hasn't been home so the girls took their blades on the balcony. S came into the kitchen to check on things and saw the girls out their and told them to come in immediately because they know they aren't supposed to be out there. The girls then said "but Joy said we could".
Sometimes I get so irritated with them. They told me that it was ok for them to do it so of course I let them, who am I to question the parents rules. To me it's not a huge deal anyway so it seemed fine which is why I didn't question it. What irks me is that the girls knew they weren't supposed to do it and told me it was ok. wekwmqwe oh well I guess I will just have to ask the parents if every little thing the girls ever do is ok...?