I get where you are coming from, but in general, I think the context clues of the conversation would help determine what/who is being referenced.

"oh yeah, just speak to the MoE; they probably have the answer you are looking for." Obviously in this situation, it would be impractical to think that talking to the ministry is best, so it's safe to assume the minister is who is being referenced.

"I'm sure the MoE has something that might suit your skill in writing." In this case, it is impractical to think that the minister personally has something that you would need to do; instead, the ministry has a writing division that would be of use, so assuming that the ministry was being referenced would be a safe bet.