Closing Doors

I don't know how to handle closing elevator doors. If someone's coming is the polite thing to do hold the elevator for them, or are you tempting fate? If the late person is nearby, say in my building's lobby, I'll always hold the doors. But when they're far away and pushing the 'door open' button would inconvenience my fellow elevator riders, I slide back away from the door and dig my phone out of my pocket like I'm getting a call. Not that I'd ever talk on the phone in the elevator. That's rude.