I believe that the pilot was killed and the others were not harmed for the following combination of reasons:
1. The smoke monster (Man In Black) was seeking people that he could destroy. The opportunity arose with smokey approaching the plane and the individuals inside making themselves known and thus directly exposed to the danger. The pilot became a target when he became visible to the smoke monster, which made its move. The motivation of the smoke monster is part of the “game” between him and Jacob – Jacob’s “guests” are here to prevent smokey from leaving the island, so smoky destorys those that he can, and instills fear and dread in those that he cannot harm via the “rules”.
2. Kate and Jack are not attacked because they cannot be attacked via the “rules” – they are both candidates and thus can only be harmed by accidental death of killing themselves (a plot that smokey ultimately unfolds with the submarine plan).
3. Charlie is not attacked because – IF he is not a candidate, which I will assume here – it is not “his time to die”. One thing to recall is that at the point that the plane crashes, Kate and Jack have already been to the island in terms of a true timeline (they were there in Dharma time – forming future events which include the crash).
Because “whatever happened, happened”, Jack and Kate (and Hugo and others) are destined to go back in time. I think it is reasonable to assume that Charlie is a link to this destined timeline – he has freewill, but he also has a destiny (actually is called predeterminism when someone has freewill but also an ultimate destiny) with the candidates that helps “course correct” them on their journeys. His relationship with them helps forge each of their predetermined paths into the past.
Another way to think of Charlie’s destiny is his impact on Hugo during his life; Hugo, in my opinion, was “fated” to become the island’s keeper. Charlie’s life and death help Hugo become the person he must – the eventual Guardian of the Island. In this way, Charlie was “destined” to guide Hugo on his path, and thus Charlie could not die until it he had fulfilled this destiny.
What is the proof of this?
Charlie’s bond to a predetermined destiny was made clear when his “time to die” had come; this is driven home by Desmond’s attempts to save him which ultimately fail (because they have to).
Not to elaborate too much more, but to clarify the predeterminism that I am referring to: think of a book with a beginning and an end. Charlie can write the book however he wants, but the ending never changes – and along the way, there are likely a few paragraphs that also must appear. Thus he has free will, but also must eventually conform to course correcting and to his ending – the ending of the book. Not everyone in Lost had a “Destiny”, but I believe Charlie did, which was why he was spared in the Pilot – because his life afterwards and eventual death were meaningful to the Fate of the island, and thus the Fate of the world.
Glad to be writing about Lost, Time Travel, Fate and Freewill again!