Given the year, make, and mileage, I'd say that's a bit high. Try to swing it for $2500-2700 if you're dead-set on it.
Just checked out my Kelley Blue Book for you:
1996 Pontiac Sunfire, 2-door coupe:
Private Party (good cond.): $2350
Retail (excellent cond): $3825
So the average is roughly $3000 (though it should be noted that the "private party" amount of $2350 is what is reasonable for private owners to expect when they sell the car, so $3000 is quite high). However, the acceptable milage range for 1996 automobiles according to the book is 82K to 87K, so you can quite reasonably ask him to come down in price at least a couple of hundred dollars (since 106K is
way beyond 87K, not just "slightly"). Do yourself a favor and buy this book (or a similar one), and have it handy when you go to inspect the car (assuming you haven't already), or during negotiations-- it'll really give you something to fall back on when he becomes stubborn. Just point to the facts.