IFF can be overridden. In fact, whomever was operating the Buk had to override the IFF in order to launch, otherwise the system would have been in lock-out mode.
EDIT: This makes the operators twice criminal. Firstly for not eyeballing whatever they were painting with the TV unit, secondly for overriding the IFF.
I don't think civilian airliners are sending an IFF signal that identifies it as a friendly to a Russian system, just a code for their flight. If it was so easy every combat jet in the world would broadcast a civilian IFF signal.
edit: from what I've read the Buk they had was just a TELAR, a Transporter Erector Launcher Air Radar that has a powerful but narrow beam fire control radar. Normally there is a search radar that would look for targets and identify them, this would be a unit that can recognize the altitude, speed, distance, possible plane type and any transponder signals. Like most military hardware the TELAR unit has an autonomous capability where the operator can direct the radar with a visual or thermal camera, in this case they aim it at the little blip in the sky with the contour trail coming out of it. They would have no really good info on speed and height, just a thing in the sky their missiles fly at.