The same happened in Germany. Then when you ask the people, they are mostly uneducated and wouldnt want to go to the fields, become a cleaning worker etc, because its "too low" for them.
Polish people arent taking cheap labor away from other people. They are taking jobs no one else did or wanted to do.
Its not fear. Its stupid racism. The same we happen in Germany when uneducated people from eastern germany mostly think polish people are "teeking our jebs", while they actually dont wanna get asparagus out of fields..