It's a restriction on an individual's right to work as many hours as they want, and on employers rights. It's EU interference with the British free market, in other words.
It didn't help that it's introduction was controversial either. Back in 1993, when the working time directive was introduced, the UK voted against it in the Council of Ministers, and refused to implement it on the basis that it was social legislation, and the UK had an opt-out from that since Maastricht in 1992. But the EU Commission instead claimed that it was based on health and safety laws, which the UK didn't have an opt-out over, so we had to abide by it. The UK lost the court case over it too.