Uh, because it switches up the gameplay? What you describe as retarded is what a lot of fans actually liked. It was a nice change of pace to go from headshotting grunts and jackals every 5 feet in a start/stop sequence.
Their behaviors weren't up to par because the technology can't handle it. What would you have the scarabs do? Jump up and down, move at a steady pace etc.? I don't see how any of the other enemy sandbox is any better. What difference is there between a hunter then? You avoid the shots, circle around etc.
What clever tactics did the enemies have? That they could move around and show random behaviors because they were smaller?
My issue isn't with them trying to change pace. It's with the change of pace being executed poorly. We all know that Bungie didn't write particularly strong AI for the Scarab due ostensibly due to tech constraints, which is precisely why the Scarabs probably should have stayed on the cutting room floor.
I remember in one of the earlier previews for The Storm, the Scarab was described as having more complex AI and "had to be chased down" dynamically across the level. Journalistic hyperbole, perhaps, but it sounded amazing in comparison to the moving BSP that was the Halo 2 Scarab. What we got was a braindead supertank that potshotted you every once in a while.
My biggest gripe with the Scarabs is that they never felt threatening. Every battle involving one becomes amusing if you let it linger on too long because they are so incompetent at taking things out. It was also a really bad idea to put Pelicans (that also did nothing) in The Covenant battle because the Scarabs would try to take them out. No dice, Pelicans didn't dynamically crash.
The AI of Halo is serviceable. Yes, they all have patterns and weaknesses. But at least every ground unit is capable of killing me. I can honestly say that I haven't been killed by the Halo 3 Scarabs since 2007 on any difficulty below Mythic unless I was screwing around. A Hunter may not be as intimidating as it could be (Reach's badass Hunters come to mind), but one could still get me.
What would I have preferred? More vehicles capable of killing me. I would take five to ten Wraiths as opposed to one or two Scarabs. A shame that the 360's tech couldn't support the massive battle level that Tip of the Spear was meant to be.