The job of a director/conductor is 99% that of a producer, and probably 100% of what makes that job more difficult than putting a drum loop on and putting a sample over it. The only distinction I like to make in production is between producers and beat makers. Producers go those extra 100 miles in every respect for making a song and beat makers forward beats via e-mail. There is relatively little ambiguity there.
What there is ambiguity in, is in who can chop samples better, who is "better" at drums, etc. That shit largely comes down to opinion/preference and just creates a circle between all of the great producers who are frankly the most qualified to even judge such things. Kanye says Dilla is the best. Dilla says Preemo is the best. Preemo says Marl. Marl says Just. Just says Kanye. You see this shit in all producer interviews, you keep expecting it to just trace back to the origin but some old head names a legend from a newer generation. If we're talking the elite, Kanye undeniably deserves a spot. If you want to narrow down a top 5, 10, or whatever, it will say more about your taste than it does the skill IMO, because there will always be someone with a milestone in something, or sheer consistency, or that one beat that makes you jayzlisteningtolucifer.gif and form a good counterpoint.