Avoid responsibility for what though, government programs? Can you point to a successful anti-poverty program that has made a significant impact? A lot of people argue government programs have made the situation worse.
I think the suggestion that things can be fixed or even helped by some ambiguous government solution is harmful. As you say it makes it easy for people to avoid responsibility.
As I said, this isn't really my area, and I'm a lot stronger on philosophy than policy, but I think it's true that many government programs have made the situation much worse. Government action has also made black people much better off, though. Certainly no one's going to argue that banning slavery didn't help, if we're going back far enough, and almost everyone thinks the Civil Rights Act was a good idea.
But, for starters, we can at least stop or modify existing programs. Mass imprisonment is an obvious one. I'd say that the way we handle redistribution needs to be looked at - it's pretty easy for poor people to face extremely high effective marginal tax rates due to the phasing out of benefits, so this should occur much more gradually and benefits should be more generous at higher incomes (maybe this counts as doing something more).
As for doing more stuff, note that police response times are much higher in areas with lots of black people, that schools are much worse, etc. Black people often feel that the police aren't really there to help them, and this is reinforced by the police actually failing to be there to help them. Sometimes when the police do show up they're much more willing to risk hurting innocents. And absolutely nobody is clamoring to send their kids to public schools in the middle of cities. Yes, sometimes more is spent per person, but children are still being cheated. Health care for all seems like a no-brainer. I'd be interested in looking into ways of encouraging small, locally-owned businesses over large corporations who don't reinvest their profits in the community.
Edit: Maybe this was obvious, but I also think that there's just too little redistribution, and that it's a lot easier to justify more of it when we understand just how thoroughly we fail to provide equality of opportunity. People don't stop being poor when they're paid so little that they still qualify for welfare and welfare benefits are as stingy as possible out of fear that people will actually end up with extra money after spending on necessities.