You are absolutely right in saying that the North did not fight the war for the sake of freeing the slaves. There was a significant number of well established people in the North who viewed the institution of slavery as being important, and furthermore, went on to argue that African americans and White americans would not be able to co-exist peacefully in the United States, and would likely need to be separated.
The issue is, however, you were looking at the lens only from the perspective of why the North fought the war and not taking it the step further to analyze why the South fought the war. The south fought the war for the protection of the institution of slavery. You can pretty it up by any other words, but once again, it was cited in the reasons for secession as the primary cause for the states to secede from the union in all declarations that were released.
Furthermore, the Confederate States Constitution explicitly detailed slavery in several articles as being fundamental to the CSA. It was apparent from the constitution that the CSA was being set up as a slave-holding republic for a significant period of time to come.