It's mentioned in the video and in the sidequest section, but the Tokugawa rulers established an official four-tier class system to divide the Japanese people based on Confucian philosophy imported from China. The highest level would be the samurai, who were the ruling warrior class. Next came the farmers or peasants, since they produced the most essential commodity - food. Third were the artisans or craftsmen, because they had special skills and produced less vital goods. Finally, at the bottom were merchants, ostensibly middlemen who produced nothing of value. Social mobility was nonexistent and one's class was determined by birth.
However, class had little to do with material wealth and comfort. While only samurai could hope to have direct political power, the merchant class was always the wealthiest while the farmers were always the poorest and lived the hardest lives. As the Edo Period wore on and the samurai's martial skills waned for lack of demand, most families became essentially parasites on Japanese society. Producing nothing and filling no positions of power, they fell ever farther into debt - debt owed to the merchant class, who grew wealthier each year. Several times during this era the shogunate declared sweeping amnesties on all samurai debt, wiping out entire merchant class fortunes and livelihoods. Of course, this had the eminently foreseeable effect of making the townsmen reluctant to issue loans to samurai families at all, ultimately putting them in even more dire straits once the money ran out again.
Quite a few members of society did not fall into any one of the four tiers. At the upper end, there were the hereditary nobles, essentially relics from the Heian Era some 500 years earlier when the Emperor had held court in Kyoto. At that time there was no such thing as "samurai" and the capital was filled with clans of rather softer and more artistic types who lived more or less oblivous to what was going on outside the city walls. When the rural lords who later became daimyou and samurai officially began running the country they gained legitimacy by simply marrying into or arranging to be adopted by these noble families, which in turn assured the nobles' continued existence in the luxury to which they had become accustomed.
Monks and priests also stood outside the system, and could have been born as members of any of the four classes. How comfortable their actual existence was and the degree of their political power varied widely by period, sect and location.
The most famous outsiders were considered the last after all other classes: the outcasts or burakumin - untouchables. These people suffered the same fate of their more familiar counterparts in India for much the same reasons. Because of religious taboos regarding defilement by association with death and blood, tanners, cobblers, executioners, butchers and others like them were considered little more than animals and segregated strictly from the rest of society. This continued well into modern times, and even today there is a very low rate of intermarriage between buraku people and other Japanese, largely due to the latter group's institutionalized distaste for the former. Given how non-religious modern Japan is as a whole, it's safe to assume that these feelings, where they do crop up, are not due to any real feeling of defilement by occupation (particularly when almost no burakumin have jobs different from anyone else) but simply to a legacy of discrimination.
This system continued until after the Meiji Emperor had asserted control and officially nullified the caste distinctions in 1871. However, although great strides have been made, burakumin still face measurable discrimination in Japan today.