To be honest, Sega always kind of struggled in the home console market. Back in the day, their main stream of revenue came from their arcade division, while their console division was always kind of lack luster. The Sega Master System did have some success in Europe and Brazil, but never took off in the NA market, or their home turf in Japan. The Sega Genesis/ Mega Drive was a run away success in North America and even sold really well in Europe and was their best selling console of all time. But even in Japan, it was a flop.
The people running Sega of Japan had a weird resentment for the success of Sega of America and the Genesis. When they developed the Saturn, they practically made it in a bubble and it was developed by one of their arcade divisions. No other part of Sega had any input on it out side of Sega of Japan. Which was a really big mistake, given that Sega od America had first dib's on the N64 hardware (which was turned down by Sega of Japan) and Sony even tried to set up a partnership with the Playstation.
The first version of the Saturn was very underpowered compared to the Playstation. Which caused Sega of Japan to go into a panic mode and redesign parts of the system on the fly, which caused all sorts of additional headaches for developers that had to make games for it. Also combined with the lack of a good tool set, and a surprise early launch that just pissed up retailers (at least in North America) , Sega of Japan really botched that system up. Though ironically, it was still the best selling Sega console in Japan.
After the Saturn flopped, Sega actually spent a lot of money on R&D for the successor. They worked with partners like NVidia, 3DFX, Lockheed Martin (apparently) and of course PowerVR2 for the final system.
By the time the Dreamcast came out, the arcade scene was slowly dying everywhere with the exception of Japan. The Dreamcast actually sold alright numbers for the short period of time that it was on the market, but it wasn't enough to turn a profit.