Sometimes, books (along with movies and other forms of art) do not always “resolve” as we’d like them to, such as ending with a spiritual conversion that leads a person to faith in God. Some characters even end up in a worse state than when they started.
Especially with fiction, that lack of resolution can actually be beneficial at times — particularly if that makes the reader want resolution. In other words, if a text ends sadly, the reader might still be moved, as the “longing” for resolution has a positive effect upon the reader.
In other words, there’s a difference between art that persuades towards wrong and art that persuades towards right. Seeing a character fail can actually do the latter, if the reader does not want to be like him/her.
One example of this is the movie The Wolf of Wall Street, which is depressing in terms of content (so I don’t recommend for that reason), but the movie does persuade against rather than towards sin. No reasonable viewers would want to be like the main character, as he is flawed to the core and the movie shows the misery of that. But the point is that a bad character can actually make a viewer want to be good (rather than bad).
I’d also include another category: art that persuades towards neither. Some of that is neutral (e.g., an abstract painting), and some is intentional, meaning that it tries to minimize right and wrong (such as an R-rated comedy). This is also why a funny sitcom can lead a person to sin, if it minimizes wrong behavior with humor, more so than an R-rated war movie, which shows the pain of sin. They lead to different effect within the viewer.
For our purposes, I mostly want to point out that a “bad ending” doesn’t end with the character; it continues in the reader’s mind. So it’s the “takeaway” that can make art good or bad for a reader. In short, I think of it this way: What does the text, or other form of art, make the reader want?