For example, it doesn't end with the leads getting together; rather, one of them dies or the relationship is left ambiguous. I know that taking this approach would be bad if the genre I was aiming for was 'romance', but what if the love story is just one subplot among many? And the book was not part of a series but a standalone? Artistically, I've always preferred bittersweet endings that leave things the teensiest bit unresolved, but if I were looking to sell, would I have to change that? I'd like to know beforehand so I can change the story before I finalize it.

In line with what Fred Bob said, I think it can feel upsetting as a reader if the romance is the main plot-line and it just ends with one of them dying or whatever.

I am not against unhappy endings, but they have to be done right. I feel like the best way to have the relationship end unhappily is if there is a greater message you are trying to portray. For example:

In his literary classic 1984, (spoilers, if you haven't read it) George Orwell has an ending that is very much so an unhappy one. In part one, you learn, through Winston, what the regime is like. Through Julia and their romance, Winston believes in rebelling against the regime. When they are both captured, you hope that their love for each other is enough to keep them strong, but it isn't. They both break, and turn on each other, demonstrating that Big Brother always wins. It wouldn't be true to Orwell's dystopian future if their love prevailed enough to upset the entire regime. By the end of the novel, you feel defeated, but that is the point. You don't feel robbed of a good ending by the author--you feel upset that Big Brother is how it is, and the general dismay was the entire point of the book and its ending.

You could also use the relationship to prove a point. Maybe X and Y, who love each other, aren't good for each other. And the story demonstrates how unhealthy their relationship is, until the characters finally realize it themselves. And they end up, not together, but still better off.

I think the main thing to keep in mind is that the ending doesn't have to be "happy" but it should still convey the overall message of your story, and it should be meaningful, even if it isn't happy.

love story as main story

No one will dislike a happy ending if it was well earned. But there are readers (like me) who feel put down by an unhappy ending and avoid that author in the future.

So any kind of downer ending will loose you some readers, while a (well done) happy ending won't.

love story as side storyline

If the unhappiness is "neutralised" by a larger happy ending, then it will be acceptable to almost anybody. It will add a bittersweet tang to the happiness, but not destroy it completely. While some readers don't like this as much as a really happy ending, it doesn't completely destroy their need for happiness, especially if this is a volume in a series where there is some chance of having the bad state turn better again (see the typical ups and downs in relationships in YA series).

Note.

Many people lead hard and unhappy lives. They read (and use other media) to escape their difficult realities, and what they want is solace, hope, a dream of happiness they can cling to. If media have an unhappy ending, for these readers it is like taking away their hope of a happy ending for their current problems.

You have to have some level of happiness in your life (or be a negative creep) to appreciate a bad ending to a story that you have invested emotions and hopes (for the characters) in. If you're down, you don't want to be put down even more. And many people are very much down in their lives.

For that reason, I like to write uplifting stories. People are very grateful for that and love you for it, no matter how many faults your writing has otherwise. That is, people will give five stars on Amazon for a badly written book with a happy ending, but earning five stars with a bad ending is pretty difficult.

But I don't know if you care for reviews and sales. If you do, think Hollywood. They don't do unhappy endings.

Frankly, there should be no problem finding an audiance. A particular Young Adult in my closer family devours such unhappy, dystopious stories. She does get very upset when people die or don't "get each other" at the end, but returns to those books again and again...

Me, personally, I'm not so much interested in love stories, but I recall some of the most intense moments when reading books where people die or get separated with finality. For example, the elves in the LOTR/Silmarillion (the books) practically exist to fulfill the endless trope of being separated by time and circumstances.

Ending a book with "$LONG_TIME_FAVOURITE then went away, and was never seen again" can have great emotional impact, and give a sense of closure as well - death obviously being the final version of that. For example, the last book of Ursula LeGuin's Earthsea Quartet ends several characters in this or a similar way, partly prolongued over many chapters, and it is a very strong work, in my opinion.

One of the most well-known romance stories of all-time ends on a downer; Romeo & Juilet. Despite being a tragedy about two ill-fated lovers in a situation doomed from the start, it has remained popular enough that people continue to perform it and write variations upon it today.

Personally, I see it going one of two ways:

Firstly, it could be refreshing. You could explore how no matter how genuine a love between two people can be, ultimately people change, so their love wasn't a destined, eternal matter, just two humans who loved each other then moved on.

Alternatively, it could be extremely dissatisfying. If you build up the romance as being the sole aim/conflict of the story, then you'd best resolve it in a satisfying way. The way to avoid it is by, say, making the conflict instead be about a protagonist doubting he was capable of loving at all, then seeing this failed love as sad, but proof of his humanity.

The key here is whether or not the romance is the entire point of the plot. If it is, then it's gonna be resolved. If the aim of the novel is to subvert the idea of romance being omni-important (akin to Frozen) or exploration of love's slow death is an important theme, then making the romance fail is likely the best way to explore this.

One of my favourite love stories, if not my favourite, ends badly not just once but repeatedly, Alan Garner's Redshift tells a number of love stories, not just those of the core characters, all of which end badly one way and another. Do broader audiences have any tolerance for such a story? Pass, but such unhappy or ambiguous endings can make for extremely compelling reading.