Whether it's something personal, a job or a writing project the answer isn't simple, as it will differ for each thing. Also, it doesn't always have to be final. You can put something on hold for a while, then come back to it.
But how do you know?
With writing, it's usually a gnawing feeling that you're not getting anywhere. The characters aren't gelling, the world won't allow itself to be built, the story makes no sense.
Sometimes you can leave it for a while, days, weeks or months, come back to it and you'll see where the problems are.
Other times, you come back to it, realise it's bad and put it in a drawer. You hope that it'll prove useful at some point in the future, even if it's just as a bad example, showing you what to avoid.
I've just quit doing my own personal blog for now. The last one was Friday (28th September) and I decided to go out on a high and do a Friday Fun.
Compiling it has become more of a chore recently and I found it difficult to keep up with the Monday/Wednesday/Friday regularity, even though I knew what I wanted to write on those days.
It's been just under ten years since I started it and it has evolved over that time. I've enjoyed doing it, but it has become a chore.
I thought about keeping it going until Christmas, but decided that as I want to stop, I may as well do it now.
There are writing projects I have on hold and some I've given up on. I intend to go back to them and see what I can make of them, and not doing the blog will free up some time and, most inportantly, some motivation.
We can't know for sure when it's time to quit something, we can only follow our head and our heart. If it feels wrong, it usually is.
Learning when to quit is a lesson we take time over and should do.
"If at first you don't succeed, try again. Just try a different way. If that fails, give up. No sense in being a fool about it."