2 or 2.5k is probably good. Like somebody else commented, perhaps those submitting works of 1.5k+ should have to comment on works of comparable length. Or, my idea, they should have to comment on more than 1 work.
I'm sure others have expressed this opinion, but I think it would be fair if you either got rid of the word limit or capped it at around 5000 and implemented a rule that you much critique a close to equal amount of words in order to get in. My piece has 2500+ words? No problem! I'll critique another long piece or a couple of poems!
I'm not certain how long my stuff is normally at this point but 1.5K is....woefully inadequate.
Two words: combined total.
When you submit for a contest or portfolio, you are sometimes given a choice. Submit 1 piece that is X number of words or multiple pieces that add up to X number of words.
In critique terms, a critique of a 3K story is worth 2 critiques of 1.5K stories. Make people add a word count. And assign points for different poem and story lengths (reviewing a haiku vs. Paradise Lost should be worth different points). I think there's a critique bias towards shorter stuff...but if I need to earn 3 points to get my long piece accepted, heck, I might be more likely to review the longer pieces since I have experience writing novellas and can offer better advice.
And that's my second point: no novella lengths. It just doesn't work...but a way to put these authors together for private crits would be awesome! Still, don't blast the roof off the word limit, just raise it. An 8-10K short story approaches oxymorondom.
I think a points system would be far too complicated to keep up. but making people who submit longer works critique a longer work isn't as complicated and is basically the same concept, which is what I think we'll do.
Yeah, I don't want no neverending story either. we'll just see.