I don't know if this is the type of answer you're looking for: As I've simply not looked at RPG attack/defense formula's:
But I probably have a few concepts you could try use to give a different balance:
If you become so strong that enemies become meaningless, then you could have another (negative) STAT ( EVIL, or CURSED): After all, no one likes a bully. So if you're farming low level enemies, someone or something curses you. Or curses your weapon.
So fighting: So same enemies, but a different approach needs to be taken to combat. Because a cursed weapon could hurt you, too. Or hurt your team. Maybe you can just punch your enemies: At least it'll increase your base STRENGTH. Or use some more passive magics.
(Of course, you could always embrace your evil... and that could lead to a different character development. Perhaps someone who embraces evil starts to lose members of their party until eventually they're alone. Or perhaps there is a black market for cursed weapons: So farming the weak to 'create' new cursed weapons could be an interesting side-hustle)
It's not really the answer I'm looking for, sadly... I have to agree with that. I feel like having a "punish the player for playing the game" system would just be counterproductive (the player would not just ruin their fun by making combat trivial, but also get berated for it). I've seen a few games that lets you juggle a dead enemy's corpse until it touches the ground, getting more drops for each "overkill" hit, but it doesn't translate all that well into a turn-based game. And I want the game to be set in a world where abusing the weak isn't just okay, it's done EVERYWHERE.
I'm not really familiar with the paper mario school, in the Disgaea school though I've seen quite a few attempts to mitigate the problems you described with a varying degree of success. It generally involves some kind of automatic level scaling, meaning that the enemies get stronger the stronger you get, effectively making every battle more fair.
The problem is that a naive application of this concept totally removes the sense of progression from the player, which is BAD!
To address this you enemies can be scaled differently, de facto creating enemies that have a "lower threat level" (as in, lower stats growth) making them easier to deal with in the long run as you level up but not total pushovers either, since they still scale to your level. In addition to this you can add a minimum (or maximum) level if you want some of those to be really tough at lower levels, or add more types of attacks at specific levels.
Yeah, removing the sense of progress is a bad thing and I'm not a huge fan of level scaling because it's almost always just plain bad. You might as well just not have scaling and increased stats at all... but of course, then you can't have "RPG elements" as a selling point. I'm thinking the action commands system would work mostly in place of level scaling (if you're good at the minigames you can deal damage to lv. 9999 enemies right after the tutorial, but if you mess up even once you're oneshot - and otherwise trivial foes can sneak in some serious hits if you don't focus enough when fighting them) but it's ultimately not going to feel REWARDING to fight enemies that are too low-level even if it's fun... so a bit of both probably is necessary.
I'm having some ideas for how to make level scaling work out better than Oblivion... thoughts?
- Certain enemy types would not scale at all, or have a max scaling level that they can't go past (e.g. rats, slime, other entry-level enemies).
- Story-based stuff maybe shouldn't scale at all, because it removes the "you can just grind to get more stats" fallback solution when a player really struggles with a boss.
- Recurring bosses would grow stronger each time you meet them, so maybe they would keep up with you better? If they're your self-appointed rival they might take keeping up with you more personally.
- Side quest stuff could scale with the area's crime rate and demonic invasion rate more than they scale with the player's level, so areas designated as being dangerous will FEEL dangerous.
- Maybe level scaling is based on your overall progression rather than your level, so you can get ahead with grinding if you want but you still will face stronger and stronger enemies the further you are into the game.
Another idea I just got... what if the player can turn level scaling on/off? Like, you get a checkbox that says "adjusted level?" which affects whether the mission uses the "recommended" level or scales enemies to your level? Disgaea 5 has a similar feature with the Cheat Shop that just lets you turn up enemy levels if you feel things get too trivial, but it's very hard to tell how many stars is enough but not TOO much... having things either be scaled to your level or "the intended way" lets you always get appropriately-levelled enemies if you want, but get that sense of progression if you just wanna tear through the game like a knife through butter.
Well, in Disgaeas I recall you could easily roll through the main game content (the story) with a character of LVL 100 or less (D3 I think I finished with the MC at LVL 70 or so). The rest, which is running through the random dungeon generator to max level of 9999, is just for funsies. However one problem in Disgaeas you notice even early on is how multiclassing becomes almost a must. Classes with poor hit point progress fare even worse as time passes, to the point where they start getting one-shotted by pretty much any attack. The only cure is to run them through several dozen levels of another class that has better HP progression.
It's even worse in Disgaea 5, I can assure you... to get a chance at some of the postgame content, you need to max out EVERY class to get the most base stats (class mastery bonuses is like 95% of your total stats, making the classic "grind to 9999 to save up levels" thing useless), and it means that not only don't you need a full team anymore in the postgame (just one or two really strong members), anyone will have access to every skill anyway AND the same base stats. Together with how the system basically gives you a single stat named The Stat, it removes basically all the strategy...
I personally would go similar to how I remember the Mario & Luigi [GBA] approach, (similar to how you described Paper Mario, although I've never played PM) where every Hitpoint matters significantly, and you slowly level your adventurer's stats throughout gameplay. Having a multitude of stats and occasionally giving the player an opportunity to upgrade one of them draws out the process, which in my mind doesn't seem like a negative issue. I believe it gives more weight to the upgrade process, which is positive.
You mention that the Paper Mario approach makes it hard to provide meaningful steady progression, and after reflecting on the subject, I agree. I think instead of keeping the HP in the single digits, it would be best topped out in the 30's - 50's, although that number would be somewhat subject to the length of the game.
What makes you feel the Fire Emblem approach can suffer from drawbacks of both methods at times?
Generally speaking, and maybe this is just speculation and off topic; I don't believe obstacles should get easier as you progress through a game just because your character leveled up. I believe that you should obtain more freedom the more your character grows. A stronger character should not defeat enemies easier, but perhaps it should instead allow the player to take multiple hits so that they have more room to deploy their own strategies. Freedom of choice is the fun part about games anyway, right?
EDIT:
No matter the approach of leveling up your stats, it may solve some issues noted above and make for interesting gameplay if you design it so that it's more; Stats = Freedom instead of Stats = Power. This way, you should not have the problem of, "Oh no, I totally obliterated this enemy with ease. That was too easy." Instead it would be, "I have upgraded my character so much that I have more freedom to play the game the way I want."
I'm just throwing ideas out there at this point.
Perhaps you can deploy the Paper Mario method and modify it so that different stats are connected together with "strings." Picture a spiderweb with your stats in the webs. (The web does not have to be visible in game) Attack Strength and Hit Points are connected, so that when your Hit Points are low, Attack Strength suffers and becomes weaker. When it comes to leveling up your character, you can choose to boost absolute stats, or you can choose to upgrade a string that is between two stats. If the string between Attack Strength and Hit Points is upgraded, the two stats become more independent, so that neither suffers as much when the other is low. This would be a deeper system to the Paper Mario style method that could keep a good sense of progression.
Fire Emblem Awakening's endgame has this weird quirk where HP is capped at 80 while total attack power easily goes far beyond that (with crits and Brave weapons) that essentially makes
every attack a oneshot. It changes the dynamic between meatshields, armor-tanky and glass cannon classes around in really weird ways (which probably is in no small part due to the ATK-DEF damage dynamic, where the possible difference range between the stats can get really massive).
I like the idea of "stats = freedom"! That's a way I've not really thought of them before, but I've seen it in some places.... like how Dark Souls has stat requirements for most equipment, and large penalties if you equip something without appropriate stats, so higher stats --> can use more equipment variety. Trying to make every number that influences the game into a stat that unlocks more options for you could be an interesting system... higher raw strength could let you use heavier gear, but you still need higher
refined strength to get more powerful attacks. Invincibility time after attacks, attack animation speed and stamina regeneration speed could all be stat-based to allow for different builds. Maybe even universal constants like the number of equip slots might actually be stat-based? You could have an "equipment tank" class which gets a lot of benefits from items' passive effects, but is much more vulnerable to armor-breaking and seduction (that forces them to remove all equipped items) than someone using their innate strength alone.
Power creep is always an issue. Just how powerful do you want a character to be? My personal opinion is the keep stat progression low. Let the power different mainly come from new skills. No matter what the formula for damage and defense, include some minimum ability for the enemy to do damage then later in the game you can have the same guys from before attack you in mass where they didn't before and still offer some sort of challenge.
Yeah, having a flat progression and having stat boosts be purely from a skill tree feels like the best option in some ways... it's easy to figure out the cap for how strong you can be (and in turn, the game balance), and it gives me an excuse to have tons of filler nodes in the skill tree for all those stat points, making it appear bigger and more impressive.
Tons of good ideas to consider. I'm still really torn on what I like most, since I'm fond of both of these schools for different reasons... I guess I should take the time to do some practical research as well.