Personally, depending on the scale/scope of the game, I always start with a good ol' spreadsheet.
Currently, I'm actually developing an RTS game with many different units. I've got 3 races, several units per race, with unique abilities for the higher tier units. And they're all on a spreadhseet combined with a schedule and progress percentages to keep me on track.
Adopting something like this, make a journal, or spreadsheet, of spells and skills. But first, what are the core components of your character system? Do you have health? Armor? Magic resistances? Endurance? etc...? What kinds of enemies and threats will the player face? Will the player face several threats at once? Or just one threat at a time? I ask because that could determine if you would bother with AOE spells or not, or just focus on single-target spells.
Come up with something interesting, or cool. Write down some notes about it. Maybe, in the end, you won't use it, but it will inspire you. Like the following:
Corrosive Blast (active ability)
> All enemies on screen (or certain radius from the mouse) have their armor reduced by
20% for X seconds and are slowed by
30% during that time.
Sharpshooter (active ability)
> The next target you click with the mouse takes
200% critical damage.
Powerful Swings (passive ability)
> For the duration of this passive spell, all mouse clicks do
120% damage and stagger enemies for
2 seconds.
Rain of Fire (active ability)
> The next mouse click sends down a barrage of firey arrows around a target location.
Finesse (passive ability)
> The player can safely disarm any traps by right-clicking on them.
I'm literally just making stuff up. But I've found in the past that before I start making skill, spells, or passive traits, I need a good grasp ont the core components of my character system. What mechanics do you plan on implementing? Luck? Stealth? Dodge or Block percentages?
Anyway, come up with a list or spreadsheet of ideas where those core mechanics are manipulated in some way. A long time ago, I filled an entire notebook with spell ideas for a Dragon Age:Origins mod I wanted to create. Spend a day or two and go wild and write stuff down!
Hope this helped in some way. Good luck with your project!