It's kind of a confusing task... I'm trying to plan ahead for allowing the player to rebind the controller to how they see fit.
I suppose most of these controllers are nearly identical - so is the strategy to avoid using buttons that are unique to the controller, and only use common buttons among all the controller you intend to incorporate?
Or you can do it like I do in my input system and allow anything and everything to be used. You shouldn't assign ANY of the controls yourself, rather you should let the player hit the keys and buttons(and move the axes) to assign them to your actions. This way, the player can do whatever they want with the controls, and you don't have to worry about anything, like trying to satisfy all the players, as letting them do it themselves is likely the best way to do that.
There is nothing wrong with having a set of defaults, as it is a stepping stone for controls. WASD or arrow keys can be good starters for directions for example, along with X/Y axes on gamepads for that default scheme. You may want to organize your controls to make less "root" controls if needed. So something like "conjure nature" would possibly be on the same button as some other "nature" operation, where you differentiate by how long it is held, or what direction is pressed at the same time. I don't know your game so I don't know the best way, but I DO know that though it isn't necessarily that important to minimize the amount of actions, its kinda important to minimize actual inputs.
Remember that plenty of times, actions are dependent on circumstance. This is why games have a single "use/interact" button and it simply does whatever applies at the time, opening a door, pushing a block, talking to someone, whatever...all on a single input. This is situation dependent. It could also be dependent on directional input in tandem like I mentioned above. Finally, if it is not "real-time" as in you can take time to choose things from a menu, then you can make a pie menu for "nature" stuff, and use a single button to open said menu. If I knew your game better I'd have better suggestions, but the advice to limit actual inputs still applies, even if you don't limit actual actions.
As far as a good amount, I'd limit actual inputs to less buttons than you find on a standard gamepad, like an XBOX one. In reality, unless your game is just that complex, that's probably too many as well, though it can be mitigated if things are optional. Optional could be for example if you have a weapon change button, and then you supply an alternative that scrolls weapons choices backwards...that's an optional one because you can work just fine with only the first weapon change button. The same applies to "groups" like the nature stuff. You could have a button bring up the pie menu for nature stuff, and then offer other inputs for the player to assign to shortcut to a specific option on that pie menu.