@hdctambien had lot of good ideas, I'll give some of my opinions:
The kill-everything spells are the balance against deathstars. They are crowd control. It is dangerous to put all your points into a mega unit because a single spell could kill almost all of them. But, casting that same spell on a smaller unit might not be worth the risk/reward (ie, miscasting could cost you more than you gain from deleting a 200 point unit, but it might be worth it to kill a 600 point unit)
The problem with the big spells is that they give you a 16% (or more) chance of killing characters, which is usually worth it (especially in a game with lots of practically unkillable characters)
There are other ways to fix this same issue in the game that aren't some arbitrary "deathstars aren't allowed" rule and nerfing the #6 spells.
You could allow Look Out Sir rolls to be given to characters that would otherwise by auto-killed by a spell. That gives all characters in a unit a 2+ ward vs them, which is pretty good protection. Also, characters nearby a similar unit get a 4+ ward, but characters riding monsters or going it alone put themselves in a dangerous spot.
This is tough, since armies have such different capabilities and strengths. I think we could move away from any kind of Herohammer. Make characters tough, but in no way unkillable. Age of Sigmar is a bit mixed pack with this: on one hand, it greatly emphasized the role of heroes, but on the other hand, they have lots of wounds, but can be brought down with sufficient force. I do like this. Look out sir saves would make the characters too survivable and penalize the monster riding characters even more. I think only a handful of monsters are worth it to take for a character, and those are the ones we see the most on the table. Balancing the characters would help other monsters to see the battle, as well.
Also, the miscast table should be adjusted so that it is always more dangerous to cast the more expensive spells. The best suggestion I've seen for that is to reorganize the miscast chart so that all the really bad effects are at the top end and the low end are weaker. Then when you roll for miscast you roll 1d6 + X where X is the number of dice you used to cast the spell. So if you throw 6 dice at a spell you miscast will be between 7 and 12, but if you only throw 1 die at a spell, your miscast can only be between 2 and 7.
Maybe the chart looks something like this:
2 wizard can't cast any more.
3-4 casting wizard takes s6 hit.
5 -6 s6 small template. d6 dice are lost.
7 all wizards take S6 hit. d6 dice are lost.
8 -9 s10 small template. d6 dice are lost.
10 - 11 Wizard loses d3 levels and can't cast any more.
12 Big template S10, wizard dies.
Now, throwing 5 or 6 dice at a spell is really risky so it better be worth it to attempt a #6 spell. And the look-out-sir rule removes the viability of the #6 spell to work against characters. BUT, it also gives you the opportunity to trade your wizard (via bad miscast results) to try to kill that unkillable flying character that has a rerollable 1+ save and 3+ ward that rerolls 1's.
This is brilliant! I wish you had made this into another post, so I could have liked it separately!
The other issue with deathstars is character walls. These could be fixed a bit by allowing attacks against non-character models even if you are only in base contact with characters. basically, the targeting rule would change to something like "You can always attack the regular models in a unit you are in contact with, and you can choose to attack a character with any model in base contact with it."
This is also quite simple but very valid point.
Also, the challenge rules need to be fixed a bit. A character on a dragon shouldn't be able to be "broken" because they completely decimated a champion in a challenge. Maybe having challenges only exist as long as both models are alive will fix that. So if a character kills the champion in the challenge at his initiative step, his dragon can still attack/thunderstomp the unit at it's initiative step (and the regular troops could still attack the character/dragon if their initiative step comes after there-is-only-one in the challenge)
Or, maybe a unit's banner/ranks can't be used towards combat resolution if they don't actually fight (ie, if the only actual combat was in the challenge, then only combat resolution generated from the challenge counts towards who wins the combat)
Or maybe, single characters that are in a challenge and win count as stubborn when rolling leadership.
In 40k, the characters in challenge are no longer "immune" to attacks coming from outside the challenge. If the troops have no one else to fight (a Daemon Prince attacking a Necron Overlord with his group of Warriors, for example), the troops can also attack the single assailant (or maybe they can attack it anyway, even if there are other targets. I'm not completely sure). Would this work in Fantasy?
I like the lone challenger who wins the challenge being stubborn or even unbreakable. If combined with the 40k ruling above, the character can still be killed by overwhelming force, but it won't just break because the Skeletons have more ranks...
I'm not really bullish on a rewritten 8th/7th edition working in the long run. 8th really has a lot of issues (lots of them can be fixed, by making the radical change of "only 1 magic item per character" or "remove all the BRB magic items" ... but I don't think anyone is on the same page with me on that) But I think the balance between the #6 spells and hordes is pretty good, as long as the risk or magic is appropriate and the side effect of auto-killing characters is mitigated.
At first I liked the idea of having just one magic item per character, but then I started thinking how unbalanced some armies are with regards to the mundane equipment/other perks. For example, Tomb Kings characters are usually more supporting ones than super-killy. They have no access to Heavy Armor, other than BRB items. They are also one of the few models in the whole game who are extra susceptible to fire (taking two wounds for each unsaved flaming wound), so the 2+ Ward against flaming attacks is a must. My point is, if the amount of magic items would be so severely restricted, more care should be put to balancing the characters across the armies.
And my 2 cents on cannons: Keep everything about cannons the same, except it only deals d6 wounds on the spot that it lands, and anything that is hit by the bounce only gets 1 wound. That way it is still dangerous, but more random (10" from the back of the base is no longer a viable strategy). It still does the same amount of damage to a ranked unit it bounces through (Monstrous Infantry excepted). Also, having the cannon randomize when it hits a rider/mount would be good (however, if you're rewriting things just rewriting all the Monster Mounted units to a single profile would be a better choice)
This sounds good to me. It can still be the monster/super character killer/wounder, but don't just mow everything in it's path. Or maybe change it so that it does d3 wounds after the bounce. This way it would still be useful against Monstrous Cavalry, that are quite problematic in some armies.
I also like
@GreenMachine's suggestion about capping the Ward saves to 4+. The armies that can get a better save than that are so few (Sorry, NIGHTBRINGER), and as you can see from this whole post, I'm an advocate of Character Enfeebling Movement.