It's sort of like one hand clapping. It's missing a vital component that is often key in fantasy. Deciding to bring it into your story shouldn't just be a given, however. Just as you would want a functional second hand, magic needs to make sense within the world you are creating. Before you introduce it, the writer must first decide how much exists and who has it. After you decide who has magic, you need to decide where it originates.
In Terry Brooks' Magic Kingdom of Landover series, wizards and witches have innate power. The only mortal with power is the king, who is protected by a magic talisman. The power of the talisman is known only to the king for most of the series. Before his enemies can destroy him, they must determine the source of his power and either acquire the talisman themselves or exploit the king's weaknesses.
If a fantasy society is structured around those who have magic, then you have to show why it is an advantage to have it. If magic users aren't in charge, they may be servants of those with economic or political power. Consider the traditional Arthurian legend, where Merlin is the subject of King Arthur. Merlin has the magic, but he uses it at the bidding of Arthur.
Is the magic in your story a gift treasured and respected by the general populace or is it feared as an evil? The attitude of the people in your world toward magic will help you in structuring your society.
In my Daermad Cycle, the Celdryans fear magic and seek to control it through the priestly organizations. The ordinary people are generally unaware of magic and terrified of it. Within their society, magic is mainly used secretly to manipulate others or political events. The Kindred, however, are naturally magical. It is common for them to have gifts and they exercise them publicly, but they do so within a regulated structure. While the Celdryans are terrified of the Kin's magic, the Kin recognize that it is the misuse of magic, made possibly by secrecy, that allows magic to be perverted. By having opposing views of magic in the series, I set up a lot of the tension between the two major races.
Be sure to determine what the non-magic users in your society do to compensate for their lack. Depending on the source of magic within your story, you have a number of options. If magic comes from the gods, your non-magic population may engage in sacrifices and prayer to obtain the favor of the gods. If the magic is evoked through the use of artifacts, there may be fights and quests to seek out the most powerful talismans. If it is innate, you can show your ordinaries seeking to curry favor with the mages.
While we tend to think of magic as being a large benefit to those who have it, it can add realism and depth to your story to show the downsides. In Kate Elliott's Spirit Walker Trilogy, the magister have great power, but their magic comes at a price. Cold mages cannot live with ordinaries because their innate magic kills combustion, which is a real problem in an alternate Europe still in the grips of an ice age. Moreover, they cannot themselves live for extended periods of time without warmth, so they must devise homes that are heated by distant combustion. That weakness adds believability to the world Elliott has created.
Although you can certainly create a magical system from your own imagination, the magic systems of human civilizations can be a great resource. You can mix and match and totally change what you learn, but the more you know, the more ideas you'll have. Strive for consistency. I use a continuity notebook to keep track of who does what to whom and how it is done. Every time magic -- or a certain type of magic, because you can have more than one if you are organized -- appears in your story, you should follow the same rules, so as not to confuse your readers.
A magical system is a wonderful, almost required, addition to any fantasy, greatly enhanced by considering all angles before a writer sets out to include it.
Lela Markham is the author of The Willow Branch, Book 1 of the Daermad Cycle, a series deeply steeped in magic.