You can mix and match only licenses of the same edition within a single org for Sales Cloud, Service Cloud, and Force.com editions.
For instance, I could add Professional Edition Service Cloud licenses to an org with existing Professional Edition Sales Cloud licenses. However, I could not add Enterprise Edition Service Cloud licenses to an existing org with Professional Edition Sales Cloud licenses.
It is often possible to purchase specific features of other editions, rather than upgrading outright. For example, workflow rules are available in Professional Edition for an additional cost.