A brief explanation for anyone curious enough to care.

Disclaimer: described below is how the Java edition of Minecraft works. Other Minecraft editions basically function the same, but there are some differences here and there which make the final cost calculations not line up.

Enchanting details

When working with item enchantments, there are a couple of things that are at play. These are described in detail on the wiki. The details relevant to OMEO are listed below.

Prior work penalty (read: 'anvil use count')

All items start at a prior work penalty of zero. Every time you do something with an item on the anvil, the item's prior work penalty is increased. The only exception is if you purely (re)name an item; this doesn't increase the item's prior work penalty. The maximum prior work penalty an item can get before it becomes too costly to process any further is five.
If you do not know an item's prior work penalty, you can find it out by placing it on an anvil and initiating a rename -- you do not have to actually rename it;

place it on the anvil,
type another name for it,
note down how much XP levels that would cost,

look up the prior work penalty using the following table:

XP level cost	Prior work penalty
1	0
2	1
4	2
8	3
16	4
32	5

Anvil cost limit

Using the anvil costs XP levels. The total cost of the operations on the anvil depends on what you do, but it cannot exceed 39 XP levels. If the total cost exceeds this limit, you will be told the operation is "Too expensive!" Prior work penalties play a significant role in the final cost, so it's important to give OMEO the correct prior work penalties when listing your items.

Enchanting an item on the enchanting table

When giving an item enchantment(s) using the enchantment table, its prior work penalty doesn't change.

Repairing items on the workbench

When you repair an item on the workbench by combining it with another identical item or with its base ingredients, the resulting item will have more durability. However, all enchantments, names and prior work penalties present on the used items will get lost. This is thus a valid way to reset an item's prior work penalty, though doing so with a grindstone is often more beneficial.

Removing enchantments with a grindstone

When removing enchantments with a grindstone, you'll get back a portion of the XP that was present in the item. All enchantments as well as the prior work penalty will be removed (except for curses), and the item's name will stay. So, if an item doesn't have a curse on it, this could be a preferred way to remove any prior work penalty from an item.

Repairing and combining items on the anvil

You can use the anvil to combine two items or an item and a book. Doing so, the left one will be the target item, while the right one will be the sacrifice item. When you extract the combined item from the anvil, the sacrifice item is destroyed in the process. If you combine two items, the target item will be repaired in the same way as when you would have done so on a workbench. The difference is that the enchantments that were present on the starting items will combine and be placed on the resulting item (if possible). This does come at the cost of XP though. The resulting item's prior work penalty is set to the highest prior work penalty of the two items being combined, plus one additional level. Also note that using books is generally a more economical way to get enchants on an item.
When solely repairing an item on the anvil using the item's repair material (iron ingots, diamonds, etc.) the item will retain its enchantments and name, but its prior work penalty will be increased nonetheless. This way of repairing items will thus become costly quite soon and thus unsustainable in the long term. Repairing items this way is out of scope for OMEO; see the wiki for further details if you are interested.

(Re)naming an item on the anvil

You can use the anvil to give an item a name or change its existing name. This will not increase the item's prior work penalty, but it will cost one XP level plus any prior work penalty present on the item. Renaming is still possible once the prior work penalty becomes too high; the XP level required for a rename is capped at 39. It is thus most economical to rename an item as soon as possible, or to rename it while also upgrading it.
When you are combining an item with another item on the anvil and rename it as well, this rename will only cost that one additional XP level. Be careful though: when that one extra XP level would make the total anvil cost pass 39 XP levels, the combine (and thus rename) will be impossible!

Anvil cost calculations

The total cost for combining items on the anvil is a combination of several pieces;

Rename cost

If you also rename the item, this adds an extra 1 XP level.

Repair cost

If an item is going to be repaired in the process (either with raw materials or by combining broken items), this will add an additional 2 XP levels.

Enchantment cost

Each enchantment the target item already has that the sacrifice item doesn't will cost no XP levels. Each enchantment the sacrifice item adds to or increases on the target item will cost XP levels;

Target cannot have the enchantment: the enchantment is dropped, costing no XP levels. Example: looting on a chestplate.
Target already has an incompatible enchantment: the sacrifice's enchantment is lost, but still adds 1 XP level. Except when combining books with Silk Touch and Luck of the Sea or Looting, which is impossible.

Target can have the enchantment: this costs the final enchantment level multiplied by the matching multiplier for the sacrifice's type. The final level depends on the enchantment level the target already has compared to the sacrifice item;

Target doesn't have the enchantment or has it at a lower level than the sacrifice item: the final level will be the same as the sacrifice item's level.
Target has the enchantment at the same level: the final level will be one higher than the target and sacrifice item's level (keeping the enchantment's max level in mind).
Target has the enchantment at a higher level: the final level is the same as the target item's level.

The multipliers are:

Book multiplier	Item multiplier	Applicable to enchantments
1	1	Efficiency, Loyalty, Piercing, Power, Protection, Sharpness
1	2	Bane of Arthropods, Feather Falling, Fire Protection, Knockback, Projectile Protection, Quick Charge, Smite, Unbreaking
2	4	Aqua Affinity, Blast Protection, Depth Strider, Fire Aspect, Flame, Fortune, Frost Walker, Impaling, Looting, Luck of the Sea, Lure, Mending, Multishot, Punch, Respiration, Riptide, Sweeping Edge
4	8	Channeling, Curse of Binding, Curse of Vanishing, Infinity, Silk Touch, Soul Speed, Swift Sneak, Thorns

Prior work penalty

Take the highest prior work penalty from both items (target and sacrifice) and use the table below to determine the XP level cost.

Prior work penalty	XP level cost
0	0
1	1
2	3
3	7
4	15
5	31

Finding the right combinations

There are two approaches when finding out the best way to plan your enchantments: start at the individual items and build your way up to the desired final item, or start at the final item and work your way down to the individual items. In this case the latter is far more complicated to manage, while the former results in not only getting at the desired item (if possible), but also all other items that can be created. OMEO therefore uses the former option.

There is no guarantee the desired item can be made from the inputs. We want to show a list of the options coming close in that case, and not just tell that it's impossible to get there. There is also the possibility something even better can be made, which would also be nice to know about. The best way to go about this is to just spit out all relevant possible combinations, and sort that list by the matching score.

To get at all possible combinations,

OMEO places all source items the user added into an item pool.
If the desired item is not a book, and if all source items of that type have enchantments on them or have a prior work penalty, OMEO also throws in a fresh unenchanted item of that type (because that could be a better start than any of the enchanted and/or prior work penaltied source items).
It then starts making all relevant combinations between all items, putting all combined items back into the pool to continue combining those even further.
OMEO takes care to combine all items in both ways (because one way could be less costly than the other).
When combining, OMEO skips those combinations where both predecessors share a common constituting item (we can't double dip here).
OMEO chugs along until there are no combinations left to be made. Due to the nature of this process, this can however take quite a while... But the chapter below describes how OMEO deals with this issue.

Strippin' the bloat

This is unfortunately a very unsatisfying process, since the number of combinations quickly skyrockets beyond what a computer can handle. E.g., if the user added 10 items, combining these could add up to 10 * 10 = 100 extra items to consider. And these in turn could add up to 110 * 110 = 12,100 extra items, going to 146,410,000 items for the next round, etc. This thus more than squares with each pass, and there can be up to six passes needed to reach the end.

However... there are lots of corners to be cut here, making the process manageable;

We don't need to combine items when the combine cost is higher than 39, since the anvil rejects these combines as "Too expensive".
Not all combinations are possible, like combining an item with one of its constituting items earlier in the list.
Some combinations are not even a consideration because of mismatching item type (using a book as target and a non-book as sacrifice, or trying to combine two different non-book items).
Some combinations are just wasteful, like combining a lower level enchant to a higher level one (making the combine useless by completely wasting one of the items while still costing XP in the process).
When combining two items the result depends on which one you use as the target and which one as the sacrifice. This also goes for the cost. But in case they both end up on the same configuration (enchantments and levels wise), we can drop the uninteresting most expensive one.
When adding lots of e.g. Efficiency 1 books, it's far easier to lump them all together and keep track of how many we had to begin with. This effectively reduces a whole swath of items back to one item, heavily curbing the growth.
We try out all item combinations. But if we later find a certain item combination that results in the same item as one we combined before from the same constituting items, we can drop one of them if it is strictly less interesting in all regards (higher prior work penalty, higher total cost and greater number of constituting ingredients). It could turn out that we can drop a previously combined item which was already used as an ingredient for further combinations. In that case all those derived items can also be thrown away, since we will recreate them in a better way with the newly found better combination as a base. Doing this however means the progress will effectively be rewound (quite) a few steps; this is the reason OMEO's progress can sometimes temporarily count down! This is thus a feature to keep the processing time manageable, and not a bug :)
Some combinations are useless, like combining with items that only have enchantments we don't need on the final item. (not implemented yet)

When also using these extra rules, the total number of possibilities is often quite manageable.

Other options

The anvil offers extra options while working with enchantments. These are not considered by OMEO;

Some damaged tools can be repaired as well by throwing ingots, leather, etc. at them. This costs 1 xp level per ingot and increases the prior work counter for the tool. However, this cannot be performed in the same step as an enchantment combine. So, if a tool needs to be repaired this way, it should be done before or after the steps from the divination result. If done before, this must be specified by correctly specifying the tool's new prior work penalty in the form. If done after, the repair cost (incl. the final combined tool's prior work counter) should be taken into account by the user. Then again costly tools are probably getting a 'mending' on them as well, in which case repairing the tool this way is an unnecessary costly step anyway.
Damaged tools will be repaired implicitly when combined with another tool. This will cost an extra 2 xp levels per repair. Implicit repairing may happen more than once; a combined tool's durability is the sum of the target + sacrifice tool's durability, plus an extra 12%. This means a combine of two quite damaged tools will result in a still damaged tool (though less so than the original tool). Depending on when in the combine process damaged tools get combined together, this might or might not influence whether the repair would increase the total cost past the max 39 xp levels. While this could be taken into account perfectly fine, it would require the user to enter the durability of the items as well (e.g. on a 0% to 100% scale). And this adds too much clutter to the form to make it worthwhile IMO.

Obligatory disclaimer

OMEO is not affiliated with Minecraft, Microsoft or Mojang. Minecraft and any artwork or screenshots relating to these are the intellectual property of Microsoft; all rights are reserved worldwide.