Comparing Solvers

Every other page in JAOT answers "what is the answer". This one answers "which solver should I run this model with".

The Solver Comparer sends the same problem to every solver you pick, with the same time limit, the same gap tolerance and the same thread count. The runs happen on one machine, one after another. Two solvers sharing a machine fight for cores, and then the seconds stop meaning anything.

There is no single best solver. A model that HiGHS finishes in two seconds can take SCIP a minute, and the next model reverses it. Measuring is the only way to know.

Two places to do it

The Solver Comparer page

Find it in the sidebar under Solve → Solver Comparer, or at /solve/compare.

1. Problem — pick one of your models, or upload a file. MPS, LP, CIP and JSON are accepted. An uploaded problem lives only inside its comparison and is deleted with it; it is never saved as a model.
2. Solvers — tick the ones to compare.
3. Time limit and Gap tolerance — every solver receives the same two values.
4. Compare.

The wait is stated before you start, because the solvers run one after another: four solvers with a 60-second limit is up to four minutes, not one.

When the comparison finishes, Run again repeats it with the same problem, the same solvers and the same limits. That is the normal thing to do after changing the model or upgrading a solver.

Reading the table

Objective and best bound together are the whole content of a run that ran out of time. A solver that stopped at 60 seconds with an answer of 1,000 and a bound of 940 is telling you that nothing better than 940 exists, so its answer is at worst 6% off. That is a usable result. The objective on its own is not.

Time and Search answer different questions. A solver that is slow because it spends the time building its own model has a different problem from one that is slow because it searches for a long time. The split is drawn as a chart below the table when it is a real share of the wait.

Nodes and iterations do not compare across solvers. Each one counts its own way — a SCIP node is not a CBC node. They explain one solver's behaviour; they do not rank two.

Times are comparable inside one comparison and nowhere else. The machine that produced them is stated above the table, and each row carries the version of the solver that produced it. A comparison you ran last month, on a machine under a different load and possibly an older build, is not a fair opponent for one you run today.

A solver that searched past the shared limit is named under the table. Every solver is asked for the same limit against the same clock, and all four normally stop within hundredths of a second of it. A solver can still overrun on a hard model, because they all check the time only between search steps. When one does, its Search time turns amber and a notice under the table names the solver and by how much. Read that row's times as "it was allowed to run longer", not as a fair loss.

A solver that cannot run your model

It still gets a row, marked Not supported, with the reason:

• the solver cannot handle integer or binary variables
• the solver cannot handle quadratic terms
• the solver is not installed on this server
• the solver cannot take part in a comparison on this server

It never reaches the queue, so it costs nothing against your quota. There is no blank cell in this table, because a blank cell reads as zero.

Hexaly is in the last group. It needs its own container image and licence, and the comparison worker runs from the base image.

When the solvers disagree

Under the table, the comparison says whether the solvers that finished actually agree:

• All solvers found the same solution. Nothing more to say.
• Same objective, different variable values. Both solvers are right. The model has more than one optimal solution, and each solver found a different one. This is normal and worth knowing before you build a process around one specific assignment.
• Different objectives. The largest difference is named. This is worth investigating: one solver stopped early, or the model has a numerical problem.

The charts

Three charts are drawn under the table, and only when they add something.

• How much each solver had left to close — the span from each solver's bound to its answer. A short bar means the solver had nearly proved its answer.
• Total time, on a logarithmic scale — a comparison routinely spans two orders of magnitude, and a linear axis presses every fast solver flat against zero. Each bar carries its number in writing, because on a log scale the length is not proportional.
• Searching against building the model — drawn only when at least one solver spent a real share of its time building. Otherwise it would repeat the time chart.

A solver that ran but does not appear in a chart is named underneath it. A missing bar otherwise reads as "I did not ask for that solver".

The solver matrix

One comparison tells you which solver is fastest on that problem. A model whose data changes every month has no single answer to that, and picking a solver from one month's numbers is how you pick the wrong one.

The matrix lives in a model's Solve tab, in the Solver matrix section. It compiles the model's JModel source against each dataset you tick and runs every row against every solver you tick: datasets down the side, solvers across the top.

It needs a JModel source and datasets. A flat or imported model has no data left to swap, so every row would be the same problem under a different name.

• One metric at a time — time, objective, gap, nodes or iterations. Pick it above the grid.
• Each cell is shaded against the best of its own row, in four steps, and only for the metrics that have a direction. There is no "better" for an objective, a node count or an iteration count, so those are left unshaded.
• Clicking a row opens the full comparison for that dataset, agreement block and charts included. Each row is a comparison in its own right.
• Below the grid: which solver came first most often, and which dataset cost the most.

The launch is instant whatever the size of the model. Each row is compiled by its own worker rather than inside the request. On a model of 22,500 variables, launching three datasets against four solvers takes a little over two seconds.

A dataset that compiles but does not fill the model fails its own row. The rest of the grid keeps running and that row stays in place saying what is wrong with it. A source that does not compile at all is refused outright, because every row would fail the same way.

Row sizes differ, and that is usually the answer. Each row shows its own problem class and variable count. Two datasets of the same model routinely ground to very different sizes, which is often why one row took ten times longer than the one above it.

Every matrix this model has run stays reachable. The newest one opens when you come back to the tab, and the picker beside the status switches to any of the last twenty. A matrix run before a model change is the only thing that says what the change cost.

Taking the table with you

Both surfaces have Download CSV and Download JSON above the table.

• CSV is one line per solver, or one line per dataset and solver for a matrix. Long rather than wide, because the grid on screen shows one metric at a time and a file shaped like the grid would drop the other four.
• JSON is exactly what the API returns, so a script reads the same shape the API reference documents.

Numbers in the file are written raw, with a dot for the decimal point, whatever your browser's language is. The table on screen follows your language; a file that did the same would put a column break inside a number.

What it costs

Each solver counts as one execution against the daily solve limit.

• A comparison of 4 solvers costs 4.
• A matrix of 3 datasets and 4 solvers costs 12 — the product, not the sum.

The matrix states the number of solves and the worst-case wait, and asks for confirmation before it starts. A comparison the quota cannot cover is refused whole rather than run halfway: half a table invites a conclusion the missing half might have contradicted.

Stopping one

Stop ends a comparison before its next solver starts. A solve already inside a solver cannot be interrupted from outside, so the run in flight finishes and nothing after it begins. Rows that never got their turn are marked Stopped. Stopping refunds no quota — the slots were charged when you launched.

Via API and MCP

Everything on these two pages is available over HTTP and to an MCP agent. See the Solver Comparison API and the MCP Overview.