Achieving Frontier Text-to-SQL with the River API

The ReViSQL Recipe

ReViSQL trains with reinforcement learning, repeating the same basic loop:

1. 01

   Sample

   Give the model a batch of text-to-SQL questions and have it attempt each question 16 times. On each attempt, the model reasons through the problem and can query the database up to five times. It uses those results to produce a final SQL query.

   Question Batch

   ????????
2. 02

   Score

   Execute the final query and score it against the gold result. ReViSQL uses VeriEQL (opens in a new tab) to catch queries that are only accidentally correct, and also penalizes attempts that ignore evidence provided with the question.

   Answers

   ✓××–×✓××
3. 03

   Compare

   For each question, compare the model's attempts against one another. Better-than-average attempts are reinforced, while worse ones are discouraged. If every attempt gets the same score, there’s nothing to learn from the comparison, so the question provides no training signal.

   Group average: (2 − 1) / 8 = 0.125

   ✓ Correct: 1 − 0.125+0.875

   × Incorrect: 0 − 0.125−0.125

   – No solution: −1 − 0.125−1.125

   No update — all scores are equal
4. 04

   Update

   Train on the model’s entire attempt, including its reasoning, exploratory queries, and final answer. The database responses provide context, but aren’t trained on. Over time, successful approaches become more likely and unsuccessful ones less likely.

   select ▏

   avg

   30%

   sum

   45%

   count

   25%

   Successful attempts raise token probabilities. Unsuccessful attempts lower them.

Each cycle follows a different question. Real updates depend on the full attempts and training objective; shaping penalties and tokenizer details are omitted.

The idea is simple: generate several candidate solutions to the same problem, test which ones actually work, and train the model to favor those approaches.

The Authors’ Code

The ReViSQL authors had released their training code (opens in a new tab), so we used it as our reference implementation on River. In our initial tests, longer runs consistently collapsed around the second or third epoch, making the reported result difficult to reproduce.

Since the training code was originally written for Tinker, we first wanted to see whether differences between training APIs affected the result. We tested this by running the same author implementation across River, Tinker, and Fireworks.

Our Implementation

Rebuilding the training code from scratch gave us a cleaner foundation for longer experiments and made it easier to isolate what was actually affecting performance. The following are some highlighted changes we made in our code to improve performance and stability.

Async RL

The released loop waits for every rollout in a batch before training. Because text-to-SQL rollouts vary widely in length, slow groups can leave training capacity idle.

Correctness and reproducibility

Grader

We sharpened the grading check inside the training loop. The authors’ grader allowed a 1% tolerance when a query returned a single numeric value. This simplifies floating-point comparisons, but also lets incorrect row-count queries receive full credit when their results are close enough to the gold answer. We tightened this check so small counting errors would not be rewarded as correct.

Kimi Response Parsing

When Kimi K2.6 hit the token limit mid-reasoning, the authors’ released code treated the unfinished reasoning as its answer and graded it. That reasoning sometimes held a draft query, so an attempt that ran out of tokens could still be graded correct.

This sent the wrong training signal. An unfinished attempt could earn the same reward as a completed one, so the model never learns to wrap up reasoning. Worse, long attempts could still be reinforced whenever a draft query happened to be correct. This would slowly lead to runaway degenerate generations on most runs.

In our implementation, an interrupted attempt has no answer and receives a −1 reward, the lowest possible score. That makes finishing within the budget strictly better.

We suspect this bug was one of the main reasons we couldn’t reproduce the reported Kimi results with the authors’ code.

Outdated VeriEQL

The authors’ released code pinned a version of VeriEQL from before a fix for a known bug with ORDER BY inside subqueries, so it “refuted” some correct queries and cost them reward. That penalty fell on a pattern used in about 3% of the training questions’ reference queries, such as picking the top row inside a subquery. We use the fixed version.

Tool-Use Demonstration

Our implementation prepends a short worked example of a tool call and final answer to the conversation history. Without it, models still follow the required format, but early responses are much longer. For example, Kimi K2.6 used about 1,000 extra tokens per answer and hit the token limit far more often at the start of training. Ablating the example did not meaningfully affect accuracy, so we keep it because it makes early training more efficient.

Optimizing for longer runs

Validation accuracy often peaked earlier than Arcwise accuracy (in five of our six strongest runs), so the late steps were chasing small improvements, and a constant learning rate proved too aggressive for them. We switched to a schedule that warms the learning rate up over the first steps and then decays it along a cosine curve, locking in early gains while holding the peak for longer.

Now that we had a reliable implementation of the recipe, we got to work probing the claims in the paper and refining the recipe for each engine on our API.

Recipe Insights

Many parts of the ReViSQL recipe had a clear impact on performance. A few stood out across our experiments.

The BIRD-Platinum Dataset

BIRD-Platinum (opens in a new tab) is a curated and repaired subset of BIRD’s training data. The authors audited roughly 2,500 examples and found errors in 61.1%, including incorrect reference SQL in more than half. They repaired the questions, supporting evidence, and reference queries to give training a more reliable reward signal.

The verified training data had the clearest measurable effect on performance. We trained Kimi K2.6, GLM-5.3 Flash, and Qwen3.8-27B with the same algorithm and questions on two versions of the annotations: the paper’s cleaned version and BIRD’s original. Across the first two epochs, the cleaned annotations scored 9.4 points higher on Arcwise-Plat on average.

With the original annotations, performance actually degraded over training, with all three models falling 7 to 10 points from their early peaks. For Kimi, the advantage also carried to Spider2-Lite, where the cleaned data scored 10.4 points higher.

We noticed that validation accuracy still seemed to plateau across runs in the 92–93% range, which suggested there may still be issues in the BIRD-Platinum dataset. We investigated and found there was still a considerable number of inconsistencies and errors remaining.

What's compelling about this categorization is, the 6.5% bucket of Incorrect Gold SQL is about the same gap as the validation accuracy plateau when training on BIRD-Platinum.

The BIRD-Diamond Dataset

We invested some considerable effort into building BIRD-Diamond (opens in a new tab), a further cleaned version built on top of the authors’ BIRD-Platinum work. We were optimistic that we would see more gains from this additional dataset cleaning, but surprisingly, we actually saw an average of about 1% lower eval scores when trained on the Diamond dataset.

We dug into our runs to understand why this second dataset cleaning harmed our eval strength. Our first clue was in the validation scores.

While fixing the incorrect gold SQL closed the gap for the models to saturate the training set, the disambiguation fixes removed some of the complexity of the dataset. We can see above that the learnable delta is essentially the same between BIRD and BIRD-Platinum. For BIRD-Diamond, however, it appears we trivialized about 4% of the problems through cleaning issues beyond incorrect gold SQL queries.

The original datasets teach the model to be more robust to such defects in the questions. We found that the 1% delta on eval from BIRD-Diamond runs vs BIRD-Platinum was dominated by a specific class of question defects: evidence errors. Training on BIRD-Diamond teaches the model to be extremely trusting of the evidence, while training on BIRD-Platinum teaches some skepticism and robustness to handle such errors in the Arcwise eval set.

Reward Shaping

Reward shaping adds smaller signals on top of the outcome reward to steer how the model reaches its answer. ReViSQL adds two.

VeriEQL Penalty

VeriEQL (opens in a new tab) asks a stricter question than the benchmark grader: not just whether two queries match on the real database, but whether they would return the same result on any valid database. It searches for a small counterexample where they disagree. If it finds one, ReViSQL treats the model’s answer as accidentally correct. In practice, that also makes VeriEQL a kind of style reward, favoring queries that follow the same conventions as BIRD’s gold SQL.

VeriEQL produced counterexamples for 29% of the model’s benchmark-correct answers, close to the paper’s reported 32.8%. But most did not survive our benchmark-aligned check. Put differently, for every 100 answers the benchmark marked correct, VeriEQL found a counterexample for about 29, but only about 5 still produced different results under the benchmark’s own grader.

We also tested three other SQL equivalence checkers, SpotIt+, ParSEval, and Polygon, and none was a clear improvement over VeriEQL. The investigation surfaced a paradox: the stricter a checker’s definition of “the same result,” the more often it rejected benchmark-correct answers over technical differences the benchmark itself does not count. In other words, a more aggressive checker does not necessarily produce a better reward signal; it can just add more noise.

Evidence Process Reward

The evidence process reward behaved differently. Models learned the required format quickly: fewer than 1% of rollouts skipped the evidence steps after the first epoch. That shows the reward successfully teaches compliance. We suspect it also interacts with the eval questions whose evidence is misleading: drawing more attention to the evidence may encourage the model to question it rather than follow it blindly.

Measuring the Effect

In the paper's ablation, removing reward shaping lowered Arcwise-Plat accuracy by 2.8 percentage points. In our own ablation runs on GLM-5.3 Flash, the run without the VeriEQL penalty peaked 0.8 percentage points lower than its shaped counterpart, and the run without the evidence reward 1.4 points lower.

The chart below follows both pairs of runs, showing each run’s accuracy at every checkpoint.

Shaping appears to be beneficial in general, and most pronounced in the 2–3 epoch window, where runs tend to start to plateau. Additionally, the evidence reward appears to have a stabilizing effect on the runs, which is interesting.

We will revisit shaping again below, as it had some interesting interactions with some of the recipe modifications we tested.

Reasoning Settings

We expected higher reasoning settings and more tokens per turn to improve performance, but the effect was more nuanced. DeepSeek V4.1 Flash benefited from its highest setting with longer turns: at max effort and 8,192 tokens per turn, it reached its peak in about half the steps it needed at 3,072. Qwen3.8 showed no difference between medium and xhigh at 8,192 tokens per turn. Our recipe defaults to xhigh to be safe, but the setting does not seem to have a strong effect on performance.

Another surprising result was that GLM-5.3 Flash did slightly better at 3,072 tokens per turn than at either longer budget we tested, and so did Nemotron 3.5 Lightning. This pattern does not hold for every model, so these settings should be tested and tuned for any new model added to the River API.

Extending the Recipe

Once we could reliably reproduce the paper’s results, we began looking for ways to extend the recipe and push performance higher.

Training Hints

During training, when all 16 attempts at a question fail, the step gets no learning signal from that question. We added a fallback to try to recover one.

We re-sample the question with a hint added to the prompt and use those results instead. We pre-generated hints (opens in a new tab) with Kimi K2.6 using the question, its evidence, the schema, and the reference query.

After the first epoch of our final runs, about 6% of questions needed a hint, and surprisingly, that share barely varied across models.

While that share was similar across models, the response to hints was not. After receiving a hint, 85% of hinted re-runs came back 16/16 correct on GLM-5.3 Flash. Yet for Nemotron 3.5 Lightning, only 43% of hinted re-runs came back 16/16 correct.

Ideally with hints, you want the model to only recover between 1 and 15 correct attempts on a question so that there is signal to learn from. So hints give stronger models less to learn from: they often solve the hinted question outright.

Hints were one of the strongest gains we found from changing the recipe. The run with hints stayed ahead for most of training, leading at 26 of the 33 evaluations both runs have reached, and its best score over those steps was about a point and a half higher, 92.8% against 91.2%.

Hints matter because they are the first part of the recipe that can teach the model something beyond what it already knows. Reinforcement learning in the ReViSQL recipe works by making the model more likely to choose correct answers it can already produce, so a question it never answers correctly gives it nothing to learn from. A hint lets the model reach a correct answer on those questions, and what it learned there carried over to evaluation, where hints are never shown.

Our hints were generated by a model with access to the question, evidence, and gold SQL. In practice, many appear to be too strong, making the solution too easy to recover. A better hint set, ideally human-written and limited to smaller, more targeted nudges, could make this fallback more effective.

Learnability Weighting

Most training questions stop teaching the model anything early on. About 60% of questions were already solved in all 16 attempts on their first visit. With a standard curriculum, the share of questions with signal falls steadily as the model solves more of the training set, so later steps spend most of their compute on questions that teach nothing. Learnability weighting counters this by steering training toward the questions that still have something to teach.

We enable learnability weighting at experimentally-tuned values for our recommended recipes.

BIRD-Platinum-Hard

After the first epoch, most questions come back 16/16 on every visit. A natural next idea is to drop them altogether and save the inference. We tried this two ways: training each model only on the questions it had not solved 16/16 on its first visit, and training on a shared set we call BIRD-Platinum-Hard (opens in a new tab), the 689 training questions that most models could not solve fully on the first visit. A pass over BIRD-Platinum-Hard costs a third of the questions sampled per epoch.

Behavior on this dataset was different. Performance rose much faster early in training, then plateaued sooner. It was also less stable for models like Qwen 3.6, as removing easier questions weakened the baseline that helps keep the model from drifting too far.

The reduced exposure to fully solved questions also weakened the effect of the VeriEQL shaping reward. Under the base reward, these questions provide no learning signal because every attempt receives the same score. Shaping reintroduces differences within them, giving the optimizer useful signal. Because the Hard dataset removes most of these solved questions, there are fewer opportunities for shaping to help, and peak evaluation performance was lower with shaping enabled.

This result also suggests that learnability weighting should be careful not to suppress the solved cohort so aggressively that it removes useful shaping signal.

Recipe Cards

Every model result in this post was trained on the same core recipe. Its preset changes only a handful of settings, and everything else is shared. Pick a model to see its card.

¹ Expert routing is only available on MoE models

Reflection on Results

One of the most exciting results was how consistently the recipe worked across models. With only small per-model adjustments, the same recipe improved six models from five families by 9 to 36 points on Arcwise-Plat. What surprised us most is where that gain comes from. A training group only teaches the model something when some of its 16 attempts succeed, so most of the improvement comes from reinforcing answers the models could already produce, even if only rarely, until they produce them reliably.

However, this left us with a question: if training mostly sharpens what each model can already do, why did models that started far apart all end up in the same place? Five of the six models finished within about a point and a half of each other on Arcwise-Plat, between 92.0% and 93.4%, even though their base models started anywhere from 71% to 83%. On Spider2-Lite, the same five models stayed spread out, finishing between 50.4% and 61.5%. That contrast made us ask whether Arcwise-Plat itself had become the limit, so we looked at how often each question was solved across 979 evaluations of those five models.

The chart shows, for every question, how often the model that handles it best gets it right in evaluations from step 60 on. Nearly 90% of the questions are solved reliably, in at least 90% of evaluations. The next 7% of questions seem to be reachable, but much less reliably, with the models capturing only a fraction of them on any given checkpoint. While around 3% of the benchmark looks out of reach for any recipe.

One particular class of questions seems to explain a lot of the behavior we saw late in the run.

Ambiguous Questions

Evaluation scores tend to move in small increments, but those shifts hide much more activity underneath. Between evaluations five steps apart, 26 questions change outcome on average. Most changes cancel out as answers flip in both directions, so the score reflects only the net difference.

Much of that churn comes from a recurring pool of questions rather than being spread evenly across the benchmark.

When we investigated these questions, we found that most had ambiguity with multiple valid interpretations. When the model learns to favor one interpretation, it unlocks some questions, while losing others. This also explains why some checkpoint evals are stronger than others. The models settled on more net-positive interpretations at those steps.

If the same ambiguity is interpreted differently across questions, no single reading can match them all. Some portion of the benchmark is therefore unreachable, even for a perfect model.

Final Thoughts

A stronger model might still push Arcwise-Plat a little higher, but we are probably close to where this benchmark saturates. Spider2-Lite, on the other hand, is still far from saturated, and our trained models rank roughly in the same order as their base models did.

The clearest gain in this recipe came from BIRD-Platinum’s cleaned annotations. But most of those questions are already easily solved by the strongest open-weight models. That leaves only a shrinking minority of the dataset providing useful training signal. Pushing further on harder benchmarks like Spider2-Lite will likely require harder training data annotated to the same standard.

Getting Started

View this recipe on GitHub →

We encourage you to try this recipe on your own data.

Adding your own data is straightforward. Each example needs a question over one of your databases paired with the correct SQL query. You can often bootstrap this data from existing production query logs, then write the corresponding questions afterward.

Evidence is optional, and we generally recommend leaving it out when adapting the recipe to your own data. BIRD-Platinum and BIRD-Diamond become harder without evidence, and that setting is also closer to how real users behave. They rarely specify the exact columns, joins, or formulas needed to answer a question.

If you want more training volume, you can also mix in BIRD-Platinum (opens in a new tab) or BIRD-Diamond (opens in a new tab). When using BIRD data, we recommend disabling shaping or applying it only to your own examples so the model is shaped toward the style you actually care about.

If you want to build on this recipe and would like a hand, contact us.

References

Yuxuan Zhu, Tengjun Jin, Yoojin Choi, and Daniel Kang. (2026). Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineering. arXiv:2603.20004v4.

Paper (opens in a new tab) · ReViSQL code (opens in a new tab)