Uncensored Models for Japanese: How to Read the Scores
Public leaderboards measure Japanese capability, but the ranking does not transfer to real tasks: one meeting-summary benchmark found a −0.52 rank correlation.
Key points
- Public leaderboards such as Nejumi Leaderboard 4 measure Japanese tasks, and the same model can appear as separate entries under different reasoning settings.
- Overall rank and real-task rank can move in opposite directions: the J-MeetEval meeting-summary benchmark found a rank correlation of -0.52, with the top overall model finishing last.
- For an uncensored build, check what the edit changed and whether the publisher reports Japanese evaluation, then compare candidates on your own seven prompts.
Contents
Search for a Japanese-capable build and you will find plenty of candidates, all labelled supported. Then the meeting notes come back in the wrong format, or a summary lands at half the requested length, and the label turns out to say very little about the work you actually do.
Public leaderboards give you a starting point: Japanese capability is measured, and the results are published. What they do not give you is a guarantee, because an overall score and a real task do not always rank models the same way. This article covers how to read the scores, one published example where the rankings inverted, what changes when the build is an uncensored edit, and the seven prompts to run before you commit.
Can you compare Japanese capability from public scores?#
Yes as a starting point, using a leaderboard that measures Japanese tasks. The one to know is Nejumi Leaderboard 4 from W&B Japan, which scores eleven areas and publishes an overall value between 0 and 1 (Nejumi Leaderboard 4).
Two cautions when you read it. First, an overall score is an aggregate: it tells you little about the areas your work depends on, so check the areas closest to your use case before the headline number. Second, the same model can appear as separate entries under different reasoning settings, so confirm you are comparing like with like.
In the snapshot of 1 September 2026, the open-weight entries near the top included Qwen3.8-Flash-Next, GLM-5.3 and members of the Qwen3.5 family. Treat this as a pattern rather than a permanent fact, and check the date on the table you are reading: these boards are updated.
Does the overall ranking carry over to real tasks?#
Not always, and there is a published measurement of that. J-MeetEval, released by Nishika as part of the GENIAC programme on 5 October 2026, scores instruction-following on Japanese meeting summaries: meet transcripts of 3,000 to 11,000 characters with several format constraints at once, and mark each output as pass or fail (the J-MeetEval announcement).
It covers 197 transcripts and 408 instructions. Across seven models in the 4B to 9B class, the rank order on J-MeetEval ran against the rank order on Nejumi, with a Spearman correlation of ρ = −0.52.
| Model (4B–9B class) | J-MeetEval pass rate | Nejumi overall |
|---|---|---|
| gemma-4-E4B-it | 83.3% | 0.6692 |
| Qwen3-VL-8B-Thinking | 80.6% | 0.7021 |
| Qwen3-8B | 76.0% | 0.6900 |
| NVIDIA-Nemotron-Nano-9B-v2-Japanese | 73.3% | 0.7111 |
| Qwen3.5-4B | 72.1% | 0.7352 |
The model with the highest Nejumi score in that group, Qwen3.5-4B, came last on the meeting task. The gap concentrates in formatting instructions: asked to compress a discussion to 20% of its length, gemma-4-E4B-it complied in 5 of 6 cases and Nemotron in none, and Nemotron's median reply was 776 characters against gemma's 1,985.
That is the mechanism to remember. A terse model can look strong in a general benchmark that rewards brevity, while summarising a meeting rewards completeness at a specified length. The overall score does not measure that. Shortlist from the score, then test the instructions your job needs.
What changes when the build is uncensored?#
An uncensored build is an edit to the base model, and the model name tells you nothing about what the edit did to Japanese ability. Read three things on the model card: what was changed (which layers or weight groups), whether Japanese evaluation is reported, and who made the edit, since the same base name from two publishers is two different models.
Where no Japanese evaluation exists, only your own test settles it. The size of the edit is a useful signal from the published audits: in a third-party comparison of Qwen3.8-27B builds, a version that added small components without cutting existing weights measured a KL divergence of 0.0389, while one that modified 841 tensors reached 1.5427. That number is not a Japanese measurement, but it shows how far an edit can move a model, and the Qwen3.8-27B audit write-up has the detail.
If you run the model locally, quantisation belongs in the same comparison: the same model at a different level changes size and answer quality together, so hold the level fixed when you compare. The quantisation explainer covers how to read the labels.
What are the seven prompts to test?#
Shortlist two or three builds, run them under the same settings, and send the same seven prompts in the same order, noting the result for each.
| Prompt | What to send | What to watch |
|---|---|---|
| 1. Terminology | Ask for a specialist term explained to a beginner in three sentences | Whether the meaning is right |
| 2. Politeness | Ask for a client email rewritten in formal Japanese | Whether the register matches the request |
| 3. Summary to length | Ask for a 3,000-character text at 20% of its length | Whether the length is honoured, and nothing dropped |
| 4. Table | Ask for three items as a table with named columns | Whether the columns and order match |
| 5. Width and form | Require full-width numerals in a text containing numbers | Whether half-width digits creep in |
| 6. Proper nouns | Ask for a summary of text with names, companies and places | Whether names and titles survive unchanged |
| 7. Reluctant topics | Send a technical question in health, law or security | Whether the answer arrives complete |
Test refusal separately from compliance: a build can start drafting and still never deliver, because the response loops in its own reasoning until the limit, a failure the audits above also recorded. Repeat each prompt a few times rather than judging on one run, and start with the prompt you use at work most often.
Should you run it locally or try a service?#
Both paths are covered elsewhere: fetching and running a build with Ollama, and the round-up of models for candidates. If preparing a machine is more than you want, a browser-based service avoids the setup, and how to choose a no-filter AI compares them; the Deepent Research page covers what ours does.
What is Deepent Research?#
Deepent Research is a browser-accessible LLM chat service built to refuse as little as possible. It keeps going with research that a standard AI would cut off mid-answer.
- It refuses nothing. We run uncensored and abliterated models with guardrails reduced as far as possible, so prompts a standard AI answers with "I can't help with that" still get a response (Qwen3.5 / Qwen3.8-27B-abliterated, depending on your plan).
- Your input is never used for training. What you look up is used only to provide the service, never to train AI or for any other purpose. Personal data is protected, and our privacy policy is public.
- Try it free. No GPU and no install, just open a browser. In the logged-out trial your conversation isn't saved, so you can start casually.
→ Try it free, no sign-up: https://deepentresearch.com
No question can stop research.
Frequently asked questions#
Where do I look for Japanese capability scores?#
Nejumi Leaderboard 4 from W&B Japan is the usual starting point, since it scores Japanese tasks across eleven areas and publishes an overall value. Check the snapshot date and whether the entries you compare share the same reasoning setting, then look at the areas closest to your work rather than the headline score alone.
Is a high overall score enough to pick a model?#
No. In the J-MeetEval study, the highest-scoring model overall finished last on Japanese meeting summaries, with a rank correlation of −0.52 between the two orders. Following length and format instructions is a separate ability, so shortlist from the score and then test your own instructions on the candidates.
Does an uncensored edit hurt Japanese ability?#
It depends on the edit. Some builds restrict their changes to specific layers or add components without cutting existing weights, while others rewrite most of the model, and the Japanese result is not uniform across them. Check whether the publisher reports Japanese evaluation, and test the candidates yourself where nothing is reported.
Does quantisation change Japanese output?#
It can. Lower quantisation levels shrink the file and tend to cost answer quality, so hold the level fixed when comparing builds and check the model card for what each label means. Our quantisation explainer covers how to read them.
Disclaimer#
This article was drafted with AI assistance and then reviewed, fact-checked, and edited by the Deepent Research team before publication.
It is provided for educational and research purposes and is not intended to encourage unlawful activity or crime.
Figures come from third-party public reports and publisher-published values and may change with measurement conditions. Japanese capability is best confirmed on your own prompts.
How this note was edited and verified
We cross-check primary sources such as official documentation and papers, and organize only what the facts support.
This note was drafted with AI, then fact-checked, expanded, and edited by the operator before publication.
Related notes
Writing With an Uncensored AI: Start a Story, Keep It Going
Tense scenes and technical drafts are where assistants stop. Here is how to write with an uncensored AI: brief it first, work scene by scene, and verify.
Why Does an AI Refuse? Four Reasons and What to Do Next
A refusal can come from four causes: a safety check, missing information, an unclear question, or a glitch or message limit. Read the message to find yours.
What Can You Hand to an AI, and What Should You Keep?
Summaries and background reading are easy to delegate. Personal data and publishing raw output are not. Here is where the line sits, and why, in five checks.
Try it yourself
Deepent Research is an LLM chat service that refuses almost nothing. Everything covered in this note can be tried without downloading anything.