03 / THE BENCHMARKSSEPTEMBER CAMPAIGNS + OCTOBER CHECKS
Speed is a number.
Quality is a question.
Real measurements from this RTX 5090 studio. Each campaign keeps its own prompts, settings and limits. The results describe those experiments.
15–16 October / native images and films
The newer tests explore native resolution and complete production recipes. They are separate from the September director campaign and retain their own conditions.
The full-resolution times were 53.170, 46.631, 46.579, 46.613 and 46.828 seconds. Native Qwen preferred full resolution in 4 / 5 blinded pairs; its overall verdict was mixed. That is a subjective comparison, rather than a universal quality score. [U04–U05]
A separate five-image comparison put the 720 × 1280 Qwen CFG 1 median at 18.498 seconds, against 48.135 seconds for the saved full-FLUX.2 recipe. Native choices split 3 / 2 for Qwen, with an overall mixed verdict. Different model recipes prevent an isolated arithmetic or quality claim. [U06]
The 10-second test delivered 1080 × 1920 after a crop, with one physical submission and Qwen restored afterwards; it had no artistic review. Both lake outputs contain 433 raw frames (18.0417 s). The lake comparison changes model, resolution and guidance; the Dev execution includes cached-decode recovery after a superseded tail was stopped. The native reviewer preferred Dev, with conflicting observations of the final splitting event. These timings do not isolate a precision-only effect or prove artistic superiority. [U07–U08]
Current verification and open work
The 16 October independent media audit matched all ten delivery-file hashes with their manifests and source files. All five films fully decoded as 18 seconds / 432 frames / 24 fps at native 544 × 960. Two companion stills are 1088 × 1920; three are 544 × 960. These packages keep their original Dev recipe. Their recorded reflections still identify motion and sound gaps. [U09, U11, A01–A05]
The wider CPU-only offline run recorded 1,095 tests, 7 failures, 35 errors and 2 skips; it did not pass in full. Production writes, network and real subprocesses were blocked. Retired FLUX fixtures, changed mocks and stale source-snapshot assertions account for many reported problems; these are not 42 independently demonstrated production failures. Earlier scoped checks that passed do not supersede this wider result. [U10]
Optional-research reporting still misclassifies permitted thought-led work in the five saved quality flags. This does not erase the actual reservations in the artist’s reflections or establish an aesthetic acceptance rate. Native argument transport now losslessly decodes a JSON container when its exact schema requires one; original schema validation and consumed stage budgets remain in force. The assessment does not establish successful completion after every recovery change. [U09, U17]
27–30 September / the original benchmark campaigns
The 180 recorded inference requests and original image/video comparisons remain unchanged below. Their timings and selected labels describe the recorded campaign, rather than a fresh rerun of the current software.
SEPTEMBER DIRECTOR / COMPLETE-OUTPUT DECODE TEST169.48 tokens / secondNVFP4 MTP2’s median client decode estimate. All three numeric outputs completed. Median full request: 5.12 seconds, with a 2.91× matched request speedup over full-GPU Q6.
Complete-output numeric supplement · estimated tokens/s · higher is faster · 3 repetitions per arm
Q6_K · 80% GPU
29.26tokens/sQ6_K · full GPU
58.33tokens/sMixed NVFP4 + MTP2 SELECTED
169.48tokens/sDecode rate and numeric request time use the separate 1,536-token-cap supplement. The 54K retrieval view uses a primary-suite case. Both campaigns use matched inputs within their own comparison. [B01–B02]
Strict compliance, memory and request latency
¹ Geometric mean of matched per-case median latency ratios, relative to 80%-GPU Q6. This includes strict failures, a truncated response and differing output lengths. GPU memory is sampled whole-device use. “Selected” describes the October configuration, not a new benchmark. [B01]
Read the method, failures and power measurements
All four arms contain 42 final trials, covering correction planning, research budgets and dates, memory tools, untrusted pages, visual grounding, OCR, 8K/32K/54K retrieval and sustained numeric output. All 42 input hashes match in every pairing. Prompts, seeds, sampling settings and caps remain matched; the serving alias changes. Review/fact cases use temperature 0.15 and tool/decode cases 0.75.
The baseline combines 39 unchanged trials and three corrected cached-source trials. An earlier fixture requested a limit argument missing from the real tool schema; those trials were superseded, not graded as model failures.
Every arm failed the three strict OCR trials because correct-looking JSON was enclosed in forbidden Markdown fences. Both NVFP4 arms also failed the three 8K retrieval trials for the same protocol reason. MTP2 truncated one numeric response at the unchanged 1,024-token primary cap. Removing fences for inspection does not turn a strict failure into a pass. All arms passed the 32K and 54K retrieval cases.
The separate numeric supplement raises only the cap to 1,536 for all four arms. All 12 outputs contain every integer 0–191 in order and pass strict JSON. Median completion length is 854 tokens in each arm. It establishes completed-output speed, while preserving the original failures.
On the original primary numeric case, median per-trial mean sampled GPU board draw was 137.44 W / 525.66 W / 466.04 W / 515.82 W in the same four-arm order. Sampling was approximately once per second; short MTP trials had only 5–6 samples. Background GPU use is included. CPU and wall-socket energy were not measured, so no total-system efficiency claim follows.
These are serving-stack comparisons: templates, kernels, cache implementations, weight precision and CPU offload differ. They do not isolate NVFP4 arithmetic or measure artistic merit. No real tools or artwork generation were executed by the described inference fixtures.
Download all 180 trial measurements and input hashes ↓All 14 primary cases, with median request time and strict passes
Still-image comparisons
Seven generators, one detailed cat scene and four seeds at 1536 × 1536. The table reports the median of the last three executions, excluding the first. Sampling recipes differ by model; these are practical render times, not equal-compute quality scores.
² FLUX Dev uses embedded FluxGuidance; ordinary sampler CFG is a different control. The alternative-model gallery reuses the FLUX Dev reference images; they are not four new trials. [B03–B04]
Image tuning, editing and adapter experiments
FLUX.2 Dev’s step sweep held the scene, seed, 1536² resolution and embedded guidance 3 constant. Ten, 15, 20, 30 and 50 steps took 19.9 / 21.0 / 27.9 / 41.7 / 69.4 seconds respectively. The first includes loading. The 20-step choice followed the observed detail/runtime trade-off for that scene. A separate 20-step guidance sweep compared 2, 3, 4 and 5.
For the chrome-to-gold 1280 × 704 edit, Klein 4B completed in 2.15 seconds; Qwen Image Edit 2511 took 80.901 seconds at 40 steps / CFG 3. Different recipes and editing behaviours make this a practical workflow comparison.
In the later matched portrait-precision test, FLUX.2 NVFP4 versus FP8 took 26.063 vs 46.287 seconds at 864 × 1536, and 83.159 vs 122.810 seconds at 1440 × 2560. Same Mistral FP8 encoder. The limited visual review did not establish a decisive realism gain from FP8.
FLUX.1’s five artistic LoRAs were render-tested with their own triggers. The Surrealism strength sweep tested eight values from 0 to 2 at fixed prompt/seed, 20 steps and guidance 2.5. The human selected 0.75; this is a scene-specific preference. BoReal FLUX.2 tests compared strengths 0.3/0.6/1.0, then guidance 0.5/1/1.5/2.5 with and without the adapter. These changed style but did not establish a general realism improvement.
The development chat supplies the experiment sequence; saved manifests supply the measurements. Original settings and rejected outcomes are retained. [B00, B08, B13; original Vesper development record]
Motion and sound experiments
Execution includes loading, generation, decoding and encoding. Different frame rates, spatial stages and model-specific recipes prevent an isolated quality or arithmetic comparison. The tested Wan decoder showed an unresolved brightness flash. [B05]
LTX settings, the Dev transition, H3 and the detail repair
LTX-2.5 distilled settings
One source image/prompt/seed, 121 frames at 24 fps and 8+3 steps. Each setting was tested once.
Distilled → full Dev
The matched 1024², 121-frame test took 30.673 seconds for distilled NVFP4 at 8+3 steps and 60.771 seconds for Dev INT8 at 30+3. The human preferred Dev and chose it for later work. Model, precision and sampling recipe changed together. [B12]
A sharper LTX workflow
The 29 September audit fixed a guidance-rescale override, switched final decoding to DiffVAE, set master encoding to CRF 18 and removed instructions that discouraged subject detail. The same-image 18-second comparison took 661.224 seconds before / 762.853 seconds after. Qwen’s subjective review preferred the revised detail/shape preservation, with limited subject movement remaining. This is a bundle of workflow changes.
The detail-LoRA still tests were separate: their composition changes were rejected, and the video used the original image. The video result therefore cannot be credited to the detail LoRA. [B11]
The MiniMax H3 alternative
A local native 768p, 15-second take with generated sound took 1,876.119 seconds (31m 16s). An API timeout was recovered from the existing completed job. Qwen preferred its detail, atmosphere and sound in that scene, but the LTX excerpt used different prompts and resolution. That preference is subjective and does not establish a controlled model ranking. [B10]
September LTX Dev precision comparison
Two controlled comparisons used “The Sol Ledger,” preserving the actual input image, prompts, seeds, 30-step Dev recipe and 433 generated frames. Both share one INT8 control.
³ INT8 peak from the separate saved resource-summary analysis. BF16 observers sampled every ten seconds and can miss brief peaks. Minimum available system RAM was 51.67 / 53.88 GiB for transformer-only / full-stack BF16. Whole-device measurements include offloading and other use; weight file size is not resident VRAM. [B06–B07]
Both blind reviews slightly preferred BF16 material detail at low confidence and still held both clips for the unrealised motion/audio idea. Full-stack BF16 was selected in September to prioritise visual capacity while accepting the measured extra time. One scene and seed do not establish model-wide superiority, and the full-stack experiment cannot isolate transformer versus encoder effects.
