Summary: NVIDIA and Hugging Face connected NeMo Automodel, a distributed training library, directly to the Diffusers ecosystem so Hub models can train and return without checkpoint conversion. The July 17 joint post describes six flow-matching recipes, YAML-driven parallelism, full or LoRA fine-tuning, and pre-encoded VAE latents. Its current limits are the six covered models and Kubernetes orchestration still on the roadmap.
GPU count was never the whole difficulty of fine-tuning image and video models. Checkpoint conversion, data preprocessing, parallelism setup, and guiding the trained weights back into an inference pipeline — each layer is fertile ground for bespoke glue code. On July 17, NVIDIA and Hugging Face published a joint post wiring NeMo Automodel, the distributed training library, directly into the Diffusers ecosystem. The pitch compresses to one line: models enter training from the Hub untouched, and they leave the same way.
A training layer that demands no conversion
NeMo Automodel is NVIDIA’s open-source, PyTorch DTensor-native training library within the NeMo framework family. The integration goes the Hugging Face-native route: aim pretrained_model_name_or_path at any Diffusers model ID on the Hub and training begins — loading happens through Diffusers model classes, and generation reuses Diffusers pipelines.
The property that matters more is the clean round trip. A trained checkpoint loads straight into DiffusionPipeline for inference or pushes back to the Hub for sharing; downstream tooling such as quantization and LoRA adapters carries on working. The training framework no longer imprisons the model inside a separate format ecosystem — and that is the most practical shift this collaboration delivers: production-grade distributed diffusion training, applied to any Diffusers-format model on the Hub, with no checkpoint conversion and no model rewrites along the way. Everything ships open source under Apache 2.0, documented in the Diffusers training guides.
Parallelism as a declaration, not a rewrite
FSDP2, tensor, expert, context, and pipeline parallel arrive here as configuration options. The same training program and recipe structure stretches from one GPU to a multi-node cluster via YAML edits alone — no model-code rewrites per parallelism strategy.
Training modes occupy the same structure: full fine-tuning chases peak quality, LoRA-style parameter-efficient fine-tuning chases peak efficiency, and the two share a single recipe stack instead of running as separately maintained paths.
Six ready recipes and where the hardware bar sits
Six ready-to-run fine-tuning recipes ship today, every one a flow-matching model — the current hard boundary; pipelines built on other training objectives sit outside coverage for now. Training happens in latent space over pre-encoded VAE outputs, while the data side relies on multiresolution bucketed loading that groups batches by resolution — a natural fit for image datasets whose sizes refuse to behave.
| Model | Modality | Notes |
|---|---|---|
| Wan 2.1 T2V | Text-to-video | The 1.3B version fits a single 40GB A100 |
| Wan 2.2 T2V A14B | Text-to-video | MoE: 27B total, 14B active per step; no LoRA recipe yet |
| FLUX.1-dev | Text-to-image | The model in the official walkthrough |
| FLUX.2-dev | Text-to-image | 32B, the largest of the six |
| HunyuanVideo 1.5 | Text-to-video | — |
| Qwen-Image | Text-to-image | — |
The scale spread deserves a pause. Wan 2.1’s 1.3B version squeezes onto a single 40GB A100, so a small team can begin on one card; FLUX.2-dev climbs to 32B; and Wan 2.2 A14B is a MoE design with 27B total parameters but only 14B active per step — the savings land on compute, not on cluster-configuration complexity, and it still lacks a LoRA recipe.
The workflow: pre-encode first, then run the recipe
The recommended install is the Docker container nvcr.io/nvidia/nemo-automodel:26.06, with PyTorch, TransformerEngine, and the rest of the CUDA stack prebuilt; or simply:
pip3 install nemo-automodelWorkflow step one is pre-encoding the dataset: VAE latents and text embeddings get computed once into a cache, and every subsequent training step reads from it rather than re-encoding. The preprocessing itself distributes across all visible GPUs. The economics only truly show themselves under repeated training: encoding happens a single time, and each later experiment or hyperparameter sweep reads the cache — the larger the dataset and the more rounds of experiments, the more repeated compute you bank, at the price of one extra up-front step and the storage the cache occupies.
The official walkthrough full-fine-tunes FLUX.1-dev over a 78-card Rider–Waite tarot dataset: samples bucketed at 384×640, 8-way FSDP2, 200 steps. Generation leans on a fixed seed for the control comparison — prompts carrying the trigger token trtcrd summon the learned tarot style, prompts without it fall back to the base output, so the effect traces to the fine-tune rather than to sampling luck.
How to read the benchmarks
The official figures come from a single node of 8× H100 80GB (NVLink), reported as the mean plus-or-minus sample standard deviation across three steady-state 10-step windows. FLUX.1-dev full fine-tuning (FSDP2) sits around 0.902 s/step, 4.44 images/s per GPU, and 63.88 GiB peak memory; the same model on LoRA r64 (DDP) reaches 6.72 images/s per GPU. The image benchmark set is 256 captioned samples; video runs use 512×512×49 frames.
Treat these numbers as a way to compare recipes within one environment — never as speed promises for other hardware. Lay the two rows side by side and a selection rule falls out: on the same model and hardware, LoRA r64 delivers higher per-GPU throughput (6.72 images/s) than full fine-tuning (4.44 images/s), while full fine-tuning purchases a complete weight update. For style or domain adaptation, throughput and cost structure both argue for LoRA; full fine-tuning budgets exist for pushing model capability somewhere new.
Trade-offs, limits, and builder guidance
Limits, stated plainly: flow-matching models only; recipes covering exactly six models; multi-node orchestration riding an HPC workload scheduler today, with Kubernetes still on the roadmap. The extension design, though, is tight — a new model needs only a data preprocessing handler plus a model adapter, two small slivers of code, while the whole recipe stack of FSDP2, bucketed loading, checkpointing, and generation stands untouched. Supporting a new model shrinks from “rebuild a pipeline” to “fill in two defined interfaces” — the single biggest maintenance-cost difference for teams chasing fresh releases.
Recipes are YAML-first today; the roadmap points to a fully typed Pythonic API for diffusion recipes coexisting with the YAML quick-start path, so notebooks and existing training code can adopt them.
A builder’s opening sequence: validate the entire flow on Wan 2.1 1.3B with a single card; pre-encode the data before training; hand style or domain adaptation to LoRA and reserve full fine-tuning for maximum quality; and read the benchmarks as relative numbers from one environment — re-measure on your own hardware.