Sparse attention¶
Sparse attention targets model-level redundancy in long video token sequences. Many spatiotemporal attention interactions contribute little to the final output, so structured sparse patterns can reduce attention cost.
The paper lists sparse attention as a model-level family. The sparse-attention agent chooses the backend, sparsified layers, sparsity pattern, and compensation policy for the deployment instance.
In Sol-Engine¶
Sol-Engine uses Sol-Attn for long video attention in HunyuanVideo-13B and Wan2.1-T2V-14B. LTX-2.3 uses PISA-style sparse video self-attention in selected stage-2 refinement work, combined with cache, fusion, NVFP4, and token pruning in the full optimization stack.
Implemented entries:
techniques/sparse_backends/sol_attn/techniques/sparse_backends/sol_attn_backend.pypython/sglang/multimodal_gen/runtime/efficiency/transforms/sparse_attention.pypython/sglang/multimodal_gen/runtime/layers/attention/backends/piecewise_attn.pypython/sglang/multimodal_gen/runtime/layers/attention/backends/sparse_video_gen_2_attn.pypython/sglang/multimodal_gen/runtime/layers/attention/backends/sparse_linear_attn.pypython/sglang/multimodal_gen/runtime/layers/attention/backends/block_sparse_attn.py
Methods¶
| Method | Open-source status | Integration | Role in the design space |
|---|---|---|---|
| Sol-Attn | open-source | HunyuanVideo and Wan2.1 | on-the-fly block thresholding with approximation correction |
| PISA | open-source | local LTX runtime | piecewise sparse attention selected for the LTX-2.3 full optimization stack |
| SpargeAttention | open-source | external adapter | training-free block-wise sparse attention family |
| Sparse VideoGen | open-source | external adapter | spatial-temporal sparse execution with online profiling |
| Sparse VideoGen2 | open-source | local backend plus SVG adapter | semantic-aware sparse-token identification and GPU layout |
Practical notes¶
- Sparse settings should be validated visually.
- The value of sparse attention depends on sequence length and stage placement.
- Report both denoise time and end-to-end time; decode and offload overhead can hide attention savings.