Model Overview

Description:

Cosmos-Embed1-448p-anomaly-detection v1.1 is a joint video-text embedder fine-tuned from Cosmos-Embed1-448p for text-to-video retrieval, event search, and zero-shot classification in physical-world video. All fine-tuning data comes from the train subset of NVIDIA PhysicalAI-Event-Videos v1.1. Retrieval evaluation uses the dataset's v1.1 test subset; Kinetics-400 is reported separately as a general-domain benchmark.

The fine-tune was produced with the NVIDIA TAO Toolkit cosmos_embed1 pipeline. TAO Toolkit is NVIDIA's low-code framework for adapting pretrained vision and multimodal models to a target domain: it provides the training, evaluation, retrieval-inference, and export (ONNX and Hugging Face) tasks used for this model, so the released checkpoint can be reproduced, re-evaluated, or further fine-tuned from the same specification files.

It produces embeddings for similarity ranking, rather than generated captions, bounding boxes, identities, or calibrated event probabilities.

Version	Base Model	Evaluation Resolution	Frames	Embedding Dim
Cosmos-Embed1-448p-anomaly-detection v1.1	Cosmos-Embed1-448p	448×448	8	768

The upstream Cosmos-Embed1 family is unchanged by this fine-tune: Cosmos-Embed1-224p (224×224, 256-dim), Cosmos-Embed1-336p (336×336, 768-dim), and Cosmos-Embed1-448p (448×448, 768-dim).

This model is ready for commercial or non-commercial use.

License/Terms of Use

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License. Additional Information: Apache 2.0 and MIT.

Built on NVIDIA Cosmos.

Deployment Geography:

Global

Use Case:

Physical AI developers and engineers building video retrieval, event-search, and classification applications across warehouse, retail, traffic, and general activity footage. Video embeddings can be indexed once; text embeddings rank candidate clips by cosine similarity at query time. The model is not validated as an autonomous safety decision system.

Release Date:

Hugging Face 11/05/2026 via Model Link.

References(s):

NVIDIA Cosmos-Embed1 research page.
NVIDIA TAO Toolkit documentation: overview of the toolkit, containers, and model pipelines.
NVIDIA TAO Toolkit Cosmos-Embed1 guide: data input format, train, evaluate, inference, and export tasks, LoRA configuration, and ONNX / Hugging Face export for this model.
Li, Junnan, et al. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML, 2023.
NVIDIA PhysicalAI-Event-Videos: public reference for the PhysicalAI sources and annotation workflow.
Kinetics dataset: general-domain action-classification evaluation.

Model Architecture:

Architecture Type: Transformer
Network Architecture: QFormer-based video-text embedder with EVA-ViT-g visual backbone
Base Model: Cosmos-Embed1-448p
Number of model parameters: Approximately 1.2 × 10^9.

The video embedder processes frames individually with the ViT backbone. Frame features are concatenated temporally and augmented with temporal embeddings. The QFormer summarizes these features through cross-attention using 32 visual query tokens, which are pooled and projected into a 768-dimensional embedding. The text embedder uses the QFormer's BERT-based self-attention branch to encode text into the same embedding space. Both outputs are normalized to unit L2 (Euclidean) norm.

Fine-tuning

Fine-tuning uses Low-Rank Adaptation (LoRA) of the vision backbone, QFormer, and projection layers with video-text contrastive and captioning losses in bfloat16 (BF16) precision.

Training uses eight-frame video inputs resized to 512×512; evaluation uses 448×448 inputs.

Input(s):

Input Type(s): Text or video; the encoders can be used independently.
Input Format(s): UTF-8 text strings or Red, Green, Blue (RGB) video frames.

Input Parameters:

Text: One-Dimensional (1D)
Video: Three-Dimensional (3D)

Other Properties Related to Input:

Evaluation samples eight frames uniformly across each supplied clip.
The Hugging Face processor accepts RGB arrays shaped [batch, frames, channels, height, width], resizes frames to 448×448, and scales pixel values to [0, 1]. The model receives video tensors shaped [batch, 3, 8, 448, 448]; model-specific normalization is handled by the model.
Text uses the Cosmos BERT tokenizer with padding/truncation to 128 tokens, including special tokens. Text tensors have shape [batch, 128].
Evaluation uses English text. Multilingual quality, alternative frame counts, and alternative evaluation resolutions have not been established for v1.1.

Output(s)

Output Type(s): Text embedding or video embedding.
Output Format(s):

Text embedding: floating-point, L2-normalized vector.
Video embedding: floating-point, L2-normalized vector.

Output Parameters: One 768-dimensional vector per video clip or text input; batched shape [batch, 768].

Cosine similarity ranks video-text pairs in the shared embedding space. Similarity scores are not calibrated probabilities or proof that an event occurred.

Other Properties Related to Output

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s):

PyTorch through the NVIDIA TAO Toolkit cosmos_embed1 pipeline (accuracy evaluation)
Open Neural Network Exchange (ONNX) Runtime (via ONNX export)
NVIDIA TensorRT (via ONNX export)

Supported Hardware Microarchitecture Compatibility:

NVIDIA Ampere
NVIDIA Blackwell
NVIDIA Hopper

Note: The Cosmos-Embed1 family has been tested with BF16 precision on NVIDIA Ampere and Hopper GPUs. Older GPU architectures (e.g., NVIDIA Volta) may require switching to 32-bit floating-point (FP32) precision.

Preferred/Supported Operating System(s):

Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.

Model Version(s):

Version	Configuration	Release Format
Cosmos-Embed1-448p-anomaly-detection v1.1	EVA-ViT-g + QFormer, vision and QFormer/projection LoRA, 8 frames, 768-dimensional embeddings	Merged Hugging Face export

v1.1 integrates into an AI system as an embedding encoder: the video tower embeds clips into a searchable index, the text tower embeds user queries at search time, and the application ranks clips by cosine similarity. Both towers can run through NVIDIA TAO Toolkit cosmos-embed1 inference or as exported ONNX models inside a retrieval or event-search service.

Training, Testing, and Evaluation Datasets:

Dataset Overview

All training data comes from the train subset of NVIDIA PhysicalAI-Event-Videos v1.1. All retrieval testing and evaluation data comes from its v1.1 test subset. Kinetics-400 is a separate general-domain action-classification benchmark.

Training Dataset:

Dataset: NVIDIA PhysicalAI-Event-Videos v1.1, train subset.
Data Modality: Video and English text.
Video Training Data Size: Less than 10,000 hours of fine-tuning video.
Text Training Data Size: Less than a billion tokens; text is tokenized to the model's 128-token context.
Data Collection Method: Hybrid: Synthetic, Manually-Collected
Labeling Method: Hybrid: Automated, Manually-Labeled

Testing Dataset:

All retrieval testing data comes from the test subset of NVIDIA PhysicalAI-Event-Videos v1.1.

Data Modality: Video and English text.
Data Collection Method: Hybrid: Synthetic, Manually-Collected
Labeling Method: Hybrid: Automated, Manually-Labeled
Properties: 393 trimmed clips from 166 source videos and 3,040 English event queries, covering everyday human activity and anomalous events in warehouse, retail, street, and indoor scenes.

Evaluation Dataset:

Labeling Method by dataset:

PhysicalAI-Event-Videos v1.1 test subset: Hybrid: Automated, Manually-Labeled
Kinetics-400: Manually-Labeled

1. PhysicalAI-Event-Videos v1.1 test subset — event-only retrieval

Data Collection Method: Hybrid: Synthetic, Manually-Collected
Labeling Method: Hybrid: Automated, Manually-Labeled

All retrieval evaluation data comes from the test subset of NVIDIA PhysicalAI-Event-Videos v1.1: 393 clips and 3,040 event queries, comprising 264 anomaly, 1,369 activity, 64 location, and 1,343 precise-moment queries. Person-attribute queries are excluded from event-only scores.

Evaluation uses eight uniformly sampled frames, 448×448 processor inputs, and BF16 precision. It ranks all gallery clips by cosine similarity for each query. mAP averages per-query average precision over all labeled relevant clips; Hit@K measures whether any relevant clip appears in the first K results and is distinct from multi-relevance Recall@K. All query records, including repeated texts, are scored.

Model	mAP	Hit@1	Hit@5
Cosmos-Embed1-448p base	49.78%	44.31%	74.74%
Cosmos-Embed1-448p-anomaly-detection v1.1	63.21%	58.88%	86.97%

Query category	Queries	Base mAP	v1.1 mAP	v1.1 Hit@1	v1.1 Hit@5
Anomaly	264	50.98%	64.21%	66.67%	89.39%
Activity	1,369	53.90%	69.07%	72.83%	93.06%
Location	64	92.97%	98.79%	100.00%	100.00%
Precise-moment search	1,343	43.29%	55.35%	41.18%	79.67%

Protocol limitation: Checkpoint selection used the same validation gallery, so the result is not an independent held-out generalization estimate.

2. Separate benchmark: Kinetics-400 Validation Set — action classification

Kinetics-400 is evaluated separately from PhysicalAI-Event-Videos to measure general-domain action recognition. It contains real-world human-action video with 400 class labels. The evaluation set contains 19,877 validation clips. Human-labeled action classes are matched with the prompt A video of a person {label} using cosine ranking, eight sampled frames, and the Cosmos 448-pixel processor.

Data Collection Method: Hybrid: Automated, Manually-Collected
Labeling Method: Manually-Labeled

Model	Top-1 Accuracy	Top-5 Accuracy
Cosmos-Embed1-448p base	88.23%	97.72%
Cosmos-Embed1-448p-anomaly-detection v1.1	83.01%	96.05%

Top-K accuracy measures whether the ground-truth action class is among the K highest-ranked prompted classes. The v1.1 row uses the same merged checkpoint as the domain retrieval results. The fine-tune improves the measured domain retrieval score (+13.43 percentage points mAP over the base) while reducing Kinetics-400 Top-1 accuracy by 5.22 percentage points versus the base model. Pretraining overlap with Kinetics-400 cannot be ruled out; interpret this evaluation as general-domain retention rather than evidence of unseen-class generalization.

Inference:

Acceleration Engine: PyTorch (accuracy results); ONNX (opset 17, separate video and text towers, FP32 weights) and TensorRT 10.9 16-bit floating-point (FP16) engines validated for accuracy parity on A100.
Test Hardware:

NVIDIA Ampere A100 80GB

Accuracy results above are from PyTorch in BF16; domain retrieval ran on one A100 GPU and the Kinetics-400 evaluation ran on NVIDIA A100 GPUs, which describes the evaluation setup, not a minimum inference requirement. TensorRT 10.9 FP16 engines built from the ONNX towers reproduce the retrieval benchmark on one A100 (event-only mAP 63.12% / Hit@1 58.98% / Hit@5 86.81% versus 63.21% / 58.88% / 86.97% in PyTorch BF16). Measured latency on that A100: video tower 742 ms per batch of 8 clips in FP16 (2,116 ms in FP32); text tower 4 ms per batch of 32 queries. These are single-GPU measurements for one TensorRT version, not a performance guarantee.

How to use this model

Using Hugging Face Transformers

The Hugging Face repository nvidia/Cosmos-Embed1-448p-anomaly-detection provides the merged v1.1 weights (sharded model-*.safetensors), the model configuration, the processor, the tokenizer, and the custom model code.

import decord
import numpy as np
import torch
from transformers import AutoProcessor, AutoModel

model_path = "nvidia/Cosmos-Embed1-448p-anomaly-detection"
model = AutoModel.from_pretrained(
    model_path, trust_remote_code=True
).to("cuda", dtype=torch.bfloat16).eval()
preprocess = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

reader = decord.VideoReader("/path/to/video.mp4")
frame_ids = np.linspace(0, len(reader) - 1, 8).round().astype(int).tolist()
frames = reader.get_batch(frame_ids).asnumpy()
batch = np.transpose(np.expand_dims(frames, 0), (0, 1, 4, 2, 3))  # BTCHW
captions = ["a person walking", "a car driving on a highway"]

with torch.inference_mode():
    video_inputs = preprocess(videos=batch).to("cuda", dtype=torch.bfloat16)
    video_embedding = model.get_video_embeddings(**video_inputs).visual_proj
    text_inputs = preprocess(text=captions).to("cuda")
    text_inputs["attention_mask"] = text_inputs["attention_mask"].to(torch.bfloat16)
    text_embedding = model.get_text_embeddings(**text_inputs).text_proj
    scores = video_embedding @ text_embedding.T

print(captions[scores[0].argmax().item()])

TAO Integration and Export

The NVIDIA TAO Toolkit cosmos-embed1 pipeline (guide) covers the full lifecycle of this model: data preparation, fine-tuning (including the PhysicalAI-Event-Videos recipe and LoRA configuration), evaluation with a retrieval benchmark, text-to-video search inference, and export to ONNX or a merged Hugging Face checkpoint. Each task takes an experiment specification (configuration reference); use one that matches this checkpoint's model configuration and tokenizer. Example task entrypoints are:

cosmos-embed1 train -e /path/to/train.yaml
cosmos-embed1 inference -e /path/to/inference.yaml
cosmos-embed1 evaluate -e /path/to/evaluate.yaml
cosmos-embed1 export -e /path/to/export.yaml

ONNX export supports separate video and text towers or a combined graph. The validated v1.1 export uses separate towers with dynamic batch: the video tower takes float32 videos [batch,3,8,448,448] scaled to [0, 1] (normalization is inside the graph) and returns video_embedding [batch,768]; the text tower takes int64 input_ids/attention_mask [batch,128] and returns text_embedding [batch,768]. Both outputs are L2-normalized. TensorRT engines are GPU- and version-specific and are not distributed; build them from the ONNX files with trtexec --fp16.

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.

For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards.

Please report model quality, risk, security vulnerabilities, or NVIDIA AI concerns here.

Bias

Field	Response
Participation considerations from adversely impacted groups protected classes in model design and testing:	None
Measures taken to mitigate against unwanted bias:	None
Bias Metric (If Measured):	None

Explainability

Field	Response
Intended Task/Domain:	Cosmos-Embed1-448p-anomaly-detection v1.1: text-to-video retrieval, event search, and zero-shot classification in physical-world video.
Model Type:	Transformer
Intended Users:	Developers building video retrieval, event-search, and classification applications for physical-world footage (warehouse, retail, traffic, and general activity).
Output:	L2-normalized text and video embedding vectors, 768 dimensions per input.
Describe how the model works:	EVA-ViT-g frame features are summarized through QFormer cross-attention. The BERT-based QFormer text branch maps text into the same embedding space. v1.1 uses LoRA to adapt the vision encoder and QFormer/projection layers for video retrieval and event search.
Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of:	None
Technical Limitations & Mitigation:	Due to the training datasets being predominantly composed of short action-focused English captions, different text prompts may not be properly aligned to video data, producing suboptimal matching. English evaluation uses eight uniformly sampled frames and 448-pixel inputs; training uses 512-pixel inputs. Prompt wording and domain shift can affect retrieval. Cosine similarity measures alignment between text and video embeddings for ranking; a score of 0.8 does not mean an 80% probability that the described event occurred. Checkpoint selection uses the evaluation gallery, so domain scores are not independent held-out estimates. General-domain action accuracy is lower than the base model (Kinetics-400 Top-1 83.01% versus 88.23%, a decrease of 5.22 percentage points). Evaluate representative application-specific data, compare with the base model when general-domain action recognition is important, and validate prompt choices and retrieval errors before deployment.
Verified to have met prescribed NVIDIA quality standards:	Yes
Performance Metrics:	Event-only retrieval mAP and Hit@1/Hit@5 on 393 clips and 3,040 queries (person-attribute queries excluded); Kinetics-400 Top-1/Top-5 on 19,877 decoded clips. v1.1: event retrieval 63.21% mAP, 58.88% Hit@1, 86.97% Hit@5 (Cosmos-Embed1-448p base 49.78% / 44.31% / 74.74%); Kinetics-400 83.01% Top-1, 96.05% Top-5 (base 88.23% / 97.72%).
Potential Known Risks:	Embeddings can represent all forms of input, including content that may be considered toxic, offensive, or indecent. The model may generate false matches and/or missed events. Checkpoint selection on the evaluation gallery limits generalization claims. The model may produce suboptimal embeddings to unsafe text prompts. A high text-video similarity score indicates a match in the embedding space; it does not identify a person or establish inferences about behavior or intent. Review the source video and relevant context before drawing such conclusions.
Licensing:	GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License. Additional Information: Apache 2.0 and MIT.

Privacy

The datasets used to train Cosmos-Embed1-448p-anomaly-detection v1.1 included personal data. The datasets used to train this model were reviewed before release. Please review NVIDIA’s Privacy Policy for more information.

Safety

Field	Response
Model Application Field(s):	Video search and retrieval for warehouse, retail, traffic, and general physical-world activity analysis; Video Understanding
Describe the life critical impact (if present).	Cosmos-Embed1-448p-anomaly-detection v1.1 produces video and text embeddings for retrieval, event search, and classification. It is not validated as an autonomous safety decision system; false alarms and missed events require application-specific evaluation and human review.
Use Case Restrictions:	Abide by the NVIDIA Open Model License. Additional Information: Apache 2.0 and MIT.
Model and dataset restrictions:	The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.