Stanford UniversityUnited States2024
OpenVLA
An AI model: a vision-language-action model (VLA) takes camera images and an instruction and turns them into movements for a robot. This page describes the software, not a robot.
Robotsnot stated
Key figures as stated by the developer
6 of 6 stated-
Model type A
vision-language-action -
Controls A
arms and hands -
Inputs A
cameralanguage - Access B open download
- Parameters C 7,000,000,000
- Base model B Prismatic prism-dinosiglip-224px (DINOv2 and SigLIP vision, Llama 2 language)
How to read the figures
- 14kgA30.9lbA
- The letter in the grey square after a figure points to the source the figure came from. Robots with several sources carry several letters.
- 74.5lb
- Imperial figures are our own conversion unless the manufacturer states an imperial figure itself. The source letter always belongs to the metric figure — a conversion has no source of its own.
Versions and notes in the key figures (6)
Model type
Developer wording: "Vision-language-action (language, image => robot actions)".
Controls
The model predicts "robot actions consisting of 7-DoF end-effector deltas".
Inputs
The developer says: "The model takes language instructions and camera images as input and generates robot actions."
Access
Developer: "We release two OpenVLA models trained as part of our work, with checkpoints, configs, and model cards available" on its Hugging Face page.
Parameters
The developer calls it "a 7B-parameter open-source VLA".
Base model
README: "trained from the Prismatic `prism-dinosiglip-224px` VLM (based on a fused DINOv2 and SigLIP vision backbone, and Llama-2 LLM)".
Full specifications
10 of 12 fields stated| Field | Value | Source and caveats |
|---|---|---|
| Input and output | ||
| Model type | vision-language-action | A Developer wording: "Vision-language-action (language, image => robot actions)". |
| Inputs | camera language | A The developer says: "The model takes language instructions and camera images as input and generates robot actions." |
| Controls | arms and hands | A The model predicts "robot actions consisting of 7-DoF end-effector deltas". |
| Degrees of freedom controlled | 7DoF | A 7 degrees of freedom for the setup "BridgeV2 environments with a Widow-X robot"; action space: "7-DoF end-effector deltas of the form (x, y, z, roll, pitch, yaw, gripper)". |
| Size and speed | ||
| Parameters | 7,000,000,000 | C The developer calls it "a 7B-parameter open-source VLA". |
| Control rate | not stated | Note For fine-tuning data the README advises: "Collect at a control frequency around 5-10Hz". B |
| Where it runs and what it is built on | ||
| Runs where | not stated | |
| Base model | Prismatic prism-dinosiglip-224px (DINOv2 and SigLIP vision, Llama 2 language) | B README: "trained from the Prismatic `prism-dinosiglip-224px` VLM (based on a fused DINOv2 and SigLIP vision backbone, and Llama-2 LLM)". |
| Openness | ||
| Access | open download | B Developer: "We release two OpenVLA models trained as part of our work, with checkpoints, configs, and model cards available" on its Hugging Face page. |
| License | MIT; Llama Community License also applies to the pretrained models | B Model card: checkpoints "are released under an MIT License"; README: "both the above models are derived from Llama-2, and as such are subject to the" Llama Community License. |
| The developer’s own claims | ||
| Learns new tasks | needs retraining | A Developer claim: OpenVLA can be fine-tuned "for new tasks and robot setups given minimal demonstration data". |
| Works across robots | yes | A Developer claim: "It supports controlling multiple robots out-of-the-box". Yes only if the developer itself says the model runs on different robot models, for example robots from two manufacturers or two of one manufacturer’s models. Several units of the same robot do not count. |
Notes
- The OpenVLA team describes OpenVLA 7B as an open vision-language-action model trained on 970K Open X-Embodiment robot manipulation episodes.
- The OpenVLA repository lists the initial release as 13 June 2024.
- The OpenVLA model card names its developers as the OpenVLA team of researchers from Stanford, UC Berkeley, Google Deepmind, and the Toyota Research Institute.
- The Stanford University page gives its address as Stanford, California.
Sources
-
A
Manufacturer product page. huggingface.
co/ · Retrievedopenvla/ openvla-7b -
B
github.
com/ · Retrievedopenvla/ openvla -
C
arxiv.
org/ · Retrievedabs/ 2406. 09246
One line per source, not per figure: a source has one retrieval date. The letter beside a figure points to the line it came from.