Color Legend​

Categoric​

• One Variable
  • ⚫ Waffle
  • 🟢 Bar Plot
  • 🟢 Lollipop
  • 🟢 Word Cloud
  • 🔴 Circular Packing
  • 🔴 Doughnut
  • 🔴 Pie
  • 🔴 Treemap
• Two or More Variables
  • Two Independent Lists
    • 🔴 Venn Diagram
  • Nested
    • 🟢 Bar Plot
    • 🔴 Circular Packing
    • 🔴 Dendrogram
    • 🔴 Sunburst
    • 🔴 Treemap
  • Subgroup
    • ⚫ Grouped Scatter
    • ⚫ Heatmap
    • 🟢 Lollipop
    • 🟢 Parallel Plot
    • Spider
    • 🔴 Grouped Bar Plot
    • 🔴 Grouped Bar Plot
    • 🟤 Sankey Diagram
  • Adjacency
    • 🟤 Arc
    • 🟤 Chord
    • 🟤 Network
    • 🟤 Sankey
    • ⚫ Heatmap

Relational​

• Network
  • ⚫ Heatmap
  • 🟢 Hive
  • 🟤 Arc
  • 🟤 Chord
  • 🟤 Network
  • 🟤 Sankey
• Nested
  • No Value
    • 🔴 Circular Packing
    • 🔴 Dendrogram
    • 🔴 Sunburst
    • 🔴 Treemap
    • 🟤 Sankey
  • Value for Leaf
    • 🔴 Circular Packing
    • 🔴 Dendrogram
    • 🔴 Sunburst
    • 🔴 Treemap
    • 🟤 Sankey
  • Value for Edges
    • 🔴 Dendrogram
    • 🟤 Chord
    • 🟤 Sankey
  • Value for Connection
    • Edge Bundling

Map​

• 🟣 Bubble Map
• 🟣 Choropleth
• 🟣 Connected Map
• 🟣 Map
• 🟣 Map Hexbin

Time Series​

• One Series
  • 🟡 Box Plot
  • 🟡 Violin
  • 🟡 Ridge Line
  • 🔵 Area
  • 🔵 Line Plot
  • 🟢 Bar Plot
  • 🟢 Lollipop
• Several Series
  • 🟡 Box Plot
  • 🟡 Violin
  • 🟡 Ridge Line
  • ⚫ Heatmap
  • 🔵 Line Plot
  • 🔵 Stacked Area
  • 🔵 Stream Graph

Categoric and Numeric​

• One Numeric + One Categoric
  • One Observation, per Group
    • ⚫ Waffle
    • 🟢 Bar Plot
    • 🟢 Lollipop
    • 🟢 Word Cloud
    • 🔴 Circular Packing
    • 🔴 Doughnut
    • 🔴 Pie
    • 🔴 Treemap
  • Several Observations, per Group
    • 🟡 Box Plot
    • 🟡 Violin
    • 🟡 Ridge Line
    • 🟡 Density
    • 🟡 Histogram
• One Category, Several Numeric
  • No Order
    • 🟡 Box Plot
    • 🟡 Violin
    • ⚫ Grouped Scatter
    • ⚫ 2D Density
    • ⚫ PCA
    • ⚫ Correlogram
  • A Numeric is Ordered
    • ⚫ Connected Scatter
    • 🔵 Area
    • 🔵 Line Plot
    • 🔵 Stacked Area
    • 🔵 Stream Graph
  • One Value Per Group
    • ⚫ Grouped Scatter
    • ⚫ Heatmap
    • 🟢 Lollipop
    • 🟢 Parallel Plot
    • 🟢 Spider Plot
    • 🔴 Grouped Bar Plot
    • 🔴 Grouped Bar Plot
    • 🟤 Sankey Diagram
• Several Categories, One Numeric
  • Subgroup
    • One Observation. per Group
      • ⚫ Grouped Scatter
      • ⚫ Heatmap
      • 🟢 Lollipop
      • 🟢 Parallel Plot
      • 🟢 Spider Plot
      • 🔴 Grouped Bar Plot
      • 🔴 Grouped Bar Plot
      • 🟤 Sankey Diagram
    • Several Observations, per Group
      • 🟡 Box Plot
      • 🟡 Violin
  • Nested
    • One Observation. per Group
      • 🟢 Bar Plot
      • 🔴 Circular Packing
      • 🔴 Dendrogram
      • 🔴 Sunburst
      • 🔴 Treemap
    • Several Observations. per Group
      • 🟡 Box Plot
      • 🟡 Violin
  • Adjacency
    • ⚫ Heatmap
    • 🟤 Arc
    • 🟤 Chord
    • 🟤 Network
    • 🟤 Sankey

Numeric​

• One Numeric Variable
  • 🟡 Density
  • 🟡 Histogram
• Two Numeric Variables
  • Not Ordered
    • Few Points
      • 🟡 Box Plot
      • 🟡 Histogram
      • ⚫ Scatter Plot
    • Many Points
      • 🟡 Density
      • 🟡 Violin
      • ⚫ 2D Density
      • 🔵 Marginal Distribution
  • Ordered
    • ⚫ Connected Scatter
    • 🔵 Area Plot
    • 🔵 Line Plot
• Three Numeric Variables
  • Not Ordered
    • 🟡 Box Plot
    • 🟡 Violin
    • ⚫ Bubble Plot
    • ⚫ 3d Scatter or Surface
  • Ordered
    • 🔵 Area
    • 🔵 Line Plot
    • 🔵 Stacked Area
    • 🔵 Stream Graph
• Several Numeric Variables
  • Ordered
    • 🔵 Area
    • 🔵 Line Plot
    • 🔵 Stacked Area
    • 🔵 Stream Graph
  • Not Ordered
    • 🟡 Box Plot
    • 🟡 Ridge Line
    • 🟡 Violin
    • ⚫ Correlogram
    • ⚫ Heatmap
    • ⚫ PCA
    • 🔴 Dendrogram

Ref​

• from Data to Viz

Vocabulary & Expressions​

Problem formulation (문제 정의)​

• The first step is to figure out what problem you want to solve.
  1. “사용자에게 어떤 문제를 해결해주고 싶은가?” → 모호하지 않고 구체적으로 정의해야 함.
  2. “그 문제 중 어떤 부분을 머신러닝으로 풀 수 있는가?” → 예: 사진을 라벨로 매핑하는 함수 학습.
• 이를 구체화하려면 ML 컴포넌트에 대해 loss function 을 지정해야 한다.
• 문제를 쪼개보면 일부는 전통적 SW 엔지니어링으로 해결 가능하고, 일부만 ML로 다뤄야 할 수 있다.
• 학습 유형은 지도·비지도·강화·준지도(semisupervised)까지 연속선상에 있음.
  • Semisupervised learning: 일부 라벨만 활용해 비라벨 데이터에서 더 많은 정보 추출.
  • Weakly supervised learning: 부정확·노이즈 라벨을 사용.
• 결론: Noise와 label 부족은 “지도 ↔ 비지도” 사이의 연속체를 형성한다.

Data collection & management (데이터 수집/관리)​

• 데이터는 직접 제작, 크라우드소싱, 사용자 행동에서 수집 가능.
• 부족할 때는 transfer learning 활용.
• Privacy 검토와 동의, 공정성, federated learning 등 고려 필요.
• Data provenance(출처 관리): 데이터 정의, 값의 범위, 생성 주체, 중단 여부, 정의 변경 이력 등 추적 → 파이프라인 안정성이 알고리즘보다 중요.
• 항상 자문: “이 데이터는 내 문제를 풀기에 적절한가? 입력과 출력 모두 충분히 담고 있는가?”
• Learning curve 로 데이터 확장 효과/학습 plateau 확인.
• 방어적 태도 필요: 입력 오류, 누락, 적대적 사용자, 철자 불일치 등 처리.
• Data augmentation (회전, 이동, 노이즈 추가 등)으로 모델 강건성 향상.
• 불균형 데이터는 undersampling, oversampling, SMOTE/ADASYN, boosting 등으로 완화.
• 아웃라이어는 로그 변환 등으로 영향 축소, 트리 모델은 상대적으로 강건.

Feature engineering (특징 엔지니어링)​

• Quantization: 연속값을 구간(bin)으로 강제.
• One-hot encoding: 범주형 속성을 다중 Boolean으로 변환.
• 도메인 지식 기반 새 특성 추가 (예: 날짜 → 주말/공휴일 여부).
• “At the end of the day, some ML projects succeed and some fail… the most important factor is the features used.” (Pedro Domingos)

Exploratory data analysis (EDA) & visualization​

• 목표: 예측/검증이 아닌 데이터 이해.
• Histograms, scatter plots 로 분포/결측/오류/이상치 확인.
• 클러스터링 → 프로토타입 시각화, 이상치 탐지 (“고양이 vs 사자 옷 입은 고양이”).
• 차원 축소 (예: t-SNE)로 고차 데이터를 2D/3D로 시각화.

Model selection & training​

• 데이터가 정리되면 모델 구축 단계.
• Random forests → 범주형 특징 많고 일부 무관할 때.
• Nonparametric methods → 데이터 많고 지식 부족, 특징 선택 고민 줄이고 싶을 때.
• Logistic regression → 선형 분리 가능(또는 feature engineering 후).
• SVM → 데이터 크기 작고 차원 높을 때.
• Deep neural nets → 패턴 인식(이미지·음성).
• 하이퍼파라미터는 경험 + 탐색으로 조율.
• 검증 데이터 남용 시 validation overfitting 위험 → 여러 검증셋 필요.
• 성능 평가: ROC curve, AUC, confusion matrix.
• 중요한 건 아이디어–실험–검증 반복 사이클을 빠르게 하는 것.

Trust, interpretability, explainability​

• 단순히 지표 성능만으로는 신뢰 부족 → 규제·언론·사용자도 신뢰성 원함.
• Accountability: 오류 발생 시 책임 주체와 항소 절차 필요.
• Interpretability: 모델 내부를 직접 이해 (트리, 선형회귀).
  • 핵심 질문: “If I change x, how will the output change?”
• Explainability: 블랙박스 모델 + 별도 모듈로 설명 (예: LIME).
• 단순 설명이 잘못된 확신을 줄 수 있음. → 테스트와 실제 성능이 더 큰 신뢰를 준다.
• “안전하다고 설명만 있는 실험기 vs 100회 무사비행한 비행기” 비유.

Operation, monitoring, maintenance​

• 운영 단계에서는 롱테일 입력(long tail) 문제 등장 → 예상 못한 입력 지속 발생. → 실시간 모니터링과 사람 평가자 필요.
• Nonstationarity: 세상과 사용자 행동 변화 → 최신 데이터 vs 안정적 모델 트레이드오프.
• 신선도 요구 다름: 어떤 문제는 매일/매시간 새 모델, 어떤 문제는 수개월 동일 모델.
• 배포 자동화 → 작은 변경은 자동 승인, 큰 변경은 리뷰.
• Online vs Offline model: 기존 모델 점진적 수정 vs 매번 처음부터 재학습.
• 데이터 자체가 바뀔 수도 있음 (스팸 이메일 → 스팸 문자, 음성, 영상 등).

Checklist​

Tests for Features and Data​

• Feature expectations are captured in a schema.
• All features are beneficial.
• No feature’s cost is too much.
• Features adhere to meta-level requirements.
• The data pipeline has appropriate privacy controls.
• New features can be added quickly.
• All input feature code is tested.

Tests for Model Development​

• Every model specification undergoes a code review.
• Every model is checked in to a repository.
• Offline proxy metrics correlate with actual metrics.
• All hyperparameters have been tuned.
• The impact of model staleness is known.
• A simpler model is not better.
• Model quality is sufficient on all important data slices.
• The model has been tested for considerations of inclusion.

Tests for Machine Learning Infrastructure​

• Training is reproducible.
• Model specification code is unit tested.
• The full ML pipeline is integration tested.
• Model quality is validated before attempting to serve it.
• The model allows debugging by observing the step-by-step computation of training or inference on a single example.
• Models are tested via a canary process before they enter production serving environments.
• Models can be quickly and safely rolled back to a previous serving version.

Monitoring Tests for Machine Learning​

• Dependency changes result in notification.
• Data invariants hold in training and serving inputs.
• Training and serving features compute the same values.
• Models are not too stale.
• The model is numerically stable.
• The model has not experienced regressions in training speed, serving latency, throughput, or RAM usage.
• The model has not experienced a regression in prediction quality on served data.

Ref​

• Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2016). What’s your ML test score? A rubric for ML production systems. NIPS Workshop on Reliable Machine Learning in the Wild.

Nearest-neighbor Models​

• 쿼리점 $x_q$에 대해 가장 가까운 $k$개의 이웃을 찾아 분류 또는 회귀에 사용한다.
  • 분류: 다수결
  • 회귀: 평균, 중앙값, 혹은 국소적 선형회귀
• 거리 척도: Minkowski 거리
  • $L_p(x_j, x_q) = \left( \sum_i |x_{j,i} - x_{q,i}|^p \right)^{1/p}$
  • $p=2$ → 유클리드 거리
  • $p=1$ → 맨해튼 거리
  • 불리언 속성 → 해밍 거리
  • 공분산 고려 → 마할라노비스 거리
• 차원의 저주 (curse of dimensionality):
  • 평균 이웃 부피: $\ell^n = k/N \;\;\Rightarrow\;\; \ell = (k/N)^{1/n}$
  • $n$이 커질수록 $\ell$ 값이 커져 이웃이 “멀어진다”.
  • 대부분의 점은 고차원 공간에서 경계(껍질)에 몰린다.
  • 저차원: 보간(interpolation) 가능
  • 고차원: 외삽(extrapolation)이 많아져 일반화 어려움

k-d trees​

• 데이터를 차원별로 분할해 만든 이진 트리.
• 각 노드에서 특정 차원의 중앙값 $m$을 기준으로 $x_i \le m$ 여부에 따라 좌/우로 분할한다.
• 탐색: 쿼리점 기준으로 한쪽 브랜치로 내려가며 후보를 찾되, 경계와 가까우면 반대편 서브트리도 확인해야 한다.
• 효율 조건: 데이터 수가 차원 수보다 훨씬 많아야 하며, 최소 $2^n$개 이상 필요하다.
• 실용 범위:
  • 약 10차원 이하에서는 수천 개 데이터
  • 약 20차원 이하에서는 수백만 개 데이터

Support Vector Machines (SVM)​

• 최대 마진 분리자(maximum margin separator)를 찾는다.
• 목표: 경험적 손실 최소화 대신 일반화 손실 최소화
• 결정 경계: $\{x : w \cdot x + b = 0\}$
• 학습은 이차계획법(QP) 최적화 문제로 정식화된다.
  • 이중 표현(dual form):
    $\arg\max_\alpha \sum_j \alpha_j - \tfrac{1}{2} \sum_{j,k} \alpha_j \alpha_k y_j y_k (x_j \cdot x_k)$
  • 제약조건: $\alpha_j \ge 0,\; \sum_j \alpha_j y_j = 0$
• 최적 해에서 대부분 $\alpha_j = 0$이고, 경계 근처의 점들(서포트 벡터)만 $\alpha_j > 0$이다.
• 예측 함수:
  $h(x) = \text{sign}\Big(\sum_j \alpha_j y_j (x \cdot x_j) - b \Big)$
• 장점:
  • 서포트 벡터만 유지하면 되므로 효율적
  • 비모수적 유연성 + 모수적 안정성(과적합 억제)

The Kernel Trick​

• 커널 트릭: 실제 고차원 특징 공간 $F(x)$를 계산하지 않고, 내적만을 커널 함수로 대체한다.
  • $K(x,z) = F(x)\cdot F(z)$
• 대표 커널 함수:
  • 다항 커널: $K(x,z) = (1 + x \cdot z)^d$
  • 가우시안 커널 (RBF): $K(x,z) = e^{-\gamma \|x-z\|^2}$
• 소프트 마진 분류기: 일부 오분류 허용, 오분류된 점을 올바른 쪽으로 이동시키는 거리만큼 패널티를 부여한다.
• 커널 기법은 내적에만 의존하는 다른 알고리즘에도 적용 가능하다.
• Mercer's theorem: “합리적인” 커널 함수는 항상 어떤 특징 공간에서의 내적에 해당한다.

단변량 선형 회귀 (Univariate Linear Regression)​

• 입력이 하나 $x$인 경우, 가설: $h(x) = w_1x + w_0$
• 손실 함수: 제곱 오차 (Squared Error)
• 경사 하강법으로 최적의 $(w_0, w_1)$ 찾기
  • $w_0 \leftarrow w_0 + \alpha (y - h(x))$
  • $w_1 \leftarrow w_1 + \alpha (y - h(x)) \cdot x$
• 손실 함수가 볼록(Convex) → 전역 최소값(Global Minimum) 보장

배치 / 확률적 경사 하강법 (Batch vs SGD)​

• 배치 경사 하강법(Batch GD): 모든 데이터 사용 → 정확하지만 느림, 대규모 데이터 비효율적
• SGD(Stochastic GD): 무작위 예시 하나(또는 작은 minibatch)만으로 업데이트 → 빠르고 효율적
• 미니배치(Minibatch): 속도 + 안정성 균형 가능
• 학습률 $\alpha$ 감소 스케줄 → 수렴 보장

다변량 선형 회귀 (Multivariable Linear Regression)​

• 입력이 $n$차원인 경우, 가설: $h(x) = w \cdot x = \sum_i w_i x_i$
• 정규 방정식 (Normal Equation): $w^* = (X^TX)^{-1}X^Ty$
• $(X^TX)^{-1}X^T$ = 유사역행렬(Pseudoinverse)
• 고차원에서는 과적합 위험이 크므로 정규화 필요

정규화 (Regularization)​

• 비용 함수: $Cost(h) = Loss(h) + \lambda \cdot Complexity(h)$
• 복잡도 함수: $Complexity(h_w) = \sum_i |w_i|^q$
• $q = 1$ → L1 정규화 (희소 모델, 많은 $w_i = 0$)
• $q = 2$ → L2 정규화 (가중치 제곱합 최소화)
• L1 → 회전 불변성 없음 (축이 중요한 경우 적합)
• L2 → 회전 불변성 있음 (축이 임의적일 때 적합)

퍼셉트론 학습 규칙 (Perceptron Learning Rule)​

• 선형 함수 + Hard Threshold → 선형 분류기
• 가중치 업데이트: $w_i \leftarrow w_i + \alpha (y - h(x)) \cdot x_i$
• 선형 분리 가능(linearly separable) → 완벽한 분리자로 수렴
• 분리 불가능한 경우 → 수렴 보장 없음, $\alpha$ 스케줄 필요

로지스틱 회귀 (Logistic Regression)​

• Hard Threshold 문제
  • 불연속, 미분 불가능 → 학습 불안정
  • 항상 0 또는 1 확정 예측 → 경계 근처 비효율적
• 해결책: 로지스틱 함수 $g(z) = \frac{1}{1 + e^{-z}}$
• 가설: $h_w(x) = g(w \cdot x) = \frac{1}{1 + e^{-w \cdot x}}$
• 출력 $\in (0,1)$ → 확률로 해석 가능, soft boundary 형성
• 경계 중앙에서 0.5, 멀어질수록 0 또는 1에 가까움

로지스틱 함수의 도함수 성질​

• 로지스틱 함수: $g(z) = \frac{1}{1+e^{-z}}$
• 미분: $g'(z) = \frac{e^{-z}}{(1+e^{-z})^2}$
• $1 - g(z) = \frac{e^{-z}}{1+e^{-z}}$
• 따라서 $g(z)(1-g(z)) = \frac{e^{-z}}{(1+e^{-z})^2}$
• 결론: $g'(z) = g(z)(1-g(z))$

로지스틱 회귀 가중치 업데이트 유도 과정​

• 손실 함수: $Loss(w) = (y - h_w(x))^2$
• $\frac{\partial}{\partial w_i} Loss(w) = \frac{\partial}{\partial w_i}(y - h_w(x))^2$
• $= 2(y - h_w(x)) \cdot \frac{\partial}{\partial w_i}(y - h_w(x))$
• $= -2(y - h_w(x)) \cdot \frac{\partial}{\partial w_i} h_w(x)$
• $h_w(x) = g(w \cdot x)$ 이므로 $\frac{\partial}{\partial w_i} h_w(x) = g'(w \cdot x) \cdot x_i$
• $g'(w \cdot x) = h_w(x)(1-h_w(x))$
• 최종: $\frac{\partial}{\partial w_i} Loss(w) = -2(y - h_w(x)) \cdot h_w(x)(1-h_w(x)) \cdot x_i$
• 경사 하강법 업데이트:
  $w_i \leftarrow w_i - \alpha \cdot \frac{\partial}{\partial w_i} Loss(w)$
• 따라서:
  $w_i \leftarrow w_i + \alpha (y - h_w(x)) \cdot h_w(x)(1-h_w(x)) \cdot x_i$

결론​

• 발전 흐름: 선형 회귀 → 경사 하강법 → 다변량 확장 → 정규화 → 퍼셉트론 → 로지스틱 회귀
• L1 vs L2 정규화
  • L1: 희소 모델 (축 중요)
  • L2: 회전 불변 (축 임의적)
• 퍼셉트론: 선형 분리 가능할 때만 완벽 동작
• 로지스틱 회귀: soft boundary 제공 → 확률적 예측 + 현실 데이터에 강함

Octo​

• Octo is a transformer-based policy with modular tokenizers (language via T5, images via CNN patches), blockwise masking, and readout tokens, trained on 800k multi-robot trajectories.
• Actions are generated through a diffusion head that produces continuous, multimodal, chunked predictions, enabling precise control and broad generalization.
• It achieves state-of-the-art zero-shot performance across 7 robots and allows efficient finetuning to new sensors and action spaces, while being fully open-source.

Time t-1: [l] [g] [o_{t-1}] [TR,t-1]
Time t:   [l] [g] [o_t]     [TR,t]
Time t+1: [l] [g] [o_{t+1}] [TR,t+1]

[TR,t-1], [TR,t], [TR,t+1]  ──►  Diffusion head  ──►  [a_t, a_{t+1}, …]

Motivation​

• Traditional robot learning trains policies from scratch on robot/task-specific datasets → costly data collection, narrow generalization.
• Generalist Robot Policies (GRPs) pretrained on diverse robots/tasks can be finetuned with little in-domain data while generalizing broadly.
• Real-world deployments face challenges across robot embodiments, sensor setups, action spaces, task specs, and environments.

Prior GRPs & Gaps​

• GRPs aim for low-level visuomotor control across tasks, environments, and robotic systems.
• Existing models often have restricted inputs (e.g., a single camera), lack efficient finetuning to new domains, and importantly, largest models are not publicly available.

Contribution (What is Octo?)​

• Octo: a large transformer-based policy trained on 800k trajectories from the Open X-Embodiment dataset.
• Accepts language instructions or goal images, and can be finetuned within hours on consumer GPUs to new sensors and action spaces.
• First GRP to support effective finetuning to new observations and actions and to be fully open-source (training pipeline, checkpoints, data).
• Novelty lies in combining: transformer backbone + language/goal image conditioning + diffusion head for expressive action distributions.

Architecture​

• Input tokenizers:
  • Language via pretrained T5-base
  • Images via shallow CNN → patch tokens
• Transformer backbone: processes unified token sequence.
• Blockwise masking + Readout tokens:
  • Nonexistent modalities are masked
  • Readout tokens only attend to past observations/tasks, not vice versa
• Diffusion action head: predicts continuous, multimodal, chunked actions.
• Modularity: new sensors/outputs can be added by only training lightweight encoders or heads; pretrained backbone remains unchanged.

Training Data & Objective​

• Mixture of 25 heterogeneous robot datasets: diverse robots, sensors (with/without wrist cams), labels (with/without language).
• Conditional diffusion decoding predicts continuous, multimodal action distributions.
  • Transformer runs one forward pass; denoising steps are contained in the small diffusion head.

Experiments​

• Evaluated on 7 robotic platforms across 4 institutions.
• Key questions:
  1. Zero-shot multi-robot control?
  2. Do Octo weights improve finetuning vs. scratch or standard pretrained representations?
  3. Which design choices matter for generalist robot policies?

Results​

• Achieves state-of-the-art zero-shot multi-robot control, competitive with RT-1-X and RT-2-X.
• Provides a versatile policy initialization: significantly outperforms baselines for data-efficient finetuning to new obs/action spaces.

Limitations / Future Work​

• Needs better language conditioning, improved wrist camera support, and data beyond optimal demonstrations.

One-line Takeaway​

• Octo = modular, efficient, open-source GRP:
  A transformer + diffusion policy trained on large-scale multi-robot data that adapts quickly with little in-domain data to new sensors and action spaces, enabling broad generalization.

Ref​

• Mees, O., Ghosh, D., Pertsch, K., Black, K., Walke, H. R., Dasari, S., Hejna, J., Kreiman, T., Xu, C., & Luo, J. (2024). Octo: An open-source generalist robot policy. First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024.

Influencing improvement​

• Agent's component
• Agent's prior knowledge, which influence that mode lit builds
• the feedback available to learn from.

Components​

• Given an intelligent agent that performs some intelligent tasks, any components of agent program can be improved by learning.

Prior knowledge​

• Inductive learning (귀납적 학습): learning a general function or rule (possibly incorrect) from specific input-output pairs
  • Bottom to top
  • Specific to general
• Deductive learning (연역적 학습): going from a general rule to a new specific rule that is logically entailed, but is useful because it allows more efficient processing
  • Top to bottom
  • general to specific

Feedback​

• feedback on its percept sequence
• no feedback on its percept sequence
• rewards for taking a sequence of actions based on its percept sequence

Supervised learning​

• the agent observes some examples of input-output pairs first and then learns a function or a relationship that maps from inputs to output.
• Attributes/Features: the inputs are independent variables in the problem domain
• Target attribute: the output is the dependent variable which is dependent on the inputs.
• Model: the learned function or relationship
• The agent learns a model using examples and uses this model to predict the outcomes for new inputs.

Unsupervised learning​

• The agent collects adequate examples in the problem domain but it does not get any explicit feedback to the examples.
• The agent can make sense of the examples through identifying clusters or frequent patterns in the data.
• When shown a large number of examples, the agent can learn to identify clusters of similar examples.

Reinforcement learning​

• the agent learns from a series of actions which can be rewards or punishments to improve its performance in completing the task under consideration.
• the feedback helps the agent to enforce positive actions and reduce the negative actions through adjusting the policy.

Supervised Learning Technique​

• Decisinon tree
• Random forests
• Linear regrasssion
• Logistic regression
• K nearest neighbours
• Support vector machines
• Neural networks

Regression problem​

• to predict a continuous value as the output for a given input
• weather temperature: solar radiation, wind direction and speed, geographic location..
• how to predict the output value of a new data instance on the basis of observed features from the existing data (historical examples) in the problem domain.
• Elements
  • Collection of existing or historical data samples which are represented by a set of attributes or independent variables
  • The output values of the existing data samples
    • the output variable or attribute must be continuous
• Regressor: a function describes the relationship between the attributes of a data sample and the output.
  • takes the values of attributes of a data sample and predicts the output value of this given data sample.

Evaluate a regressor​

• R Square/Adjust R Square
• MSE Mean Square Error/RMSE Root Mean Square Error
• MAD Mean Absolute Error

Examples of regression problems​

• Predict the fuel price using the Brent crude oil price, financial performance of the oil related companies (cash flow, projects lined up, etc.) and/or geopolitical risks (OPEC announcements, government sanctions, etc.)
• Predict the house price of a suburb from the suburb's profile
• Predict the blood pressure of a patient based on the patient's health profile
• Predict the electricity price using temperature, demand and time.

Classification problem​

• to predict discrete or categorical value as the output for a given input
• Pass or Failed given learning outcomes, student ID, prior learning, attitude, commitment and attendance.
• how to put a new data instance into one of predefined categories or classes on the basis of observed features from the existing data in the problem domain.
• Elements
  • Collection of existing or historical data samples with class labels
  • Predefined categories or classes
  • Adequate samples in each category or class in the existing or historical data.
• Logic-based techniques
  • Decision tree
  • Learning set of rules
• Perceptron-based techniques
  • Single-layer perceptron
  • Multi-layer perceptron
  • RBF network
• SVM
• Statistical learning techniques
  • Naive Bayes classifier
  • Bayesian networks
• Instance-based learning
  • K-nearest neighbor (KNN)

Evaluate a classifier​

• Confusion matrix
• Precision
• Recall/Sensitivity
• Specificity
• F1-Score
• Area Under Curve & Receiver Operating Characteristics Curve (AUC-ROC)

Examples of classification problems​

• banking, healthcare, medical diagnosis, marketing (sentiment aalysis), telecommunication, agriculture, security (fraud detection).
• e-mails into spam or non-spam class
• loan applications into an approved or a rejected class.
• patients into having a certain disease or not having that disease groups.
• text into positive or negative sentiment.
• customers into churn or non-churn classes.

Overfitting​

• general phenomenon with all types of learning models.
• a modeling error that occurs when a function is too closely or exactly fit to a limited set of data points.
• more likely as the complexity of models and the number of input attributes increase
• less likely as the number of training examples is large.

Decision Tree​

• if-then statements to define patterns in data
• A if-then statement splits the training data into two or more branches based on some values
• Best Split: The results of each branch should be as homogeneous as possible, or has the lowest impurity possible.
  • Information gain
  • Gini index

Implement Decision Tree​

• the split (a feature and a condition) that leads to the lowest impurity in the resulting child nodes, in a greedy manner
• For categorical features: each unique value can be a split condition.
• For continuous features: midpoints between consecutive sorted unique values are used as split conditions.
• For each potential split condition, the algorithm calculates the impurity of the resulting child nodes.
• The lowest impurity node becomes the split point for that branch.
• The process is then repeated recursively for each child node until all leaf nodes are pure, or the stopping criteria are met.

The selectino of best split attributes​

• ID3: employs a top-down, greedy search through the space of possible branches with no backtracking using information gain
• C4.5: using information gain ratio
• CART: using Gini Index
• Gini Index
• Chi-Square
• Reduction in Variance

Entropy​

the fundamental quantity in information theory. It is a measure of the uncertainty of a random variable.

• the fundamental quantity in information theory. It is a measure of the uncertainty of a random variable
• A more homogeneous node with a clear majority class has low impurity and low entropy, while a more mixed distribution of classes has high impurity and high entropy.

Information gain​

the decrease in entropy.

• The information gain from the attribute test on (split on A) is the expected reduction in entropy.
• Information gain computes the difference between entropy before the split and average entropy after the split of the dataset based on given attribute values
• $Entropy(S) = - \sum_{} p_i \log_2 p_i$
• $Gain(S,A)=Entropy(S) − Entropy_{remain}(S,A)$

Gini index​

• For classification, another impurity measure commonly used for classification tasks in decision trees.
• a lower Gini index indicates lower impurity, meaning that the samples in the node predominantly belong to a single class
• a bit more computational efficient than entropy as it does not involve logarithm calculations. but results are quite similar.
• $Gini(S) = 1 - \sum_{i=1}^{K} p_i^2$

Variance​

• For regression tree, the target variable is continuous rather than categorical.
• use variance as a measure of impurity in regression trees.
• lower variance indicates that the data points are closely clustered around the mean
• $Var(S) = \frac{1}{N} \sum_{} (y_i - \mu)^2$

Prediction​

• For classification tasks: the predicted class label is the majority class among the training samples in the leaf node.
• For regression tasks: the predicted value is the mean of the target values of the training samples in the leaf node.

Dealing with Overfitting​

• Overfitting: a common issue in decision trees, where the model captures noise or outliers in the training data rather than the underlying pattern.
• the model performs poorly when applied to new, unseen data.

Setting Stopping Criteria​

• prevent the tree from becoming overly complex, which may lead to overfitting
• applied during the tree construction process
• limiting the maximum depth of the tree
• setting a minimum number of samples per leaf node
• requiring a minimum impurity decrease for a split

Pruning Strategies​

• applied after the tree has been fully grown
• removing branches from the fully grown tree to simplify its structure
• ensure that it captures the underlying patterns in the data rather than noise or outliers
• Pruned trees perform significantly better than unpruned trees when the data contain a large amount of noise.

Ensemble Methods​

• combine multiple decision trees to form a more robust and accurate model.
• address overfitting by averaging the predictions of the individual trees, reducing variance and improving generalization.
• Random Forests
• Gradient Boosted Trees

Random Forest​

combines multiple weak decision tree models to create a stronger learning model.

• two types of randomness are introduced to ensure that the individual decision trees are diverse and less prone to overfitting.
• Random sampling of the input data
• Bootstraping:
  • involves sampling with replacement 복원추출 (meaning that some instances appearing multiple times and others not appearing) from the original dataset, creating a new dataset.
  • each decision tree is trained on a slightly different set of data points, reducing the likelihood of overfitting.
• Random selection of features at each split
  • At each split in each decision tree, a random subset of features is considered when determining the best split.
  • each tree in the ensemble does not rely on the same set of features for making decisions, resulting in a more diverse set of trees.
  • By considering only a subset of features at each split, the model is less likely to be influenced by a small number of dominant features, leading to a more balanced and accurate prediction.

Predict with Random Forest​

• aggregating the predictions of all individual decision trees in the forest.
• Majority voting: For classification, Count the number of times each class is predicted by the individual decision trees. The class with the highest count is considered as the final prediction.
• Averaging: For regression, Calculate the mean of the predictions made by the individual decision trees. The mean value is considered as the final prediction.

Linear regression​

a learning technique that finds a linear relationship between input variables and the target variable based on a fundamental assumption that there is a linear relationship between input variables and the target variable

• e.g. the input variables (engine size, weight and car age) ➡️ target variable (car fuel efficiency)
  • assumption that there is a linear relationship
• A linear regression technique learns a set of coefficients to estimate the linear relationship between $x$ and $y$, denoted as $h_w$, which can be represented by the following equation.
  • $h\_w(x) = w_0 + w_1x_1 + ... + w_nx_n = \sum_{i=0}^{n} w_ix_i$
  • $w$ is a weight vector
  • $\hat{y} = \sum_{i=0}^{n} w_ix_i$
• linear regression model is an approximate function between the input variables and the target variable, there will be an error between the output of the model and the actual output value for a data sample
  • This error can be represented by a loss function, which calculates the mean square error
  • $Loss(h_w) = \frac{1}{2m}\sum_{j=1}^{m}(h_w(x_j) - y_j)^2 = \frac{1}{2m}\sum_{j=1}^{m}(y_j - \sum_{i=0}^{n} w_ix_{j,i})^2$
• for solving regression problems

Solving a linear regression problem​

• to find the best linear relationship $h_w$that best fits the training data of $m$ data samples.
  • makes the loss to be minimised.
• to find the best weight vector $w^*$, such that for a given training dataset of $m$ data samples.
  • $w^* = \arg\min_{w} Loss(h_w)$
• gradient descent: continuously resamples the gradient of the weight coefficients in the opposite direction depending on the weight $w$.
  • Until the loss function $Loss(h_w)$ reaches the global minimum
  • to change the individual components of $w$ a little bit in the direction that minimises $Loss(h_w)$, and to do this many times.
• $w_i \;\leftarrow\; w_i + \alpha \sum_{j=1}^{m} x_{j,i} \Big( y_j - h_w(x_j) \Big)$
  • $\alpha$: the step size, the learning rate
• Training model: the process of iteratively updating weights with a learning rate to minimise loss, where the final weight vector defines the model used for predicting new data.
• use regularisation on a multivariate linear function to avoid overfitting.
• Batch gradient descent: consider the entire training dataset $(X, y)$ at once.
  • $w_0 \;\leftarrow\; w_0 + \alpha \sum_{j=1}^{m} \Big(y_j - (w_0 + w_1 x_j)\Big)$
  • $w_1 \;\leftarrow\; w_1 + \alpha \Big(\sum_{j=1}^{m} (y_j - (w_0 + w_1 x_j)) \cdot x_j\Big)$
• Stochastic gradient descent (SGD): consider only a single training data sample $(x_j, y_j)$ at a time.
  • $w_0 \leftarrow w_0 + \alpha \big( y_j - (w_0 + w_1 x_j) \big)$
  • $w_1 \leftarrow w_1 + \alpha \big( (y_j - (w_0 + w_1 x_j)) \cdot x_j \big)$
  • can be used in an online setting, where new data is coming one at a time, or offline, where we cycle through the same data as many times as is necessary, taking a step after considering each single example.
  • With a fixed learning rate $\alpha$, the stochastic version does not guarantee convergence.
  • often faster than batch gradient descent.
  • With a schedule of decreasing learning rates (SA), the stochastic version does guarantee convergence.
• These update rules are derived as the next weight update equations by taking the partial derivatives of the loss function with respect to $w_0$ and $w_1$.

Logistic Regression​

an extension of linear regression in such a way that the output of a linear regression model goes through a logistic function

• $y(x) = \frac{1}{1 + e^{-x}}$
• The output value of this logistic function is between 0 and 1.
• 0 is for certainly being labeled "0" and 1 is for certainly being labeled "1", and a value between 0 and 1 represents the probability of being labeled "1"
• a logistic regression model: a linear regression model + a logistic function
• mainly for solving classification problems

Nearest Neighbor​

a technique to predict the output of a given new sample based on a collection of existing samples.

• is to find the k-nearest neighbours of given sample in the collection and determine the output based on these k neighbours.
• k is always chosen to be an odd number.
• can be used for both classification and regression problems.
  • For classification: majority vote of the neighbours.
  • For regression: mean/median (or regression) of the neighbours.
• Instance-based learning
  • KNN does not learn a separate model.
  • Instead, it stores all training data and uses them directly at prediction time.
• Non-parametric model
  • KNN has no parameters (like weights in linear regression) to train.
  • The model is essentially the full dataset plus a distance measure.

Distance measures​

• Minkowski distance or $L^p$ norm
  • $L^p(x_j, x_q) = \left( \sum_i |x_{j,i} - x_{q,i}|^p \right)^{1/p}$
  • Euclidean distance: $p = 2$, for the dimensions are measuring similar properties, such as the width, height and depth of 3D objects.
  • Manhattan distance: $p = 1$, for the dimensions are measuring dissimilar properties, such as age, weight, and gender of a patient.
  • Hamming distance: the number of attributes on which the two points differ, for Boolean attribute values

Nomarlization​

• use the raw data from each dimension then the total distance will be affected by a change in scale in any dimension
• To avoid this, apply normalization to the measurements in each dimension.
• to compute the mean $\mu_i$ and standard deviation $\sigma_i$ of the values in each dimension, and rescale them
• The rescaling is done using the formula:
  • $x'_{j,i} = \frac{x_{j,i} - \mu_i}{\sigma_i}$ where $x'_{j,i}$ is the normalized value, $x_{j,i}$ is the original value, $\mu_i$ is the mean, and $\sigma_i$ is the standard deviation.

Time complexity​

• Conceptually trivial: Given a set of N examples and a query $x_q$, iterate through the examples, measure the distance to $x_q$ from each one, and keep the best k.
• $NN(k, x_q)$'s time complexity is $O(N)$, N is the number of examples in the training dataset.
• Use a k-dimensional tree: a balanced binary tree with an arbitrary number of dimensions.
  • Time complexity can be improved to $O(\log N)$
  • appropriate only when there are many more examples than dimensions
  • It works well with up to 10 dimensions with thousands of examples.
• Use a Hash table with a locality-sensitive hash (LSH)
  • Time complexity can be improved to $O(1)$

SVM​

a framework for finding a boundary that distinctly classifies the data points in an optimal way.

• supervised learning, binary classification
• SVM chooses the boundary with the maximum possible geometric margin, which has the largest distance to the nearest training data points of any class
• initially designed for binary classification problems but can also be applied for solving multi-class classification problems

Linear discriminant​

• $X_i$ is multiplied by its matching weight $w_i$
• all these products are added together and passed to a threshold function
• Decision surface: if $g(x) = w \cdot x \gt 0$ then $f(x) = +1 (class1)$ else $f(x) = -1 (class2)$
• Decision function: $f(x) = \text{sign}(g(x)) = \text{sign}(w_0 + w_1x)$
  • To make a decision, the continuous value $g(x)$ is passed through the sign function so that it outputs either +1 or -1.
• If the data from the two classes can be separated with a hyperplane, linearly separable.

Hyperplane​

• separates the data in 2D by a line or in 3D by a plane
• The orientation of the hyperplane is given by the vector $w$
• the location of the hyperplane is given by $w_0$
• The distance from the origin to the hyperplane is $\frac{|w_0|}{\|w\|}$
• If a given data sample $x^*$ and $g(x^*) = 0$, then this data sample is on the separation boundary. It can normally be assigned to any class.
• geometric margin: the minimum distance between the samples and the hyperplane by constructing and solving a constrained optimization problem
  • $\gamma_i = y_i \left( \frac{w}{\|w\|} \cdot x_i + \frac{w_0}{\|w\|} \right)$
• primary optimization problem: to maximize the minimal geometric distance across the training dataset of m samples.
  • $\max_{w, w_0} \Big( \min_{i=1,\ldots,N} \gamma_i \Big) = \max_{w, w_0} \Big( \min_{i=1,\ldots,N} \Big( y_i \Big( \frac{w}{\|w\|} \cdot x_i + \frac{w_0}{\|w\|} \Big) \Big) \Big)$
  • $\min_{w, w_0} \; \frac{1}{2}\|w\|^2$
  • $\text{s.t. } \; y_i (w \cdot x_i + w_0) \geq \min_{i=1,\ldots,N} \big( y_i (w \cdot x_i + w_0) \big)$
• dual optimization problem: easier to solve. More importantly the dual optimisation problem enables the so-called kernel trick in SVM
  • $\max_{\alpha} \; \sum_{i=1}^N \alpha_i - \frac{1}{2} \sum_{i=1}^N \sum_{j=1}^N \alpha_i \alpha_j y_i y_j (x_i \cdot x_j)$
  • $\min_{\alpha} \; \frac{1}{2} \sum_{i=1}^N \sum_{j=1}^N \alpha_i \alpha_j y_i y_j \,(x_i \cdot x_j) \;-\; \sum_{i=1}^N \alpha_i$
  • $\text{s.t.} \quad \sum_{i=1}^m \alpha_i y_i = 0, \quad \alpha_i \geq 0, \quad i=1,2,\ldots,N$

Attractive Properties​

• SVMs construct a maximum margin separator
  • the largest possible distance to example points, helping to improve generalization
• SVMs create a linear separating hyperplane
  • kernel trick: to embed the data into a higher-dimensional space
  • Often data that are not linearly separable in the original input space are easily separable in a higher-dimensional space
  • In general (excepted some special cases) if we have $N$ data points then they will always be separable in spaces of $N$ dimensions or more
• SVMs are a nonparametric method
  • retain training examples and potentially need to store them all
  • In practice, they often end up retaining only a small fraction of examples
  • have the flexibility to represent complex functions, but they are resistant to overfitting
• not usually expect to find a linear separator in the input space $x$, but we can find linear separators in the high-dimensional feature space $F(x)$ simply by replacing $(x_j x_k)$ in
  • $argmax_{\alpha} \sum_{j}\alpha_j - \frac{1}{2} \sum_{j,k}\alpha_j \alpha_k y_j y_k (x_j \cdot x_k)$
    • with $K(x_j, x_k) = F(x_j) \cdot F(x_k)$
    • $F(x_j) \cdot F(x_k)$ can often be computed without first computing $F$ for each point.
• In a higher dimensional feature space, which is created by transformation $F(x)$, if we can express $K(x_j \cdot x_k) = F(x_j) \cdot F(x_k)$, the kernel function $K(x_j \cdot x_k)$ can be applied to pairs of input data to evaluate dot product in some corresponding feature space.
  • kernel trick is to plug a kernel function $K(x_j \cdot x_k)$ into the dual optimisation problem to replace $(x_j \cdot x_k)$
  • Optimal linear separators can be found efficiently in feature spaces with billions of (or, in some cases, infinitely many) dimensions.
  • we can learn in the higher-dimensional space, but we compute only kernel functions rather than the full list of features for each data point.

Classification evaluation metrics​

Accuracy​

the proportion of correctly classified instances (data points or samples) among the total instances.

$Accuracy = \frac{TP + TN}{\text{Total number of predictions}}$

• a ratio of the number of correct predictions
• it may not be suitable for imbalanced datasets where the class distribution is skewed
  • a model that always predicts the majority class will have high accuracy but may not be useful in practice.

Precision​

the proportion of true positives among all positive predictions.

$Precision = \frac{TP}{\text{number of positive predictions}} = \frac{TP}{TP + FP}$

• the model's ability to not mistakenly view negatives as positives
• A high precision value indicates that the model has made fewer false positive predictions.
• it's useful to minimize the number of false positives.

Recall​

Sensitivity, True Positive Rate, the proportion of true positive instances among the actual positive instances

$Recall = \frac{TP}{\text{number of positive instances}} = \frac{TP}{TP + FN}$

• the model's ability to not mistakenly view actual positives as negatives
• A high recall value indicates that the model has successfully identified a large portion of the actual positive instances
• it's useful when the cost of false negatives is high
  • e.g. in medical diagnosis, where failing to identify a disease can have severe consequences

F1 Score​

the harmonic mean of precision and recall

$F1 = 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall}$

• a balanced evaluation of the model's performance
• It is particularly useful when dealing with imbalanced datasets, where one class is significantly more prevalent than the other.
• It's useful for imbalanced datasets to balance precision and recall.

Regression evaluation metrics​

Mean Absolute Error (MAE)​

the average of the absolute differences between the predicted values and the actual values

$MAE = \frac{1}{n} \sum_{i=1}^n \lvert y_i - \hat{y}_i \rvert$

• It measures the average magnitude of errors made by the model, without considering their direction
• A lower MAE value indicates that the model has made smaller prediction errors

Mean Squared Error (MSE)​

the average squared difference between the predicted and actual values

$MSE = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2$

• It's useful when you want to penalise larger errors more heavily, making it more sensitive to outliers at the same time
• often used as a loss function when training regression models
• A lower MSE value indicates that the model has smaller prediction errors, with a strong preference for avoiding large errors.

Root Mean Squared Error (RMSE)​

the square root of MSE

$RMSE = \sqrt{MSE}$

• it more interpretable as it is in the same units as the dependent variable.

R-Squared (Coefficient of Determination)​

how well the regression model approximates the actual data

$R^2 = 1 - \frac{SSR}{SST} = 1 - \frac{\sum_{i=1}^n (y_i - \hat{y}_i)^2}{\sum_{i=1}^n (y_i - \bar{y})^2}$

• the proportion of "sum squared regression (SSR)" and "total sum of squares (SST)"
  • SSR obviously captures the model's prediction errors
  • SST is the variance of the target variable.
    • can be viewed as a naive model ($\hat{y} = \bar{y}$) using the average value of the target variable as the prediction
• $R^2 = 1$: The model predicts perfectly (the error is 0).
• $R^2 = 0$: The model does no better than a naive model that always predicts the mean of the target variable.
• $R^2 < 0$: The model performs worse than simply predicting the mean, meaning its predictions increase the error compared to the naive baseline.

Method​

• a block of code grouped together and has a name
• can be invoked by its name to perform certain action
• can have parameters that represent the values needed for the method to run
• can have local variables usable only within its own code block.

Function vs Procedure​

• Procedure: no return value, perform an action
  • Example: move(), run(), deposit(), eat()
• Function: have a return value, do not perform any action
  • Example: total(), sum(), area()
• a function and behaves as a combined function procedure, but not recommended.

Method Overloading​

• Java allows methods in the same class to have the same name but different parameters.
• method signature: The method name together with the number and types of a method's parameter.

Parameter vs Arguments​

• Parameter: placeholder variables used at method definition, indicate the type and order of argument
• Arguments: data values passed to the method when the method is invoked or called.

Patterns​

The read pattern​

def <name>():
  <prompt>;
  return <type>

The update read-loop pattern​

<read function>
while (<value> != <end value>):
   <use the value>
   <read function>

The array-loop pattern​

for <value> in <range>:
  >use the item from array>

The any-pattern​

for <item> in <collection>:
  if (<test>):
    return True
return False

The every-pattern​

for <item> in <collection>:
  if (not(<test>)):
    return False
return True

The none-pattern​

for <item> in <collection>:
  if (<test>):
    return False
return True

Boolean Functions​

def isEven(number):
  if number % 2 == 0:
    return True
  else:
    return False

def isEven(number):
  return (number % 2 == 0)

Recursion​

• a technique where a method calls itself repeatedly.
• to provide a termination logic for a recursive method to avoid infinite execution.

def factorial(n):
  return 1 if (n == 1 or n == 0) else n * factorial(n - 1)

def factorial(n):
  F = lambda n: n * F(n-1) if n > 1 else 1
  return F(n)

Process in Programming​

• process is the method used to solve a problem
• Break it down-Build it up is a technique structured approach to handle complex problems.

• Trains a Vision-Language-Action (VLA) model by co-fine-tuning web-scale VLMs with robot trajectories, and treats robot actions as text tokens.
• Yields strong generalization and emergent capabilities (symbol understanding, reasoning, human recognition) beyond what appears in robot data.
• Runs in direct closed-loop control; largest evaluated model (55B) executes at ~1–3 Hz via a cloud (multi-TPU) inference setup.

What RT-2 Is​

• A family of VLA models (RT-2-PaLI-X, RT-2-PaLM-E) that fine-tune large VLMs on robot trajectories to output low-level actions.
• Target: generalizable, semantically aware manipulation policies that map images + instructions → actions end-to-end.
• RT-2 does not rely on a restricted 2D action space or calibrated cameras.
• The unified output space lets language and action tokens share the same model weights, without action-only layers.

Core Recipe​

• Directly train open-vocabulary VQA/dialogue VLMs to output robot actions while they still solve standard vision-language tasks.
• Build on RT-1 protocol/data, but replace the policy backbone with a large VLM.

Action as Language (Tokenization)​

• Discretize continuous action dims (Δpos/Δrot, gripper, terminate) into 256 bins; represent each dimension with an integer token.
• PaLI-X: reuse numeric tokens (≤1000). PaLM-E: overwrite 256 least-frequent tokens as action vocabulary (symbol tuning).
• Form a single output string per step (e.g., terminate Δposx Δposy Δposz Δrotx Δroty Δrotz gripper).

Co-Fine-Tuning & Output Constraint​

• Mix robot data with original web VQA/caption data in training batches (up-weight robot samples) to prevent forgetting and improve generalization.
• During decoding on robot tasks, restrict sampling to valid action tokens so outputs are always executable.

Closed-Loop Control & Real-Time Inference​

• RT-2 is trained and deployed for direct closed-loop control (camera → action → camera …), not just high-level planning.
• For large models, inference runs via a multi-TPU cloud service; RT-2-PaLI-X-55B reaches ~1–3 Hz; smaller models ~5 Hz.

Generalization & Benchmarks​

• Matches RT-1 on seen tasks but far exceeds baselines on unseen objects/backgrounds/environments (~2× vs RT-1/MOO; up to ~6× vs others).
• Open-source Language-Table sim: co-fine-tuned PaLI-3B outperforms baselines, showing the approach transfers to other robots/sims.

Emergent Capabilities​

• Symbol understanding (e.g., “move apple to 3 / heart / star”).
• Reasoning (visual matching, simple math like “sum of two plus one”, multilingual commands).
• Human recognition (e.g., “person with glasses”); none of these were present as low-level actions in robot data.
• Chain-of-thought (CoT) variant adds a Plan step before actions → supports multi-stage semantic reasoning (e.g., pick a rock as an improvised hammer; pick an energy drink for a tired person).

Scaling & Ablations​

• From-scratch training (even 5B) performs poorly; fine-tuning helps; co-fine-tuning helps most.
• Bigger models (55B > 5B) generalize better.
• PaLM-E variant shows an edge on math reasoning; PaLI-X stronger on symbols/vision reasoning on average.

Limitations​

• Does not learn fundamentally new motor skills beyond the distribution in robot data; mainly transfers semantic/visual knowledge.
• Compute/latency costly; real-time control can bottleneck. Limited availability of strong open VLMs and convenient FT APIs.

Future Directions (from the text)​

• Acquire new skills from human videos or richer datasets.
• Quantization/distillation for faster/cheaper inference.
• More open VLMs / FT APIs to make VLA models broadly buildable.

Ref​

• Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., Vuong, Q., Vanhoucke, V., Tran, H., Soricut, R., Singh, A., Singh, J., Sermanet, P., Sanketi, P. R., Salazar, G., Ryoo, M. S., Reymann, K., Rao, K., Pertsch, K., Mordatch, I., Michalewski, H., Lu, Y., Levine, S., Lee, L., Lee, T.-W. E., Leal, I., Kuang, Y., Kalashnikov, D., Julian, R., Joshi, N. J., Irpan, A., Ichter, B., Hsu, J., Herzog, A., Hausman, K., Gopalakrishnan, K., Fu, C., Florence, P., Finn, C., Dubey, K. A., Driess, D., Ding, T., Choromanski, K. M., Chen, X., Chebotar, Y., Carbajal, J., Brown, N., Brohan, A., Arenas, M. G., & Han, K. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control Proceedings of The 7th Conference on Robot Learning, Proceedings of Machine Learning Research. https://proceedings.mlr.press/v229/zitkovich23a.html

PaLM-E​

• ViT (e.g., ViT-4B, ViT-22B) extracts image embeddings.
• OSRT builds object-centric slot representations.
• These are injected into the LLM embedding space (PaLM variants: 8B, 62B, 540B) for high-level abstraction and planning, with execution delegated to low-level policies (e.g., RT-1).

Core idea​

• Build embodied language models by injecting continuous sensor inputs (images, states, other modalities) directly into a pretrained LLM’s embedding space, linking words ↔ percepts.
• Inputs are multimodal sentences that interleave text tokens with encoded visual/state tokens; outputs are text (answers or high-level plans).

Architecture & representations​

• Start from a decoder-only, autoregressive LLM (PaLM) and condition on a prefix that mixes text and encoder-produced vectors.
• Provide multiple encoder options:
  • State vectors (simplest).
  • ViT features with a learned projector ψ to match LLM embedding dimensionality.
  • Object-centric, 3D-aware OSRT (neural scene representations). Supports entity-label tokens (<obj j>) so the model can refer to specific objects in generated plans.

Training setup​

• Train end-to-end (encoders + projector + optionally the LLM) to output sequential decisions as natural text or answers (VQA, captioning).
• Dataset items contain (continuous observations, text sequence, prefix index); loss is cross-entropy on non-prefix tokens.
• Explore freezing the LLM (train encoders/projection only), and co-training across diverse tasks ("full mixture"; only ~9% is embodied data).

Planning & control loop​

• For planning/control, PaLM-E emits textual subgoals/skills drawn from a small skill vocabulary; a separate low-level policy executes them.
• The system runs closed-loop: execute → observe → (re)plan; PaLM-E acts as a high-level policy sequencing low-level skills.

Why not text-only LLMs or affordance-only grounding?​

• Prior work that feeds only text to the LLM (and uses external affordance models) is insufficient when spatial layout matters.
• PaLM-E instead grounds inside the LLM by injecting continuous observations, enabling direct plan generation while leveraging the LLM’s world knowledge.

Environments & use cases​

• Three domains: TAMP (grasp/stack planning), Language-Table (multi-object tabletop pushing), Mobile manipulation (kitchen tasks).
• Use cases to test embodied reasoning: affordance prediction, failure detection, long-horizon planning (low-level policies from RT-1).

Results (high level)​

• Transfer via co-training: One model trained on mixed tasks/embodiments achieves higher performance than task-specialists; "full mixture" yields >2× gains (Fig. 3).
• Few-shot/data efficiency: Solves robotics tasks with very few examples (e.g., 10–80 for Language-Table, 320 for TAMP). OSRT further improves data efficiency.
• Mobile manipulation: End-to-end embodied planning works in real kitchens, robust to disturbances; PaLM-E beats PaLI (zero-shot) and QT-OPT/CLIP baselines on affordance/failure detection.
• General V+L: The 562B generalist achieves state-of-the-art on OK-VQA and strong VQAv2/COCO without task-specific finetuning.
• Language retention & scaling: Freezing LLM preserves language ability but can struggle on some robotics tasks; unfrozen + scale up significantly reduces catastrophic forgetting.
• Emergent behaviors: Multimodal chain-of-thought and multi-image reasoning emerge in PaLM-E-562B, despite training on single-image prompts.

Takeaways​

• Injecting neural scene representations (OSRT) and entity-labeled multimodal tokens is effective even without massive embodied data.
• Diverse, joint training transfers vision-language knowledge into embodied decision-making, enabling data-efficient robot planning.
• Two viable paths to retain language skills during multimodal finetuning:
  1. Freeze the LLM, train encoders (max language retention, sometimes weaker robotics),
  2. Unfreeze and scale the LLM (much less forgetting, strong embodied performance).

Ref​

• Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., & Florence, P. (2023). PaLM-E: An Embodied Multimodal Language Model Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research. https://proceedings.mlr.press/v202/driess23a.html