RT-X​

• RT-X trains generalist robot policies by co-training RT-1/RT-2 on an X-embodiment mix of multi-robot, multi-task data, enabling efficient adaptation to new robots, tasks, and environments.
• It standardizes 1M+ trajectories from 22 embodiments into the Open X-Embodiment (RLDS/tfrecord) repository, unifying observations and 7-DoF actions via coarse alignment.
• Experiments show strong positive transfer and emergent skills (≈3× with RT-2-X on cross-robot tasks); performance scales with model capacity, short image histories, and web pretraining, while sensing/actuation diversity and frame alignment remain open problems.

Motivation​

• Seeks a generalist X-robot policy that can be efficiently adapted to new robots, tasks, and environments.
• Mirrors a trend from CV/NLP where general-purpose, web-scale pretrained models outperform narrow, task-specific models.
• Robotics lacks comparably large, diverse interaction datasets, making direct transfer of these lessons challenging.

Objectives​

1. Positive transfer: Test whether co-training on data from many robots improves performance on each training domain.
2. Ecosystem building: Organize large robotic datasets to enable future X-embodiment research.

Core Approach​

• Train RT-1 and RT-2 on data from 9 different manipulators, producing RT-X variants that outperform policies trained only on the evaluation domain and show better generalization and new capabilities.

What’s Different From Prior Transfer Methods​

• Many prior works reduce the embodiment gap via specialized mechanisms (shared action spaces, representation learning objectives, policy adaptation using embodiment metadata, decoupled robot/environment representations, domain translation).
• RT-X directly trains on X-embodiment data without explicit gap-reduction machinery and still observes positive transfer.

Dataset & Format (Open X-Embodiment)​

• 1M+ real robot trajectories, 22 embodiments (single-arm, bimanual, quadrupeds), pooled from 60 datasets / 34 labs, standardized for easy use.
• Uses RLDS (serialized tfrecord), supporting varied action spaces and input modalities (RGB, depth, point clouds), and efficient parallel loading across major DL frameworks.
• Language annotations are leveraged; PaLM is used to extract objects/behaviors from instructions.

Data Format Consolidation (Coarse Alignment)​

• Observations: History of recent images + language instruction. One canonical camera view per dataset is resized to a common resolution.
• Actions: Convert original controls to a 7-DoF end-effector vector (x, y, z, roll, pitch, yaw, gripper or their rates). Actions are normalized before discretization; outputs are de-normalized per embodiment.
• Deliberate non-alignment: Camera poses/properties are not standardized; action frame alignment across datasets is not enforced. The same action vector may cause different motions on different robots (absolute/relative, position/velocity allowed).

Policy Architectures​

• RT-1 (≈35M params): Transformer for control. Inputs: 15-frame image history + natural-language instruction.
  • Vision via ImageNet-pretrained EfficientNet; language via USE embedding.
  • Fuse via FiLM → 81 vision–language tokens → decoder-only Transformer outputs tokenized actions.
• RT-2 (VLA family): Internet-scale VLM co-fine-tuned to output action as text tokens (e.g., 1 128 91 241 5 101 127).
  • Any pretrained VLM can be adapted; this work uses RT-2–PaLI-X (ViT backbone + UL2 LM; primarily pretrained on WebLI).

Training Setup​

• Robotics data mixture: Data from 9 manipulators (a union of multiple well-known robotics datasets).
• Loss: Standard categorical cross-entropy over tokenized actions.
• Regimes:
  • RT-1-X: Trained solely on the robotics mixture.
  • RT-2-X: Co-fine-tuned on a ~1:1 mix of original VLM data and the robotics mixture.

Experimental Questions​

1. Does X-embodiment co-training improve in-domain performance (positive transfer)?
2. Does it improve generalization to unseen tasks?
3. How do model size, architecture, and dataset composition influence performance/generalization?

Key Results​

• Small-scale domains: RT-1-X outperforms the Original Method (the authors’ per-dataset baselines) on 4/5 datasets with a large average gain → limited data domains benefit greatly from X-embodiment co-training.
• Large-scale domains:
  • RT-1-X does not beat an RT-1 trained only on the embodiment-specific large dataset (suggests underfitting for this class).
  • RT-2-X (larger capacity) outperforms both Original Method and RT-1 → X-robot training helps even in data-rich regimes when using sufficient capacity.

Generalization & Emergent Skills​

• Unseen objects/backgrounds/environments: RT-2 and RT-2-X perform on par (VLM backbone already strong here).
• Emergent skills (transfer across robots): On Google Robot tasks that do not appear in RT-2’s dataset but exist in Bridge (for WidowX), RT-2-X ≈ 3× RT-2.
  • Removing Bridge from RT-2-X training significantly reduces hold-out performance → skills likely transferred from WidowX data.

Design Insights (Ablations)​

• Short image history notably improves generalization.
• Web pretraining is critical for large models’ high performance.
• Model capacity matters: 55B model succeeds more than 5B on emergent skills → greater capacity ⇒ greater cross-dataset transfer.
• Co-fine-tuning vs. fine-tuning: Similar performance in this study (attributed to the greater diversity of robotics data in RT-2-X vs. prior works).

Limitations (Open Problems)​

• Does not cover robots with very different sensing/actuation modalities.
• Does not study generalization to new robots nor define a decision criterion for when positive transfer will occur.
• Camera pose/properties and control frame remain unaligned; a deliberate but still challenging domain gap to address in future work.

Ref​

• O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., & Jain, A. (2024). Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. 2024 IEEE International Conference on Robotics and Automation (ICRA).

OOP vs Procedural Programming​

OOP​

• a programming paradigm built around the concept of objects, which contain data and code to manipulate data.
• The idea to model real-world entities and their interactions.
• Global Data (fields) are enclosed in the objects.
• Program components/tasks are easily divided across the development team / Requires more planning and design preparation
• Easier to manage and maintain dependencies between objects / OOP programs are much larger and complex
• Objects export the interface and hide the implementation and data / Tend to use more memory and GPU
• Code is highly reusable and easy to scale and distribute / Making changes in one class potentially impact others, which can complicate the development of the code.

Procedural Programming​

• the concept of procedure calls by structuring the program around procedures. (or functions/subroutines)
• a sequential manner unless directed otherwise.
• Global data (elements) is exposed to all the functions.
• Easier to compile and interpret / Difficult to scale or extend
• Straightforward and simpler to code / Dependencies between elements are unclear and not well-structured.
• Less memory requirements / Data is exposed and insecure due to its exposure across the whole program
• Easy to track the program flow / Hard to divide the work among programmers in a team.

Classes​

• A class is a template/blueprint used to create objects

class <class-name> (<extend - superclass>):
    <variable-name> = <value> #Class fields - data members

    def __init(self, <parameters>): #class constructor - object sbuilder
        <code>

    <method-name> (self, <parameters>): #methods
        <code>

Classes Py​

• Accessors: functions (with no parameters) in a Python class that provide access to the data attributes of an object.
  • known as getter methods, are named starting with the verb get, followed by the field name, which should start with an uppercase letter.
• Mutators: procedures (with parameter) in a Python class that enable the developer to modify the values of object attributes.
  • known as setter methods, are named starting with the verb set, followed by the field name, which should start with an uppercase letter.

def get<Variable> ():
    return self.<field>

def set<Variable> (self, value):
    self.<field> = value

Classes Java​

public class Bank {
  private Customer customer;
  private String branch;

  public Bank() {
    customer = new Customer();
  }

  public Bank(String name) {
    this();
    this.branch = name;
  }

  public boolean find(Bank bank) {
    return this.branch.equals(bank.branch);
  }
}

Packages​

Packages Java​

• used to group related classes
• like folders containing files (classes)
• either Java defined or user-defined
• used to write maintainable and portable code and to avoid class name conflicts.

Modules Py​

• used to grou prelated functio nand classes together
• normal Python scripts that are used into other scripts
• either Python defined or user-defined
• used to write maintainable and portable code to improve reusability

π0​

Problem & Motivation​

• Achieving real-world generality in robot learning is blocked by data scarcity, generalization, and robustness limits.
• Human intelligence most outpaces machines in versatility—solving diverse, physically situated tasks under constraints, language commands, and perturbations.
• In NLP/CV, foundation models pre-trained on diverse multi-task data, then fine-tuned (aligned) on curated datasets, outperform narrow specialists; the same paradigm is hypothesized for robotics.

Core Proposal​

• A novel flow-matching architecture built on a pre-trained Vision-Language Model (VLM) to inherit Internet-scale semantics.
• Further training adds robot actions, turning the model into a Vision-Language-Action (VLA) policy.
• Use cross-embodiment training to combine data from many robot types (single/dual-arm, mobile), despite differing configuration/action spaces.
• Employ action chunking + flow matching (diffusion variant) to model complex, continuous, high-frequency actions.
• Introduce an Action Expert (separate weights for action/state tokens), akin to a Mixture-of-Experts, augmenting the standard VLM.

Training Recipe (Pre- vs Post-Training)​

• Pre-training on highly diverse data builds broad, general physical abilities.
• Post-training on curated, task-specific data instills fluent, efficient strategies.
• Rationale: high-quality-only training lacks recovery behaviors; low-quality-only training lacks efficiency/robustness; combining both yields desired behavior.

Data & Backbone​

• ~10,000 hours of demonstrations + the OXE dataset; data spans 7 robot configurations and 68 tasks.
• VLM backbone initialized from PaliGemma (3B); add ~300M parameters for the action expert (total ~3.3B).
• Pre-training mixture: weighted combination of internal datasets + full OXE; n^0.43 weighting to down-weight overrepresented task-robot pairs.
• Unify interfaces: zero-pad qt/at to the largest robot dimension (18); mask missing image slots; late-fusion encoders map images/states to the same token space as language.

Modeling Details​

• Conditional flow matching models the continuous distribution over action chunks.
• Train with a diffusion-style loss on individual sequence elements (instead of cross-entropy), with separate weights for diffusion-related tokens.
• Flow path uses a linear-Gaussian schedule; sample noisy actions with ε∼N(0, I); predict denoising vector field; Euler integration from τ=0→1 at inference.
• Efficient inference by caching K/V for the observation prefix; action tokens recomputed per integration step.

High-Level Language Policy​

• Because the policy consumes language, a high-level VLM can decompose tasks (e.g., bussing) into intermediate language subgoals (SayCan-style planning), improving performance on complex, temporally extended tasks.

Evaluation Setup & Baselines​

• Out-of-box (direct prompting), fine-tuning on downstream tasks, and with high-level VLM providing intermediate commands.
• Compare against OpenVLA (7B, autoregressive discretization; no action chunks/high-frequency control) and Octo (93M; diffusion), trained on the same mixture.
• Include a compute-parity π0 (160k steps vs 700k) and a π0-small variant (no VLM init).

Key Results​

• Out-of-box: π0 outperforms all baselines; even compute-parity π0 beats OpenVLA/Octo; π0-small still surpasses them—highlighting the benefits of expressive architectures + diffusion/flow matching + VLM pre-training.
• Language following: π0 clearly exceeds π0-small across conditions:
  • π0-flat: only overall task command.
  • π0-human: human-provided intermediate steps.
  • π0-HL: high-level VLM-provided steps (fully autonomous).
  • Better language-following accuracy directly translates into stronger autonomous performance with high-level guidance.
• New dexterous tasks (e.g., bowls stacking, towel folding, microwave, drawer items, paper towel replacement):
  • Fine-tuned π0 generally outperforms OpenVLA, Octo, and small-data methods ACT / Diffusion Policy.
  • Pre-training helps most when tasks resemble pre-training data; pretrained π0 often beats from-scratch by up to 2×.
• Complex multi-stage tasks (laundry folding, table bussing, box building, to-go box, eggs):
  • π0 solves many tasks; full pre-training + fine-tuning performs best.
  • Gains from pre-training are especially large on harder tasks; absolute performance varies with task difficulty and pre-training coverage.

Takeaways & Limitations​

• π0 mirrors LLM training: pre-train for knowledge, post-train for alignment (instruction-following and execution).
• Limitations/open questions:
  • Optimal composition/weighting of pre-training data remains unclear.
  • Not all tasks work reliably; difficult to predict how much/what kind of data is needed for near-perfect performance.
  • Uncertain positive transfer across very diverse tasks/robots and to distinct domains (e.g., driving, navigation, legged locomotion).

Ref​

• Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., Levine, S., Li‑Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Shi, L. X, … Zhilinsky, U. (2025, June 21). π₀: A vision‑language‑action flow model for general robot control Robotics: Science and Systems (RSS), Los Angeles, CA, United States. https://roboticsconference.org/program/papers/10/

VIMA​

• Unified Multimodal Prompts: Reformulates diverse robot tasks (language, images, video) into a single sequence modeling problem.
• Object-Centric Tokenization: Uses object-level tokens (Mask R-CNN + ViT) instead of raw pixels, improving data efficiency and semantic generalization.
• Cross-Attention Conditioning: Conditions the policy on prompts via cross-attention, maintaining strong zero-shot performance even with small models or novel tasks.

Motivation​

• Robot task specification comes in many forms: one-shot demonstrations, language instructions, and visual goals.
• Traditionally, each task required distinct architectures and pipelines, leading to siloed systems with poor generalization.

Key Contributions​

1. Multimodal Prompting

   • A novel formulation that unifies diverse robot manipulation tasks into a sequence modeling problem.
   • Prompts are defined as interleaved sequences of text and images, enabling flexibility across task formats.

2. VIMA-BENCH

   • A large-scale benchmark with 17 tasks across six categories (object manipulation, goal reaching, novel concept grounding, video imitation, constraint satisfaction, visual reasoning).
   • Provides 650K expert trajectories and a four-level evaluation protocol for systematic generalization.

3. VIMA Agent

   • A transformer-based visuomotor agent with encoder-decoder architecture and object-centric design.
   • Encodes prompts with a pre-trained T5 model, parses images into object tokens via Mask R-CNN + ViT, and decodes actions autoregressively using cross-attention.

Design Insights​

• Object-Centric Representation: Passing variable-length object token sequences directly to the controller is more effective than pixel-based tokenization.
• Cross-Attention Conditioning: Stronger prompt focus and efficiency compared to simple concatenation (e.g., GPT-style).
• Robustness: Minimal degradation under distractors or corrupted prompts, aided by T5 backbone and object augmentation.

Results​

• Performance:

  • Outperforms baselines (VIMA-Gato, VIMA-Flamingo, VIMA-GPT) by up to 2.9× success rate in hardest zero-shot generalization.
  • With 10× less training data, still 2.7× better than best competitor.

• Scaling:

  • Sample-efficient: with just 1% of data, matches baselines trained with 10× more.
  • Generalization holds across L1–L4 evaluation, with smaller regression than alternatives.

Conclusion​

VIMA demonstrates that multimodal prompting is a powerful unifying framework for robot learning.
It achieves strong scalability, data efficiency, and generalization, establishing a solid starting point for future generalist robot agents.

Ref​

• Jiang, Y., Gupta, A., Zhang, Z., Wang, G., Dou, Y., Chen, Y., Fei-Fei, L., Anandkumar, A., Zhu, Y., & Fan, L. (2023). VIMA: Robot Manipulation with Multimodal Prompts Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research. https://proceedings.mlr.press/v202/jiang23b.html

RoboFlamingo​

• RoboFlamingo decouples vision-language understanding and control, using OpenFlamingo for perception and a lightweight policy head for sequential decision-making.
• Unlike prior VLM-based approaches, it requires only small-scale imitation fine-tuning on language-conditioned manipulation data, without large-scale co-fine-tuning.
• This design enables data-efficient, zero-shot generalizable, and deployable robot manipulation policies on modest compute resources.

Key Idea​

• Proposes RoboFlamingo, a simple framework to adapt existing VLMs for robotic manipulation with lightweight fine-tuning.
• Built on OpenFlamingo, decoupling vision-language understanding from decision-making.
• Pre-trained VLM handles language and visual comprehension, while a dedicated policy head models sequential history.
• Fine-tuned only on language-conditioned manipulation datasets using imitation learning.

Advantages​

• Requires only a small amount of demonstrations to adapt to downstream manipulation tasks.
• Provides open-loop control capability → deployable on low-performance platforms.
• Can be trained/evaluated on a single GPU server, making it a cost-effective and accessible solution.

Benchmarks​

• Evaluated on CALVIN benchmark (34 tasks, 1000 instruction chains).
• RoboFlamingo achieves 2× performance improvements over previous state-of-the-art methods.

Performance​

• Imitation Learning: Outperforms all baselines across all metrics.
• Zero-shot Generalization:
  • Vision: Stronger generalization in ABC→D setting.
  • Language: Robust to GPT-4 generated synonymous instructions.
• Ablation Studies:
  • Ignoring history (MLP w/o hist) gives worst results.
  • LSTM and GPT-based policy heads perform best (LSTM chosen as default).
  • VL pre-training is crucial for downstream manipulation.
  • Larger VLMs show better data efficiency.
  • Instruction fine-tuning improves both seen and unseen tasks.

Flexibility of Deployment​

• Supports open-loop control by predicting entire action sequences with a single inference → reduces latency and test-time compute.
• Direct open-loop use without retraining can degrade performance; mitigated with jump-step demonstrations.

Conclusion​

• Demonstrates that pre-trained VLMs enable data efficiency and strong zero-shot generalization in robotic manipulation.
• RoboFlamingo is presented as an intuitive, efficient, and open solution, with high potential when combined with large-scale real robot data.

Ref​

• Li, X., Liu, M., Zhang, H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., & Liu, H. (2024). Vision-language foundation models as effective robot imitators. International Conference on Learning Representations (ICLR 2024), Vienna, Austria.

OpenVLA​

• OpenVLA is a 7B open-source VLA model built on Llama2 + DINOv2 + SigLIP, trained on 970k demos, achieving stronger generalization and robustness than closed RT-2-X (55B) and outperforming Diffusion Policy.
• It introduces efficient adaptation via LoRA (1.4% params, 8× compute reduction) and 4-bit quantization (half memory, same accuracy), enabling fine-tuning and inference on consumer GPUs.
• Limitations remain (single-image input, <90% reliability, limited throughput), but OpenVLA provides the first open, scalable framework for generalist robot policies.

Motivation​

• Training robot policies from scratch struggles with robustness and generalization.
• Fine-tuning vision-language-action (VLA) models offers reusable, generalizable visuomotor policies.
• Barriers: prior VLAs are closed-source, lack best practices for adaptation, and need server-class hardware.

Model & Training​

• OpenVLA: 7B parameters, open-source.
• Built on Llama 2 with fused DINOv2 + SigLIP vision encoders.
• Trained on 970k robot demonstrations from Open-X Embodiment dataset.
• Represents robot actions as tokens (discretized into 256 bins, replacing unused Llama tokens).
• Standard next-token prediction objective.

Architecture & Approach​

• End-to-end fine-tuning of VLM to generate robot actions as tokens.
• Differs from modular methods (e.g., Octo) that stitch separate encoders/decoders.
• Vision features are obtained by encoding the same input image with both SigLIP and DINOv2, then channel-wise concatenated and passed through an MLP projector. This preserves SigLIP’s semantic alignment with language and DINOv2's spatial reasoning, giving the VLM richer multimodal context for manipulation tasks.
• Uses Prismatic VLM backbone with multi-resolution features (spatial reasoning + semantics).

Performance​

• Outperforms closed RT-2-X (55B) by +16.5% task success with 7× fewer parameters.
• Beats Diffusion Policy (from-scratch imitation learning) by +20.4% on multi-task language-grounded settings.
• Demonstrates robust behaviors (distractor resistance, error recovery).

Efficiency​

• Introduces parameter-efficient fine-tuning:
  • LoRA updates only 1.4% of parameters yet matches full fine-tuning.
  • Can fine-tune on a single A100 GPU in ~10–15 hours (8× compute reduction).
• Quantization:
  • 4-bit inference matches bfloat16 accuracy while halving memory footprint.
  • Runs at 3Hz on consumer GPUs (e.g., A5000, 16GB).

Evaluations​

• Tested across 29 tasks and multiple robots (WidowX, Google robot, Franka).
• Strong generalization on:
  • Visual (unseen backgrounds/distractors).
  • Motion (new object positions/orientations).
  • Physical (new object shapes/sizes).
  • Semantic (unseen tasks, instructions).
• First generalist open-source VLA achieving ≥50% success rate across all tested tasks.

Design Insights​

• Fine-tuning the vision encoder (vs. freezing) crucial for robotic control.
• Higher image resolution (384px vs. 224px) adds 3× compute without performance gains.
• Training required 27 epochs, far more than typical VLM runs, to surpass 95% action token accuracy.

Limitations & Future Work​

• Supports only single-image observations (no proprioception, no history).
• Inference throughput (~6Hz on RTX 4090) insufficient for high-frequency control (e.g., ALOHA at 50Hz).
• Success rates remain below 90% in challenging tasks.
• Open questions:
  • Impact of base VLM size on performance.
  • Benefits of co-training with Internet-scale data.
  • Best visual features for VLAs.

Contributions​

1. First open-source generalist VLA with strong performance.
2. Scalable end-to-end training pipeline (action-as-token).
3. Demonstrates LoRA + quantization for consumer-grade GPU adaptation.
4. Provides code, checkpoints, and data curation recipes to support future research.

Ref​

• Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E. P., Sanketi, P. R., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., & Finn, C. (2025). OpenVLA: An Open-Source Vision-Language-Action Model Proceedings of The 8th Conference on Robot Learning, Proceedings of Machine Learning Research. https://proceedings.mlr.press/v270/kim25c.html

Color Legend​

Categoric​

• One Variable
  • ⚫ Waffle
  • 🟢 Bar Plot
  • 🟢 Lollipop
  • 🟢 Word Cloud
  • 🔴 Circular Packing
  • 🔴 Doughnut
  • 🔴 Pie
  • 🔴 Treemap
• Two or More Variables
  • Two Independent Lists
    • 🔴 Venn Diagram
  • Nested
    • 🟢 Bar Plot
    • 🔴 Circular Packing
    • 🔴 Dendrogram
    • 🔴 Sunburst
    • 🔴 Treemap
  • Subgroup
    • ⚫ Grouped Scatter
    • ⚫ Heatmap
    • 🟢 Lollipop
    • 🟢 Parallel Plot
    • Spider
    • 🔴 Grouped Bar Plot
    • 🔴 Grouped Bar Plot
    • 🟤 Sankey Diagram
  • Adjacency
    • 🟤 Arc
    • 🟤 Chord
    • 🟤 Network
    • 🟤 Sankey
    • ⚫ Heatmap

Relational​

• Network
  • ⚫ Heatmap
  • 🟢 Hive
  • 🟤 Arc
  • 🟤 Chord
  • 🟤 Network
  • 🟤 Sankey
• Nested
  • No Value
    • 🔴 Circular Packing
    • 🔴 Dendrogram
    • 🔴 Sunburst
    • 🔴 Treemap
    • 🟤 Sankey
  • Value for Leaf
    • 🔴 Circular Packing
    • 🔴 Dendrogram
    • 🔴 Sunburst
    • 🔴 Treemap
    • 🟤 Sankey
  • Value for Edges
    • 🔴 Dendrogram
    • 🟤 Chord
    • 🟤 Sankey
  • Value for Connection
    • Edge Bundling

Map​

• 🟣 Bubble Map
• 🟣 Choropleth
• 🟣 Connected Map
• 🟣 Map
• 🟣 Map Hexbin

Time Series​

• One Series
  • 🟡 Box Plot
  • 🟡 Violin
  • 🟡 Ridge Line
  • 🔵 Area
  • 🔵 Line Plot
  • 🟢 Bar Plot
  • 🟢 Lollipop
• Several Series
  • 🟡 Box Plot
  • 🟡 Violin
  • 🟡 Ridge Line
  • ⚫ Heatmap
  • 🔵 Line Plot
  • 🔵 Stacked Area
  • 🔵 Stream Graph

Categoric and Numeric​

• One Numeric + One Categoric
  • One Observation, per Group
    • ⚫ Waffle
    • 🟢 Bar Plot
    • 🟢 Lollipop
    • 🟢 Word Cloud
    • 🔴 Circular Packing
    • 🔴 Doughnut
    • 🔴 Pie
    • 🔴 Treemap
  • Several Observations, per Group
    • 🟡 Box Plot
    • 🟡 Violin
    • 🟡 Ridge Line
    • 🟡 Density
    • 🟡 Histogram
• One Category, Several Numeric
  • No Order
    • 🟡 Box Plot
    • 🟡 Violin
    • ⚫ Grouped Scatter
    • ⚫ 2D Density
    • ⚫ PCA
    • ⚫ Correlogram
  • A Numeric is Ordered
    • ⚫ Connected Scatter
    • 🔵 Area
    • 🔵 Line Plot
    • 🔵 Stacked Area
    • 🔵 Stream Graph
  • One Value Per Group
    • ⚫ Grouped Scatter
    • ⚫ Heatmap
    • 🟢 Lollipop
    • 🟢 Parallel Plot
    • 🟢 Spider Plot
    • 🔴 Grouped Bar Plot
    • 🔴 Grouped Bar Plot
    • 🟤 Sankey Diagram
• Several Categories, One Numeric
  • Subgroup
    • One Observation. per Group
      • ⚫ Grouped Scatter
      • ⚫ Heatmap
      • 🟢 Lollipop
      • 🟢 Parallel Plot
      • 🟢 Spider Plot
      • 🔴 Grouped Bar Plot
      • 🔴 Grouped Bar Plot
      • 🟤 Sankey Diagram
    • Several Observations, per Group
      • 🟡 Box Plot
      • 🟡 Violin
  • Nested
    • One Observation. per Group
      • 🟢 Bar Plot
      • 🔴 Circular Packing
      • 🔴 Dendrogram
      • 🔴 Sunburst
      • 🔴 Treemap
    • Several Observations. per Group
      • 🟡 Box Plot
      • 🟡 Violin
  • Adjacency
    • ⚫ Heatmap
    • 🟤 Arc
    • 🟤 Chord
    • 🟤 Network
    • 🟤 Sankey

Numeric​

• One Numeric Variable
  • 🟡 Density
  • 🟡 Histogram
• Two Numeric Variables
  • Not Ordered
    • Few Points
      • 🟡 Box Plot
      • 🟡 Histogram
      • ⚫ Scatter Plot
    • Many Points
      • 🟡 Density
      • 🟡 Violin
      • ⚫ 2D Density
      • 🔵 Marginal Distribution
  • Ordered
    • ⚫ Connected Scatter
    • 🔵 Area Plot
    • 🔵 Line Plot
• Three Numeric Variables
  • Not Ordered
    • 🟡 Box Plot
    • 🟡 Violin
    • ⚫ Bubble Plot
    • ⚫ 3d Scatter or Surface
  • Ordered
    • 🔵 Area
    • 🔵 Line Plot
    • 🔵 Stacked Area
    • 🔵 Stream Graph
• Several Numeric Variables
  • Ordered
    • 🔵 Area
    • 🔵 Line Plot
    • 🔵 Stacked Area
    • 🔵 Stream Graph
  • Not Ordered
    • 🟡 Box Plot
    • 🟡 Ridge Line
    • 🟡 Violin
    • ⚫ Correlogram
    • ⚫ Heatmap
    • ⚫ PCA
    • 🔴 Dendrogram

Ref​

• from Data to Viz

Vocabulary & Expressions​

Problem formulation (문제 정의)​

• The first step is to figure out what problem you want to solve.
  1. “사용자에게 어떤 문제를 해결해주고 싶은가?” → 모호하지 않고 구체적으로 정의해야 함.
  2. “그 문제 중 어떤 부분을 머신러닝으로 풀 수 있는가?” → 예: 사진을 라벨로 매핑하는 함수 학습.
• 이를 구체화하려면 ML 컴포넌트에 대해 loss function 을 지정해야 한다.
• 문제를 쪼개보면 일부는 전통적 SW 엔지니어링으로 해결 가능하고, 일부만 ML로 다뤄야 할 수 있다.
• 학습 유형은 지도·비지도·강화·준지도(semisupervised)까지 연속선상에 있음.
  • Semisupervised learning: 일부 라벨만 활용해 비라벨 데이터에서 더 많은 정보 추출.
  • Weakly supervised learning: 부정확·노이즈 라벨을 사용.
• 결론: Noise와 label 부족은 “지도 ↔ 비지도” 사이의 연속체를 형성한다.

Data collection & management (데이터 수집/관리)​

• 데이터는 직접 제작, 크라우드소싱, 사용자 행동에서 수집 가능.
• 부족할 때는 transfer learning 활용.
• Privacy 검토와 동의, 공정성, federated learning 등 고려 필요.
• Data provenance(출처 관리): 데이터 정의, 값의 범위, 생성 주체, 중단 여부, 정의 변경 이력 등 추적 → 파이프라인 안정성이 알고리즘보다 중요.
• 항상 자문: “이 데이터는 내 문제를 풀기에 적절한가? 입력과 출력 모두 충분히 담고 있는가?”
• Learning curve 로 데이터 확장 효과/학습 plateau 확인.
• 방어적 태도 필요: 입력 오류, 누락, 적대적 사용자, 철자 불일치 등 처리.
• Data augmentation (회전, 이동, 노이즈 추가 등)으로 모델 강건성 향상.
• 불균형 데이터는 undersampling, oversampling, SMOTE/ADASYN, boosting 등으로 완화.
• 아웃라이어는 로그 변환 등으로 영향 축소, 트리 모델은 상대적으로 강건.

Feature engineering (특징 엔지니어링)​

• Quantization: 연속값을 구간(bin)으로 강제.
• One-hot encoding: 범주형 속성을 다중 Boolean으로 변환.
• 도메인 지식 기반 새 특성 추가 (예: 날짜 → 주말/공휴일 여부).
• “At the end of the day, some ML projects succeed and some fail… the most important factor is the features used.” (Pedro Domingos)

Exploratory data analysis (EDA) & visualization​

• 목표: 예측/검증이 아닌 데이터 이해.
• Histograms, scatter plots 로 분포/결측/오류/이상치 확인.
• 클러스터링 → 프로토타입 시각화, 이상치 탐지 (“고양이 vs 사자 옷 입은 고양이”).
• 차원 축소 (예: t-SNE)로 고차 데이터를 2D/3D로 시각화.

Model selection & training​

• 데이터가 정리되면 모델 구축 단계.
• Random forests → 범주형 특징 많고 일부 무관할 때.
• Nonparametric methods → 데이터 많고 지식 부족, 특징 선택 고민 줄이고 싶을 때.
• Logistic regression → 선형 분리 가능(또는 feature engineering 후).
• SVM → 데이터 크기 작고 차원 높을 때.
• Deep neural nets → 패턴 인식(이미지·음성).
• 하이퍼파라미터는 경험 + 탐색으로 조율.
• 검증 데이터 남용 시 validation overfitting 위험 → 여러 검증셋 필요.
• 성능 평가: ROC curve, AUC, confusion matrix.
• 중요한 건 아이디어–실험–검증 반복 사이클을 빠르게 하는 것.

Trust, interpretability, explainability​

• 단순히 지표 성능만으로는 신뢰 부족 → 규제·언론·사용자도 신뢰성 원함.
• Accountability: 오류 발생 시 책임 주체와 항소 절차 필요.
• Interpretability: 모델 내부를 직접 이해 (트리, 선형회귀).
  • 핵심 질문: “If I change x, how will the output change?”
• Explainability: 블랙박스 모델 + 별도 모듈로 설명 (예: LIME).
• 단순 설명이 잘못된 확신을 줄 수 있음. → 테스트와 실제 성능이 더 큰 신뢰를 준다.
• “안전하다고 설명만 있는 실험기 vs 100회 무사비행한 비행기” 비유.

Operation, monitoring, maintenance​

• 운영 단계에서는 롱테일 입력(long tail) 문제 등장 → 예상 못한 입력 지속 발생. → 실시간 모니터링과 사람 평가자 필요.
• Nonstationarity: 세상과 사용자 행동 변화 → 최신 데이터 vs 안정적 모델 트레이드오프.
• 신선도 요구 다름: 어떤 문제는 매일/매시간 새 모델, 어떤 문제는 수개월 동일 모델.
• 배포 자동화 → 작은 변경은 자동 승인, 큰 변경은 리뷰.
• Online vs Offline model: 기존 모델 점진적 수정 vs 매번 처음부터 재학습.
• 데이터 자체가 바뀔 수도 있음 (스팸 이메일 → 스팸 문자, 음성, 영상 등).

Checklist​

Tests for Features and Data​

• Feature expectations are captured in a schema.
• All features are beneficial.
• No feature’s cost is too much.
• Features adhere to meta-level requirements.
• The data pipeline has appropriate privacy controls.
• New features can be added quickly.
• All input feature code is tested.

Tests for Model Development​

• Every model specification undergoes a code review.
• Every model is checked in to a repository.
• Offline proxy metrics correlate with actual metrics.
• All hyperparameters have been tuned.
• The impact of model staleness is known.
• A simpler model is not better.
• Model quality is sufficient on all important data slices.
• The model has been tested for considerations of inclusion.

Tests for Machine Learning Infrastructure​

• Training is reproducible.
• Model specification code is unit tested.
• The full ML pipeline is integration tested.
• Model quality is validated before attempting to serve it.
• The model allows debugging by observing the step-by-step computation of training or inference on a single example.
• Models are tested via a canary process before they enter production serving environments.
• Models can be quickly and safely rolled back to a previous serving version.

Monitoring Tests for Machine Learning​

• Dependency changes result in notification.
• Data invariants hold in training and serving inputs.
• Training and serving features compute the same values.
• Models are not too stale.
• The model is numerically stable.
• The model has not experienced regressions in training speed, serving latency, throughput, or RAM usage.
• The model has not experienced a regression in prediction quality on served data.

Ref​

• Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2016). What’s your ML test score? A rubric for ML production systems. NIPS Workshop on Reliable Machine Learning in the Wild.

Nearest-neighbor Models​

• 쿼리점 $x_q$에 대해 가장 가까운 $k$개의 이웃을 찾아 분류 또는 회귀에 사용한다.
  • 분류: 다수결
  • 회귀: 평균, 중앙값, 혹은 국소적 선형회귀
• 거리 척도: Minkowski 거리
  • $L_p(x_j, x_q) = \left( \sum_i |x_{j,i} - x_{q,i}|^p \right)^{1/p}$
  • $p=2$ → 유클리드 거리
  • $p=1$ → 맨해튼 거리
  • 불리언 속성 → 해밍 거리
  • 공분산 고려 → 마할라노비스 거리
• 차원의 저주 (curse of dimensionality):
  • 평균 이웃 부피: $\ell^n = k/N \;\;\Rightarrow\;\; \ell = (k/N)^{1/n}$
  • $n$이 커질수록 $\ell$ 값이 커져 이웃이 “멀어진다”.
  • 대부분의 점은 고차원 공간에서 경계(껍질)에 몰린다.
  • 저차원: 보간(interpolation) 가능
  • 고차원: 외삽(extrapolation)이 많아져 일반화 어려움

k-d trees​

• 데이터를 차원별로 분할해 만든 이진 트리.
• 각 노드에서 특정 차원의 중앙값 $m$을 기준으로 $x_i \le m$ 여부에 따라 좌/우로 분할한다.
• 탐색: 쿼리점 기준으로 한쪽 브랜치로 내려가며 후보를 찾되, 경계와 가까우면 반대편 서브트리도 확인해야 한다.
• 효율 조건: 데이터 수가 차원 수보다 훨씬 많아야 하며, 최소 $2^n$개 이상 필요하다.
• 실용 범위:
  • 약 10차원 이하에서는 수천 개 데이터
  • 약 20차원 이하에서는 수백만 개 데이터

Support Vector Machines (SVM)​

• 최대 마진 분리자(maximum margin separator)를 찾는다.
• 목표: 경험적 손실 최소화 대신 일반화 손실 최소화
• 결정 경계: $\{x : w \cdot x + b = 0\}$
• 학습은 이차계획법(QP) 최적화 문제로 정식화된다.
  • 이중 표현(dual form):
    $\arg\max_\alpha \sum_j \alpha_j - \tfrac{1}{2} \sum_{j,k} \alpha_j \alpha_k y_j y_k (x_j \cdot x_k)$
  • 제약조건: $\alpha_j \ge 0,\; \sum_j \alpha_j y_j = 0$
• 최적 해에서 대부분 $\alpha_j = 0$이고, 경계 근처의 점들(서포트 벡터)만 $\alpha_j > 0$이다.
• 예측 함수:
  $h(x) = \text{sign}\Big(\sum_j \alpha_j y_j (x \cdot x_j) - b \Big)$
• 장점:
  • 서포트 벡터만 유지하면 되므로 효율적
  • 비모수적 유연성 + 모수적 안정성(과적합 억제)

The Kernel Trick​

• 커널 트릭: 실제 고차원 특징 공간 $F(x)$를 계산하지 않고, 내적만을 커널 함수로 대체한다.
  • $K(x,z) = F(x)\cdot F(z)$
• 대표 커널 함수:
  • 다항 커널: $K(x,z) = (1 + x \cdot z)^d$
  • 가우시안 커널 (RBF): $K(x,z) = e^{-\gamma \|x-z\|^2}$
• 소프트 마진 분류기: 일부 오분류 허용, 오분류된 점을 올바른 쪽으로 이동시키는 거리만큼 패널티를 부여한다.
• 커널 기법은 내적에만 의존하는 다른 알고리즘에도 적용 가능하다.
• Mercer's theorem: “합리적인” 커널 함수는 항상 어떤 특징 공간에서의 내적에 해당한다.