Method​

• a block of code grouped together and has a name
• can be invoked by its name to perform certain action
• can have parameters that represent the values needed for the method to run
• can have local variables usable only within its own code block.

Function vs Procedure​

• Procedure: no return value, perform an action
  • Example: move(), run(), deposit(), eat()
• Function: have a return value, do not perform any action
  • Example: total(), sum(), area()
• a function and behaves as a combined function procedure, but not recommended.

Method Overloading​

• Java allows methods in the same class to have the same name but different parameters.
• method signature: The method name together with the number and types of a method's parameter.

Parameter vs Arguments​

• Parameter: placeholder variables used at method definition, indicate the type and order of argument
• Arguments: data values passed to the method when the method is invoked or called.

Patterns​

The read pattern​

def <name>():
  <prompt>;
  return <type>

The update read-loop pattern​

<read function>
while (<value> != <end value>):
   <use the value>
   <read function>

The array-loop pattern​

for <value> in <range>:
  >use the item from array>

The any-pattern​

for <item> in <collection>:
  if (<test>):
    return True
return False

The every-pattern​

for <item> in <collection>:
  if (not(<test>)):
    return False
return True

The none-pattern​

for <item> in <collection>:
  if (<test>):
    return False
return True

Boolean Functions​

def isEven(number):
  if number % 2 == 0:
    return True
  else:
    return False

def isEven(number):
  return (number % 2 == 0)

Recursion​

• a technique where a method calls itself repeatedly.
• to provide a termination logic for a recursive method to avoid infinite execution.

def factorial(n):
  return 1 if (n == 1 or n == 0) else n * factorial(n - 1)

def factorial(n):
  F = lambda n: n * F(n-1) if n > 1 else 1
  return F(n)

Process in Programming​

• process is the method used to solve a problem
• Break it down-Build it up is a technique structured approach to handle complex problems.

• Trains a Vision-Language-Action (VLA) model by co-fine-tuning web-scale VLMs with robot trajectories, and treats robot actions as text tokens.
• Yields strong generalization and emergent capabilities (symbol understanding, reasoning, human recognition) beyond what appears in robot data.
• Runs in direct closed-loop control; largest evaluated model (55B) executes at ~1–3 Hz via a cloud (multi-TPU) inference setup.

What RT-2 Is​

• A family of VLA models (RT-2-PaLI-X, RT-2-PaLM-E) that fine-tune large VLMs on robot trajectories to output low-level actions.
• Target: generalizable, semantically aware manipulation policies that map images + instructions → actions end-to-end.
• RT-2 does not rely on a restricted 2D action space or calibrated cameras.
• The unified output space lets language and action tokens share the same model weights, without action-only layers.

Core Recipe​

• Directly train open-vocabulary VQA/dialogue VLMs to output robot actions while they still solve standard vision-language tasks.
• Build on RT-1 protocol/data, but replace the policy backbone with a large VLM.

Action as Language (Tokenization)​

• Discretize continuous action dims (Δpos/Δrot, gripper, terminate) into 256 bins; represent each dimension with an integer token.
• PaLI-X: reuse numeric tokens (≤1000). PaLM-E: overwrite 256 least-frequent tokens as action vocabulary (symbol tuning).
• Form a single output string per step (e.g., terminate Δposx Δposy Δposz Δrotx Δroty Δrotz gripper).

Co-Fine-Tuning & Output Constraint​

• Mix robot data with original web VQA/caption data in training batches (up-weight robot samples) to prevent forgetting and improve generalization.
• During decoding on robot tasks, restrict sampling to valid action tokens so outputs are always executable.

Closed-Loop Control & Real-Time Inference​

• RT-2 is trained and deployed for direct closed-loop control (camera → action → camera …), not just high-level planning.
• For large models, inference runs via a multi-TPU cloud service; RT-2-PaLI-X-55B reaches ~1–3 Hz; smaller models ~5 Hz.

Generalization & Benchmarks​

• Matches RT-1 on seen tasks but far exceeds baselines on unseen objects/backgrounds/environments (~2× vs RT-1/MOO; up to ~6× vs others).
• Open-source Language-Table sim: co-fine-tuned PaLI-3B outperforms baselines, showing the approach transfers to other robots/sims.

Emergent Capabilities​

• Symbol understanding (e.g., “move apple to 3 / heart / star”).
• Reasoning (visual matching, simple math like “sum of two plus one”, multilingual commands).
• Human recognition (e.g., “person with glasses”); none of these were present as low-level actions in robot data.
• Chain-of-thought (CoT) variant adds a Plan step before actions → supports multi-stage semantic reasoning (e.g., pick a rock as an improvised hammer; pick an energy drink for a tired person).

Scaling & Ablations​

• From-scratch training (even 5B) performs poorly; fine-tuning helps; co-fine-tuning helps most.
• Bigger models (55B > 5B) generalize better.
• PaLM-E variant shows an edge on math reasoning; PaLI-X stronger on symbols/vision reasoning on average.

Limitations​

• Does not learn fundamentally new motor skills beyond the distribution in robot data; mainly transfers semantic/visual knowledge.
• Compute/latency costly; real-time control can bottleneck. Limited availability of strong open VLMs and convenient FT APIs.

Future Directions (from the text)​

• Acquire new skills from human videos or richer datasets.
• Quantization/distillation for faster/cheaper inference.
• More open VLMs / FT APIs to make VLA models broadly buildable.

Ref​

• Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., Vuong, Q., Vanhoucke, V., Tran, H., Soricut, R., Singh, A., Singh, J., Sermanet, P., Sanketi, P. R., Salazar, G., Ryoo, M. S., Reymann, K., Rao, K., Pertsch, K., Mordatch, I., Michalewski, H., Lu, Y., Levine, S., Lee, L., Lee, T.-W. E., Leal, I., Kuang, Y., Kalashnikov, D., Julian, R., Joshi, N. J., Irpan, A., Ichter, B., Hsu, J., Herzog, A., Hausman, K., Gopalakrishnan, K., Fu, C., Florence, P., Finn, C., Dubey, K. A., Driess, D., Ding, T., Choromanski, K. M., Chen, X., Chebotar, Y., Carbajal, J., Brown, N., Brohan, A., Arenas, M. G., & Han, K. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control Proceedings of The 7th Conference on Robot Learning, Proceedings of Machine Learning Research. https://proceedings.mlr.press/v229/zitkovich23a.html

PaLM-E​

• ViT (e.g., ViT-4B, ViT-22B) extracts image embeddings.
• OSRT builds object-centric slot representations.
• These are injected into the LLM embedding space (PaLM variants: 8B, 62B, 540B) for high-level abstraction and planning, with execution delegated to low-level policies (e.g., RT-1).

Core idea​

• Build embodied language models by injecting continuous sensor inputs (images, states, other modalities) directly into a pretrained LLM’s embedding space, linking words ↔ percepts.
• Inputs are multimodal sentences that interleave text tokens with encoded visual/state tokens; outputs are text (answers or high-level plans).

Architecture & representations​

• Start from a decoder-only, autoregressive LLM (PaLM) and condition on a prefix that mixes text and encoder-produced vectors.
• Provide multiple encoder options:
  • State vectors (simplest).
  • ViT features with a learned projector ψ to match LLM embedding dimensionality.
  • Object-centric, 3D-aware OSRT (neural scene representations). Supports entity-label tokens (<obj j>) so the model can refer to specific objects in generated plans.

Training setup​

• Train end-to-end (encoders + projector + optionally the LLM) to output sequential decisions as natural text or answers (VQA, captioning).
• Dataset items contain (continuous observations, text sequence, prefix index); loss is cross-entropy on non-prefix tokens.
• Explore freezing the LLM (train encoders/projection only), and co-training across diverse tasks ("full mixture"; only ~9% is embodied data).

Planning & control loop​

• For planning/control, PaLM-E emits textual subgoals/skills drawn from a small skill vocabulary; a separate low-level policy executes them.
• The system runs closed-loop: execute → observe → (re)plan; PaLM-E acts as a high-level policy sequencing low-level skills.

Why not text-only LLMs or affordance-only grounding?​

• Prior work that feeds only text to the LLM (and uses external affordance models) is insufficient when spatial layout matters.
• PaLM-E instead grounds inside the LLM by injecting continuous observations, enabling direct plan generation while leveraging the LLM’s world knowledge.

Environments & use cases​

• Three domains: TAMP (grasp/stack planning), Language-Table (multi-object tabletop pushing), Mobile manipulation (kitchen tasks).
• Use cases to test embodied reasoning: affordance prediction, failure detection, long-horizon planning (low-level policies from RT-1).

Results (high level)​

• Transfer via co-training: One model trained on mixed tasks/embodiments achieves higher performance than task-specialists; "full mixture" yields >2× gains (Fig. 3).
• Few-shot/data efficiency: Solves robotics tasks with very few examples (e.g., 10–80 for Language-Table, 320 for TAMP). OSRT further improves data efficiency.
• Mobile manipulation: End-to-end embodied planning works in real kitchens, robust to disturbances; PaLM-E beats PaLI (zero-shot) and QT-OPT/CLIP baselines on affordance/failure detection.
• General V+L: The 562B generalist achieves state-of-the-art on OK-VQA and strong VQAv2/COCO without task-specific finetuning.
• Language retention & scaling: Freezing LLM preserves language ability but can struggle on some robotics tasks; unfrozen + scale up significantly reduces catastrophic forgetting.
• Emergent behaviors: Multimodal chain-of-thought and multi-image reasoning emerge in PaLM-E-562B, despite training on single-image prompts.

Takeaways​

• Injecting neural scene representations (OSRT) and entity-labeled multimodal tokens is effective even without massive embodied data.
• Diverse, joint training transfers vision-language knowledge into embodied decision-making, enabling data-efficient robot planning.
• Two viable paths to retain language skills during multimodal finetuning:
  1. Freeze the LLM, train encoders (max language retention, sometimes weaker robotics),
  2. Unfreeze and scale the LLM (much less forgetting, strong embodied performance).

Ref​

• Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., & Florence, P. (2023). PaLM-E: An Embodied Multimodal Language Model Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research. https://proceedings.mlr.press/v202/driess23a.html

RT-1​

• RT-1 discretizes robot actions into 256-bin tokens, creating a shared "action language" across robots.
• It absorbs heterogeneous data from simulation and other robot morphologies without losing performance.
• It generalizes robustly to new tasks, environments, and long-horizon scenarios (up to 50 steps).

Introduction & Motivation​

• Leveraging large, diverse, task-agnostic datasets enables high performance in zero-shot or small task-specific settings.
• Data collection and curation is a critical bottleneck in robotics ("the unsung hero" of large-scale ML).
• Transformer-based controllers are powerful but inefficient for real-time robotics, requiring architectural adaptations.

Model & Architecture​

• RT-1 architecture: EfficientNet + FiLM layers + TokenLearner for compact vision-language tokenization.
• Action tokenization: 11 action dimensions (7 arm, 3 base, 1 mode) discretized into 256 bins each.
• This abstraction converts continuous robot actions into a discrete "token language", enabling cross-domain and cross-robot transfer.
• Real-time feasibility: optimized design achieves ~3Hz inference speed suitable for real-world control.

Experiments & Results​

General Performance​

• RT-1 executes over 700 unique instructions at 97% success rate.
• On unseen instructions: 76% success, outperforming next-best baseline by +24%.
• Robustness: 83% success with distractors, 59% with background changes (significantly higher than baselines).

Absorbing Simulation Data​

• Adding sim data does not degrade real-task performance.
• Objects/tasks only seen in simulation: performance boosted 23% ⇒ 87%.
• Unseen instructions with sim objects: 7% ⇒ 33%, showing strong sim-to-real domain transfer.

Absorbing Multi-Robot Data​

• Mixed RT-1 + Kuka datasets: only 2% drop in original tasks.
• Bin-picking eval: RT-1 only 22% ⇒ mixed training 39% (almost 2×).
• Kuka-only training: 0% on EDR robots ⇒ morphology transfer alone fails.
• Mixed data enables RT-1 to leverage cross-robot experiences without explicit demonstrations.

Long-Horizon Scenarios (SayCan Integration)​

• Evaluated in two kitchens:
  • Kitchen1: 67% execution success.
  • Kitchen2 (novel environment): also 67% execution success.
• Outperforms Gato (0% in Kitchen2) and BC-Z (13% in Kitchen2).
• Demonstrated execution of ultra-long tasks up to 50 steps.

Data Quantity vs Diversity​

• Reducing dataset size ⇒ gradual performance/generalization decline.
• Reducing task diversity ⇒ much sharper decline, especially in generalization.
• Key takeaway: Data diversity is more critical than data quantity.

Conclusions & Limitations​

• RT-1 proves large-scale data absorption and strong generalization in robotics.
• Limitations:
  • Based on imitation learning ⇒ cannot surpass demonstrator performance.
  • Generalization limited to recombinations of known concepts ⇒ fails on truly novel motions.
  • Dataset is large but not dexterous (fine manipulation limited).

Future Directions​

• Enable non-experts to collect training data and prompt models for faster skill scaling.
• Increase environmental diversity to strengthen robustness to backgrounds/environments.
• Improve reaction speed and context retention via scalable attention and memory.

Ref​

• Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., & Hsu, J. (2022). Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817.

Say Can​

• The core of SayCan is using an LLM to decompose high-level instructions into low-level skills, and reinforcement-learned affordance value functions to evaluate whether each skill is feasible in the current environment.
• The Say × Can structure is modular: different LLMs or affordance models can be swapped in, but each module’s inherent biases are carried into the system.
• To mitigate limitations, loop-based strategies are essential — CoT and RLHF provide feedback loops for LLMs, while closed-loop feedback enables affordance functions to adapt during execution.

Motivation (Why LLMs alone fall short)​

• LLMs lack embodiment. They haven’t acted in the physical world, so using them for decision-making on a specific robot is unreliable.
• LLMs don’t know robot’s abilities or state. They may split instructions into subtasks, but without context of capabilities and environment, plans can be irrelevant.
• Prompting alone isn’t enough. Structured prompts help, but they don’t guarantee admissible or executable steps.

Core Proposal (What SayCan adds)​

• Ground with pretrained skills. Constrain LLM to propose actions that the robot can actually perform in context.
• Say × Can factorization.
  • Say (task-grounding): LLM estimates relevance of each skill to the instruction.
  • Can (world-grounding): Affordance functions estimate probability of success from current state.

Probabilistic Formulation​

• Two probabilities multiplied:
  • $p(\ell_\pi|i)$: LLM score of relevance.
  • $p(c_\pi|s,\ell_\pi)$: affordance score of success.
  • Select: $\pi = \arg\max p(c_\pi|s,\ell_\pi)\,p(\ell_\pi|i)$.

Planning Procedure​

• Planning is structured as a dialog: user gives high-level instruction, LLM produces a step sequence, loop until "done."
• Benefit: Interpretability—scores provide transparency.
• Caveat: Without affordances, chosen steps may be irrelevant to the current scene.

Affordances via RL​

• Affordance = value function. In sparse reward settings, value ≈ success probability.
• TD RL and MDP formalism used to learn $Q_\pi(s,a)$.

Implementation​

• Skill training:
  • BC-Z (behavioral cloning) and MT-Opt (reinforcement learning).
  • Multi-task BC/RL amortizes training cost.
• Language conditioning: Pretrained sentence encoder frozen, text embeddings as input.
• Action space: 6-DoF end-effector, gripper open/close, base x-y & yaw deltas, terminate.

Metrics​

• Plan success rate: 2/3 human raters agree that the plan is valid.
• Execution success rate: 2/3 raters agree robot achieved the task.

Key Results​

• Grounding nearly doubles performance vs non-grounded baselines.
• Understands sequence order (approach → pick → bring).
• Failures: Long-horizon tasks (early termination), negation, ambiguous references.
• Error split: ~65% LLM, 35% affordance.

Ablations​

• Remove LLM (task-grounding):
  • BC-NL: 0% all tasks.
  • BC-USE: 60% on single primitives, 0% otherwise.
• Remove affordances (world-grounding):
  • No-VF: 67%, Generative: 74% vs 84% (SayCan).

Scaling & Models​

• PaLM > FLAN. PaLM-SayCan achieves 84% plan / 74% execute.
• Stronger LMs improve robotics performance.

Extensibility​

• Add new skills easily: register skill, affordance, prompt example.
• Chain-of-Thought: Add "Explanation" → helps with negation and reasoning-heavy queries.
• Multilingual: Almost no performance drop (English, Chinese, French, Spanish).

Open-Source Variant​

• CLIPort for pick-and-place.
• Affordances approximated by ViLD open-vocabulary object detector.
• GPT-3 as language model.

Limitations & Future Work​

• Limits: Inherits LLM biases; skill library is bottleneck; hard to react to skill failures.
• Closed-loop extensions: Huang et al. use environment feedback + inner monologue for replanning.
• Future directions: Expand/robustify skills, explore new grounding sources (non-robotic), test if natural language is the right ontology, combine planning + language, use LMs for policy pretraining.

Ref​

• ichter, b., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., Kalashnikov, D., Levine, S., Lu, Y., Parada, C., Rao, K., Sermanet, P., Toshev, A. T., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Yan, M., Brown, N., Ahn, M., Cortes, O., Sievers, N., Tan, C., Xu, S., Reyes, D., Rettinghouse, J., Quiambao, J., Pastor, P., Luu, L., Lee, K.-H., Kuang, Y., Jesmonth, S., Joshi, N. J., Jeffrey, K., Ruano, R. J., Hsu, J., Gopalakrishnan, K., David, B., Zeng, A., & Fu, C. K. (2023). Do As I Can, Not As I Say: Grounding Language in Robotic Affordances Proceedings of The 6th Conference on Robot Learning, Proceedings of Machine Learning Research. https://proceedings.mlr.press/v205/ichter23a.html

Data Preparation​

• In real world applications, data can be inconsistent, incomplete, and noisy.
• Data Collection problems: when data is collected incorrectly
• Incomplete Data: when information is missing
• Data entry problems: when data is entered incorrectly
• Contradictions in data: when the data says something in one place, and then says a different thing elsewhere in the dataset. We can think of this data as noisy.
• Discrepancy in naming conventions: when data descriptions are unclear, people may misinterpret their meaning.
• Duplicated records: when integrating data from different sources, the same data may get entered multiple times.
• Data transmission problems: when data is sent between different people or databases or companies, things can get lost in the process.

Data mining tasks​

• Classification
• Estimation
• Prediction
• Characterisation
• Discrimination
• Affinity grouping
• Clustering
• Time series analysis

Data Cleaning​

• Missing data
  • Ignore the record
  • Fill the missing value manually
  • Fill missing values with calculated values
    • The missing values can be filled using the average value for a particular attribute
    • or by using attribute mean for all samples belonging to the same class as the given record.
    • also be filled using methods such as Bayesian classification or decision trees to automatically infer the values.
• Noisy data: a meaningless variation that cannot be interpreted properly by machines
  • Binning
    • binning methods use the neighbour's data, this is referred to as local smoothing
    • can replace all data in a segment by its mean or boundary values
  • Clustering
    • grouping of data points according to a distance measure
    • use a clustering algorithm to classify each data point into a specific group
    • can detect outliers
  • Regression
    • a data mining function that deals with the prediction of a continuous value rather than a class
    • maps data values to a function
    • Using regression to fit data by finding a mathematical equation may be used to smooth noisy data.

Binning​

• Equi-width: Bins have equal width.
• Equi-depth: Bins have the same number of values in them or almost the same number if they don't divide equally.

Equi-width binning​

Equal-interval binning, split the whole range of numbers into intervals with equal size.

• Price: 4, 8, 9, 15, 21, 21, 22, 26, 27, 28, 29, 36
• Equal-width binning
  • Bin1 [4, 12]: 4, 8, 9
  • Bin2 (12, 20]: 15
  • Bin3 (20, 28]: 21, 21, 22, 26, 27, 28
  • Bin4 (28, 36]: 29, 36
• Smoothing by bin means
  • Bin1: 7, 7, 7
  • Bin2: 15
  • Bin3: 24, 24, 24, 24, 24, 24
  • Bin4: 33, 33
• Smoothing by bin boundaries
  • Bin1: 4, 9, 9
  • Bin2: 15
  • Bin3: 21, 21, 21, 28, 28, 28
  • Bin4: 29, 36

Equi-depth binning​

Equal-frequency binning, use intervals containing an equal number of values.

• Price: 4, 8, 9, 15, 21, 21, 22, 26, 27, 28, 29, 36
• Equal-depth binnning
  • Bin1: 4, 8, 9
  • Bin2: 15, 21, 21
  • Bin3: 22, 26, 27
  • Bin4: 28, 29, 36
• Smoothing by bin means: each value in a bin is replaced by the mean value of the bin.
  • Bin1: 7, 7, 7
  • Bin2: 19, 19, 19
  • Bin3: 25, 25, 25
  • Bin4: 31, 31, 31
• Smoothing by bin boundaries: each bin value is replace by the closest boundary value.
  • Bin1: 4, 9, 9
  • Bin2: 15, 21, 21
  • Bin3: 22, 27, 27
  • Bin4: 28, 28, 36

Data Integration​

provides unified data by combining data from various heterogeneous data sources into a coherent data store

• The sources can include flat files, databases or multiple data cubes.
• Careful integration may help to avoid and reduce inconsistencies and redundancies in the final dataset.
• Building an enterprise's data warehouse is considered one of the most popular data integration implementations.
• Redundant attributes: An attribute (feature or column of a dataset) is called redundant if it can be derived from any other attribute or set of attributes.
  • In the process of data integration in data mining, the use of multiple data stores may lead to the problem of redundancy in data.
  • Dimension naming or inconsistencies in an attribute can also lead to redundancies in the dataset.

Pearson correlation coefficient​

• Correlation analysis can be used to detect redundancies in Numerical data
• It can measure how strongly one attribute implies the other on the basis of the available data.
• > 0.5: a strong positive correlation, A⬆️ B⬆️
• < -0.5: a strong negative correlation, A⬆️ B⬇️
• 0: no correlation. A and B are independent.
• correlation != causation

$r_{A,B} = \frac{n\sum{}xy - (\sum{}x)(\sum{}y)} {\sqrt{(n\sum_{} x^2 - (\sum_{} x)^2) \, (n\sum_{} y^2 - (\sum_{} y)^2)}}$

$r_{A,B} = \frac{\sum_{} (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{} (x_i - \bar{x})^2} \, \sqrt{\sum_{} (y_i - \bar{y})^2}}$

• step-by-step derivation
  • 분산: $Var(X) = \frac{1}{n} \sum_{} (x_i - \bar{x})^2$
  • 공분산: $Cov(X,Y) = \frac{1}{n} \sum_{} (x_i - \bar{x})(y_i - \bar{y})$
  • 상관계수 (정규화): $\rho = \frac{\mathrm{Cov}(X,Y)}{\sigma_X \sigma_Y}$
  • 평균: $\bar{x} = \frac{1}{n} \sum_{}x_i \quad \bar{y} = \frac{1}{n} \sum_{} y_i$
  • 분자 전개
    • $\sum_{} (x_i - \bar{x})(y_i - \bar{y})$
    • $\sum_{} (x_i y_i - x_i \bar{y} - y_i \bar{x} + \bar{x}\bar{y})$
    • $\sum_{} (x_i y_i ) - \bar{y}\sum_{} x_i - \bar{x}\sum_{} y_i + n\bar{x}\bar{y}$
    • 평균 대입
      • $\sum_{}x_iy_i - \frac{1}{n}(\sum_{}y_i)(\sum{}x_i) - \frac{1}{n}(\sum{}x_1)(\sum{}y_1) + \frac{1}{n}(\sum{}x_1)(\sum{}y_1)$
      • $\sum{}x_iy_i - \frac{1}{n}(\sum{}x_i)(\sum{}y_i)$
  • 분모 전개
    • $\sqrt{\sum_{} (x_i - \bar{x})^2} \, \sqrt{\sum_{} (y_i - \bar{y})^2}$
    • $\sqrt{\sum_{} x_i^2 - 2\bar{x}\sum_{} x_i + n\bar{x}^2} \, \sqrt{\sum_{} y_i^2 - 2\bar{y}\sum_{} y_i + n\bar{y}^2}$
    • 평균 대입
      • $\sqrt{\sum_{} x_i^2 - \frac{2}{n}(\sum_{} x_i)(\sum_{} x_i) + \frac{1}{n}(\sum{}x_i)^2} \, \sqrt{\sum_{} y_i^2 - \frac{2}{n}(\sum_{} y_i) + \frac{1}{n}(\sum{}y_i)^2}$
      • $\sqrt{\sum_{} x_i^2 - \frac{1}{n}(\sum_{} x_i)^2} \, \sqrt{\sum_{} y_i^2 - \frac{1}{n}(\sum_{} y_i)^2}$
  • 재정의 $r_{A,B} = \frac{\sum{}x_iy_i - \frac{1}{n}(\sum{}x_i)(\sum{}y_i)} {\sqrt{(\sum_{} x_i^2 - \frac{1}{n}(\sum_{} x_i)^2) \, (\sum_{} y_i^2 - \frac{1}{n}(\sum_{} y_i)^2)}}$
  • 분자/분모에 n 곱하고 인덱스 생략 $r_{A,B} = \frac{n\sum{}xy - (\sum{}x)(\sum{}y)} {\sqrt{(n\sum_{} x^2 - (\sum_{} x)^2) \, (n\sum_{} y^2 - (\sum_{} y)^2)}}$

Data Transformation​

The data is consolidated or transformed so that the patterns found are easier to understand, and the consequent mining process is more efficient.

• Smoothing: smoothing is used to remove noise from the data to improve clarity around the important features in the dataset
• Normalization: the method of scaling your data, into a regularized range, so that you can compare and represent it more accurately
• Discretization & Concept hierarchy generation
  • Discretisation is the process of putting values into buckets so that there are a limited number of possible states.
  • Discretisation transforms a continuous attribute into a categorical attribute, usually happens after the data is cleaned.
  • This process includes replacing lower-level data (primitive) with higher-level concepts through the use of concept hierarchies.
  • Street may be replaced with city, country or region.
  • Age may be replaced with senior, adult, younger and youth.
• Binarization: transforming data into binary numbers (e.g. 0, 1).
  • This helps make classifier algorithms more efficient.

Data Nomalization​

• the data should be standardised or normalised in order to avoid dependency on the selection of measurement units.

• This constitutes transforming data to lie within a common or smaller range, like [0.0, 1.0] or [−1, 1].

• Min-max normalization

  • Min-max normalisation maps a value of $K$, indicated by $v_n$, to a new value $v'_n$ within the range $[new\_min_K, new\_max_K]$
    • $v'_n = \frac{(v_n - min_K)}{(max_K - min_K)} \cdot (new\_max_K - new\_min_K) + new\_min_K$
    • calculates the relative position within the original range and reflects it in the new range accordingly.
  • preserves the relationships between the original data values.
  • It will encounter an out-of-bounds error if a future input case for normalisation falls outside of the original data range for K.

• Z-Score normalization

  • normalises attribute values using the average (i.e., mean) and standard deviation of $K$.
    • $v'_n = \frac{(v_n - \mu_K)}{\sigma_K}$
    • It converts the distance of a data point from the mean into a unitless measure.
  • is useful when there are outliers that dominate the min-max normalisation
  • is useful when the actual minimum and maximum of attribute $K$ are unknown.

• Decimal scaling normalization

  • The number of decimal points moved is based on the maximum absolute value of $K$.
    • $v'_n = \frac{v_n}{10^j}$
    • where $j$ is the smallest integer such that $max(|v'_n|) < 1$.
    • divides all values by the power of 10 just larger than the maximum absolute value, bringing them into the range $(-1, 1)$.

• Softmax normalization

  • a nonlinear transformation that yields an 's'-shaped curve that approaches 0 and 1 asymptotically.
  • New values will be mapped between 0 and 1 even if they are beyond the range of your existing data.
  • $\alpha = \frac{\nu - \mu}{\lambda \, (\sigma / 2\pi)}, \qquad \nu' = \frac{1}{1 + e^{-\alpha}}$
    • Center the data around the mean :: use $(\nu - \mu)$
    • Remove units by scaling with the standard deviation :: divide by $\sigma$.
    • Control how steep or flat the curve is :: adjust with $\lambda$.
    • Add $(2\pi)$ as a conventional constant to better match the logistic curve with statistical distributions.
    • the formula naturally arises by centering at the mean, standardizing by the spread, letting the user control the slope, and refining with a scaling constant.

• Sigmoid normalization

  • a nonlinear transformation similar to softmax. It ranges between −1 and 1 (asymptotically), and has a fixed linear portion within $±mu$.
    • $\alpha = \frac{\nu - \mu}{\lambda \, (\sigma / 2\pi)}, \qquad \nu' = \frac{1 - e^{-\alpha}}{1 + e^{-\alpha}}$
    • Center the data around the mean :: use $(\nu - \mu)$
    • Remove units by scaling with the standard deviation :: divide by $\sigma$.
    • Control how steep or flat the curve is :: adjust with $\lambda$.
    • Add $(2\pi)$ as a conventional constant to refine scaling with respect to statistical distributions. Apply the hyperbolic tangent :: map the result smoothly into the range $[-1, 1]$.
    • the formula naturally arises by centering at the mean, standardizing by spread, letting the user control the slope, and using tanh to compress all values into $[-1, 1]$.

Discretization & Concept hierarchy generation​

• Data discretisation is a form of numerosity reduction that transforms a continuous attribute into a categorical attribute.
• Higher concept labels or a smaller number of intervals (i.e. binning) are used to replace the raw data in order to simplify the original data and increase the efficiency of mining.
• Discretisation is very beneficial for generating concept hierarchies automatically, which allow data mining at multiple levels of data abstraction.
• One or more concept hierarchy can be defined for the single attribute for accommodating the requirements of various users.

• If dependent and independent variables have only a few values, a wide range of classification algorithms can be used.

Data Binarizaion​

• maps a categorical or continuous attribute into one or more binary variables.
• Binarisation can convert a continuous attribute to a categorical attribute which can then be converted into set of binary attributes.
• only possible to keep the meaning of one categorical value at one time, losing the meaning of the others.

Data Reduction​

• to acquire a reduced data set representation which is much smaller in quantity and maintains the quality of the data close to the original data.
• to reduce data storage and analysis costs while increasing storage efficiency

Aggregation​

• storing and presenting data as a summary, using statistical metrics like means, median and variance.
• Data aggregation is often used to construct a data cube for data analysis at multiple levels of abstraction.
• Multidimensional aggregated information is stored in data cubes

Dimensionality reduction​

• to minimize the number of features
• feature subset selection or feature selection detects and removes weakly relevant, redundant, or irrelevant dimensions or attributes
• to determine a minimum set of attributes so that the resulting probability distribution of the data classes is as near as possible to the original distribution obtained using all attributes.
• Feature subset selection: uses only available subsets of the features to reduce the dimensionality of the data
  • Redundant features: Duplicates of all or much of the information present in one or more attributes.
    • the amount of sales tax paid / purchase price of a product
  • Irrelevant features: Contain no information that is important for the data mining process at hand.
    • the color of a product when predicting its price
  • While some redundant and irrelevant attributes can be eliminated immediately by considering the domain knowledge or common sense.
  • The ideal approach to feature selection is to try all possible subsets of features in the input for the data mining algorithm of interest, and then consider the subset that gives the best outcome.
• Feature subset selection techniques
  • Brute-force approach
  • Embedded approaches:
    • Feature selection occurs naturally as part of the data mining algorithm.
    • The algorithm decides by itself which attributes are to be ignored.
  • Filter approaches:
    • Features are chosen before running the data mining algorithm by taking some of the approaches which are independent of the data mining process.
    • can be selected with pairwise correlation as low as possible.
  • Wrapper approaches:
    • consider the target data mining algorithm as a black box to determine the best subset of attributes.
    • Instead of evaluating all possible combinations, it intelligently searches only a subset to find a near-optimal feature set.
    • Heuristic methods: Forward selection, backward elimination, genetic algorithm, greedy search.
    • Decision tree induction

Numerosity reduction​

• Regression, clustering, histograms, sampling
• reducing the volume of the data, without any loss of data
  • parametric models: store only the model parameters rather than the actual data, regression, log-linear models
  • non-parametric approaches: clustering, sampling, histograms
• Histograms
  • unsupervised techniques that does not use a class label
  • Singleton bucket: each of the buckets shows only a single frequency pair/attribute value
  • Equal-width histogram: divided into equal ranges
  • Equal-frequency(depth) histogram: each bucket has the similar number of data
• Sampling
  • a large dataset to be denoted by a smaller random subset (or sample) of the data
  • often used in preliminary exploration as well as final analysis.
  • useful when processing the entire dataset is too large or expensive.
  • If the sample preserves the important properties of the original dataset (e.g., the mean), the sample is said to be representative
  • Simple random sampling: every data point has an equal probability of being chosen.
  • Sampling without replacement: once a data point is chosen, it cannot be selected again.
  • Sampling with replacement (bootstrap): the same data point can be picked multiple times, since it is placed back into the dataset after selection.
  • Cluster sampling: the dataset is divided into clusters (groups), and sampling is performed at the cluster level.
  • Stratified sampling: the dataset is split into strata (partitions), and random samples are drawn from each stratum. This is especially useful when the data is imbalanced, e.g., sampling customers across different age groups.

• Strata: Youth, Middle-aged, Senior

Vocabulary & Expressions​

Key Idea​

• CLIPort proposes a two-stream architecture for vision-based manipulation:
  • Semantic pathway (what): leverages CLIP for broad semantic understanding.
  • Spatial pathway (where): leverages Transporter for fine-grained spatial reasoning.
• This design is inspired by the two-stream hypothesis in cognitive psychology (ventral/dorsal pathways).

Framework Contributions​

• Benchmark Extension: Expanded the Ravens benchmark with language-grounding tasks for manipulation.
• Two-Stream Architecture: Uses pre-trained vision-language models (CLIP) to condition precise manipulation policies with language goals.
• Empirical Results: Demonstrates robustness on diverse manipulation tasks, including multi-task settings and real-robot experiments.

Architectural Design​

• CLIPort integrates semantic (CLIP) with spatial (Transporter) features by lateral fusion.
• The semantic stream is conditioned with language features from CLIP’s text encoder and fused with intermediate spatial features.
• Enables end-to-end learning of affordance predictions (pick-and-place) without explicit object models, segmentations, or symbolic states.

Key Insights​

• Formulates manipulation as action detection (where to act), instead of object detection.
• Tabula rasa systems (like plain Transporter) require new demonstrations for every goal/task. CLIPort addresses this with a strong semantic prior (from CLIP) to generalize across tasks and concepts.
• Language-conditioned policies provide an intuitive interface for specifying goals and transferring concepts.

Experimental Results​

• Simulation (PyBullet, UR5 robot with suction gripper):
  • 10 language-conditioned tasks with thousands of unique instances.
  • Multi-task CLIPort outperformed or matched single-task models, even with fewer demonstrations.
  • CLIP-only or Transporter-only baselines saturate, while CLIPort exceeds 90% success with just 100 demos.
• Generalization:
  • CLIPort generalizes to unseen attributes (e.g., new colors, shapes, object categories).
  • Struggles with completely novel attributes (e.g., “pink” or “orange” never seen in training).
• Real-World Robot Experiments (Franka Panda):
  • Achieved ~70% success on real tasks with just 179 demonstrations.
  • Performance trends were consistent with simulation, validating sim-to-real transfer.

Conclusion​

• CLIPort shows that multi-task, language-conditioned policies generalize across tasks better than object-centric or tabula rasa methods.
• With action abstraction and spatio-semantic priors, end-to-end models can learn new skills without requiring hand-engineered pipelines.
• Limitations remain for dexterous 6-DoF manipulation and complex continuous control.

Ref​

• Shridhar, M., Manuelli, L., & Fox, D. (2022). Cliport: What and where pathways for robotic manipulation. Conference on robot learning.

Overview​

• Introduces a HoOA benchmark that isolates hallucinations on object attributes (color, shape) from existence/relationship errors.
• Proposes MIAVLM: leverages multiview images (generated from a single image’s 3D representation) and a Multiview Attributes Perceiver (MAP) to make fusion order-invariant.
• Adds negative instructions during tuning to counter LVLMs’ tendency to answer "Yes".
• Results: best HoOA metric (0.775 / 0.787) with fastest inference (0.071 / 0.105 s). "9in1" tiling is ineffective; separate multiview inputs help.
• Training: LM loss, Adam (lr=0.001), cosine annealing, 20 epochs, single NVIDIA 3090.

Hallucinations on Object Attributes (HoOA)​

Issues​

• HoOA = incorrect attribute descriptions for existing objects (distinct from HoOE/HoOR).
• Root causes analyzed:
  • Single-view insufficiency: fine-grained details can be invisible from a single viewpoint.
  • Instruction bias: overexposure to positive/affirmative patterns → "Yes" bias.
  • Order sensitivity: multi-image inputs change predictions when view order changes.

Mitigation Methods (this paper)​

• Multiview prompts: sample views from a single image’s 3D reconstruction to recover missed details.
• MAP (order-invariant fusion): learn view weights and fuse per-view features via weighted sum; input order has no effect; supports any number of views.
• Negative instructions: incorporate "No"-answerable questions in tuning to suppress "Yes" bias.

Benchmark (HoOA)​

Construction​

• Based on CelebAText-HQ; manual attribute descriptions rewritten into Yes/No questions.
  • Positive questions → correct answer "Yes".
  • Negative questions → attribute flipped/opposite → correct answer "No" (to expose "Yes" bias).
• Scale: 1,430 images, 14,291 positive + 14,291 negative questions.
• Split: 9:1 train:test.
• Metric: average of accuracy on positive and negative questions (balanced HoOA score).

Model: MIAVLM​

Visual Extractor (VE)​

• 6 stacked Transformer decoder blocks.
• Soft prompts $P \in \mathbb{R}^{l \times d}$ are queries; image embeddings $e_i$ are keys/values.
• Per-view cross-attention computed in parallel (no autoregressive chaining; no assumed order).
• Per-view output: $$o_i = \mathrm{softmax}\!\left(\frac{(P W_Q)(e_i W_K)^\top}{\sqrt{d}}\right) e_i W_V,\quad O_{VE}=\{o_1,\dots,o_n\}.$$

Multihead Sampler (MS)​

• Learns view weights for fusion.
• Decomposer (2-layer MLP) maps each view’s $[CLS]$ to $m=4$ tokens $\{e_i^{1},\dots,e_i^{m}\}$.
• For each token/head $j$: compute attention scores vs. $P$ → mean over prompt tokens → $\mathrm{weights}^j \in \mathbb{R}^n$.
• Average across heads: $$w_{MS} = \tfrac{1}{m}\sum_{j=1}^{m}\mathrm{weights}^j \in \mathbb{R}^n.$$

MAP (Multiview Attributes Perceiver)​

• Order-invariant weighted fusion: $$\text{Output}=\sum_{i=1}^{n} w_i\,o_i.$$
• Properties: supports any number of views; permutation-invariant to input order.
• By learning weights for each view, MAP highlights informative perspectives and suppresses less useful ones, ensuring consistent predictions even when the view order changes. This directly addresses the input-order sensitivity observed in baselines such as OpenFlamingo.

Benchmarks​

Baselines & Input Modes​

• Baselines: BLIP3, OpenFlamingo (4 variants), OPERA, Idefics2, LLaVA-UHD.
• Two input modes:
  1. Original image only.
  2. Original + 8 generated views.
     • Models that accept only one image use 9in1 tiling (nine images stitched into one).

Main Results​

• MIAVLM:
  • HoOA metric: 0.775 / 0.787 (modes 1 / 2)
  • Positive accuracy: 0.752 / 0.762
  • Negative accuracy: 0.797 / 0.812
  • Inference time: 0.071 / 0.105 s (fastest)
• 9in1 tiling did not improve results (likely harder to interpret).
• Nine separate multiview images generally improved performance.

Ablations​

• Negative instructions: boost negative-question accuracy but slightly reduce positive-question accuracy; overall HoOA increases (approx. 0.665 → 0.787).
• Input-order sensitivity:
  • MIAVLM is order-invariant
  • OpenFlamingos accuracy varies when shuffling view order.

Limitations & Notes​

• Trade-off from negative instructions (negatives ↑, positives ↓).
• Effectiveness depends on the quality of generated views.

Insights​

• This approach seems especially suitable for perception, where multiple scene views may arrive in arbitrary order, ensuring consistent attribute recognition.

Ref​

• Tan, Z., Li, Y., Meng, S., Yuan, X., Li, W., Mo, T., Wang, B., & Chu, X. (2025, 6–11 April 2025). Mitigating Hallucinations on Object Attributes using Multiview Images and Negative Instructions. ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

match​

match term:
  case pattern-1:
    action-1
  case pattern-2:
    action-2
  case pattern-3:
    action-3
  # the underscore _ case executes the default code
  case _:
    action-default

Repetition Statements​

• The count-controlled repetition: a fixed number of times.
• The sentinel-controlled repetition: a designated value that ends the loop.
• The infinite repetition: continues until externally stopped.

The For Loop​

for <value> in <range of values>:
  <code>

sum = 0;

# [1, 2, ..., 19]
# adds values from 1 to 19 to sum
for e in range(1, 20):
  sum += e

print(f"The sum is: {sum}")

Loop-And-A-Half​

n = 5
sum = 0

while n < 10:
  sum += n

  if sum > 100:
    break