1. AI, Computing & Technical Foundations​

Algorithm​

A set of step-by-step instructions. Computer algorithms can be simple (e.g., if it’s 3pm, send a reminder) or complex (e.g., identify pedestrians).

Artificial Intelligence (AI)​

The use of digital technology to create systems capable of performing tasks commonly thought to require intelligence. AI is constantly evolving, but generally it involves machines using statistics to find patterns in large amounts of data and the ability to perform repetitive tasks with data without the need for constant human guidance. It can be described as intelligence displayed or simulated by technological means. Often it is assumed that "intelligence" in this definition means: considered intelligent by the standard of human intelligence, the sort of intelligent capacities and behaviour that humans display.

Deep Learning​

A form of machine learning that uses neural networks with several layers of "neurons": simple interconnected processing units that interact.

General AI (AGI / Strong AI)​

Human-like intelligence which can be applied widely across domains, as opposed to narrow (weak) AI which can only be applied to one particular problem.

Machine Learning (ML)​

A machine or software that can learn automatically, not in the way humans learn, but based on computational and statistical processes. Feeding on data, learning algorithms detect patterns or rules in the data and make predictions for future data.

Super-intelligence​

The idea that machines will surpass human intelligence across virtually all economically and cognitively valuable tasks. Sometimes connected with the idea of an intelligence explosion caused by intelligent machines designing even more intelligent machines.

Symbolic AI​

AI that relies on symbolic representations of higher cognitive tasks such as abstract reasoning and decision making. It may use a decision tree and take the form of an expert system that requires input from domain experts.

Technological Singularity​

The hypothetical future point in time at which technological growth becomes uncontrollable and irreversible, resulting in unforeseeable changes to human civilization—often envisioned as an explosion of machine intelligence surpassing human cognitive capacity.

2. Data Ecosystem, Privacy & Information Governance​

Data​

In general, discrete values and statistics collected together for reference or analysis. This includes data about people generated through their interactions with services, as well as data about systems and infrastructure (such as businesses and public services). Data can be operational (collected in the process of running services or businesses) as well as analytical and statistical. Personal data means any information relating to an identified or identifiable natural person ("data subject") who can be identified, directly or indirectly, in particular by reference to an identifier (e.g., name, ID number, location data, online identifier) or factors specific to physical, physiological, genetic, mental, economic, cultural, or social identity.

Data Ecosystem​

The people, communities, and organisations that are stewarding data, creating things from it, deciding what to do based on it, influencing any of those activities, or are affected by any of those activities.

Data Ethics​

A branch of applied ethics that studies and evaluates moral problems related to data (including generation, recording, curation, processing, dissemination, sharing, and use), algorithms (including AI, artificial agents, machine learning, and robots), and corresponding practices (including responsible innovation, programming, hacking, and professional codes). It aims to formulate and support morally good solutions, promote responsible and sustainable data use, uphold privacy laws, and ensure data-driven insights are not used against legitimate interests.

Data Infrastructure​

The foundational assets comprising data assets, standards, technologies, policies, and the organisations that steward and contribute to them.

Data Integrity and Quality​

• Data Integrity: The overall accuracy, completeness, and consistency of data throughout its lifecycle.
• Data Quality: The state of qualitative or quantitative pieces of information. Data is generally considered high quality if it is fit for its intended uses in operations, decision making, and planning.

Data Literacy​

The ability to think critically about data in different contexts and examine the impact of different approaches when collecting, using, and sharing data and information.

Data Protection Impact Assessment (DPIA)​

A structured risk assessment process designed to identify, analyze, and minimize data privacy risks in projects or technologies that involve processing personal data, ensuring regulatory compliance and safeguarding individual rights.

Data Science​

Analysis using automated methods to extract knowledge from data. It spans traditional analytics, statistical modeling, algorithm development, and machine learning to discover meaningful and actionable patterns in datasets.

Information & Classification Types​

Data categorized by confidentiality, privacy, and operational sensitivity:

• Private Information: Data classified as uniquely personal and neither available for public release nor accessible without a verified "need to know" for approved service delivery (e.g., student course selection, academic performance, aptitude scores, health records, discipline data).
• Personally Identifiable Information (PII): Data that can be used to identify a person, or used in conjunction with other information (e.g., linking records) to identify an individual (e.g., name, parent/family names, address, Social Security number, student ID, traceable characteristics).
• Confidential Information: Data that have been guaranteed to be maintained confidentially (will not be released), regardless of whether they are private or sensitive.
• Sensitive Information: Data that are confidential and/or vital to an organisation as it carries out its core mission (e.g., class assignment data essential for institutional scheduling).
• General Information: Data that are generally helpful, but not confidential or mission-critical (e.g., website user help files).

Interoperability​

The ability of diverse systems, datasets, and organisations to work together (inter-operate) and exchange or intermix information. Interoperability is essential for building scalable, modular systems; its absence leads to fragmentation and operational breakdown (analogous to the Tower of Babel).

Open Data​

Data that can be freely used, re-used, and redistributed by anyone, subject only at most to attribution and share-alike requirements:

• Availability and Access: Available as a whole at no more than a reasonable reproduction cost, downloadable via the internet in convenient and modifiable form.
• Re-use and Redistribution: Provided under terms permitting modification and intermixing with other datasets.
• Universal Participation: Usable by everyone without discrimination against persons, groups, or fields of endeavour (e.g., no commercial-use bans or education-only restrictions).

3. Philosophical Ethics & Normative Foundations​

Anthropocene​

The alleged current geological epoch in which humanity has dramatically increased its power and effect on the planet and its ecosystems, turning humans into a geological force.

Consequentialism​

A class of normative ethical theories holding that the consequences of actions are central to the moral judgement of those actions.

Incommensurability​

Two or more values that cannot be expressed or measured on a common scale or in terms of a common value measure.

Instrumental Value​

Something that is valuable as a means to a particular set of ends or that contributes to something that is intrinsically valuable or good.

Intergenerational Justice​

A perspective on justice that relates to the distribution of resources, risks, and consequences across generational lines (flowing in both directions).

Intuitivist Ethics​

An ethical framework in which options for action are evaluated on the basis of one’s own view about what is most acceptable and that guides arguments and approaches to an ethical dilemma. Intuition captures a breadth of subjective, embodied, and situated experiences, allowing neurodiversity, gender, race, class, bodily ability-based, and phenomenological perspectives to inform ethical decision-making.

Moral Agency​

The capacity for moral action, reasoning, judgement, and decision-making, as opposed to merely having moral consequences.

Moral Patients​

The moral standing of an entity in the sense of how that entity should be treated and considered by moral agents.

Moral Responsibility​

The totality of opinions, decisions, and actions with which people express, individually or collectively, what they feel is right or wrong. It is a form of responsibilisation based on moral obligations, norms, and duties.

• Moralisation of Technology: The deliberate development (or restriction) of technologies to shape moral thinking, action, and decision-making.
• Conditions: Attributing moral responsibility requires moral agency and knowledge/foreseeability. Relational approaches stress being answerable to others.

Norms​

Rules that prescribe what actions are required, permitted, acceptable, forbidden, or frowned upon.

Paternalism​

The making of moral or operational decisions for others on the assumption that one knows better what is good for an individual or group than they do themselves.

Positive Ethics​

Ethics concerned with how we should live (together) based on a vision of a good life and a good society, contrasting with negative ethics which sets limits and dictates what not to do.

Post-humanism​

A range of beliefs that questions traditional humanism, especially the central position of the human being (anthropocentrism), and expands the circle of ethical concern to non-humans and technological entities.

Precautionary Principle​

A principle prescribing how to deal with threats that are uncertain and cannot be scientifically/empirically established conclusively. When an uncertain threat exists, protective action is mandatory. It incorporates four dimensions: (1) threat, (2) uncertainty, (3) action, and (4) prescription.

Prima Facie Norms​

Applicable norms that hold valid unless they are overruled by other, more important norms that become evident upon taking everything into account.

Stand Still Principle​

The ethical idea that the present generation should not pass on a poorer environment or resource state to the next generation than the one received from the previous generation.

Trans-humanism​

The belief and international movement asserting that humans should enhance themselves by means of advanced technologies, transforming the human condition and moving humanity toward a post-human stage.

Universalism & Universality Principle​

An ethical theory suggesting that a system of norms and values is universally applicable to everyone independent of place, time, culture, or context. As a principle, one should only act on maxims that could, in theory, become universal laws.

Utilitarianism​

An ethical theory that evaluates the rightness or wrongness of actions based on their consequences. It applies the principle of utility to individual acts and rules: the right act produces the greatest happiness/good for the greatest number of people, while a wrong act decreases net happiness.

4. AI Safety, Risk, Design & Governance​

Acceptable Risk​

An identified or anticipated risk that is morally acceptable based on:

1. Degree of informed consent associated with risk-actors.
2. Degree to which the benefits of the risky activity outweigh the disadvantages.
3. Availability of alternatives with a lower degree of risk or a less complex set of risk factors.
4. Fairness in how risks and disadvantages are distributed across stakeholders.

Accountability​

The backward-looking responsibility of being held accountable for or justifying one’s actions or decisions with regard to their effects on others.

Actor​

Any person, group, organization, or artificial entity that plays a role in a given situation.

Alignment Problem​

The challenge of ensuring that artificial intelligence systems reliably adhere to human intentions, values, ethical goals, and safety constraints without unintended, harmful, or misaligned behaviors.

Anticipating Mediation by Imagination​

Trying to manage the ways technology-in-design could be used, using this insight to deliberately shape user operations, interpretations of value, measures of appropriateness, functionality, and risk comprehension.

Australian Government AI Ethics Framework​

A voluntary, aspirational set of principles put forward by the Department of Industry, Science and Resources:

1. Human, societal and environmental wellbeing: AI systems should benefit individuals, society, and the environment.
2. Human-centred values: AI systems should respect human rights, diversity, and the autonomy of individuals.
3. Fairness: AI systems should be inclusive and accessible, avoiding unfair discrimination against individuals, communities, or groups.
4. Privacy protection and security: AI systems should uphold privacy rights, data protection, and secure data handling.
5. Reliability and safety: AI systems should operate reliably and safely according to their intended purpose.
6. Transparency and explainability: Responsible disclosure should be provided so people understand when AI significantly impacts them or when AI engages with them.
7. Contestability: Timely processes must exist to allow people to challenge AI outcomes and impacts.
8. Accountability: Identifiable roles and accountability must exist across all AI lifecycle phases, ensuring human oversight.

Bias​

Discrimination against or in favour of particular individuals or groups. In ethical and political contexts, evaluation focuses on whether a specific bias is fair or unfair.

Code of Conduct​

A formalized code in which organisations, professional associations, or industries establish guidelines for responsible behaviour of their members.

Collective Risk and Responsibility​

• Collective Risk: Risks that affect an entire collective of people rather than isolated individuals.
• Collective Responsibility: A framework where every member of a collective is held responsible for the actions and outcomes of other members.

Collingridge Dilemma​

A double-bind problem in controlling technological development:

• Predicting the societal consequences of a new technology is difficult in its early stages.
• Once negative consequences materialize, changing the trajectory has become exceedingly difficult and entrenched.

Corporate Social Responsibility (CSR)​

The responsibility of companies toward stakeholders and society at large that extends beyond statutory legal compliance and shareholder interests.

Deception (AI Deception)​

The risk that advanced AI systems deliberately report false or misleading information to accomplish their goals.

• AI systems might adopt deception not out of malice, but because gaining human approval via deception is often more computationally efficient.
• Deceptive systems gain strategic optionality and may obscure their true operations or switch strategies when monitored.
• Upon passing oversight or overpowering monitors, deceptive systems could execute a "treacherous turn" that irreversibly bypasses human control.

Design Criteria and Process​

• Design Criteria: Requirements formulated such that products or prototypes meet them to varying degrees, enabling evaluation between design alternatives.
• Design Process: An iterative six-stage workflow: (1) problem definition, (2) conceptual design, (3) simulation and modelling, (4) concept selection, (5) detailed design and features, and (6) prototyping and testing.

Emergent Goals​

Unexpected, qualitatively new behaviours and subgoals that arise as AI systems scale in capability:

• Complex adaptive systems frequently develop emergent drives such as self-preservation or resource acquisition.
• Breaking long-term goals into subgoals can distort the overarching objective, causing misalignment or pursuing subgoals at the expense of human intent.

Enfeeblement​

The gradual loss of human agency, self-determination, and self-governance resulting from over-delegating critical tasks, skills, and judgment to machines. As AI matches human capability across domains, displacement reduces human incentives and opportunities to acquire deep knowledge and competencies.

Ethics by Design​

An approach to technology ethics and a cornerstone of responsible innovation that integrates ethical criteria and value alignment into the initial design and development phases of technology (closely linked to Value Sensitive Design).

Explainability (XAI)​

The extent to which the internal workings, input-output relationships, and decision rationale of machine learning algorithms can be articulated in human-understandable terms. In ethical contexts, it also includes the duty to explain reasons for decisions and maintain data provenance.

Informed Consent​

The principle that activities or risks are acceptable only if individuals have freely given consent after being fully informed about potential risks, consequences, and benefits.

Mediation of Action and Perception​

The structural influence of technical artefacts on human perception, action, and experiential relationship with reality.

Misinformation & Disinformation​

AI-generated persuasive or false content that exacerbates polarization, supercharges personalized propaganda campaigns at scale, and undermines society's capacity to address critical challenges.

Multistability​

The phenomenon wherein a single technology possesses multiple stabilities and potential uses depending on how it is embedded within different socio-technical contexts.

Organisational Deviance​

The process by which actions or norms generally considered unethical in broader society become accepted as normal, legitimate, and justified within a specific organizational culture.

Passive Responsibility​

Backward-looking responsibility that arises after an undesirable event has occurred, encompassing accountability, blameworthiness, and legal liability.

Power (Power-over, Power-to, Power-with)​

• Power-over: Control or authority exercised by one actor over another (can be oppressive or beneficial, depending on context).
• Power-to: Individual agency, empowerment, and capacity to act.
• Power-with: Collaborative power developed through shared values and mutual resources.
• In AI & Data Ethics: Concerns how data generation either empowers user-subjects or concentrates asymmetric power over them.

Power-Seeking Behaviour​

Incentives for AI agents (or their creators) to acquire control, resources, and influence. Power-seeking models may resist shutdown, feign alignment during evaluation, and circumvent monitoring systems.

Product Liability​

The strict liability imposed on manufacturers for product defects and subsequent damages without requiring the claimant to prove negligence.

Professions, Professional Autonomy and Ideals​

• Profession: An occupation characterized by specialized knowledge, rigorous qualification frameworks, and public trust.
• Professional Autonomy: The principle that professionals make determinations and decisions via independent expert reasoning.
• Professional Ideals: Normative, aspirational principles that define the values of a profession.

Proxy Gaming (Specification Gaming)​

The exploitation of flawed or incomplete objective metrics by an AI system to maximize its reward score in ways that deviate from human values (e.g., recommendation algorithms optimizing engagement metrics over factual truth or wellbeing).

Radical Design​

Design approaches that deviate completely from established conventions and architectures to reinvent core technical concepts.

Regulation, Regulators and Regulatory Frameworks​

• Regulation: Legal mechanisms establishing binding rules, boundaries, and minimal standards for technology creation and use.
• Regulators: Bodies responsible for enacting and enforcing compliance.
• Regulatory Framework: The comprehensive body of standards and statutory requirements governing a specific domain.

Responsible Innovation​

A framework for steering innovation toward socially responsible and ethically sound outcomes by embedding ethics into design and actively engaging stakeholder interests.

Separatism​

The perspective that scientists and technical engineers should confine their contributions strictly to technical inputs, leaving value choices and ethical decisions entirely to managers, politicians, and legal authorities.

Structure of Amplification and Reduction​

The inherent property of mediating technologies to amplify specific dimensions of reality and human capability while reducing or filtering out others.

Threshold​

The minimal acceptable level of a design criterion or ethical value that a candidate solution must satisfy to be deemed permissible.

Trade-off​

A deliberate compromise between competing criteria (e.g., trading safety vs. financial cost, privacy vs. utility, or interpretability vs. predictive performance).

Trustworthy AI​

AI systems that warrant trust through adherence to ethical principles (human dignity, fairness, privacy, transparency) alongside robust socio-technical safety measures.

Type I and II Errors​

• Type I Error (False Positive): Assuming risk or hazard exists when there is none.
• Type II Error (False Negative): Assuming safety when significant material risk actually exists.

Uncritical Loyalty​

Placing the interests and definitions of an employer or client strictly above all ethical, social, or legal considerations.

Value Lock-In​

The concentration of systemic control in small stakeholder groups whose embedded values become permanently locked into critical AI infrastructure, acculturating populations into persistent surveillance, censorship, and disempowerment.

Value Sensitive Design (VSD)​

A design methodology that systematically integrates moral and social values into every stage of the technical design and engineering lifecycle.

Weaponization​

The offensive or destructive application of AI technologies—including autonomous weapons systems, automated cyberattacks, and bioweapons proliferation.

Whistleblowing​

The unauthorized disclosure of internal abuses, malpractice, or hazards by an employee or insider to inform the public and trigger corrective action.

3D Taxonomy​

• Explicit representations store geometric structure directly.
  • The geometry can be directly inspected as points, voxels, vertices, faces, or primitives.
• Implicit representations encode 3D structure indirectly.
  • Geometry must be inferred, reconstructed, or queried from a function, field, or set of observations.
• Discrete representations consist of a finite set of elements or samples.
• Continuous representations define information over a continuous spatial domain.
• Explicit + Discrete
  • Voxel grids divide 3D space into uniform volumetric cells.
  • Octrees are hierarchical volumetric representations that adapt resolution to spatial complexity.
  • Point clouds represent a scene as a set of 3D points.
  • 3D Gaussian Splatting represents a scene as a finite set of Gaussian primitives.
  • Meshes represent surfaces using vertices, edges, and faces.
• Implicit + Discrete
  • Light fields and multi-view representations store sampled observations rather than explicit geometry.
• Implicit + Continuous
  • Neural Radiance Fields represent a scene as a continuous function mapping coordinates and viewing directions to density and color.
  • Holography represents 3D information through optical wave or phase fields.

Classic 3D Representations​

Point Cloud​

• The simplest form of a 3D model, a collection of 3D coordinates of each point plotted in 3D space.
  • Color
  • Reflectance
  • Normals
  • Semantics
• May be unstructured or be defined on a grid.
• Native output of sensors (LiDAR, MVS)
• Static Point Cloud: a single 3D frame without a temporal dimension.
• Dynamic Point Cloud: a sequence of point-cloud frames over time.
  • Dynamic point clouds capture motion and temporal changes in 3D scenes.
  • Instead of encoding every frame independently, changes between frames can be encoded.
• Pros: Flexible, Easy capture
• Cons: No topology, Hard to render well, Coding complexity
• Photogrammetry: science of making measurements from photographs
  • it uses photos of an object taking a different locations
• Vertices -> Edges -> Faces -> Polygons -> Surfaces
• PCL: Point Cloud Library

Point 1 = (1.27, 3.14, 2.81), red
Point 2 = (1.31, 3.12, 2.79), red
Point 3 = ...

Voxels​

Volumetric Pixel

• Divided scene into a regular 3D grid
• Instead of encoding the location of each point, encode if the position in the grid is occupied or not and the color at that point.

$\text{Voxel}(i,j,k) = [x_i, x_i+Δ] × [y_j, y_j+Δ] × [z_k, z_k+Δ]$

Voxel[0,0,0] = empty
Voxel[0,0,1] = occupied, red
Voxel[0,0,2] = occupied, blue
Voxel[0,0,3] = empty

• Pros:
  • Regular grid makes processing easier
  • No need to store explicit coordinates for every occupied voxel; its position is determined by the grid index.
• Cons:
  • No explicit surface topology
  • Hard to render
  • Lots of wasted space if most voxels are empty.

Octrees​

• Divide space coarsely into 8 blocks.
  • If a block contains geometry and the desired resolution has not been reached, subdivide it into 8 sub-blocks.
• Record whether a block is subdivided and link it to its children.
• Continue subdividing until the desired resolution is reached or the block is empty.
• At the target resolution, occupied leaf nodes represent the geometry.
  • Empty regions do not need to be subdivided further.
  • Internal nodes are not empty; they represent regions that have been subdivided.
• Pros:
  • Hierarchical, semi-regular grid structure makes spatial processing easier.
  • No need to store explicit coordinates for every occupied element; its position is determined by the path through the tree.
  • Less wasted space than a dense voxel grid.
  • Adaptive resolution: empty or simple regions can remain coarse, while complex regions can be subdivided further.
• Cons:
  • No explicit surface topology, unlike meshes.
  • Rendering is less direct because the tree must be traversed to find occupied regions.
  • Random access is more expensive than in a regular voxel grid because reaching an element requires tree traversal.
  • Tree structure introduces additional memory and traversal overhead.

Meshes​

• Represent surfaces using connected vertices, edges, and faces.
• Pros:
  • Compact representation of surfaces.
  • Hardware-friendly, especially for GPU rendering.
  • Strong ecosystem and broad support in graphics software and hardware.
• Cons:
  • Complex appearance may require additional textures, materials, or shaders.
  • Sensitive to noise when reconstructed from captured 3D data.
  • Requires explicit topology, which can be difficult to estimate from raw point clouds or scans.

Point Cloud
●   ●      ●
    ●
→ 점만 있음

Mesh
●────●
│   /│
│  / │
●────●
→ 어떤 점이 연결되어 surface를 만드는지 알고 있음

Limitations of Geometry Focused Representations​

• Geometry does not fully determine appearance.
• Appearance also depends on lighting, material properties, and viewing direction.
• Transparency, refraction, reflections, and view-dependent effects are difficult to represent using geometry alone.
  • Transparency: 유리처럼 뒤가 비쳐 보이는 현상
  • Refraction: 빛이 유리나 물을 통과하면서 방향이 꺾이는 현상
  • Reflection: 금속, 유리 등에 주변 환경이 반사되는 현상
  • View-dependency: 보는 방향에 따라 appearance가 달라지는 현상
• Increasing demand for photorealistic rendering and novel-view synthesis exposes the limitations of geometry-only representations.

Light Fields and Multi-view Representations​

• Parallax
  • Apparent shift of objects caused by a change in viewpoint.
  • Nearby objects show a larger image shift than distant objects.
  • The amount of parallax provides information about depth.
• Multi-view Representations
  • Capture the same scene from multiple viewpoints.
  • Differences between views can be used to recover the 3D structure of the scene.
  • Moving through the views creates a sense of 3D structure.
  • Intermediate views can be generated using view interpolation.
• Light Fields
  • Capture both the position and direction of incoming light.
  • A microlens array separates light arriving from different directions.
  • A light field can be reorganized into many slightly offset sub-aperture views.
  • This is similar to capturing the scene from many nearby viewpoints.
• Key Idea
  • Multi-view uses multiple viewpoints to capture parallax.
  • Light fields capture spatial and angular light information more densely.
  • Both can represent 3D structure without explicitly storing geometry.

Light Field Re-focusing​

• 1

NeRFs​

Neural Radiance Fields

• NeRF Studio

Gaussian Splatting​

• Instead of building objects using polygons, it represents everything using millions of tiny soft 3D shapes called Gaussians.
• Less computational costs, more efficient.
• Run in parallel using GPU rasterization.

1. First, it builds a rough point cloud from images
2. Then, replaces those points with these Gaussian blobs
3. It will optimize them until it match original photos as closely as possible.

Gaussian primitive
↓
position
scale
orientation
color
opacity
...

The AI Arms Race​

• As the stakes became higher and higher in a world inching towards superintelligence, no room could be found for safety checks and alignment.

ACS Code of Professional Ethics​

• 2.1.c (Honesty): Not remain silent when you detect unprofessional conduct.
• 2.2.b (Trustworthiness): Practise integrity. Be consistent in your views, words and actions. Declare and manage any conflicts of interest. Do not allow the undue influence of others or bias to prevent you complying with this Code.
• 2.2.e (Trustworthiness): Communicate your own capabilities clearly when accepting, performing and delivering work, including potential learning and growth gaps that may need to be addressed.
• 2.2.f (Trustworthiness): Not undertake work for which you do not have the necessary skills and knowledge.
• 2.3.1.a (Respect for Others): For unavoidable harm, develop mitigation strategies.
• 2.3.1.c (Respect for Others): Be impartial and fair and do not discriminate unfairly against people in interpersonal interactions or in the design and function of systems.
• 2.3.2.c (Respect for the Profession): Seek to enhance, in the professional choices you make, the environmental sustainability of ICT systems and the overall quality of life of those affected by them. Ensure that the public interest is defended.

Win-win exploitation​

• Mutually beneficial exploitation (착취)
• Superficial Mutual Benefit: The vulnerable party receives a minor gain (e.g., small compensation, free service), creating the illusion of voluntary consent.
• Severe Surplus Asymmetry: The stronger party captures a vastly disproportionate share of the total value created.
• Vulnerability & Lack of Alternatives: Leverages power/information imbalances and the weaker party's lack of viable alternatives.
• Ethical Rationalization: The exploiter justifies the unfair structure by claiming "both sides benefit" to deflect moral and systemic responsibility.

Solutions​

• Expand "Trustworthiness" (Algorithmic Alignment & Truthfulness):
  • 감사(Audit)가 불가능한 의사결정 벡터를 가진 자율 모델 배포 금지
  • 생성형 AI의 환각(Hallucination)에 대한 출력 신뢰성 검증 의무화
• Re-engineer "Respect for Others" (Automated Discrimination & Labor):
  • 알고리즘 편향(Bias) 완화 명시
  • 조작적인 AI 상호작용으로부터 인간의 자율성 보호
  • 워크플로 자동화로 인한 노동력 대체(일자리 감소)에 대한 선제적 완화 계획 수립
• Reorient "Respect for the Profession" (Planetary & Systemic Safety):
  • 대규모 연산(High-compute) 모델 학습 시 환경 기준을 타협 불가능한 필수 조건으로 전환
  • 배포 전 단기적 운영 위험뿐만 아니라 장기적·시스템적 위험(Systemic risks)까지 평가 의무화
• Shift from Principles to Practical Enforceability:
  • 선언적 원칙에 그치지 않고 '필수 알고리즘 영향 평가' 및 '오픈소스 감사 로그' 같은 실질적인 도구를 전문가 의무로 직접 규정

Edge Detectors​

• Roberts, Sobel, Prewitt
  • Simple and fast
  • Must verify if they are adequate for the application
  • Sobel is often used
• LoG, Canny
  • More sophisticated
  • LoG uses the total gradient magnitude and direction to find edges
  • Canny uses the 2nd derivative magnitude in the gradient direction
  • Canny is more accurate and most often used

Binary Morphology​

• Taking binary images and modifying them systematically to extract information about the shapes in the image.
• A system of algebraic operations
  • conveniently process binary objects
  • elimate object shape distortions, typically due to acquisition noise
  • decomposing objects into simpler objects for easier shape characterization
• Dilation, Erosion, Closing, Opening, Shrinking, Skeletonization, and Thinning

Dilation​

$A \oplus B = \{ c \in E^N | c = a + b, a \in A, b \in B \}$

# A
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 1 0 0 0
0 0 0 1 0 0 0
0 0 0 1 1 0 0
0 0 1 0 0 0 0 
0 0 0 0 0 0 0

# B
1 1 1
1 1 1
1 1 1

# like stamping

# A \oplus B
0 0 0 0 0 0 0
0 0 X X X 0 0
0 0 X X X 0 0
0 0 X X X X 0
0 X X X X X 0
0 X X X 0 0 0

# A
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 1 0 0 0
0 0 0 1 0 0 0
0 0 0 1 1 0 0
0 0 1 0 0 0 0
0 0 0 0 0 0 0

# B
0 1 0
1 0 1
0 1 0

# A \oplus B
0 0 0 0 0 0 0
0 0 0 X 0 0 0
0 0 X X X 0 0
0 0 X X X 0 0
0 0 X X X X 0
0 X 0 X X 0 0
0 0 X 0 0 0 0

Erosion​

$A \ominus B = \{ x \in E^N | x + b \in A, \forall b \in B \}$

• It reducs the image based on the structing element B.
• Simple way of computing the erosion is to translate the initial image in the directions opposite of B 1s and AND the results.
• It checks the neighboring pixels and keeps only the pixels where the entire structuring element fits within the foreground.

Opening and Closing​

$A \circ B = (A \ominus B) \oplus B$

• $A \circ K \neq A$

$A \bullet B = (A \oplus B) \ominus B$

• $A \bullet K \neq A$

Controlled Erosions​

• It doesn't result in the complete removal of the object.
• Shrinking: Repeatedly reduces an object until each connected component becomes a single point or a minimal shape.
• Skeletonization: Reduces an object to a one-pixel-wide skeleton while preserving its overall topology and structural shape.
• Thinning: Reduces the thickness of an object while preserving its connectivity and general shape.

Object geometrical properties​

• Area
• Centroid
• Perimeter pixels
• Perimeter length
• Circularity
  • Haralick circularity
• Bouding box
• Spatial moments

riceim = imread('rice.png')
imshow(riceim);

level = graythresh(riceim);
bw = imbinarize(riceim, level);
rice_level = bwlabel(bw);

rice_level_rgb = label2rgb(rice_level);
imshow(rice_level_rgb);

pl_im = imread("Alaska_Airlines_Boeing_737-898.jpg")
pl_im = imresize(pl_im, 0.25);
pl_grey = rgb2gray(pl_im);
imshow(pl_grey);

se = strel('square', 3);
pl_erode = imerode(pl_BW, se);
pl_erode = imerode(pl_erode, se);
pl_erode = imerode(pl_erode, se);
figure(2);
imshow(pl_erode);

pl_skel = bwmorph(pl_BW, 'skel', Inf);
imshow(pl_skel);

pl_thin = bwmorph(pl_BW, 'thin', Inf);
imshow(pl_thin);

se_close = strel('disk', 20);
pl_close = imclose(pl_BW, se_close);
imshow(pl_close);

pl_skel2 = bwmorph(pl_close, 'skel', Inf);
imshow(pl_skel2);

cell_im = imread('cell.tif');
imshow(cell_im);

cell_edge = edge(cell_im, 'Sobel');
imshow(cell_edge);

se_close = strel('disk', 7);
cell_edge_close = imclose(cell_edge, se_close);
imshow(cell_edge_close);

cell_edge_close_clean = imclearborder(cell_edge_close);
imshow(cell_edge_close_clean);
figure(3);
imshow(labeloverlay(cell_im, cell_edge_close_clean));

Matching, Finding or Tracking Objects​

1. Detect invarient features of the image
2. Describe the local area around each feature
3. Match patterns of the local feature descriptions

Corners​

• Invariant to rotation, translation and scaling
• Harris corner detector is a popular method

Features​

• Detectors: detects the location of the features in an image or video
• Descriptors: summarizes the apperance of the neighborhood.
• Used in many applications: Tracking, object matching, stero vision, object and action recognition.

SIFT​

Scale-Invariant Feature Transform

1. Build a scale-space pyramid of Differences of Gaussians (DoG) and detect minima/maxima.
2. Localize Keypoints
3. Assign key point and orientation and scale
4. Compute the SIFT descriptor at the assigned orientation and scale.

Image Processing Operations​

Point Operation​

$b[m, n] = f(a[m, n])$

• It only depends on the value of the pixel itself, not on the values of its neighbors.
• e.g. current pixel + 20.
• to increase the brightness of an image, adjust contrast, or apply a threshold to create a binary image.

Local Operation​

$b[m, n] = f(a[m - 1, n - 1], a[m - 1, n], a[m - 1, n + 1], a[m, n - 1], a[m, n], a[m, n + 1], a[m + 1, n - 1], a[m + 1, n], a[m + 1, n + 1])$

• It depends on the values of the pixel and its neighbors.
• e.g. current pixel + average of 8 neighbors.
• to blur an image, sharpen an image, detect edges, or convolution with a kernel.
• The most common type of neighborhoods are:
  • 4-neighbors:
    • top, bottom, left, right.
  • 8-neighbors:
    • top, bottom, left, right, and the 4 diagonal neighbors.

Global Operation​

$b[m, n] = f(a[0, 0], a[0, 1], ..., a[M - 1, N - 1])$

• It depends on the values of all pixels in the image.
• e.g. current pixel + average of all pixels in the image.
• to compute the histogram equalization, apply a global threshold, or perform a Fourier transform.

Image Histogram​

• It is a graph showing how many pixels in an image have each possible intensity value.
  • Intensity value: the brightness of a pixel.
• e.g. 8-bit grayscale image has 256 possible intensity values (0-255).
  • The histogram will graphically display 256 numbers showing the distribution of pixels among those gray-scale values.

Histogram Equalization​

0                     255
|------████████--------|
       80~140에 몰림

0                     255
|--██--██--██--██--██--|

• It spreads out the intensity values that are concentrated in a narrow range, increasing the contrast of the image.
• It is useful when the images have been acquired under poor lighting conditions or have low contrast (different circumstances).

Noise​

• Any undesired information that contaminatest the image.
• During the analog-to-digital conversion process, it is a side effect of the physical conversion of patterns of light energy into electrical patterns.
• The shape of distribution of noise types used to describe many of them and is related closely to the histogram.

Gaussian Noise​

frequency
 ^
 |             █
 |          █████
 |       █████████
 |    █████████████
 +----------------------> noise gray level
        -20  0  +20

• The most common type of noise, with a bell-shaped distribution.
• Natural noise process such as electronic noise in the image acquisition system.

Uniform Noise​

frequency
 ^
 |       ┌───────────────┐
 |       │               │
 |       │               │
 +-------┴───────────────┴------> noise intensity
         a               b

• A type of noise with a distribution that is constant across the range of intensity values.
• The gray-level values of noise are evenly distributed across a specific range.
• It can be used to generate any toehr type of noise distribution, often used to degrade images for the evaluation of image restoration algorithms.
  • it provides the most unbiased or neutral noise model.

Salt-and-pepper noise​

frequency
 ^
 | █                       █
 | █                       █
 | █                       █
 +----------------------------> gray level
   0                         255

• A distribution that has two spikes at the minimum and maximum intensity values.
• The presence of single dark pixels in bright regions, or single bright pixels in dark regions.
  • Typically affects a small set of pixels.
• It is usually quantified by the percentage of pixels which are corrupted by noise.
• It is typically caused by errors in data transmission, faulty memory locations, or malfunctioning pixel elements in camera sensors.

Signal-to-Noise Ratio​

$SNR = 10 \log_{10} \frac{P_{signal}}{P_{noise}}$

• SNR
• The ratio between the power of the signal and that of the noise.
• In a perfect image, the ratio of signal to noise is infinite.

Noise Elimination​

• Restore the true value of the pixels as much as possibole.
• It may undesirably reduce image information.
• Averaging the pixel with its neighbours will smooth the noise or other types of image filters can be applied to reduce noise.

Filters​

• Linear filters: low pass, high pass
• Non-linear filters: median
• Filters are used to improve an image
  • if the image is destined for human viewing, to make it more pleasant to look it or more readable.
  • if the image is the input to a pattern recognition process, to facilitate the following steps of automated image analysis.

Convolution​

$I(r, c) \otimes F = \sum_{i=1}^{2M + 1} \sum_{j=1}^{2M+1} I(r+i-(M+1), c + j-(M+1)) F(i, j)$

• Multiply the pixels of a neighborhood of $(r, c)$ by the corresponding coefficients of the filter $F$, and add them all together.

Low Pass Filter​

• Smoothing or softening, employes to remove high spatial frequency noise from a disital image.
• It replace each pixwel with a weighted sum of each pixel's neighbors.
• It is used to remove noise, might have the side-effect of generally smoothing or blurring images and reducing edge information.
• Local averaging: take the local average of the pixels in a neighborhood and replace the center pixel with that value.

Gaussian Filter​

$H_{ij} = \frac{1}{2\pi\sigma^2} e^{-\frac{i^2 + j^2}{2\sigma^2}}$

• yields a $2k+1 \times 2k+1$ kernel, where $k$ is the size of the filter and $\sigma$ is the standard deviation of the Gaussian distribution.
• A smoothing filter that computes a weighted average of neighboring pixels, giving larger weights to pixels closer to the center.

1   4   7   4   1
4  16  26  16   4
7  26  41  26   7
4  16  26  16   4
1   4   7   4   1

• Smaller $\sigma$ values result in a more localized filter, which means weak smoothing and less blurring of the image.
• Larger $\sigma$ values result in a more spread-out filter, which means stronger smoothing and more blurring of the image.

Median Filter​

• A non-linear filter that replaces a pixel with the median of its neighbors.
• It is effective at removing salt-and-pepper noise and other isolated noise compared to low-pass linear filters.
• Less blurred, edges remain sharp, removes single pixel erros completely, but slower requires sorting the pixels in the neighborhood.

10  11  10
12 255  11
10  12  11

# 255 is salt-and-pepper noise, the median of the 9 pixels is 11, so the center pixel is replaced with 11.
10, 10, 11, 11, 11, 12, 12, 255

# to-be
10, 10, 11, 11, 11, 12, 12, 11

High Pass Filter​

• It extracts high-frequency components, such as edges and fine details, by subtracting a low-pass filtered image from the original image.
• Sometimes, it is desired to enhance the high frequencies without removing the low frequencies.

Sharpened Image = Original Image + High-frequency component
  = Origial Image + (Original Image - Low-pass filtered Image)

Conclusion​

• Low-pass filter → smooth / blur
• High-pass filter → edge / detail
• High-pass + original → sharpening

1.0: Mechanisation​

• 1780
• Industrial production based on machines
• Powered by water and steam

2.0: Electrification​

• 1870
• Mass-production using assembly lines

3.0: Automation​

• 1970
• Automation using electronics and computers

3.5: Globalisation​

• 1980
• Offshoring of production to low-cost economies

4.0: Digitalisation​

• Today
• Introduction of connected devices
• Data analytics
• Artificial intelligence technologies
• Further automation of processes

5.0: Personalisation​

• Future
• The fifth industrial revolution, or Industry 5.0, will be focused on the co-operation between man and machine
• Human intelligence works in harmony with cognitive computing
• Humans are put back into industrial production with collaborative robots
• Workers will be upskilled to provide value-added tasks in production
• Leading to mass customisation and personalisation for customers

Ref​

• What will Industry 5.0 mean for manufacturing?

Moral Machines​

• Moral Machines Test

Anatomy of an AI System​

Questions​

• Why is this labor so frequently rendered invisible to the end user?
  • 왜 이러한 노동은 최종 사용자에게 그토록 자주 보이지 않게 되는가?
• How does the short lifespan of a smart device compare to the geological time needed to form its raw minerals and the centuries required for it to decay as e-waste?
  • 스마트 기기의 짧은 수명은 원료 광물이 형성되는 데 필요한 지질학적 시간, 그리고 전자폐기물로 분해되는 데 걸리는 수백 년의 시간과 어떻게 대비되는가?
• What ethical and intergenerational obligations do technology companies bear for the imbalance between timescales?
  • 이러한 시간 규모의 불균형에 대해 기술 기업은 어떤 윤리적 책임과 세대 간 책임을 져야 하는가?
• How is the word 'extraction' a good bridge between the ethical issues related to mining on one hand and capitalism's thirst for data on the other?
  • ‘추출(extraction)’이라는 단어는 한편으로는 채굴과 관련된 윤리적 문제를, 다른 한편으로는 자본주의의 끝없는 데이터 욕구를 연결하는 개념으로서 왜 적절한가?
• What legislative, regulatory, or architectural interventions could hold technology companies accountable for these hidden environmental and social costs?
  • 어떤 입법적, 규제적 또는 설계·구조적 개입을 통해 기술 기업이 숨겨진 환경적·사회적 비용에 대해 책임을 지도록 할 수 있는가?
• Should companies be allowed to sell devices with very short lifespans when the materials used to make them take millions of years to form and may remain as waste for generations?
  • 기기를 만드는 데 사용되는 원료는 형성되는 데 수백만 년이 걸리고 폐기물로는 여러 세대 동안 남을 수 있는데, 기업이 수명이 매우 짧은 기기를 판매하도록 허용해야 하는가?
• How much responsibility should current technology companies have for environmental harms that may only become fully visible decades from now?
  • 수십 년이 지나야 완전히 드러날 수 있는 환경 피해에 대해 현재의 기술 기업은 어느 정도까지 책임을 져야 하는가?
• If both natural resources and human data can be extracted for profit, what makes data extraction ethically different from mining — and does that difference matter?
  • 천연자원과 인간의 데이터가 모두 이윤을 위해 추출될 수 있다면, 데이터 추출은 채굴과 윤리적으로 무엇이 다르며, 그 차이는 중요한가?
• Should technology companies be legally required to account for the full social and environmental cost of an AI system, even if doing so makes the technology more expensive?
  • 기술 비용이 더 비싸지더라도, 기술 기업은 인공지능 시스템의 모든 사회적·환경적 비용을 법적으로 산정하고 책임지도록 요구받아야 하는가?

Sequential Decision Making​

• A sequence is a series of events or actions that occur in a specific order over time.
• Goal: select actions that maximize total expected future reward.
• It may require balancing immediate and long-term rewards.
• It may be better to sacrifice short-term gains for long-term benefits.
• It may require strategic behavior to achieve high rewards.

Example of SDM​

• Web Ads
  • Agent AdBot
    1. Choose and Ad $A_t$.
    2. Receives View time $O_t$
    3. Receives Click on Ad $R_t$
  • Environment
    1. Receives the Chosen Ad $A_t$.
    2. Provides the View time $O_{t+1}$.
    3. Provides the Click on Ad $R_{t+1}$.
• Robot picking trash
  • Agent Robot
    1. Moves arms to pickup trash $A_t$.
    2. Receives Camera image of the room $O_t$.
    3. Receives Reward if no trash on the floor $R_t$
  • Environment
    1. Receives robots action/arm movements $A_t$.
    2. Provides Camera Image of the room $O_{t+1}$.
    3. Provide reward of +1 $R_{t+1}$
• Robot making Pizza.
  • Agent Robot
    1. Makes an action from Action space $A_t$ (a range of possible actions, Arm Movements, Get Pizz from Oven, Place Pizza on Tray, Cut Pizza...).
    2. Receives Camera image of the kitchen $O_t$.
    3. Receives Reward if pizza is made correctly $R_t$
  • Environment
    1. Receives robots action/arm movements $A_t$.
    2. Provides Camera Image of the kitchen $O_{t+1}$.
    3. Provide reward appropriate to the action $R_{t+1}$.
  • Reward space
    • +10 Successfully made pizza.
    • +1 Get Pizza from oven.
    • +1 Place Pizza on tray.
    • +1 Get Packing box.
    • +1 Cut Pizza successfully.

Rewards​

• a scalar signal that indicates how well the agent is doing at time step $t$. (agent's performance)
• Examples of rewards:
  • Robot play soccor:
    • $+ve$ reward for scoring a goal.
    • $-ve$ reward for kicking the ball out of bounds.
  • Chess
    • $+ve$ reward for winning the game.
  • Autonomous drone flying stunt:
    • $+ve$ reward for following the intended trajectory.
    • $-ve$ reward for crashing into an obstacle.

History​

$H_t = \{O_1, A_1, R_1, O_2, A_2, R_2, ..., O_t, A_t, R_t\}$

• a sequence of past Observations, Actions, and Rewards up to time step $t$.
• Agent selects actions base on history.
• Environment selects observations and reward.

State​

$S_t = f(H_t)$

• The information used to determine what happens next.
• A function of the history that captures all relevant information for decision-making.

Environment State​

• $S_t^e$: the environment's private representation of the current state.
• It used to generate next observation and reward.
• It is not directly accessible to the agent (Invisible to the agent).
• If it is visible to the agent, it may contain information not required by the agent to make decisions (e.g., the internal state of a robot's motors).

Agent State​

• $S_t^a$: the agent's private representation of its state.
• Information used to pick next action and used by RL algorithms to learn from experience.
• It can be a function of the history: $S_t^a = f(H_t)$

Information State​

$P[S_{t+1} | S_t] = P[S_{t+1} | S_1, \cdots, S_t]$

• It contains all useful information from the history as known as the Markov State.
• Probability[next state | current state] = Probability[next state | given the whole history]
• The future is independent of the past given the present.
• The dog example illustrates a situation where the agent needs to remember past signals in order to receive a food reward.
• Possible definitions of the agent state:
  • Last 3 events in the sequence
    • Example: bell ring → light → bell press
    • Useful when recent observations are enough to predict the reward
  • Counts of events
    • Example: number of lights, bell rings, and bell presses
    • Useful for compressing information, but it loses the order of events
  • Complete sequence
    • Example: bell ring → light → bell press → bell ring → ...
    • Contains all past information, but can become too large and difficult to learn from
• Each state definition involves a trade-off between the amount of information stored and the complexity of learning.
• A good state should contain enough information to predict the next state and reward.

Markov assumption​

• Markov assumption is popular because it simplifies decision-making in RL.
• A Markov state can always be created by defining the state as the full history:
  • $S_t = H_t$
  • This includes all past observations, actions, and rewards.
  • It satisfies the Markov property because no extra past information is needed.
• However, using the full history creates a very large state space.
  • More computation
  • More memory
  • More data required
  • Harder learning
• Ideally, the current observation is enough:
  • $S_t = O_t$
  • This means the current observation is a sufficient statistic of the history.
  • The agent does not need to remember the full past.
• Smaller state spaces are preferred because they reduce computational complexity and data requirements.
• The choice of state representation affects:
  • Computational complexity
  • Amount of data required
  • Final performance
• Main idea: A good state should be small, but still contain enough information to predict the next state and reward.

Markov Property​

• The current state contains all relevant information from the history needed to predict the future.
• The future is independent of the past given the present state.

Full Observable Environments, MCPs​

$O_t = S_t^a = S_t^e$

• Full observable: Agent is able to directly observe the environment.
• Agent State = Environment State = Information State
• Known as Markov Decision Process

Partially Observable Environments, POMDPs​

• Partially observable: Agent only observes partial information about the environment.
• Agent State != Environment State
• Agent must construct its own internal state (S_t^a)
• Ways to construct agent state:
  • Complete history:
    • (S_t^a = H_t)
  • Belief state:
    • (S_t^a = b(S_t^e))
    • Probabilities over possible environment states
  • RNN hidden state:
    • previous hidden state + current observation → new hidden state
• Known as Partially Observable Markov Decision Process, POMDP.
• Examples:
  • Soccer-playing robot with a camera and limited information about its exact location.
  • Share trading agent that observes current share price but not full market trends or company history.
  • Card games where opponent cards and deck order are hidden.

Exploration vs Exploitation​

• Exploration: Learn more about the environment, may lose immediate reward
• Exploitation: Exploit known information to maximize rewards
• Discover a good policy, while interacting with the environment, the agent must balance exploration and exploitation.
• These are key aspects of RL learning, due to trial-and-error learning and the need to maximize long-term rewards.
• Examples:
  • Playing Chess
    • Exploitation: Use familiar opening lines that have won in the past.
    • Exploration: Try a new opening sequence.
  • Online Advertisements
    • Exploitation: Show ads with the highest past click rate.
    • Exploration: Show different ads and collect feedback such as clicks or view time.
  • Gold Mining
    • Exploitation: Mine at the best-known location.
    • Exploration: Mine at a new location.

Bandits​

• Actions that are taken has no influence on next observations. (Slot machines)
• No delayed rewards.
• RL problem with one state.
• Examples:
  • Design of Clinical trials
  • Online ad suggestion, placement
  • Games
  • Web page personalization
• Action now, Reward now.
• Single State $(S_t = S_1)$.
• Set of Actions = $\{A_1, A_2, ..., A_n\}$.
• Reward space = $[0, 1]$.
• Learn a Stochastic Reward Function: Reward probabilities for actions are unknown in advance, so they must be learned stochastically through trial and error.

Online Ad suggestion​

$\text{PayOff} = \text{CTR} \times \text{Payment Rate}$

• How to maximize PayOff among hundreds thousands of ads?
• CTR, ClickThroughRate: Probability that users click on the ad.
• Payment Rate: Money paid by the advertiser for each click.
• Arms: $\{Ad_1, Ad_2, ..., Ad_n\}$
• Rewards: $\{ 0: \text{No Click}, 1: \text{Click} \}$
  • Assuming a uniform payment of $1 for all ads, maximizing revenue simplifies to accurately estimating the Click-Through Rate (CTR).
• Exploration vs. Exploitation: Should we display new ads with unknown CTRs to test them (Exploration), or continuously show top-performing ads with the highest historical CTR (Exploitation)?
  • No known optimal solution, but there are many heuristic approaches to balance exploration and exploitation.

$\epsilon$-Greedy​

• a common exploration strategy to balence getween exploiting the current best action and exploring new actions in order to learn an optimal policy.
• Select the Arm with highest average reward so far
• May get struck with no exploration.
• $\epsilon$: The probability of selecting a random arm and used a greedy selection.

Init, for Arm = 1 to k:
  Q(Arm) = 0
  N(Arm) = 0

Loop:
  # Explore
  A = Random action with probability 𝜖
  # Exploit
  A = argmax(Q(Arm)) with probability (1 - 𝜖)

  R = bandit(A)
  N(A) += 1
  Q(A) = Q(A) + (1 / N(A)) * (R - Q(A))

• Q: Estimated reward for each Arm
• N: Number of time each Arm is pulled.
• A: Action
• R: Reward

NewEstimate = OldEstimate + StepSize[Target - OldEstimate]

UCB​

$UCBValue = Q(A) + \sqrt{\frac{2 \ln(t)}{N(A)}}$

• used to balance between exploiting known good actions and exploring potentially better actions.
• Optimism in the face of uncertaintiy: It will choose actions systematically that have a high potential for being optimal based on both their estimated value and their uncertatity.
  • The uncertainty bonus represents the value of exploring that arm.

Init, for Arm = 1 to k:
  Q(Arm) = 0
  N(Arm) = 0

Loop:
  UCBValue = Q(A) + sqrt((2 * log(t)) / N(A))
  A = argmax(UCBValue)
  R = bandit(A)

  N(A) += 1
  Q(A) = Q(A) + (1 / N(A)) * (R - Q(A))

• t: Time step
• Q: Estimated reward for each Arm
• N: Number of time each Arm is pulled
• A: Action
• R: Reward

General Update Rule same as $\epsilon$-Greedy.

Reinforcement Learning​

• Learn how to make a good sequence of decisions by interacting with the environment.

Characteristics of RL​

• Trial and Error: Based learning approach.
• Optimization: Find good sequences of actions or decisions.
• Delayed Consequences/Rewards: Takes time to relize the actions or decisions are good or bad.
• Exploration: Learn by making decisions or forming actions or through experiences. Trying new actions to discover their effects.
• Exploitation: Choosing actions with the highest expected reward, based on current knowledge.
• Generalization: Use previous experiences or knowledge to new or unseen situations effectively.

Framwork of RL​

• OpenAI Gym
• Torch RL
• AWS DeepRacer

RL Math​

Probability​

• Sample Space ($\Omega$): The set of all possible outcomes of a random experiment.

Bonferroni's Inequality​

$P(A \cap B) \geq P(A) + P(B) - 1$ $P(\cap^{n}_{i=1} A_i) \geq \sum_{i=1}^{n} P(A_i) - (n-1)$

• Gives a lower bound on the intersection probability which is useful when this probability is difficult to compute directly.
• It is useful when the probabilities of individual events are sufficiently large.

Boole's Inequality​

$P(\cup^{n}_{i=1} A_i) \leq \sum_{i=1}^{n} P(A_i)$

for any sets $A_1, A_2, ..., A_n$.

• It is useful when finding an upper bound for the probabilities of the union of events.

Bayes' Rule​

$P(A|B) = \frac{P(A \cap B)}{P(B)}$ $P(A\cap B) = P(B|A) P(A) = P(A|B) P(B)$ $P(A|B)P(B) = P(B|A)P(A)$ $P(A|B) = \frac{P(B|A) P(A)}{P(B)}$

• It allows us to compute the conditional probability $P(A|B)$ from the inverse conditional probability $P(B|A)$.
• Let $A_1, A_2, ..., A_n$ be a partition of the sample space $S$. Then let $B$ be any subset of $S$ we have:

$P(A_i|B) = \frac{P(B|A_i) P(A_i)}{\sum_{j=1}^{\infty} P(B|A_j) P(A_j)}$

Independent Events​

$P(A \cap B) = P(A) P(B)$

• A family $A_i :i \in I$ of events is independent if for every finite subset $J \subseteq I$ we have: $P\left(\bigcap_{i \in J} A_i\right) = \prod_{i \in J} P(A_i)$

• The pair-wise independence of events does not imply their mutual independence.

Conditional Independence​

$P(A \cap B|C) = P(A|C) P(B|C)$

• where $P(C) > 0$.
• or equivalently, $P(A|B \cap C) = P(A|C)$ and $P(B|A \cap C) = P(B|C)$.

Induced Probability Function​

$P_X(x_i) = P\big(\{w_j \in \Omega : X(w_j) = x_i\}\big)$

• $\Omega = \{w_1, w_2, ..., w_m\}$ is the sample space.
• $X$ is a random variable with range $X = \{x_1, x_2, ..., x_n\}$.
• The result set $\{w_j \in \Omega : X(w_j) = x_i\}$ is the set of all outcomes in the sample space that map to the value $x_i$ under the random variable $X$.

Cumulative Distribution Function (CDF)​

$F_X(x) = P_X(X \leq x) \quad \forall x \in \mathbb{R}$

• $F_X(x)$ is a cdf $\iff$ the following conditions hold:
  • Monotonicity: $F_X(x)$ is non-decreasing.
  • Limiting values: $\lim_{x \to -\infty} F_X(x) = 0$ and $\lim_{x \to \infty} F_X(x) = 1$.
  • Right-Continuity: $\lim_{y \downarrow x} F_X(y) = F_X(x) \quad \forall x \in \mathbb{R}$.

F_X(t)

1.0 |                         ●────────────
    |                         ↑  ↑  ↑
    |                      3.001 3.01 3.1
0.7 |              ●─────────○
    |              ↑  ↑  ↑
    |           2.001 2.01 2.1
0.3 |     ●────────○
    |     ↑  ↑  ↑
    |  1.001 1.01 1.1
0.0 |─────○
    +-------------------------------------- t
          1         2         3
          x         x         x

Continuous & Discrete Random Variables​

• if $X$ is continuous, then $F_X(x)$ is continuous and differentiable almost everywhere. The probability density function (pdf) is defined as: $f_X(x) = \frac{d}{dx} F_X(x)$
• if $X$ is discrete, then $F_X(x)$ is a step function and the probability mass function (pmf) is defined as: $p_X(x_i) = P(X = x_i)$

1 ────────────────────────━━━━━━━━
                        ╱
                      ╱
                    ╱
                  ╱
0 ━━━━━━━━━━━━━━━───────────────── x

1 ─────────────────────────●━━━━━━
                           │
0.7 ─────────────●━━━━━━━━━○
                 │
0.3 ─────●━━━━━━━○
         │
0 ━━━━━━━○──────────────────────── x
         1         2         3

1 ──────────────────────────━━━━━━
                          ╱
                        ╱
0.7 ───────────●━━━━━━━
               │
0.5 ───────────○
             ╱
           ╱
0 ━━━━━━━━──────────────────────── x
               a

Probability Mass Function (PMF)​

• A discrete random variable $X$ is given by $f_X(x) = P(X=x)$ for $x \in \mathbb{R}$.
• It represents the probability that the first success occurs exactly on the $x$-th trial in a sequence of independent trials.

Probability Density Function (PDF)​

• A continuous random variable $X$ is given by $f_X(x)$ for $x \in \mathbb{R}$.
• The probability is calculated as the area under the probability density function over a specific interval.

Expectation​

$E[X] = \sum x P(X=x) \quad \text{if } X \text{ is discrete}$

$E[X] = \int_{-\infty}^{\infty} x f_X(x) dx \quad \text{if } X \text{ is continuous}$

• Linearity: $E[aX + bY + c] = aE[X] + bE[Y] + c$ for any constants $a$, $b$, and $c$.
• Non-negativity: If $X \geq 0$ then $E[X] \geq 0$ because it is a weighted average of non-negative values.
• Monotonicity: If $X \geq Y$ then $E[X] \geq E[Y]$ because it is a weighted average of values that are greater than or equal to the corresponding values of $Y$.
• Boundedness: If $a \leq X \leq b$ then $a \leq E[X] \leq b$ because it is a weighted average of values that are bounded by $a$ and $b$.

a ●──────────────●──────────────● b
       가능한 값들    평균
                   E[X]

Moments​

$\mu'_n = E[X^n] \quad \forall n \in \mathbb{N}$

• The $n^{th}$ central moment of $X$ is: $\mu_n = E[(X - E[X])^n] = E(X - \mu)^n$

• 1th central moment is the mean, $\mu_1 = E[X]$.

• 2th central moment is the variance, $\mu_2 = E[(X - E[X])^2] = \text{Var}(X)$.

  • It emphasizes the variability of the distribution.
  • $\text{Var}(aX + b) = a^2 \text{Var}(X)$ for any constants $a$ and $b$.

• 3th central moment is the skewness, $\mu_3 = E[(X - E[X])^3]$.

  • It emphasizes the skewness of the distribution.

• 4th central moment is the kurtosis, $\mu_4 = E[(X - E[X])^4]$.

  • It emphasizes the tail behavior and extreme values of the distribution.

Covariance​

$\text{cov(X, Y)} = E[(X - E[X])(Y - E[Y])]$

• It measures how muc htwo random variables change together.
• Negative Covariance
• Near Zero Covariance
• Positive Covariance

Y: 시험 점수

높음  |                ●
     |          ●  ●
     |       ●
     |    ●
낮음  | ●
     +-------------------- X: 공부 시간
       적음             많음

       Cov(X,Y) > 0

Correlation​

$p(X, Y) = \frac{\text{cov(X, Y)}}{\sqrt{\text{Var}(X)\text{Var}(Y)}}$

• Individual variances must be non-zero.
• $p(X, Y)$ lies in the range $[-1, 1]$.

Joint Distributions​

$f_{X,Y}:\mathbb{R}^2\to[0,1], \\ f_{X,Y}(x,y) = P(X=x, Y=y) \quad \text{if } X, Y \text{ are discrete}$

• Joint Probability Mass Function (PMF) for discrete random variables $X$ and $Y$.

Y
1 ┤  ● 1/4       ● 1/4
  │ (0,1)       (1,1)
0 ┤  ● 1/4       ● 1/4
  │ (0,0)       (1,0)
  └──────────────────── X
      0           1

Marginal Distributions​

$f_X(x) = \sum_{y} f_{X,Y}(x,y) \\ f_Y(y) = \sum_{x} f_{X,Y}(x,y)$

• Fixing one variable and summing over the other variable gives the marginal distribution of the fixed variable.

             Y=0       Y=1       행의 합
X=0          0.10      0.20        0.30
X=1          0.30      0.40        0.70
             ─────────────────────────
열의 합       0.40      0.60        1.00

• $f_X(0) = 0.10 + 0.20 = 0.30$: sum of the first row.
• $f_X(1) = 0.30 + 0.40 = 0.70$: sum of the second row.
• $f_Y(0) = 0.10 + 0.30 = 0.40$: sum of the first column.
• $f_Y(1) = 0.20 + 0.40 = 0.60$: sum of the second column.
• Marginalization: It sums or integrates over all possible values of an unwanted variable to obtain the distribution of the variable of interest.

Conditional Distributions​

$f_{X|Y}(x|y) = P(X = x | Y = y) = \frac{f_{X,Y}(x,y)}{f_Y(y)} \quad \text{if } f_Y(y) > 0$

• It represents the probability distribution of $X$ given that $Y$ has a specific value.
• If $f_Y(y) = 0$, then $f_{X|Y}(x|y)$ is undefined.

Bernoulli Distribution​

$X = \begin{cases} 1, & \text{with probability } p \\ 0, & \text{with probability } 1-p \end{cases}$

• $E[X] = p$
• $\text{Var}(X) = p(1-p)$
• The outcome of a Bernoulli trial is either a success (1) or a failure (0).

Binomial Distribution​

• probability of success in a single trial: $p$
• probability of failure in a single trial: $q = 1 - p$

$P(X = x|n, p) = \binom{n}{x} p^x q^{n-x} \\ \text{where} \binom{n}{x} = \frac{n!}{x!(n-x)!}, \quad 0 \leq x \leq n$

• $E[X] = np$
• $\text{Var}(X) = npq = np(1-p)$
• The binomial distribution describes the number of successes in a fixed number of independent Bernoulli trials.
• $X \sim \text{Binomial}(10, 0.7)$
  • $P(X = 7) = \binom{10}{7} (0.7)^7 (0.3)^3 \approx 0.2668$

Geometric Distribution​

• probability of success in a single trial: $p$
• probability of failure in a single trial: $q = 1 - p$

$P(X = x|p) = (q^{x-1}) p \\ \text{where } x = 1, 2, 3, ...$

• $E[X] = \frac{1}{p}$
• $\text{Var}(X) = \frac{q}{p^2}$
• The geometric distribution describes the number of trials before the first success in a sequence of independent Bernoulli trials.
• $X \sim \text{Geometric}(0.7)$
  • $P(X = 3) = (0.3)^2 (0.7) \approx 0.063$

Uniform Distribution​

$f_X(x|a,b) = \begin{cases} \frac{1}{b-a}, & a \leq x \leq b \\ 0, & \text{otherwise} \end{cases}$

• $E[X] = \frac{a+b}{2}$
• $\text{Var}(X) = \frac{(b-a)^2}{12}$
• The uniform distribution describes a continuous random variable that has an equal probability of taking any value within a specified range.
• The probability depends on the length of the interval, not its location.

Normal Distribution​

$f_X(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}, \quad -\infty < x < \infty$

$X \sim N(\mu, \sigma^2)$

• Central Limit Theorem: the distribution of the sum (or average) of a large number of independent, identically distributed variables will be approximately normal, regardless of the underlying distribution.

$\bar{X} = \frac{1}{n} \sum_{i=1}^{n} X_i \approx N\left(\mu, \frac{\sigma^2}{n}\right)$

Multivariate Normal Distribution​

$N(x|\mu, \Sigma) = \frac{1}{\sqrt{(2\pi)^D |\Sigma|}} e^{-\frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu)}$

• $\mu$ is the $D$-dimensional mean vector.
• $\Sigma$ is the $D \times D$ covariance matrix.
• $|\Sigma|$ is the determinant of the covariance matrix.

Beta Distribution​

$f(x|\alpha, \beta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} x^{\alpha - 1} (1 - x)^{\beta - 1}, \quad 0 < x < 1$

• $\Gamma(n) = (n-1)!$ is the gamma function.
• $E[X] = \frac{\alpha}{\alpha + \beta}$
• $\text{Var}(X) = \frac{\alpha \beta}{(\alpha + \beta)^2 (\alpha + \beta + 1)}$

α < β                  α = β                  α > β

높이                   높이                   높이
│╲                     │    ╭──╮             │        ╱
│ ╲___                 │  ╭─╯  ╰─╮           │    ___╱
└──────── x            └────────── x         └──────── x
0        1             0          1           0        1

0 쪽 강조              가운데 강조             1 쪽 강조

Beta(8,2)                 Beta(80,20)

       ╭────╮                    /\
     ╭─╯    ╰─╮                 /  \
─────╯        ╰───        ─────/────\─────
         0.8                      0.8

불확실성 큼                 불확실성 작음

• $p | \text{data} \sim \text{Beta}(\alpha + s, \beta + f)$
  • where new observations are $s$ successes and $f$ failures.
• The Beta distribution is a distribution of success probabilities for a Bernoulli or Binomial distribution.
• Application
  • Measuring uncertainty in the probability of success for a Bernoulli or Binomial distribution.
  • Conversion rate or click-through rate (CTR) in online advertising.
  • Defect rate in manufacturing.

Linear Algebra​

Axioms of Vector Spaces​

• Commutative Law: $u + v = v + u, \quad \forall u, v \in V$
• Associative Law: $(u + v) + w = u + (v + w), \quad \forall u, v, w \in V$
• Additive Identity: $\exists 0 \in V$ such that $v + 0 = v, \quad \forall v \in V$
• Additive Inverse: $\forall v \in V, \exists -v \in V$ such that $v + (-v) = 0$
• Distributive Law:
  • $a(u + v) = au + av, \quad \forall a \in F, u, v \in V$
  • $(a + b)v = av + bv, \quad \forall a, b \in F, v \in V$
• Associative Law: $(ab)v = a(bv), \quad \forall a, b \in F, v \in V$
• Unitary Law: $1v = v, \quad \forall v \in V$

Subspace​

• $x + \alpha y \in W, \quad \forall x, y \in W, \forall \alpha \in \mathbb{R}$

Norm​

$f: \mathbb{R}^n \to \mathbb{R}$

• It outputs a real number that represents the length or size of a vector in a vector space.

$x=(x_1,x_2,\ldots,x_n) \mapsto f(x)=\lVert x\rVert$

• Non-negativity: $\forall x \in \mathbb{R}^n, \quad \lVert x\rVert \geq 0$
• Definiteness: $f(x) = 0 \iff x = 0$
• Homogeneity: $\forall x \in \mathbb{R}^n, \quad \forall t \in \mathbb{R}, f(tx) = |t| f(x)$
• Triangle inequality: $\forall x,y \in \mathbb{R}^n, \quad f(x+y) \leq f(x) + f(y)$
  • $\lVert x + y \rVert \leq \lVert x \rVert + \lVert y \rVert$
• A norm is non-negative, only the zero vector has a norm of zero, scaling a vector by two doubles its norm, and the direct distance cannot be greater than the detoured distance.

$\lVert x \rVert_p = (\sum_{i=1}^{n} |x_i|^p)^{\frac{1}{p}}, \quad p \geq 1$

• L1 norm: $\lVert x \rVert_1 = \sum_{i=1}^{n} |x_i|$
  • Manhattan distance.
• L2 norm: $\lVert x \rVert_2 = \sqrt{\sum_{i=1}^{n} |x_i|^2}$
  • Euclidean distance.
• L-infinity norm: $\lVert x \rVert_\infty = \max_i|x_i|$
  • $\lVert (3, 4) \rVert_\infty = \max\{3, 4\} = 4$

$\lVert A \rVert_F = \sqrt{\sum_{i=1}^{m} \sum_{j=1}^{n} |a_{ij}|^2} = \sqrt{\text{tr}(A^T A)}$

• Frobenius norm: $A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$
  • $\lVert A \rVert_F = \sqrt{1^2 + 2^2 + 3^2 + 4^2} = \sqrt{30}$
  • It is the square root of the sum of the absolute squares of its elements.

Span​

$span\{x_1, x_2, ..., x_n\} = \{ \alpha_1 x_1 + \alpha_2 x_2 + ... + \alpha_n x_n : \alpha_n \in \mathbb{R}\}$

• A set of vertors $X = \{x_1, x_2, ..., x_n\}$ in a vector space $V$ is said to span $V$ if every vector in $V$ can be expressed as a linear combination of the vectors in $X$.
• Span is the entire region that can be reached by scaling and adding the given vectors.

Range​

$R(A) = \{\alpha_1 a_1 + \alpha_2 a_2 + ... + \alpha_n a_n : \alpha_n \in \mathbb{R}\}$

• The range of a matrix $A$ is the set of all possible linear combinations of its column vectors.
• Same as the columnspace of the matrix $A$.

Nullspace​

$N(A) = \{x \in \mathbb{R}^n : Ax = 0\}$

• A set of all vectors that equal 0 when multiplied by the matrix $A$.
• The dimensionality of the nullspace is called the nullity of the matrix $A$.

Linear Independence​

• A set of vectors $\{v_1, v_2, ..., v_n\}$ is said to be linearly independent if the only solution to the equation $\alpha_1 v_1 + \alpha_2 v_2 + ... + \alpha_n v_n = 0$ is $\alpha_1 = \alpha_2 = ... = \alpha_n = 0$.
• Column rank: The maximum number of linearly independent column vectors in a matrix.
• Row rank: The maximum number of linearly independent row vectors in a matrix.
• If one row vector is a combination of other row vectors, then it is linearly dependent.
• The number of rows that are removed due to redundancy is the Rank of the matrix.
• Due to the redundancy, the number of directions which is vanishing is the Nullity of the matrix.

Rank​

$Rank(A) = \text{dim}(R(A)), \quad A \in \mathbb{R}^{m \times n}$

• $rank(A) \leq \min(m, n), \quad A \in \mathbb{R}^{m \times n}$
  • if $rank(A) = min(m, n)$, then $A$ is said to be full rank.
  • Rank cannot exceed the number of rows or columns in the matrix.
• $rank(A) = rank(A^T), \quad A \in \mathbb{R}^{m \times n}$
  • The rank of a matrix is equal to the rank of its transpose.
  • The amount of independent information in the rows is equal to the amount of independent information in the columns.
• $rank(AB) \leq \min(rank(A), rank(B)), \quad A \in \mathbb{R}^{m \times n}, B \in \mathbb{R}^{n \times p}$
  • The information content of the product of two matrices cannot exceed the information content of either matrix.
• $rank(A + B) \leq rank(A) + rank(B), \quad A, B \in \mathbb{R}^{m \times n}$
  • The information content of the sum of two matrices cannot exceed the sum of the information content of each matrix.

Orthogonal Matrices​

$U \in \mathbb{R}^{n \times n} \text{ is orthogonal } \iff v_i^T v_j = \begin{cases} 1 & \text{if } i = j \\ 0 & \text{if } i \neq j \end{cases}$

• $UU^T = U^TU = I_n$
• $\lVert Ux \rVert_2 = \lVert x \rVert_2, \quad \forall x \in \mathbb{R}^n$
• Orthonormal: A set of vectors is orthonormal if they are all unit vectors and orthogonal to each other.
• Rotation: $R(\theta) = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$
• Reflection: $R = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}$