Skip to content

What is CLIP?

  • Model developed by OpenAI in 2021
  • Learns relationships between images and natural language
  • Enables matching an image with the most relevant text description
  • Can classify images without task-specific retraining (Zero-Shot)

Motivation

  • Many computer vision models relied on manually labeled datasets such as ImageNet
  • These models learned only predefined fixed classes
  • New tasks often required expensive fine-tuning with labeled data
  • CLIP replaces manual labels with natural language supervision

CLIP’s Approach

  • Trained on 400 million image-text pairs collected from the internet
  • Uses natural language as supervision signal
  • Learns a shared representation space for text and images
  • Matching image-text pairs are pulled closer together in embedding space
  • Non-matching pairs are pushed apart

Architecture: Shared Embeddings

  • Two separate encoders are used:
    • Image encoder
    • Text encoder
  • Both project their inputs into the same latent embedding space
  • This allows direct comparison between images and text

Architecture: Encoders

Image Encoder

  • ResNet variants
  • Vision Transformers (ViT)

Text Encoder

  • Transformer-based models

In the original CLIP paper

  • Modified ResNet-50
  • Vision Transformer
  • GPT-2 style text transformer

Architecture: Contrastive Pre-training

  • A mini-batch contains matching image-text pairs
  • Each image embedding is compared with every text embedding in the batch
  • Similarities are computed efficiently using matrix multiplication: S = I*T.T

Where:

  • \(I\) = image embeddings
  • \(T\) = text embeddings
  • \(S\) = similarity matrix

Architecture: Similarity Matrix

  • Embeddings are normalized before comparison
  • Similarity becomes cosine similarity:
    • 1 = highly similar
    • 0 = unrelated / orthogonal
  • Cross-entropy loss is applied
  • Goal: maximize diagonal entries (correct pairs)

Use Cases

  • Zero-shot image classification
  • Image search / retrieval
  • Multimodal embeddings
  • Text-to-image systems
  • Stable Diffusion guidance and alignment

Own Implementation: Chess Thesis Example

Goal

  • Use CLIP-style contrastive learning for image-to-image similarity
  • Detect whether two chessboard tile images contain the same piece
  • Robust against:
    • Lighting changes
    • Noise
    • Small geometric distortions

Data

  • Synthetic chessboard data generated in Blender
  • Each tile extracted as a 3x3 patch with surrounding context
  • Around 70,000 samples
  • 13 classes:
    • Empty square
    • 6 white pieces
    • 6 black pieces

Encoder

  • ResNet18 used as image encoder
  • Final classification layer replaced by projection head
  • Output is a compact embedding vector

Training

  • Batch of paired images
  • Encode both inputs into latent vectors z_1, z_2
  • Normalize vectors
  • Compute similarity matrix
  • Apply symmetric cross-entropy loss

Results

  • Clear clustering of chess piece classes in embedding space
  • PCA visualization shows separable groups
  • Useful as fast pre-filter (gate model) before more expensive classification

Key Takeaway

CLIP demonstrates that large-scale contrastive learning can replace classical supervised labeling pipelines. The same principle can also be transferred to specialized domains such as chessboard recognition.

Sources

  • Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I. 2021, 'Learning Transferable Visual Models From Natural Language Supervision', arXiv preprint arXiv:2103.00020.
  • https://openai.com/index/clip/
  • https://lilianweng.github.io/posts/2021-05-31-contrastive/
  • https://medium.com/rectlabs/clip-contrastive-language-image-pre-training-dce66ae18fe1 (Image source)
  • Heatmap of cosine similarity & PCA 3D plot generated on my own