Benchmarks/Models
Predicting embeddings vs tokens with less data
Does predicting embeddings degrade more gracefully than predicting tokens?
Short answer: In a small text stand-in, cutting training data 10x made the token-space task collapse (micro-F1 0.230 to 0.001), while the embedding-space task degraded far less. Absolute scores are low, so read it as a direction, not a result.
- task A MRR
- task B micro-F1
What the numbers say
- An untrained baseline beat the trained embedding predictor on two of three retrieval metrics.
- This is a linear stand-in on text, not a real JEPA model.
Results (task A = embedding-space, task B = token-space)
| 10% (204 pairs) | 0.011 | 0.026 | 0.025 | 0.001 |
| 25% (510 pairs) | 0.012 | 0.051 | 0.039 | 0.105 |
| 50% (1,020 pairs) | 0.018 | 0.061 | 0.049 | 0.120 |
| 100% (2,041 pairs) | 0.020 | 0.083 | 0.062 | 0.230 |
Setup
Analysis post. Own experiment: Ridge regression and multi-label LogisticRegression (scikit-learn 1.9.1), Python 3.12.14, on 2,041 train / 1,187 test context-to-next-chunk pairs built from the FastAPI English docs corpus (151 files, split by file). This is a small text-only stand-in inspired by the same idea VL-JEPA applies to vision-language -- predict a target's embedding instead of its tokens -- built because no JEPA model was run. Not tested: I-JEPA, V-JEPA, V-JEPA 2, V-JEPA 2.1, VL-JEPA, LeJEPA or LeWorldModel. No GPU and no torch/timm/decord were available in this environment, and V-JEPA 2's checkpoints alone run 300MB-8GB, so no official JEPA code or weights were downloaded or run here.