Training AI Models With Limited Data

We trained a humanoid with 22-DoF dexterous hands to assemble model cars, operate syringes, sort poker cards, fold/roll shirts, all learned primarily from 20,000+ hours of egocentric human video with no robot in the loop. Humans are the most scalable embodiment on the planet. We discovered a near-perfect log-linear scaling law (R² = 0.998) between human video volume and action prediction loss, and this loss directly predicts real-robot success rate. Humanoid robots will be the end game, because they are the practical form factor with minimal embodiment gap from humans. Call it the Bitter Lesson of robot hardware: the kinematic similarity lets us simply retarget human finger motion onto dexterous robot hand joints. No learned embeddings, no fancy transfer algorithms needed. Relative wrist motion + retargeted 22-DoF finger actions serve as a unified action space that carries through from pre-training to robot execution. Our recipe is called "EgoScale": - Pre-train GR00T N1.5 on 20K hours of human video, mid-train with only 4 hours (!) of robot play data with Sharpa hands. 54% gains over training from scratch across 5 highly dexterous tasks. - Most surprising result: a *single* teleop demo is sufficient to learn a never-before-seen task. Our recipe enables extreme data efficiency. - Although we pre-train in 22-DoF hand joint space, the policy transfers to a Unitree G1 with 7-DoF tri-finger hands. 30%+ gains over training on G1 data alone. The scalable path to robot dexterity was never more robots. It was always us. - Website: https://lnkd.in/gxzgeP-2 - Paper: https://lnkd.in/g7PJdz_8

If you're working on AI projects with limited training data, building domain-specific AI applications, or struggling with the economics of data labeling, you should know about this new approach from the DeepSeek team. Reinforcement Fine-Tuning (RFT) is a new technique for fine-tuning large language models, cutting the required labeled data from thousands to just tens of examples. Traditional supervised fine-tuning (SFT) approaches have always been hampered by their dependence on vast amounts of labeled data. RFT takes a fundamentally different approach by utilizing a reward function to evaluate response correctness, enabling the model to learn more effectively than through simple mimicry of examples. The same technique that was used to develop DeepSeek-R1. This method proves particularly powerful in three key scenarios: (1) When no labeled data exists but correctness can be verified - such as code transpilation where outputs can be automatically tested. (2) When only limited labeled examples are available - fewer than 100 examples, where traditional methods typically overfit. (3) For tasks that benefit from chain-of-thought reasoning - where step-by-step logical thinking significantly improves results. A well-written post from Predibase here (they also added support for RFT on their platform recently!) https://lnkd.in/gHBdW5De P.S. Predibase just released an open-source model that outperforms OpenAI o1 by 67% for PyTorch-to-Triton transpilation tasks, enabling more efficient and intelligent AI models (link in comments).

Massachusetts Institute of Technology researchers just dropped something wild; a system that lets robots learn how to control themselves just by watching their own movements with a camera. No fancy sensors. No hand-coded models. Just vision. Think about that for a second. Right now, most robots rely on precise digital models to function - like a blueprint telling them exactly how their joints should bend, how much force to apply, etc. But what if the robot could just... figure it out by experimenting, like a baby flailing its arms until it learns to grab things? That’s what Neural Jacobian Fields (NJF) does. It lets a robot wiggle around randomly, observe itself through a camera, and build its own internal "sense" of how its body responds to commands. The implications? 1) Cheaper, more adaptable robots - No need for expensive embedded sensors or rigid designs. 2) Soft robotics gets real - Ever tried to model a squishy, deformable robot? It’s a nightmare. Now, they can just learn their own physics. 3) Robots that teach themselves - instead of painstakingly programming every movement, we could just show them what to do and let them work out the "how." The demo videos are mind-blowing; a pneumatic hand with zero sensors learning to pinch objects, a 3D-printed arm scribbling with a pencil, all controlled purely by vision. But here’s the kicker: What if this is how all robots learn in the future? No more pre-loaded models. Just point a camera, let them experiment, and they’ll develop their own "muscle memory." Sure, there are still limitations (like needing multiple cameras for training), but the direction is huge. This could finally make robotics flexible enough for messy, real-world tasks - agriculture, construction, even disaster response. #AI #MachineLearning #Innovation #ArtificialIntelligence #SoftRobotics #ComputerVision #Industry40 #DisruptiveTech #MIT #Engineering #MITCSAIL #RoboticsResearch #MachineLearning #DeepLearning

You cannot train AI on reality alone anymore. There is not enough of it. Jensen Huang explains why NVIDIA built Cosmos, an AI world model that generates synthetic training data grounded in physics. The problem is simple. Teaching physical AI like robotics requires vast amounts of diverse interaction data. Videos exist, but not nearly enough to capture the variety of situations robots will encounter. So NVIDIA transformed compute into data. Using synthetic data generation grounded by laws of physics, they can selectively generate training scenarios that would be impossible to capture otherwise. The example Huang shows is remarkable. A basic traffic simulator output gets fed into Cosmos. What emerges is physically plausible surround video that AI can learn from. This solves a fundamental limitation. You cannot train autonomous systems on every possible scenario by recording reality. There are not enough cameras or time. But you can simulate physics accurately enough that AI trained on synthetic data generalises to real environments. This applies beyond robotics. Any AI learning physical interactions, from manufacturing to logistics to infrastructure monitoring, faces the same data scarcity problem. Synthetic data generation grounded in physics laws is how you create training sets reality cannot provide. The organisations building AI for physical systems will either master synthetic data generation or get limited by whatever reality they can record. Watch the full presentation to hear Huang explain how Cosmos generates training data for physical AI. What physical AI application needs synthetic data because reality cannot provide enough examples? #AI #SyntheticData #Robotics #NVIDIA #MachineLearning

Thank you to the University of Toronto Machine Intelligence Student Team for inviting me to present a keynote on augmenting human-labeled datasets using Large Language Models (LLMs). Human-labeled data is crucial for testing, tuning, customizing, and validating LLMs in organizations. This is because human labeled data provides the ground truth for developing trustworthy #GenerativeAI applications and #AgenticAI systems. Yet acquiring sufficient human labeled data is often a bottleneck in many organizations. Subject matter experts and domain specialists typically have limited time for labeling tasks due to competing professional demands, making large-scale manual labeling difficult to sustain. My talk focused on how LLMs can be used not to substitute human labels, but to systematically augment them—extending the utility of existing human labeled data and improving model robustness without proportionally increasing manual labeling effort. I described practical methods for implementing two augmentation techniques with strong empirical grounding: • Negative Reinforcement with Counterfactual Examples – This technique involves analyzing labeled examples to generate counterfactual examples—outputs that are intentionally incorrect or undesirable—and using them to teach the model about what not to generate. By guiding the model using these negative samples, the model learns sharper decision boundaries, increasing robustness against hallucinations and confabulations. • Contrastive Learning with Controlled Perturbations – This technique creates diverse, label-preserving variants of human-labeled examples by introducing controlled modifications to the prompts and/or completions. These perturbations maintain core semantic meaning while varying surface-level features such as syntax, phrasing, or structure, encouraging the model to generalize beyond shallow lexical or syntactic cues. These techniques have been shown to drive measurable improvements in model behavior: • Lower Perplexity → More predictable completions and improved alignment with ground-truth targets. • Reduced Token Entropy → More focused and efficient completions, reducing inference complexity. • Higher Self-Consistency → More stable completions across repeated generations of the same prompt—a key requirement for dependable downstream use. These are not theoretical constructs—they are practical techniques for overcoming constraints in human-labeled data availability and scaling of #LLM applications with greater efficiency and rigor. Appreciate the University of Toronto Machine Intelligence Student Team (UTMIST) for a well-curated conference, and the UofT AI group for their initiatives in the space. Grateful to my research partner, Olga, for her contributions in collaboratively developing content for this presentation. Kudos to my PwC Canada teammates including Michelle B, Annie, Chris M, Michelle G, Chris D, Brenda, Bahar, Danielle, and Abhinav for their partnership on our PwC #AI portfolio.

If you are building Physical AI, your workflow just got a lot shorter, smarter. NVIDIA dropped a big collection of open-source agent skills for Physical AI last week, and I want to break down why this matters if you're the one actually doing the work. Here's the problem every Physical AI developer knows too well: the pipeline is brutal. Generate synthetic data, set up simulation, configure sensors, train, evaluate, deploy — and most of it is boilerplate, glue code, and tribal knowledge buried in someone's head. The hard part was never the idea. It was the 200 lines of setup before you could test the idea. Skills change that. Think of them as repeatable, agent-executable instructions. Instead of hand-writing the setup, you tell your coding agent — Claude Code, Cursor, OpenAI Codex, whatever you use — what you want, and the skill tells the agent which NVIDIA tools to call, what outputs to produce, and how to validate the result. The knowledge that used to live in a senior engineer's head is now portable. A few that stand out: → ovrtx skills (renderer creation, USD loading, stepping & rendering) — spin up physically accurate camera/LiDAR/radar simulation programmatically, straight into CUDA memory at thousands of frames/sec for RL loops. Check out this skill in action. Its super easy. https://lnkd.in/gF6d6cYh → Realtime Viewer — a browser-based OpenUSD viewer that streams RTX-rendered output straight to the web. No heavy desktop client, no install friction. You — or your customer, or a teammate three time zones away — can see the live simulation state in a browser tab. This is the "see, share and validate fast" half. Here is a live walkthrough https://lnkd.in/gSYb9kjh → Neural Reconstruction & Video Augmentation — turn real-world fleet captures into simulation environments. → Defect Image Generation — synthetic defect data for inspection models. And this isn't theoretical. Pegatron cut model training and deployment time by 67%. Inventec dropped defect data collection effort by 30%. Delta improved detection rate by 17%. Those are real numbers from teams shipping today. What this really means for a developer: less time fighting setup, more time on the actual problem. The barrier to standing up a digital twin or a sim loop went from "spin up a heavy desktop project" to "ask your agent." That's a different speed of iteration. 110 skills, 24 products, all open source — on GitHub (NVIDIA/skills) and skills.sh, usable with any coding agent. Github: https://lnkd.in/gikb6-KA Some are runnable instantly on NVIDIA Brev as preconfigured Launchables, so you can try before you commit a single line. If you're building robots, AVs, vision AI, or industrial twins, this is worth an afternoon of your time. What would you automate first? Curious where other builders see the biggest time savings. #PhysicalAI #Omniverse #OpenUSD #Robotics #DigitalTwins #NVIDIA Jessica Ji Edmar MendizabalMara Mahoney Kristen Rumley Meaghan Fitzpatrick

NVIDIA’s Physical AI Data Factory Blueprint is Designed to Improve Robot Training Data One of the biggest hurdles standing between physical AI and its “ChatGPT moment” is a lack of quality data. A big part of the reason LLMs have been such a massive – and often surprising – success is the fact that humans have essentially been creating training data for 100,000 years or so. The same can’t be said for the input required to train robots. NVIDIA is among the companies working to address the gap, and this morning at GTC the company announced Physical AI Data Factory Blueprint, an open reference architecture designed to improve how both real-world and simulated data is gathered, shaped, and assessed. The company has already recruited some big names from across autonomous driving and robotics, including FieldAI, Hexagon AB Robotics, Linker Vision, Milestone Systems, Skild AI, Uber, and Teradyne Robotics. The platform is host to number of processes designed to do right by the real and synthetic robot data. There’s Cosmos Curator, which processes and annotates datasets, Cosmos Tranffer, which is designed to address edge cases and long tail scenarios, and Cosmos Evaluator, which, you know, evaluates data. “Physical AI is the next frontier of the AI revolution, where success depends on the ability to generate massive amounts of data,” says Omniverse VP, Rev Lebaredian. “Together with cloud leaders, we’re providing a new kind of agentic engine that transforms compute into the high-quality data required to bring the next generation of autonomous systems and robots to life. In this new era, compute is data.” #nvidia #gtc #nvidiagtc #robotics #physicalai

How do we build AI for science? Augment with AI or replace with AI? Popular prescription is to augment AI into existing workflows rather than replace them, e.g., keep the approximate numerical solver for simulations, and use AI only to correct its errors in every time step. The other extreme is to completely discard the existing workflow and replace it fully with AI. We have seen this approach win in areas like weather forecasting. Such end-to-end AI is significantly better for speed: 1000-million x faster. In our latest paper, we show end-to-end learning also wins in data efficiency, which is counterintuitive. Where do these savings come from? The former approach that augments AI relies only on fully accurate training data that is expensive. But end-to-end learning can use both approximate and accurate training data, if the model can learn how to mix them correctly. In many physical systems, coarse-grid numerical solvers yield approximate data while fine-grid solvers fully resolve the scales and yield exact answers. It turns out that Neural Operators offer a perfect solution when such multi-fidelity and multi-resolution data is available, and can learn with high data efficiency requiring only a small amount of fully resolved data, since it can also utilize approximate training data. In contrast, the standard approach of augmenting AI to a coarse-grid numerical solver (closure model) can only train on fully-resolved simulations, making it very expensive and hard to train. Our results are applicable in multi-scale chaotic systems that have traditionally required running long simulations at high resolution such as climate change or plasma in nuclear fusion and astrophysics. Now you can replace expensive simulation fully with AI (Neural Operators), and also train it without requiring such simulations in large numbers for training in many scenarios.

In the last blog I talked about the importance of Classical ML/DL.  This post focuses on Finetuning image models. (Part 4 of #ArchitectingAI) Pre-trained image models are powerful. Fine-tuning them correctly is the real skill. Transfer learning lets you start with a backbone — ResNet, MobileNet, EfficientNet — already trained on millions of images, and adapt it to your problem. Less data, faster training, better results. I applied this to classify surface defects in industrial steel project. It works well and rewards a meticulous approach. Each wrong decision compounds! 5 key things that actually matter: 1. 𝐓𝐡𝐞 𝐟𝐫𝐨𝐳𝐞𝐧 𝐥𝐚𝐲𝐞𝐫𝐬 𝐚𝐫𝐞 𝐚𝐥𝐫𝐞𝐚𝐝𝐲 𝐝𝐨𝐢𝐧𝐠 𝐦𝐨𝐬𝐭 𝐨𝐟 𝐭𝐡𝐞 𝐰𝐨𝐫𝐤. 𝑭𝒊𝒏𝒆-𝒕𝒖𝒏𝒊𝒏𝒈 𝒊𝒔 𝒓𝒆𝒇𝒊𝒏𝒆𝒎𝒆𝒏𝒕, 𝒏𝒐𝒕 𝒓𝒆𝒍𝒆𝒂𝒓𝒏𝒊𝒏𝒈. Early layers in a pre-trained model capture universal patterns — edges, textures, shapes — that transfer across domains. Later layers are task-specific. Freezing the backbone and training only a new classification head gets you most of the way there. Unfreezing the whole network is rarely worth it. 2. 𝐃𝐨𝐦𝐚𝐢𝐧 𝐠𝐚𝐩 𝐝𝐞𝐭𝐞𝐫𝐦𝐢𝐧𝐞𝐬 𝐲𝐨𝐮𝐫 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐬𝐭𝐫𝐚𝐭𝐞𝐠𝐲. 𝑻𝒉𝒆 𝒇𝒖𝒓𝒕𝒉𝒆𝒓 𝒚𝒐𝒖𝒓 𝒅𝒂𝒕𝒂 𝒇𝒓𝒐𝒎 𝑰𝒎𝒂𝒈𝒆𝑵𝒆𝒕, 𝒕𝒉𝒆 𝒎𝒐𝒓𝒆 𝒚𝒐𝒖 𝒏𝒆𝒆𝒅 𝒕𝒐 𝒖𝒏𝒇𝒓𝒆𝒆𝒛𝒆. Natural images transfer easily. Industrial textures, medical scans, satellite imagery — larger domain gap, less directly applicable features. Know your gap before deciding how many layers to unfreeze. More gap = more fine-tuning needed. 3. 𝐃𝐚𝐭𝐚 𝐚𝐮𝐠𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐢𝐬 𝐧𝐨𝐭 𝐨𝐩𝐭𝐢𝐨𝐧𝐚𝐥 𝐰𝐢𝐭𝐡 𝐬𝐦𝐚𝐥𝐥 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬. 𝑻𝒉𝒆 𝒎𝒐𝒅𝒆𝒍 𝒘𝒊𝒍𝒍 𝒎𝒆𝒎𝒐𝒓𝒊𝒔𝒆 𝒚𝒐𝒖𝒓 𝒅𝒂𝒕𝒂 𝒊𝒇 𝒚𝒐𝒖 𝒈𝒊𝒗𝒆 𝒊𝒕 𝒕𝒉𝒆 𝒄𝒉𝒂𝒏𝒄𝒆. Augmentation creates diversity the model hasn't seen — rotations, skews, flips, contrast shifts, brightness changes, blur. It forces generalisation over memorisation, critical when domain-specific data is limited. Apply it before model-specific preprocessing — wrong order means augmenting corrupted inputs, silently. 4. 𝐅𝐫𝐞𝐞𝐳𝐞 𝐟𝐢𝐫𝐬𝐭. 𝐅𝐢𝐧𝐞-𝐭𝐮𝐧𝐞 𝐬𝐞𝐜𝐨𝐧𝐝. 𝑨𝒍𝒘𝒂𝒚𝒔 𝒊𝒏 𝒕𝒉𝒂𝒕 𝒐𝒓𝒅𝒆𝒓. A randomly initialised head will undo what you borrowed. New classifier weights start random. Early gradients are large and noisy — if the backbone is already unfrozen, they overwrite representations learned from millions of images. Train the head first, stabilise it, then unfreeze selectively. 5. 𝐓𝐫𝐞𝐚𝐭 𝐩𝐫𝐞-𝐭𝐫𝐚𝐢𝐧𝐞𝐝 𝐰𝐞𝐢𝐠𝐡𝐭𝐬 𝐚𝐬 𝐟𝐫𝐚𝐠𝐢𝐥𝐞 𝐝𝐮𝐫𝐢𝐧𝐠 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠. 𝑼𝒔𝒆 𝒂 𝒎𝒖𝒄𝒉 𝒔𝒎𝒂𝒍𝒍𝒆𝒓 𝒍𝒆𝒂𝒓𝒏𝒊𝒏𝒈 𝒓𝒂𝒕𝒆. A standard learning rate undoes the representations you were trying to preserve. Unfreeze only the last few layers — more layers at a higher learning rate tend to overfit or underfit fast. Transfer learning is powerful because it builds on what's already been learned. Know your domain gap. Know what to freeze. Know when to fine-tune. Do it with care.