Encord, a company that builds data tooling for AI training, is experimenting with brain wave sensors to generate training data for robots. Andrew Ceja, a pilot at Encord, is using a headset with sensors to measure his brain waves as he disassembles a block tower. This initiative is part of Encord's effort to address the scarcity of real-world physical training data for humanoid and warehouse robotics. The brain wave headset was developed by Zander Labs, a German neuroscience startup, which believes measuring brain activity can provide insights into mental states like error and intent. Encord is currently running a trial to create a brain wave-tagged dataset, which it plans to test with customer robotics models before scaling up. Lucas Gehrke, a Zander neuroscientist, explained that brain activity during tasks can help model builders determine when to deploy their highest-effort models. Vineeth Velmurugan, Encord’s head of robot learning, described this effort as the 'bleeding edge' of solving the robotics data bottleneck. Velmurugan, a veteran of OpenAI’s robot lab and Berkshire Grey, joined Encord to build its internal data-creation team. Encord was founded to help companies building machine-vision applications annotate data and evaluate models.
As customers began applying end-to-end learning to robotic manipulation tasks, executives realized they would have to produce training data themselves. Velmurugan noted that the data simply does not exist, and the bet that generative AI can do for robots what it has done for chatbots keeps running into this same wall. LLMs were built on the text of the entire internet, but finding similar raw materials to teach neural networks about physical manipulation is challenging. Self-driving car companies collect this data themselves, but it is hard to scale. Velmurugan estimates that a dataset five times the size of YouTube’s video corpus would be needed to break through this barrier. This scale helps explain why data-generation has become a business and not just a research problem. Feed your egocentric data needs Companies building robot brains are now turning to two main sources: 'Egocentric' video collected by workers wearing cameras and data from robots operated remotely. Encord does both, drawing egocentric data from several factories around the globe and using its San Leandro facility to experiment with new modalities, like brain waves, or collect data sets around specific skills for fine-tuning. When TechCrunch visited, pilots were using leader-follower rigs — paired robotic arms, one controlled directly by a human operator and one that mimics its movements — to create data about tasks like pouring coffee from a pot into mugs and stacking poker chips.
Velmurugan said every humanoid company has asked for these pieces. Storage racks held cartons of fake flowers in vases, books, plastic vegetables, kitty litter trays, and scoops, bags, and bundles of wires, the stock in trade for training manipulators for household tasks. At one of these stations, another pilot, Sofia Infante, maneuvers robotic arms to plug and unplug ethernet cables from the back of a server — the kind of work data center operators would love to be automated, if only robots could manipulate them with the required precision. Taking a spin behind the controls, I was able to see why that’s still out of reach: Pincers are far less dextrous than human fingers and lack the degrees of freedom we take for granted in our arms. Another new data modality that Encord is developing uses a set of sensors strapped to the forearm to detect electrical signals in muscles. Video taken of human hands manipulating objects typically doesn’t capture the entire hand, but Velmurugan hopes to build a 3D depiction of where the hand is at any time based on the arm sensors, creating a more robust understanding for models. Encord’s data sets are annotated with physical descriptions of what each video contains — 'right hand tightens bolt' — to aid LLM-based models in understanding what is happening. Velmurugan estimates this kind of dense annotation is worth 100 times as much as 'junky ego data' for training specific tasks, and it only costs 20 times more to produce, which is a good trade, on paper. But '20 times more' is still real money, and that’s the catch: scraping text off the internet, the way LLM makers built their models by pulling from Stack Overflow and the rest of the web, cost frontier labs next to nothing.
Generating physical training data does not, and that’s the limit of the physical-AI-as-LLM comparison. This kind of data has to be manufactured, not just collected, and that changes the economics of building these models. Velmurugan says that progress is being made — with Encord’s visibility into programs across the industry, he’s able to see start-ups and frontier labs alike figure out what works and what doesn’t to improve physical AI models. That vantage point — sitting between many robotics companies at once — is also part of Encord’s pitch. It can spot which data techniques are gaining traction industry-wide before any single customer can. That will keep the dozen or so pilots at Encord’s facility busy. Both Infante and Ceja are part of a burgeoning workforce developing the building blocks for neural networks; they previously worked at Scale, another AI data annotation firm, before joining Encord. Ceja had worked at a waste management company where his interest in technology found him in charge of keeping a robotic trash sorter in good working order. Now, as the Jenga tower topples, he says he enjoys the challenge of solving training tasks for robots — 'It’s something new every day!'
