PRAXIS: Physical Intelligence for Robots That Act in the World

1The manipulation problem

Modern AI can describe a photograph of a cluttered bin in fluent prose. It still cannot reliably reach in and pick one specific object out. The gap between recognizing the world and acting in it, physical intelligence, is where robotics is hard and where value is unlocked.

The problem has four distinct faces. First, seeing to act: perception for manipulation is not classification, it must recover metric 6-DoF pose and where an object can actually be grasped, under occlusion, reflection, and clutter. Second, contact: the moment a gripper touches an object, position control gives way to physics, friction, deformation, slip, and force must be regulated, not just pose. Third, generalization: a useful robot handles objects and tasks it was never explicitly programmed for, which demands learning, not scripting. Fourth, coordination: a facility of robots must be scheduled and deconflicted, a combinatorial problem that grows with fleet size.

These are different problems, geometric estimation, real-time force control, machine learning, and combinatorial optimization, and PRAXIS pairs each with the appropriate method rather than forcing one tool across all four.

2System overview

PRAXIS runs a loop from perception to fleet and back. Each robot perceives, grasps, and controls contact in hard real time; the facility layer schedules the fleet; every grasp attempt feeds the learning loop that sharpens the next one. The safety-critical control path is classical and local.

2.1On-robot plane and facility plane

PRAXIS separates an on-robot plane (perception, grasp execution, and force control, running at hard real-time rates on each robot) from a facility plane (fleet scheduling, deconfliction, and the learning pipeline). The on-robot plane is self-sufficient: if the network or a remote solver is unavailable, each robot completes its current task under local control and force limits. Nothing on the safety-critical control path depends on the network or a quantum backend.

3Perceive — 3D perception for manipulation

Perception for manipulation is not "what is this?" but "where is it, exactly, and where can I grasp it?" Perceive recovers metric 6-DoF pose and grasp affordance from RGB-D, under the occlusion and clutter of a real bin.

3.16-DoF pose estimation

Each object's pose is a rigid transform T ∈ SE(3) (a rotation and translation). PRAXIS estimates it by aligning a learned model of the object to the observed depth and appearance, minimizing a residual over the group of rigid motions:

where m_k are model points, o_k the matched observations, and ρ a robust kernel that tolerates outliers from clutter. A learned front end proposes correspondences and an initial pose; the optimization refines it and returns a calibrated confidence.

3.2Grasp affordance

Geometry alone does not say where a grasp will hold. A learned model predicts, for each candidate grasp g over the scene, the probability that executing it succeeds:

trained on millions of simulated and real grasp outcomes. The output is a dense affordance map that turns raw geometry into ranked, graspable regions.

3.3Scene reconstruction

Depth from one or more views is fused into a consistent occupancy model of the workspace, distinguishing free space, the target, and obstacles, so the motion planner in §4 has a clean model to plan collision-free approaches against.

4Manipulate — grasp synthesis and force control

A grasp is a hypothesis about physics. Manipulate ranks candidate grasps by a quality that predicts whether they will resist disturbance, then executes under force control so contact-rich tasks succeed without crushing the object.

4.1Grasp quality

Among candidate grasps, PRAXIS prefers those that can resist wrenches (forces and torques) from any direction. The classical grasp-quality metric is the radius of the largest wrench-space ball contained in the grasp's convex wrench set W, the worst-case disturbance the grasp can resist:

A learned model predicts this quality directly from perception, so ranking is fast, and the top grasp is validated against reachability and collision before execution.

4.2Force and impedance control

Once in contact, controlling position alone is brittle, a millimetre of misalignment becomes a large force. PRAXIS renders the end-effector as a programmable spring-damper (Cartesian impedance), commanding joint torques that regulate the contact interaction:

where J is the Jacobian, K_p, K_d the stiffness and damping, and g(q) gravity compensation. Making the robot compliant rather than stiff is what lets it insert, mate, and assemble, and why a target force envelope, not a position, defines success.

4.3Slip detection and regrasp

Tactile and visual feedback monitor for incipient slip; when detected, the controller increases grip within a safe limit or triggers a regrasp, before the object is dropped. Force is always bounded by a hard limit that no learned component can override.

5Learn — sim-to-real skill acquisition

A new task should not require months of engineering. Learn seeds a policy from a few demonstrations, refines it across massively-parallel simulation, and transfers it to hardware, so skills are acquired in hours.

5.1Learning from demonstration

A handful of teleoperated or kinesthetic demonstrations bootstrap a policy π_θ by behaviour cloning, matching the expert's actions:

This gives a competent starting policy with no reward engineering, the bottleneck that makes classical RL impractical for most tasks.

5.2Reinforcement learning in simulation

The policy is then refined in thousands of parallel simulated environments to maximize expected return:

with a modern policy-gradient method. Simulation is millions of times cheaper and safer than real trials, so the policy can practise a lifetime of attempts overnight.

5.3Sim-to-real transfer

The reality gap is closed by domain randomization: physical parameters ξ (mass, friction, latency, lighting) are randomized so the policy is optimized to be robust across the whole distribution rather than to one idealized simulator:

A policy that works across a wide band of simulated physics tends to work on the real robot, with minimal on-hardware tuning.

6Orchestrate — fleet scheduling

A facility of robots must decide who does what, in what order, without arms colliding or work stalling. This scheduling problem is combinatorial and grows explosively with fleet size, and it is where quantum and hybrid solvers earn their place.

6.1Job-shop scheduling as a QUBO

Assigning and sequencing tasks across robots to minimize completion time (makespan) is a job-shop problem. Its decisions are encoded as binary variables in a QUBO, with precedence and no-collision constraints as quadratic penalties:

and solved with QAOA, a depth-p circuit alternating a cost and a mixing unitary whose angles a classical optimizer tunes:

A classical relaxation warm-starts the solve, and the CVaR of the sampled makespans is minimized to concentrate on the best schedules. Small cells are handled by a classical solver; the quantum path engages as the fleet grows.

6.2Collision-free coordination

Robots that share a workspace are deconflicted in space and time, so two arms never claim the same volume and mobile robots never contend for the same aisle. Schedules that pass optimization are checked against a physics-aware coordinator before dispatch.

7Hybrid orchestration

The "quantum only where it pays" principle is enforced in code by a router that chooses, per job, between classical and quantum solvers and between on-robot and facility execution, always guaranteeing a usable result in time.

7.1The router

Every job carries a class, a size, and a deadline. Perception, grasp execution, and force control are always classical and on-robot. Scheduling jobs are sized: small cells go to a classical solver; large fleets to the quantum/hybrid solver when a backend and time budget allow; and when a deadline looms, the router returns the best classical result. The router is free to choose classical, and for a single cell it does.

Manipulation / scheduling router (simplified)def route(job):
    if job.kind in ("perceive", "grasp", "force_ctrl"):
        return OnRobot(job.kind)          # hard real-time, never remote
    if job.kind == "schedule":
        n = job.robots * job.tasks
        if n <= CLASSICAL_LIMIT:
            return Classical(solver="cp_sat")
        if backend.available() and job.deadline > QPU_MIN:
            return Hybrid(qaoa_depth=6, warm_start=True)
        return Classical(solver="heuristic")   # graceful fallback

7.2Backends and mitigation

One circuit interface runs across simulators and gate-model QPUs, so operators are not locked to a vendor and benefit as hardware improves. On real hardware PRAXIS applies readout-error calibration and zero-noise extrapolation, and records which backend and mitigations produced each schedule in the log.

8Deployment and integration

PRAXIS runs on the operator's robots and inside the operator's boundary, arm-agnostic on-robot, with the fleet layer at the edge or on-premises.

8.1On-robot and facility

The on-robot stack targets common edge accelerators and standard robot interfaces (for example ROS 2 and vendor motion APIs), so PRAXIS drives the operator's existing arms and grippers rather than requiring new hardware. The facility layer runs on-premises or at the edge; task data stays within the operator's boundary, and only abstracted scheduling instances are brokered to a quantum backend over an encrypted channel.

8.2Simulation and shadow deployment

Every skill is validated first in high-fidelity simulation, then in a hardware-in-the-loop cell that replays recorded tasks and stress cases (worst-case clutter, sensor noise, perturbed objects). New policies are shadow-deployed, proposing without commanding, until they earn authority on a task.

8.3Logging and audit

Every perception output, grasp, force trace, control intervention, and schedule, with the compute path that produced it, is logged for review. The system is auditable end to end, as operators and safety assessors require.

9Validation and benchmarks

We benchmark against strong baselines, tuned grasp planners and classical schedulers, and report where quantum does not yet win.

9.1Methodology

• Perception is scored by 6-DoF pose error against motion-capture ground truth and by grasp-affordance accuracy on held-out objects.
• Manipulation is scored by pick success on unseen objects, insertion success, and force-limit compliance.
• Scheduling is measured by makespan and feasibility at a fixed wall-clock budget.

The verdict column is the point: learning and force control carry perception and manipulation; quantum optimization earns its place as fleet scheduling grows, and on a single cell PRAXIS routes to the classical solver.

10Limitations and current maturity

This section is deliberately blunt. Manipulation is unforgiving, and credibility with operators depends on candor.

• NISQ reality. Quantum hardware is noisy and limited in scale. PRAXIS's quantum advantage on fleet scheduling is demonstrated in simulation and on small instances; classical solvers carry production scheduling today, and the router falls back automatically.
• Perception has failure modes. Transparent, reflective, and deformable objects remain hard for depth sensing; pose confidence drops and PRAXIS slows or re-perceives rather than grasping blind.
• Contact is still hard. Deformable and articulated objects, and tight-tolerance assembly, need real-robot fine-tuning; sim-to-real narrows the gap but does not erase it.
• Generalization has limits. A skill library covers a growing but finite space; genuinely novel tasks require new demonstrations, not zero-shot magic.
• Simulated figures. Every number here is illustrative of the target. Production performance depends on the arm, gripper, sensors, objects, and backend, and is established per platform and validated on the operator's own tasks.

11Responsible use and safety

Robots that share space with people are safety-critical. PRAXIS is deployed under operator agreements and within applicable machinery-safety standards (for example ISO 10218 and ISO/TS 15066 for collaborative operation). Safety is enforced by verifiable classical control with hard force and velocity limits that no learned component can override; the system defaults to a safe stop under uncertainty, and keeps human operators in supervisory authority. Partner task data and learned skills remain the partner's property and stay within their boundary; models are not shared across partners without consent. Full provenance is retained so every grasp, force trace, and schedule can be traced and reviewed.

12Roadmap

—References and further reading

1. Ferrari & Canny. Planning Optimal Grasps. ICRA 1992.
2. Mahler et al. Dex-Net 2.0: Deep Learning to Plan Robust Grasps. RSS 2017.
3. Hogan, N. Impedance Control: An Approach to Manipulation. J. Dyn. Sys. Meas. Control 107 (1985).
4. Khatib, O. A Unified Approach for Motion and Force Control of Robot Manipulators. IEEE J. Robotics & Automation 3 (1987).
5. Tobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS 2017.
6. OpenAI et al. Learning Dexterous In-Hand Manipulation. IJRR 39 (2020).
7. Levine et al. Learning Hand-Eye Coordination for Robotic Grasping (QT-Opt). IJRR 37 (2018).
8. Schulman et al. Proximal Policy Optimization Algorithms. arXiv:1707.06347.
9. Lucas, A. Ising Formulations of Many NP Problems. Frontiers in Physics 2 (2014).
10. Farhi, Goldstone, Gutmann. A Quantum Approximate Optimization Algorithm. arXiv:1411.4028.

This brief describes a system and design intent. All performance figures are illustrative and simulated for demonstration; production characteristics are established per platform and validated on the operator's own tasks. © Ace Hacker Research & Development Lab.