Emergent Locomotion Patterns of a Snake Robot through Reinforcement Learning

Authors: Yuya Shimizu¹, Yongdong Wang^{1,2}, So Shimooka¹, Tetsushi Kamegawa¹
¹ Graduate School of Environmental, Life, Natural Science and Technology, Okayama University
² Department of Precision Engineering, The University of Tokyo
Project Page: https://yuya-0411.github.io/project-crawl/
Year: 2026

This work was supported by OU-SPRING, JSPS KAKENHI Grant Number 26K07423, and JSPS KAKENHI Grant Number 26K21350.


Abstract

Traditional gait design for snake robots has long relied on geometric models inspired by biological snakes. However, due to the inherent morphological discrepancies between biological snakes and robotic systems, these conventional models are not always optimal for snake robots.

In this study, we trained a 10-DoF snake robot on random step fields using reinforcement learning. The learned policy autonomously emerged dynamic gaits capable of traversing unstructured terrains where conventional sine-wave-based locomotion typically fails. The robot effectively utilized environmental protrusions as pivot points for propulsion, three-dimensionally transforming its body to overcome obstacles.

Although these gaits deviate from predefined periodic motions, they exhibit characteristics approximating biological sidewinding as a direct result of physical constraints. Comparative experiments with geometric models confirm that the learned policy offers superior robustness and adaptability in rugged terrains.


Robot Model

  • Structure: 11 links connected by 10 alternating yaw/pitch joints (orthogonally arranged)
  • Length: 9 central units × 0.16 m + 2 terminal units × 0.08 m = 1.6 m total
  • Sensors: Each link is equipped with a tactile sensor for contact-aware state feedback

Method

Observation & Action Space

The policy takes a 56-dimensional observation as input and outputs 10 joint position targets.

Category Item Dimension
Observation Joint positions 10
Joint velocities 10
Per-link contact forces (11 links × 3 axes, in local frame) 33
Global orientation (roll, pitch, yaw) 3
Total 56
Action Joint position targets 10

Per-link contact forces are the key embodiment-sensing component — each link's resultant contact force is expressed in its local coordinate frame, capturing normal and tangential friction. Joint targets are scaled as: q_target = (π/2) × a, where a ∈ [−1, 1].

Reward Function

The total reward combines six terms:

Term Weight Purpose
Forward velocity 5.0 × exp(−(0.5 − v_y)² / 0.25) Encourage target speed of 0.5 m/s
Lateral deviation −0.5 × |x_com| Penalize drifting sideways
Trajectory correction +1.0 × max(0, −x_com × v_x) Self-correct when off-track
Ground contact +0.2 × N_contact Reward multi-link ground contact
Head height −10.0 if z_head > 0.2 m Enforce low center-of-gravity posture
Action smoothness −0.005 × Σ(Δa_i)² Penalize large joint movements

Training Setup

Parameter Value
Algorithm PPO (RSL-RL)
Simulator NVIDIA Isaac Lab (GPU-accelerated PhysX, 50 Hz)
Parallel environments 4,000 (10 × 20 terrain field, 20 agents/area)
Network MLP [256, 256], ELU activation
Learning rate 2.5 × 10⁻⁵
Discount factor (γ) 0.99
GAE lambda (λ) 0.95

Training Terrains (4 types)

Type Description Parameter Range
Pyramidal Stairs & Pits Multi-level elevation changes Step height 0.05–0.23 m
Random Grid (Boxes) Discontinuous rigid obstacles Grid height 0.05–0.20 m
Random Rough Surface Stochastic height-field noise Noise range 0.02–0.10 m
Pyramidal Slopes Continuous inclined surfaces Slope 0.0–0.4 rad

Checkpoint

model_3000.pt — PPO policy weights at 3,000 training iterations.

import torch

checkpoint = torch.load("model_3000.pt", map_location="cpu")
print(checkpoint.keys())

For full simulation and evaluation setup, see the project page.


Key Findings

  • The policy emerged a gait approximating biological sidewinding purely from physical constraints — no gait template was provided.
  • The robot exploits terrain protrusions as pivot points for propulsion, reconfiguring its body in 3D to overcome obstacles.
  • Against the geometric sidewinding baseline (amplitude 1.2 rad, frequency 0.50 Hz), the learned policy shows superior robustness on unstructured terrain, avoiding the "stacking phenomenon" caused by the open-loop geometric model.
  • On flat ground, the learned gait achieves reliable forward locomotion driven by the full 56-D observation, with independent per-joint phase control rather than a prescribed waveform.

Citation

@inproceedings{shimizu2026emergent,
  title={Emergent Locomotion Patterns of a Snake Robot through Reinforcement Learning},
  author={Shimizu, Yuya and Wang, Yongdong and Shimooka, So and Kamegawa, Tetsushi},
  year={2026}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading