learning curve

lives round 0 survived 0:00 trained 0

You fly the white ship.

Arrows or WASD to turn and thrust, space to fire.

← → turn↑ thrustspace fireP pauseV policy vision

Hits / min (self-play)0

How often it hits in its own practice games.

rate = 60 · H / (N · Δt)

H: hits in self-play
N: decision steps
Δt: 1/15 s

TD loss0

How wrong its value estimates turn out to be a step later.

L = mean_B [ clip(δ, ±1)² ] δ = R + γ³ · Q′(s′, a*) − Q(s, a) a* = argmax_a Q(s′, a) R = r_t + γ r_t+1 + γ² r_t+2 plotted: L̄ ← L̄ + α (L − L̄), α = 0.01

δ: TD error, clipped to ±1
B: batch, 64
γ: discount, 0.97
Q, Q′: live net, target net
s, s′: state now, and three decisions on
r: reward from one decision
L̄: running mean of L, the plotted line

Mean max Q0

What it expects the best move available to be worth.

mean max Q = mean_B [ max_a Q(s, a) ] Q(s, a) = V(s) + A(s, a) − mean_a A(s, a)

V: state value
A: advantage
B: batch, 64
s, a: state, action

Epsilon0

How often it moves at random instead of following its policy.

ε = max(ε_min, 1 − (1 − ε_min) · min(1, n / 3000))

ε: share of decisions taken at random
ε_min: 0.06, floor
n: transitions

Reward per stepby term

What it is being paid for, term by term.

r = Σ_i w_i · f_i bar_i = w_i f_i / max_i |w_i f_i|

f_i: term i, per decision
w_hit: +2.0, paid to the step that fired
w_aim: +0.04 a step on the intercept
w_got: −1.5
w_fire: −0.04

Policy network