starla
ENES+ v0.3.1 AGPL-3.0 rust 2024

← starla · 04

04 · protocol

Referencia del protocolo.

How probes and controllers actually talk: lifecycle, command formats, result lines, upload envelopes, and the rigid little quirks the controller demands.

This document describes the RIPE Atlas probe-controller communication protocol as implemented by starla. It was reverse-engineered from the official C probe (busybox) and verified against live RIPE Atlas controllers.

1Connection lifecycle

Registration

On startup, the probe SSHes to a registration server (e.g. reg-05.prod.atlas.ripe.net:443) and runs INIT with probe identification on stdin:

stdinP_TO_R_INIT
P_TO_R_INIT TOKEN_SPECS fluffy 1000 5120 debian/13/x86_64/starla/0.4.0 REASON_FOR_REGISTRATION NEW

The server responds with a controller assignment:

stdoutresponse
OK CONTROLLER ctr-dub-sw01.atlas.prod.ripe.net 443 ssh-rsa AAAA... REREGISTER 3000 PROBE_ID 1015186

If the probe is not yet registered, the server responds with plain OK (no CONTROLLER line). The probe retries with backoff until approved at atlas.ripe.net/register/swprobe.

Controller INIT

After receiving the controller assignment, the probe connects to the controller via SSH and sends another INIT (no stdin data this time):

stdoutsession
OK SESSION_ID fa0b28e5f26291ad4a41ceecffc73457cbb4180291f53abdcda5667357440a9f REMOTE_PORT 2023

The SESSION_ID is used for telnet authentication and the result-upload footer. REMOTE_PORT is the port the controller listens on for the reverse tunnel.

KEEP session

The probe opens a long-lived SSH connection and runs KEEP. This channel blocks indefinitely and serves as a health monitor; if it closes, the connection is lost and the probe reconnects.

On this same SSH connection:

  • Reverse tunnel (tcpip_forward): controller connects to REMOTE_PORT to send measurement commands via telnet.
  • Direct-tcpip channels: probe opens channels to 127.0.0.1:8080 on the controller for result uploads.

2Telnet command protocol

Authentication

When the controller connects via the reverse tunnel, the probe sends:

telnetbanner + login
IAC DO ECHO IAC DO NAWS IAC WILL ECHO IAC WILL SGA Atlas probe, see http://atlas.ripe.net/ Probe 1015186 (hostname) login:

The controller sends C_TO_P_TEST_V1 as the login, then the SESSION_ID as the password. On success: OK\r\n\r\n. On failure: BAD_PASSWORD\r\n\r\n and disconnect.

CRONLINE — recurring

grammarcronline
CRONLINE <interval> <offset> <end_time> <spread_type> <spread> <command> [args...]
FieldTypeDescription
intervalu64Seconds between executions (0 = one-shot)
offsetu64Initial delay offset (ignored by starla)
end_timei64Unix timestamp to stop (0 = never)
spread_typestringUsually UNIFORM
spreadu32Random spread in seconds
commandstringMeasurement tool name
examplecronline ping
CRONLINE 240 269 1777902395 UNIFORM 3 evping -4 -c 3 -A "1001" -O /home/atlas/data/new/7 193.0.14.129

ONEOFF — one-shot

grammaroneoff
ONEOFF <path> <command> [args...]

User-triggered via the RIPE Atlas API. The path is ignored (it's where the C probe writes results). Internally converted to a CRONLINE with interval=0.

JSON

jsondirect command
{"type":"ping","msm_id":1001,"target":"8.8.8.8","af":4,"packets":3}

Host-telemetry commands

The controller also schedules small applets that report the probe's runtime environment rather than running measurements. starla emits matching RESULT JSON so the RIPE backend sees the same shape an embedded probe would.

CommandidWhat it reports
buddyinfo9001/proc/buddyinfo block counts, derived freemem KB, and uptime. The official applet self-reboots on low memory if LOWMEM_REBOOT is set; starla never reboots — the host's memory manager owns that decision.
rptaddrs9104 (from -A)inet-addresses, inet-routes, inet6-addresses, inet6-routes, and dns resolvers, derived from /proc/net/{route,if_inet6,ipv6_route} and /etc/resolv.conf. Cached: re-emits only when state changes or after 1 hour, matching the C applet.

Ignored commands

Four CRONLINE commands are accepted but produce no output. Each is parsed deliberately (not as Unknown) so the absence of a reply is intentional:

CommandWhat the embedded probe does with itWhy starla ignores it
CRONTABNames which on-disk cron directory the following CRONLINEs belong to.starla doesn't persist schedules to a filesystem cron layout; jobs live in memory keyed by msm_id.
httppostPOSTs queued result files to the RIPE backend over the SSH tunnel.starla uploads natively from starla-results::uploader; this CRONLINE is a redundant trigger.
condmv / dfrmRotates result files between data/new/ → data/out/ → data/storage/ and clears stale spool when disk is low.starla queues results in a bounded in-memory queue; no on-disk spool to manage.
conntrackReports kernel nf_conntrack table fullness (self-watchdog for the probe's tiny kernel).The applet isn't present in our vendored reference busybox tree at reference/probe-busybox/, so the canonical RESULT format is unknown. Held rather than guessed; revisit if a reference impl surfaces.

3Measurement commands

Ping — evping

grammar
evping [-4|-6] [-c count] [-s size] [-A msm_id] [-O output] [-I interval] <target>
FlagDescriptionDefault
-4/-6Address family4
-cPacket count3
-sPacket size (bytes)64
-AMeasurement IDrequired
jsonresult · PreFormatted array
RESULT { "id":"1001", "fw":5120, "mver": "2.6.4", "lts":10, "time":1700000000, "dst_name":"193.0.14.129", "af":4, "dst_addr":"193.0.14.129", "src_addr":"10.0.0.1", "proto":"ICMP", "ttl":56, "size":64, "result": [ { "rtt":10.500000 }, { "rtt":11.200000 }, { "rtt":10.800000 } ] }

Timeouts: { "x":"*" }. RTT has 6 decimals. ttl is omitted if all packets timed out. Requires CAP_NET_RAW.

Traceroute — evtraceroute

grammar
evtraceroute [-4|-6] [-I|-U|-T] [-f first_hop] [-m max_hops] [-p paris_id] [-S size] [-A msm_id] [-O output] <target>
FlagDescriptionDefault
-I/-U/-TICMP / UDP / TCPUDP
-fFirst hop1
-mMax hops32
-pParis traceroute ID0
-SPacket size40
jsonresult · FullLine
RESULT { "id":"5001", "fw":5120, "mver": "2.6.4", "lts":0, "time":1700000000, "endtime":1700000030, "dst_name":"192.112.36.4", "dst_addr":"192.112.36.4", "src_addr":"10.0.0.1", "proto":"UDP", "af": 4, "size":40, "paris_id":5, "result": [ { "hop":1, "result": [ { "from":"10.0.0.1", "ttl":255, "size":76, "rtt":3.723 } ] }, { "hop":2, "result": [ { "x":"*" }, { "x":"*" }, { "x":"*" } ] }, { "hop":13, "result": [ { "from":"192.112.36.4", "ttl":54, "rtt":31.981, "size":88 } ] } ] }

RTT 3 decimals. 3 probes per hop. Requires CAP_NET_RAW.

DNS — evtdig

grammar
evtdig [-4|-6|-a] [-t] [-R] [-d] [-n] [-e size] [--qbuf] [--resolv] [--soa NAME | --a NAME | --aaaa NAME | … | --type N --class N --query NAME | -h | -b | -i | -r] [--retry N] [--timeout ms] [--port N] [-A msm_id] [-O output] server

Parsed as eperd/evtdig.c (firmware 5130) parses it, with getopt_long: options in any order, long ones by a unique prefix. The name must end with a dot, and there must be exactly one server unless --resolv is given. A line the reference refuses (an unknown option, no final dot, no server) is refused.

FlagDescription
-4 / -6 / -aThe server's address family: IPv4 (default), IPv6, either
-tTCP (default UDP); the result adds qt
-h, -b, -i, -rhostname.bind, version.bind, id.server, version.server, CH TXT
--soa NAME, --a NAME, --aaaa NAME, …That type, class IN
--type N, --class N, --query NAMENumeric type and class (default TXT CH)
-RSet RD. evtdig never sets it otherwise, --resolv aside
-dThe DO bit
-nAn NSID option
-e NEDNS0 UDP size (512). An OPT record goes out only for -n, -d, -c, --cookies, -e above 512, or a non-zero --edns-version/--edns-flags/--edns-option
--qbufReport the query sent (base64) as qbuf
--resolvAsk each resolver in /etc/resolv.conf (at most 3), a second apart, with RD; one resultset entry each. Nobody else is asked
--retry N, --timeout msRepeats after a timeout (0); the UDP wait (5000). TCP has 5 s for the whole exchange
--ttlReport the TTL (hop limit) a UDP reply arrived with, as "ttl" before "proto": beside the addresses, not inside result where the result-format documentation (5010) puts it
--tls, --httpsNot supported: the line is refused

Template variables in query names

VariableExpansionPurpose
$r16 random lowercase hex digitsPrevent DNS caching
$pProbe ID (decimal), as registration assigned itPer-probe uniqueness
$tUnix timestamp (decimal)Per-query uniqueness

With --resolv the name is expanded afresh for each resolver. Any other $ fails the expansion, which is reported.

jsonresult · FullLine · abuf
RESULT { "id" : "11201" , "fw":5130, "mver": "2.6.4", "time" : 1791871219 , "lts" : 8 , "dst_addr" : "2001:db8:7769:6e00::20" , "dst_port" : "53" , "af" : 6 , "src_addr" : "2001:db8:7769:6e00::31" , "proto" : "UDP" ,"result" : { "rt" : 0.649,"size" : 53 , "abuf" : "sPqEAAABAAEAAAAAB3ZlcnNpb24EYmluZAAAEAADwAwAEAADAAAAAAALCmF0bGFzLXR3aW4=", "ID" : 45306 , "ANCOUNT" : 1 , "QDCOUNT" : 1 , "NSCOUNT" : 0 , "ARCOUNT" : 0, "answers" : [ {"TYPE" : "TXT" , "NAME" : "version.bind" , "RDATA" : [ "atlas-twin" ] } ] } }

The spacing is printReply()'s ("key" : value , ), kept as it is (wrapped here). Only the first two answers are decoded, and only TXT and SOA (as TYPE, NAME, TTL, MNAME, RNAME, SERIAL); the rest is in abuf. With --resolv the line carries "time", "lts" and a "resultset" whose entries add subid and submax.

On timeout: "proto" : "UDP" , "error" : { "timeout" : 5000}, and no result. Every error is such an object beside the addresses (per resultset entry with --resolv), never inside result.

Where firmware 5130 differs from the result-format documentation (version 5000), Starla writes what 5130 writes, which is what RIPE stores: a server given by name is "name" beside dst_addr once a UDP query is sent, and "dst_name" over TCP or when nothing was sent; a name that does not resolve is "error" : {"evdns_getaddrinfo": "NAME nodename nor servname provided, or not known"} (or non-recoverable failure in name resolution), not "getaddrinfo", in libevent's words and not the system resolver's; dst_addr is there for a timeout, a send error or a refused TCP connect, and missing only when no address was ever known.

A server on loopback, a private or link-local network, ULA or multicast, given as an address or as a name that resolves to one, is not asked, as atlas_check_addr() has it (IPv4 127/8, 10/8, 172.16/12, 192.168/16, 169.254/16, 224/4; IPv6 ::1, ::ffff:0:0/96, e000::/3): UDP reports "error" : { "reason" : "address not allowed"} with the address, except with --resolv; TCP reports "error" : { "TU_BAD_ADDR" : true}, --resolv or not. ATLAS_DISABLE_CHECK_ADDR=yes in Starla's environment lets them through, as it does the reference. A link-local src_addr, and TCP's dst_addr, carry their zone (fe80::1%eth0) as getnameinfo() writes them; UDP's dst_addr does not. As in 5130, a query that fails before it has a source address (a socket or connect error, a refused address) reports the src_addr the query before it left, from the previous resolver or the previous run; with --ttl, the last reply's "ttl" stays on later timeouts and errors; a server name with several addresses of the family is asked at each, and the last one is reported.

⚠ abuf

abuf is the base64-encoded raw DNS wire response. The RIPE Atlas API parses it server-side. Without it, results are accepted but incomplete.

HTTP — evhttpget

grammar
evhttpget [-4|-6] [-A msm_id] [-O output] [-M max_body] <url>
jsonresult
RESULT { "id":"12023", "fw":5120, "mver": "2.6.4", "lts":19, "time":1775317966, "dst_name":"example.com", "af":4, "dst_addr":"93.184.216.34", "proto":"TCP", "result": [ { "method":"GET", "af": 6, "dst_addr":"2600:140f:3::17df:2fba", "src_addr":"10.0.0.1", "rt":106.612889, "res":200, "ver":"1.1", "hsize":296, "bsize":0 } ] }

TLS — evsslgetcert

grammar
evsslgetcert [-4|-6] [-p port] [-h hostname] [-A msm_id] [-O output] <target>
jsonresult · FullLine
RESULT { "id":"14002", "fw":5120, "mver": "2.6.4", "lts":0, "time":1775325499, "dst_name":"atlas.ripe.net", "dst_port":"443", "ttr":0.390310, "method":"TLS", "ver":"1.3", "dst_addr":"193.0.11.37", "af": 4, "src_addr":"10.0.0.1", "ttc":0.191244, "rt":0.390310, "server_cipher":"0x1302", "cert":[ "-----BEGIN CERTIFICATE-----\nMIIG...\n-----END CERTIFICATE-----", "-----BEGIN CERTIFICATE-----\nMIIF...\n-----END CERTIFICATE-----" ] }

Times (ttr, ttc, rt) in seconds. ver is "1.2" or "1.3" (not "Tls12"). server_cipher is hex ("0xc030"). Certificates are full PEM with escaped newlines.

NTP — evntp

grammar
evntp [-4|-6] [-c count] [-A msm_id] [-O output] <target>
jsonresult · 3 samples
RESULT { "id":"99998", "fw":5120, "mver": "2.6.4", "lts":12, "time":1775323252, "dst_addr":"213.239.234.28", "src_addr":"204.168.188.81", "proto":"UDP", "af": 4, "li":"no", "version":4, "mode":"server", "stratum":2, "poll":8, "precision":5.96046e-08, "root-delay":0.00956726, "root-dispersion":0.0142212, "ref-id":"7cd8a40e", "ref-ts":3984311600.292619705, "result": [ { "origin-ts":3984312052.977718830, "receive-ts":3984312052.989268303, "transmit-ts":3984312052.989336491, "final-ts":3984312053.002320766, "rtt":0.024534, "offset":0.011981 }, ... ] }
FieldFormat
li"no" · "61" · "59" · "unknown"
pollRaw exponent (NOT 2^n)
precisionScientific notation (2^n seconds)
ref-idASCII for stratum ≤ 1, hex for stratum ≥ 2
TimestampsNTP epoch (seconds since 1900-01-01) · 9 decimal places

4Result upload protocol

Results are uploaded via HTTP POST through SSH direct-tcpip channels to 127.0.0.1:8080 on the controller side.

httpPOST · over SSH channel
POST /?PROBE_ID=1015186&SESSION_ID=fa0b28e5... HTTP/1.1 Host: 127.0.0.1 Content-Type: application/x-www-form-urlencoded User-Agent: httppost for atlas.ripe.net Connection: close Content-Length: 1234 P_TO_C_REPORT RESULT { "id":"9018", "fw":5120, ... disk stats ... } RESULT { "id": "7001", "fw":5120, ... uptime ... } RESULT { "id": "9002", "fw":5120, ... interface stats ... } RESULT 9901 ongoing 1775483034 starla RESULT { "id":"1001", "fw":5120, ... measurement result ... } RESULT { "id":"1002", "fw":5120, ... measurement result ... } SESSION_ID fa0b28e5...

The controller responds 200 OK with body OK\n on success. 429 means rate limiting; honor the Retry-After header. The official httppost uploads once per 60 seconds.

⚠ Mandatory system status

The controller requires system status results (9018, 7001, 9002, 9901) alongside measurement results. Without them, the controller rate-limits with persistent 429 responses that survive reconnections.

System status results

Included in every upload batch, before measurement results:

IDDescriptionFormat
9018Disk statsbsize, blocks, bfree, free from statvfs("/")
7001Probe uptimeuptime seconds since start, lts
9002Network interfacesPer-iface bytes_recv, pkt_recv, bytes_sent, pkt_sent from /proc/net/dev
9901Ongoing statusPlain: RESULT 9901 ongoing <ts> starla

5RESULT line quirks

⚠ Match exactly

The controller may do rigid string parsing. These quirks must be matched.

  • mver spacing — "mver": "2.6.4" has a space after the colon. All other fields use "key":value (no space). Matches the C probe's fprintf format string.
  • Field order — id, fw, mver, lts, time, [bundle], [dst_name], af, [dst_addr], [dst_port], [src_addr], proto, [ttl], [size], [endtime], [paris_id], result.
  • Array spacing — [ { "rtt":10.5 }, { "rtt":11.2 } ] with spaces inside brackets and braces.
  • prb_id — NOT in the RESULT line; controller reads it from the URL query.
  • fw — must be 5120 (or a real deployed version). Arbitrary values may be rejected.
  • FullLine bypass — TLS, NTP, traceroute use the FullLine variant; the complete body sits between RESULT { and }, bypassing the standard envelope formatter.

6Quirks

v0.6.5 release withdrawn

The v0.6.5 GitHub release was tagged on a commit where Cargo.toml still said 0.6.4, so the packaged binaries internally reported 0.6.4 even though the release was labelled 0.6.5. The release has been deleted; use v0.6.6 or later. The git tag is preserved for history. The release workflow now fails fast if tag and workspace version don't match (see #66).

Tagged vs untagged enums

The MeasurementData enum must use #[serde(tag, content)] (tagged), not #[serde(untagged)]. With untagged, FullLine("body") deserializes back as Generic(String("body")), losing the variant discriminator. This caused TLS / NTP / traceroute results to silently produce invalid RESULT lines.

HTTP af:0

The HTTP measurement computed the address family correctly but hardcoded af: 0 in the result struct. The controller silently drops results with af:0 while returning OK.

DNS timeouts must be results, not errors

When a DNS query times out (e.g. ISP blocks outbound UDP 53), the official probe reports "error" : { "timeout" : 5000} as a valid measurement result. Treating timeouts as execution errors loses them.

Upload rate: 60 seconds, not 10

The official httppost runs once per 60 seconds. Uploading more often triggers 429 rate limiting from the controller. The penalty is per-probe (tied to SSH key / probe ID) and persists across reconnections.

System status results are mandatory

The controller expects system health data (disk stats, uptime, network interface counters, ongoing status) in every upload batch. Sending only measurement RESULT lines causes persistent 429 rate limiting that survives reconnections and lasts hours.

DNS template variables

Query names may contain $r, $p, and $t templates (measurement 30001 uses them). $p must be the probe ID registration assigned: the scheduler starts before registration, and until 2026-10-13 Starla asked for 0.ripe-tld-experiment.… where the reference asked for 1018212.ripe-tld-experiment.….