Posts / What ros2 daemon start Really Does: Domain ID, RMW and a Stale Graph

What ros2 daemon start Really Does: Domain ID, RMW and a Stale Graph

Your node is running. Its log says it is publishing. And ros2 topic list does not show the topic. You run the command again, same result. You check the spelling, the namespace, the terminal you sourced.

Then someone tells you to run ros2 daemon stop, then ros2 daemon start, and the topic is there. Nothing was wrong with your node, and nothing in your code changed. A single stop and start was enough.

That is a strange fix for a tool that is supposed to just read the network, so it is worth knowing what is being stopped and started. The short answer is that ros2 has a background process that keeps a ROS node alive and answers the command line on its behalf, and that this process keeps the environment it was started with. This post follows what ros2 daemon start does, step by step, and what it inherits from ROS_DOMAIN_ID, RMW_IMPLEMENTATION and the discovery settings.

Key takeaway
The daemon is a cache of the ROS graph for the command line. It lives outside your terminal, it is started once with the environment of whoever launched it first, and it keeps that environment until you stop it. Publishers and subscribers never go through it.

The problem the daemon solves

A ROS 2 graph has no central server. Nodes find each other with the discovery mechanism of the RMW (the middleware layer, for example Fast DDS or Cyclone DDS), and a new participant needs some time to hear about the others.

The command line is a new participant every time. A command like ros2 topic list has to create a node, wait for discovery and read the graph. Without a daemon, that waiting time is paid by every single command: the --spin-time option of the CLI, 0.5 seconds by default, is exactly this waiting time.

The daemon moves that cost out of the command. It is one long-lived node that has already discovered the graph, and each ros2 command asks it for the answer instead of discovering everything again.

Discovery itself does not go through XML-RPC. The daemon’s hidden node listens to the RMW announcements continuously, like any other node, so its picture of the graph is updated as nodes appear and disappear. XML-RPC only carries the answer between the CLI and the daemon.

%%{init: {"flowchart": {"rankSpacing": 24, "nodeSpacing": 20, "padding": 5}}}%% flowchart TB A["ros2 topic list"] -->|"XML-RPC
127.0.0.1:11511 + domain id"| D["ros2 daemon
hidden node _ros2cli_daemon_*"] D -->|"discovery
through the RMW"| G["ROS graph
nodes, topics, services"] A -.->|"--no-daemon
own node, waits --spin-time"| G classDef run fill:#FFEFC2,stroke:#E5D28A,color:#6B4A00 class D run
Fig. 1 - With a daemon, the CLI asks a node that already knows the graph.

What ros2 daemon start does

Here is the sequence, as it appears in the ros2cli code [5].

Step 1
It computes a port from the domain ID
The daemon listens on 127.0.0.1, on the port 11511 + ROS_DOMAIN_ID. With ROS_DOMAIN_ID unset, the port is 11511. With ROS_DOMAIN_ID=5, it is 11516. This is how the CLI knows which daemon to talk to, and it means that each domain has its own daemon [2].
Step 2
It binds that port before doing anything else
The socket is the lock. If the port is already taken, a daemon already runs for this domain, and the command prints The daemon is already running and stops there. Otherwise it goes on.
Step 3
It launches a detached Python process
The command line of this process is python3 -c "from ros2cli.daemon.daemonize import main; main()", followed by three tags that exist only to make it recognisable in the process list: --name ros2-daemon, --ros-domain-id <N> and --rmw-implementation <name>. Its output is sent to /dev/null (unless you pass --debug), and it is no longer attached to your terminal, which is why it survives when you close that terminal. The already-open socket is handed over to it.
Step 4
The new process creates a ROS nodeKey step
The node is hidden, and named _ros2cli_daemon_<domain>_<random id>. Creating it loads the RMW and joins the domain, with the environment the process inherited from the command that launched it. This is the step that matters for the rest of the post.
Step 5
It serves the graph over XML-RPC
At http://127.0.0.1:<port>/ros2cli/ the daemon exposes a list of functions, such as get_topic_names_and_types, get_node_names_and_namespaces, get_publishers_info_by_topic or count_subscribers. Each one is a thin wrapper around the question you would ask a node.
Step 6
It reports that it is ready
ros2 daemon start waits up to 10 seconds for the daemon to be up, then prints The daemon has been started.

ros2 daemon status only checks whether something answers on the port (The daemon is running). ros2 daemon stop sends a shutdown request and waits until the port is free again (The daemon has been stopped) [3].

You rarely type ros2 daemon start yourself. When a command needs the graph and finds no daemon on the port, it starts one on its own, then carries on [1]. The first ros2 topic list of the day is the one that starts it. In the source, there is even a comment about shell tab completion: pressing Tab on a ros2 command can be enough to spawn the daemon.

Who talks to the daemon, and who does not

The CLI commands that read the graph go through a small selector, NodeStrategy: if a daemon is reachable, ask it, otherwise create a direct node. ros2 topic list, ros2 node list, ros2 service list and ros2 topic info belong to that group.

Four things never involve the daemon:

  • Your nodes. ros2 run, ros2 launch and anything built with rclcpp or rclpy create their own participant. Publishing and subscribing do not pass through the daemon, so stopping it never interrupts a running system.
  • The data. ros2 topic echo asks the daemon, over XML-RPC, for the type of the topic and the QoS of its publishers: a few local calls at start-up. Then it creates its own node and subscribes through the RMW. The messages never pass through the daemon, so streaming has no XML-RPC cost. That node also waits its --spin-time when it is created, so for echo the daemon saves little. ros2 topic hz uses a direct node altogether.
  • Other graph tools, such as rqt_graph. It runs as an rqt plugin with its own node, and calls the same graph functions directly (list the nodes, the publishers of a topic, and so on). No daemon and no XML-RPC: it takes the environment of the terminal that launched it, like any other node.
  • Commands with --no-daemon. The option skips the daemon, does not start one, and creates a direct node that waits --spin-time seconds for discovery.

Fig. 2 shows the life of a ros2 topic echo /chatter, from the command to the last message. The daemon (in yellow) only takes part in one step, at the beginning, and only to answer two questions about the graph:

%%{init: {"flowchart": {"rankSpacing": 24, "nodeSpacing": 20, "padding": 5}}}%% flowchart TB A["1. ros2 topic echo /chatter"] --> B["2. NodeStrategy
is a daemon reachable?"] B -->|"yes"| C["3. ask the daemon over XML-RPC
type of the topic, QoS of its publishers"] B -->|"no, or --no-daemon"| C2["3. ask a direct node
after waiting --spin-time"] C --> D["4. create its own node
(waits --spin-time)"] C2 --> D D --> E["5. subscribe through the RMW"] E --> F["6. messages arrive straight from the publisher
no daemon, no XML-RPC"] F --> G["7. Ctrl+C: the node is destroyed"] classDef run fill:#FFEFC2,stroke:#E5D28A,color:#6B4A00 class C run
Fig. 2 - Life of ros2 topic echo: the daemon helps with discovery, then steps aside.

Steps 4 to 7 are the same with or without a daemon. Only step 3 changes, and the daemon makes it faster because its node has already discovered the graph.

This split gives the first diagnosis. If ros2 topic list does not show a topic but ros2 topic list --no-daemon does, the system is fine and the daemon’s picture of it is wrong. rqt_graph gives the same hint: if it shows your node while ros2 topic list does not, the daemon is the one that is out of date.

So why keep the daemon if --no-daemon exists? Speed and completeness. Without the daemon, every command pays the discovery wait, and if discovery takes longer than the 0.5 s default (a loaded network, many nodes, remote machines), the answer can be partial unless you raise --spin-time. The daemon is fast and complete, at the price of a view that can be stale. It is a trade-off, and --no-daemon is best used as a diagnostic tool.

What the daemon keeps from your environment

The daemon is a process, and a process takes its environment when it starts. Whatever was set in the terminal that launched it first is what it uses until it dies. Variables set later, in other terminals, change nothing for it.

VariableIn the daemon’s identity?What happens if it differs between two terminals
ROS_DOMAIN_IDYes, it gives the portEach terminal reaches its own daemon, so there is no conflict. Two domains mean two daemons.
RMW_IMPLEMENTATIONNoBoth terminals reach the same daemon, which uses the RMW of the first one.
ROS_AUTOMATIC_DISCOVERY_RANGE, ROS_STATIC_PEERS (ROS_LOCALHOST_ONLY on older distributions) [4]NoSame: the daemon keeps the discovery settings it started with.
Vendor configuration, such as CYCLONEDDS_URI or FASTRTPS_DEFAULT_PROFILES_FILENoSame: it was read when the daemon’s node was created.

The domain ID is the only setting that is part of the daemon’s address. All the others are invisible to the CLI, and that is the origin of the story at the top. The daemon’s discovery is continuous, but it follows these start-up settings. If they do not match the ones of your nodes, it keeps listening in the wrong place, and its view stays incomplete however long you wait.

A typical case:

  1. In the morning, a terminal runs ros2 topic list with the default RMW. This starts a daemon with that RMW.
  2. Later, you open another terminal, export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp and start your robot stack with it.
  3. In that second terminal, ros2 topic list finds the daemon from the morning on the same port and trusts it. That daemon speaks another RMW, so it does not see your nodes, and the topic is missing.

This shows an important consequence: your node and the daemon are two independent participants, each with its own environment and its own view of the graph. A node, rqt_graph or a command run with --no-daemon can very well receive the data of a topic that ros2 topic list does not show, because the command asks a daemon that was started under other conditions. The topic exists, the data flows, and only the daemon’s list is wrong.

The same thing happens with a switch of discovery range, such as from SUBNET to LOCALHOST, or with a configuration file you only just exported. The nodes are fine, the environment of the CLI is fine, and the only wrong element is a process that was started under other conditions.

ros2 daemon stop removes that process. The next command that needs the graph starts a fresh daemon, with the environment of the terminal you are typing in. That is also why the start is often optional: the first command after the stop does it for you.

The daemon also checks the network
The daemon node watches the network interfaces of the machine. When the list of addresses changes (a Wi-Fi network joined, a VPN switched on), it recreates its node on the next request and logs that it was reset. So the interface changes are handled. The variables above are not, because they are read once, at start-up.

How long it lives

A daemon is detached, so closing the terminal that started it changes nothing, and it stays active until something stops it. Recent versions of ros2cli also stop it after two hours without any request. The delay is set with ros2 daemon start --timeout <seconds>, and a negative value disables it.

Two consequences:

  • A daemon that is two hours old and still used is not unusual. It may have been started by a session you no longer remember, with a configuration you changed since.
  • A daemon that stops by itself overnight explains why a problem can “fix itself” the next morning.

Checking what you are talking to

When a topic is missing, these commands tell you which side is wrong, from the cheapest to the most direct:

# 1. Is a daemon running, and for which domain?
echo $ROS_DOMAIN_ID
ros2 daemon status

# 2. Which RMW and domain was it started with? Both are in its command line.
ps aux | grep ros2-daemon

# 3. Does the graph look different without it?
ros2 topic list --no-daemon

# 4. The fix: stop it, and let the next command start a new one.
ros2 daemon stop
ros2 topic list

In step 2, compare --ros-domain-id and --rmw-implementation with echo $RMW_IMPLEMENTATION in your current terminal. A mismatch is the whole explanation. The daemon’s own node, _ros2cli_daemon_..., can also be seen with ros2 node list --all, since hidden nodes are filtered out by default.

What breaks where

SymptomLikely causeFirst thing to run
Topic missing in ros2 topic list, present with --no-daemonThe daemon was started with another RMW or other discovery settingsros2 daemon stop, then retry
Everything missing, on a machine where it worked yesterdayThe daemon belongs to another ROS_DOMAIN_ID than the one you export nowecho $ROS_DOMAIN_ID, then ps aux | grep ros2-daemon
Missing with and without the daemonNot a daemon problem: the nodes do not see each otherCheck the RMW, the domain and the discovery range of the nodes themselves
ros2 daemon start prints The daemon is already running but the graph looks wrongSomething owns the port, and it was started under other conditionsros2 daemon stop, then ros2 daemon start
A Docker or systemd stack sees nothing from the host CLIDifferent network namespace or different environment between the twoCompare the variables on both sides before blaming the daemon

The rule to keep: the CLI shows you the graph as seen by the process that answers, and that process is not necessarily running with the settings of the terminal you are looking at.

Quick remember

  • The daemon is a cache of the ROS graph for the command line. It is a long-lived hidden node, and ros2 commands ask it instead of discovering the graph again.
  • There is one daemon per ROS_DOMAIN_ID, on the port 11511 + ROS_DOMAIN_ID. The domain ID is the only setting that is part of its address.
  • It keeps the environment of the terminal that started it (RMW, discovery settings, vendor configuration) until it is stopped, and it stops by itself after two hours without a request.
  • Your nodes never go through it. Publishing, subscribing and the data of ros2 topic echo do not, so stopping it never interrupts a running system.
  • A topic missing from ros2 topic list? Run ros2 topic list --no-daemon. If the topic shows up, the system is fine: run ros2 daemon stop and retry.

References

The user documentation says that a daemon exists, how to control it and that ROS_DOMAIN_ID offsets its port [2]. The rest of the sequence (the detached process, the hidden node, the two-hour timeout, NodeStrategy) is read in the ros2cli code, listed in reference 5.

  1. ROS 2 documentation, About Command Line Tools (Jazzy): the CLI starts a daemon in the background when none is running. docs.ros.org
  2. ROS 2 documentation, Introspection with command line tools (Lyrical): ROS_DOMAIN_ID must match the daemon’s domain, and how to run the daemon in the foreground. docs.ros.org
  3. ros2cli documentation, ros2cli.verb.daemon.stop. docs.ros.org
  4. ROS 2 documentation, Improved Dynamic Discovery (Jazzy): ROS_AUTOMATIC_DISCOVERY_RANGE and ROS_STATIC_PEERS. docs.ros.org
  5. ros2/ros2cli on GitHub, package ros2cli: daemon/ (port and server), node/daemon.py (start and stop), node/strategy.py (daemon or direct node). github.com