Skip to main content

Node synchronization errors

Synchronization problems between nodes manifest in the observation that apps on different nodes show inconsistent information, i.e. the events of nodes have not been synchronized properly.

This is mostly caused by communication errors between nodes. To debug and resolve them, you can try these steps:

Configuration errors​

If your nodes were synchronizing previously, you can skip this step. If your nodes were never synchronizing, please check the following:

  • Check that you have configured the same swarm key as described in this guide
  • Check that your nodes are publishing events to the same topic. The topic can be configured in the node settings. To query the topic with the CLI, use ax settings get /swarm/topic <node>.
  • Check that your nodes are not configured as readOnly. This can be configured in the node settings under /api/events/readOnly.
  • If mDNS is disabled on your device or network, or you are running Actyx on Docker, check that you have configured the right node(s) to connect to on all devices as described in this guide
  • If you are running Actyx on Docker without network=host and your initial peers are not running in the same local network as your other nodes, check that you have configured the announceAddresses settings on your Docker nodes as described in this guide

If you fixed any configuration issues and your nodes are still not synchronizing, go through the steps below.

Connection errors​

Test the network requirements​

Next, check if the nodes cannot connect due to firewall issues as described in this guide. If, after following the described steps, you either found no issues or fixed issues with your firewall, check if the nodes are peered.

Test if the nodes are peered​

Next, check if your nodes are actually connected to each other as described in this guide. Nodes not being able to peer with one another can in some cases be caused by binding to the IP address of an internal interface. This can happen for example on hosts that run Docker or VPN tunnels. In this case it is advisable to explicitly bind to that address which belongs to the interface via which the other Actyx nodes can be reached, e.g. actyx --bind-swarm=1.2.3.4:4001.

Event synchronisation takes up to a minute​

Sending events from one Actyx node to the other usually takes less than 20ms, but this only works while the two nodes directly connected to each other. If two nodes are not directly connected, their events will still reach the respective other node, but via a much slower path — this is a fallback path that is super-reliable and ensures eventual synchronisation.

The mechanism used on this fallback path is that each node transmits to the rest of the swarm its current knowledge of all event streams. Since there can be many streams and this is done by every node, the interval at which this information is sent is kept intentionally large, especially compared to real-time latencies. The process by which events traval on this path is also pull-based, so the listening node asks the sender for the events after it has heard the gossip message.

For this reason, it is important that you keep those nodes that shall interact in (soft) real-time always connected to each other. Sending events via one intermediary usually takes 10–20 seconds, sending via two intermediaries could take 20–40 seconds — the latency for event dissemination strongly depends on how well your swarm is connected.

Did not find what you were looking for?​

If you couldn't a solution to your problem, please don't hesitate to contact us in our community forum or on our Discord server. We'll do our best to get back to you with answers as fast as possible.