22.3. How Neo4j HA operates

A Neo4j HA cluster operates cooperatively, coordinating activity through Zookeeper.

On startup a Neo4j HA instance will connect to the coordinator service (ZooKeeper) to register itself and ask, "who is master?" If some other machine is master, the new instance will start as slave and connect to that master. If the machine starting up was the first to register — or should become master according to the master election algorithm — it will start as master.

When performing a write transaction on a slave each write operation will be synchronized with the master (locks will be acquired on both master and slave). When the transaction commits it will first occur on the master. If the master commit is successful the transaction will be committed on the slave as well. To ensure consistency, a slave has to be up to date with the master before performing a write operation. This is built into the communication protocol between the slave and master, so that updates will happen automatically if needed.

You can make a database instance permanently slave-only by including the ha.slave_coordinator_update_mode=none configuration parameter in its configuration.

Such instances will never become a master during fail-over elections though otherwise they behave identically to any other slaves, including the ability to write-through permanent slaves to the master.

When performing a write on the master it will execute in the same way as running in normal embedded mode. Currently the master will by default try to push the transaction to one slave. This is done optimistically meaning if the push fails the transaction will still be successful. This push is not like replication factor that would cause the transaction to fail. The push factor (amount of slaves to try push a transaction to) can be configured to 0 (higher write performance) and up to amount of machines available in the cluster minus one.

Slaves can also be configured to update asynchronously by setting a pull interval.

Whenever a server running a neo4j database becomes unavailable the coordinator service will detect that and remove it from the cluster. If the master goes down a new master will automatically be elected. Normally a new master is elected and started within just a few seconds and during this time no writes can take place (the write will throw an exception). A machine that becomes available after being unavailable will automatically reconnect to the cluster. The only time this is not true is when an old master had changes that did not get replicated to any other machine. If the new master is elected and performs changes before the old master recovers, there will be two different versions of the data. The old master will move away the branched database and download a full copy from the new master.

All this can be summarized as: