Netbox usage in different regions

[gr8dim]

Hello!
Dear experts, can you suggest a solution for implementing netbox operation from two different countries/regions, but with one database. It is necessary to keep records of servers of one organization, but from two different netboxes in real time, and it would be good if there was fault tolerance in case of a connection failure, but the postgress database cannot work in master-master mode and this is a problem. Thank you.

[candlerb]

If you're running a single Netbox instance then you must have a single database as your "source of truth". Netbox doesn't support multiple disconnected instances with eventual-consistency.

Your best bet would be to build a postgres database in a place where it's always available. AWS RDS would be an example of that. It can run replicas between multiple availability zones in a region, so you get extremely high availability. Pick a region which has good connectivity to both locations of interest.

If you want disconnected operation: site A can use site A's database while site B can use site B's database, even if site A can't see site B - then you need two separate Netbox instances with their own data.

[gr8dim]

Thanks for the advice, but we would like to avoid the extra costs and do everything self-hosted. I saw articles that someone is doing something similar on cockroachdb, but there are a lot of flaws there.

[candlerb]

I was going to mention cockroachdb, but apart from the fact that it's not sufficiently compatible with the way Netbox uses postgres (which uses the special data types for IP addresses and MAC addresses), it wouldn't solve your speed or availability problems at all.

Cockroachdb is fully synchronous, using the Raft protocol. That is, the replicas are kept in lock-step sync: in the CAP model, it chooses "consistency" over "availability". Every write, and indeed every read, requires a quorum of nodes to agree. If you lose the quorum then it stops completely.

Building a cockroachdb cluster within one data centre, or several very nearby data centres (availability zones) would be fine. However, building a globally-distributed cockroachdb cluster would give you the worst of all worlds: unless you have more than half the cluster nearby to you, every read and write would require global round-trips. In which case, you might as well just talk to a remote postgres server.

we would like to avoid the extra costs and do everything self-hosted

Then build your own postgres server and host it yourself in one of your locations. Use replication to a standby, and if the main one fails, you switch over to the standby.

It is possible to deploy automatic failover mechanisms too - e.g. there are various kubernetes operators for postgres which claim to be able to do this - but to be honest, it's incredibly risky. The probability of loss of data due to a bug in the failover mechanism (or it kicking in at the wrong time) is higher than the probability of your main server going down.

When evaluating costs, don't ignore the costs of your time, nor the business cost if you screw things up. If you can run a database with higher availability than AWS, or it's not critical if it's not always available, then by all means do so.

But if you're looking for a solution that chooses "availability" over "consistency", then Netbox/Postgres is not it. Systems which do this need mechanisms to resolve the inevitable split-brain conflicts which happen when the same data is updated independently in two different locations.

[gr8dim]

Then build your own postgres server and host it yourself in one of your locations. Use replication to a standby, and if the main one fails, you switch over to the standby.

Sir candlerb, I apologize if I don’t understand something, that if changes were made to Netbox B, but the connection with the database was lost, then these changes will no longer arrive in the database, and Netbox A will not know that Item_1 no longer exists ( even if the connection with the database is restored later) and in this case the Slave will no longer help us, right?

[a084ed22]

In that case, the second instance will notify the user of the failed deletion

[candlerb]

If Netbox A and Netbox B connect to the same Postgres instance, then they will see exactly the same data.

If, as in the second part of your diagram, Netbox B loses connection to this postgres database, then it will stop working. The user will see an error for any attempted read or update operation. Their attempt to delete Item 1 will fail, and they will see it fail. It's then up to the user to retry the request later when the database connection is back up. (While the database is unreachable, they won't even be able to see that item 1 exists, so there will be no button to click to delete it anyway, unless they browsed to it while the database connection was up)

I'd suggest you simplify this further, and just have a single "Netbox A" instance only, which is accessed remotely from site B over HTTPS.

If you choose to have two separate Netbox instances then you have additional problems such as how to synchronize the media and reports directories. You could do this using S3 storage, but since you say you don't want to use cloud services then you'd have to build your own S3 store for this (e.g. using Minio). You'll need to apply version updates simultaneously on both sides, and ensure plugins and configuration.py are synchronized.

You may also end up with two sets of job workers (netbox-rq). I think that should work, but it would be a bit odd because jobs submitted in site A may be executed on the instance in site B, or vice versa. (EDIT: This also requires you have a central, shared redis job queue. I don't know if multiple redis queues would be safe; I suspect not and it could confuse the GUI)

[gr8dim]

Thank you very much for your recommendations, I’ll start doing something and if I succeed, I’ll come back here with the result)