Implement drop-at-a-distance semantics. (#734)

* Implement drop-at-a-distance semantics.

This allows running `pg_autoctl drop node --name foo` from the monitor, or
even using the SQL API directly, and have the node realise it's been
dropped. And then stop.

The configuration file, state file, and PGDATA are not touched, for that the
command pg_autoctl drop node --pgdata ... --destroy can be used on the node
itself.

* Allow pg_autoctl create node on top of a dropped node.

* Allow pg_autoctl drop node to finish removing a lost node.

When a node won't come back up again, the first call to pg_autoctl drop node
sets the assigned role to DROPPED, but the node is not in a position to call
node_active() to clean this entry.

Now, another call to pg_autoctl drop node from the monitor allows to
clean-up the node for real, with the force option set to true.

If the node still comes back up again, the situation is properly detected
and the command pg_autoctl create node --run is now able to re-register the
node with its old nodeid and continue from there.

* Per review, introduce pg_autoctl drop node --force.

* Per review, fix some pg_autoctl drop node cases.

In particular dropping a local node that is running in the background was
broken. Refactor and simplify the code to make all cases work:

  1. stop local node and drop
  2. drop local node while it's running in the background
  3. drop node from the monitor while it's running
  4. drop node from the monitor after having stopped it

In case 4. we need to then resort to pg_autoctl drop node --force, as expected.

* Allow Postgres 14 failures.

* Improve handling of node being drop during init phases, per review.

* Per review, allow re-creating a primary node from a dropped node.

Author: DimCitus

.travis.yml

@@ -35,6 +35,11 @@ matrix:
     - env: PGVERSION=13 TEST=ssl
     - env: PGVERSION=14 TEST=ssl
     - env: LINTING=true
+  allow_failures:
+    - env: PGVERSION=14 TEST=multi
+    - env: PGVERSION=14 TEST=single
+    - env: PGVERSION=14 TEST=monitor
+    - env: PGVERSION=14 TEST=ssl
 before_install:
   - git clone -b v0.7.18 --depth 1 https://github.com/citusdata/tools.git
   - sudo make -C tools install

docs/conf.py

@@ -71,9 +71,9 @@ def __init__(self, **options):
 # built documents.
 #
 # The short X.Y version.
-version = "1.5"
+version = "1.6"
 # The full version, including alpha/beta/rc tags.
-release = "1.5.2"
+release = "1.6.0"
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/failover-state-machine.rst

@@ -255,6 +255,18 @@ Missing WAL bytes are fetched from one of the most advanced standby nodes by
 using Postgres cascading replication features: it is possible to use any
 standby node in the ``primary_conninfo``.
 
+Dropped
+^^^^^^^
+
+The dropped state is assigned to a node when the ``pg_autoctl drop node``
+command is used. This allows the node to implement specific local actions
+before being entirely removed from the monitor database.
+
+When a node reports reaching the dropped state, the monitor removes its
+entry. If a node is not reporting anymore, maybe because it's completely
+unavailable, then it's possible to run the ``pg_autoctl drop node --force``
+command, and then the node entry is removed from the monitor.
+
 Failover logic
 --------------
 

docs/fsm.png

@@ -19,6 +19,7 @@ This command drops a Postgres node from the pg_auto_failover monitor::
   --hostname    drop the node with given hostname and pgport
   --pgport      drop the node with given hostname and pgport
   --destroy     also destroy Postgres database
+  --force       force dropping the node from the monitor
 
 Description
 -----------
@@ -37,6 +38,11 @@ options ``--hostname`` and ``--pgport`` or the pair of options
 monitor database, and get it removed from the known list of nodes on the
 monitor.
 
+Then option ``--force`` can be used when the target node to remove does not
+exist anymore. When a node has been lost entirely, it's not going to be able
+to finish the procedure itself, and it is then possible to instruct the
+monitor of the situation.
+
 Options
 -------
 
@@ -75,20 +81,32 @@ Options
   Postgres database for the monitor. When using ``--destroy``, the Postgres
   installation is also deleted.
 
+--force
+
+  By default a node is expected to reach the assigned state DROPPED when it
+  is removed from the monitor, and has the opportunity to implement clean-up
+  actions. When the target node to remove is not available anymore, it is
+  possible to use the option ``--force`` to immediately remove the node from
+  the monitor.
+
 Examples
 --------
 
 ::
 
    $ pg_autoctl drop node --destroy --pgdata ./node3
-   17:49:42 12504 INFO  Removing local node from the pg_auto_failover monitor.
-   17:49:42 12504 INFO  Removing local node state file: "/Users/dim/dev/MS/pg_auto_failover/tmux/share/pg_autoctl/Users/dim/dev/MS/pg_auto_failover/tmux/node3/pg_autoctl.state"
-   17:49:42 12504 INFO  Removing local node init state file: "/Users/dim/dev/MS/pg_auto_failover/tmux/share/pg_autoctl/Users/dim/dev/MS/pg_auto_failover/tmux/node3/pg_autoctl.init"
-   17:49:42 12504 INFO  Removed pg_autoctl node at "/Users/dim/dev/MS/pg_auto_failover/tmux/node3" from the monitor and removed the state file "/Users/dim/dev/MS/pg_auto_failover/tmux/share/pg_autoctl/Users/dim/dev/MS/pg_auto_failover/tmux/node3/pg_autoctl.state"
-   17:49:42 12504 INFO  Stopping PostgreSQL at "/Users/dim/dev/MS/pg_auto_failover/tmux/node3"
-   17:49:42 12504 INFO  /Applications/Postgres.app/Contents/Versions/12/bin/pg_ctl --pgdata /Users/dim/dev/MS/pg_auto_failover/tmux/node3 --wait stop --mode fast
-   17:49:42 12504 INFO  /Applications/Postgres.app/Contents/Versions/12/bin/pg_ctl status -D /Users/dim/dev/MS/pg_auto_failover/tmux/node3 [3]
-   17:49:42 12504 INFO  pg_ctl: no server running
-   17:49:42 12504 INFO  pg_ctl stop failed, but PostgreSQL is not running anyway
-   17:49:42 12504 INFO  Removing "/Users/dim/dev/MS/pg_auto_failover/tmux/node3"
-   17:49:42 12504 INFO  Removing "/Users/dim/dev/MS/pg_auto_failover/tmux/config/pg_autoctl/Users/dim/dev/MS/pg_auto_failover/tmux/node3/pg_autoctl.cfg"
+   17:52:21 54201 INFO  Reaching assigned state "secondary"
+   17:52:21 54201 INFO  Removing node with name "node3" in formation "default" from the monitor
+   17:52:21 54201 WARN  Postgres is not running and we are in state secondary
+   17:52:21 54201 WARN  Failed to update the keeper's state from the local PostgreSQL instance, see above for details.
+   17:52:21 54201 INFO  Calling node_active for node default/4/0 with current state: PostgreSQL is running is false, sync_state is "", latest WAL LSN is 0/0.
+   17:52:21 54201 INFO  FSM transition to "dropped": This node is being dropped from the monitor
+   17:52:21 54201 INFO  Transition complete: current state is now "dropped"
+   17:52:21 54201 INFO  This node with id 4 in formation "default" and group 0 has been dropped from the monitor
+   17:52:21 54201 INFO  Stopping PostgreSQL at "/Users/dim/dev/MS/pg_auto_failover/tmux/node3"
+   17:52:21 54201 INFO  /Applications/Postgres.app/Contents/Versions/12/bin/pg_ctl --pgdata /Users/dim/dev/MS/pg_auto_failover/tmux/node3 --wait stop --mode fast
+   17:52:21 54201 INFO  /Applications/Postgres.app/Contents/Versions/12/bin/pg_ctl status -D /Users/dim/dev/MS/pg_auto_failover/tmux/node3 [3]
+   17:52:21 54201 INFO  pg_ctl: no server running
+   17:52:21 54201 INFO  pg_ctl stop failed, but PostgreSQL is not running anyway
+   17:52:21 54201 INFO  Removing "/Users/dim/dev/MS/pg_auto_failover/tmux/node3"
+   17:52:21 54201 INFO  Removing "/Users/dim/dev/MS/pg_auto_failover/tmux/config/pg_autoctl/Users/dim/dev/MS/pg_auto_failover/tmux/node3/pg_autoctl.cfg"

docs/ref/pg_autoctl_drop_node.rst

@@ -41,9 +41,6 @@ int ssl_flag = 0;
 /* stores --node-id, only used with --disable-monitor */
 int monitorDisabledNodeId = -1;
 
-static void stop_postgres_and_remove_pgdata_and_config(ConfigFilePaths *pathnames,
-													   PostgresSetup *pgSetup);
-
 /*
  * cli_common_keeper_getopts parses the CLI options for the pg_autoctl create
  * postgres command, and others such as pg_autoctl do discover. An example of a
@@ -1361,171 +1358,6 @@ cli_pprint_json(JSON_Value *js)
 }
 
 
-/*
- * cli_drop_local_node drops the local Postgres node.
- */
-void
-cli_drop_local_node(KeeperConfig *config, bool dropAndDestroy)
-{
-	Keeper keeper = { 0 };
-
-	/*
-	 * Now also stop the pg_autoctl process.
-	 */
-	if (file_exists(config->pathnames.pid))
-	{
-		pid_t pid = 0;
-
-		if (read_pidfile(config->pathnames.pid, &pid))
-		{
-			log_info("An instance of pg_autoctl is running with PID %d, "
-					 "stopping it.", pid);
-
-			if (kill(pid, SIGQUIT) != 0)
-			{
-				log_error(
-					"Failed to send SIGQUIT to the keeper's pid %d: %m", pid);
-				exit(EXIT_CODE_INTERNAL_ERROR);
-			}
-		}
-	}
-
-	/* only keeper_remove when we still have a state file around */
-	if (!config->monitorDisabled)
-	{
-		if (file_exists(config->pathnames.state))
-		{
-			/* keeper_remove uses log_info() to explain what's happening */
-			if (!keeper_remove(&keeper, config))
-			{
-				log_fatal("Failed to remove local node from the pg_auto_failover "
-						  "monitor, see above for details");
-
-				exit(EXIT_CODE_BAD_STATE);
-			}
-
-			log_info("Removed pg_autoctl node at \"%s\" from the monitor and "
-					 "removed the state file \"%s\"",
-					 config->pgSetup.pgdata,
-					 config->pathnames.state);
-		}
-		else
-		{
-			log_warn("Skipping node removal from the monitor: "
-					 "state file \"%s\" does not exist",
-					 config->pathnames.state);
-		}
-	}
-	else
-	{
-		/* when the monitor is disabled, just remove the state files */
-		if (!unlink_file(config->pathnames.init))
-		{
-			log_error("Failed to remove state init file \"%s\"",
-					  config->pathnames.init);
-		}
-
-		if (!unlink_file(config->pathnames.state))
-		{
-			log_error("Failed to remove state file \"%s\"",
-					  config->pathnames.state);
-		}
-	}
-
-	/*
-	 * Either --destroy the whole Postgres cluster and configuration, or leave
-	 * enough behind us that it's possible to re-join a formation later.
-	 */
-	if (dropAndDestroy)
-	{
-		(void)
-		stop_postgres_and_remove_pgdata_and_config(
-			&config->pathnames,
-			&config->pgSetup);
-	}
-	else
-	{
-		/*
-		 * We need to stop Postgres now, otherwise we won't be able to drop the
-		 * replication slot on the other node, because it's still active.
-		 */
-		log_info("Stopping PostgreSQL at \"%s\"", config->pgSetup.pgdata);
-
-		if (!pg_ctl_stop(config->pgSetup.pg_ctl, config->pgSetup.pgdata))
-		{
-			log_error("Failed to stop PostgreSQL at \"%s\"",
-					  config->pgSetup.pgdata);
-			exit(EXIT_CODE_PGCTL);
-		}
-
-		/*
-		 * Now give the whole picture to the user, who might have missed our
-		 * --destroy option and might want to use it now to start again with a
-		 * fresh environment.
-		 */
-		log_warn("Configuration file \"%s\" has been preserved",
-				 config->pathnames.config);
-
-		if (directory_exists(config->pgSetup.pgdata))
-		{
-			log_warn("Postgres Data Directory \"%s\" has been preserved",
-					 config->pgSetup.pgdata);
-		}
-
-		log_info("drop node keeps your data and setup safe, you can still run "
-				 "Postgres or re-join the pg_auto_failover cluster later");
-		log_info("HINT: to completely remove your local Postgres instance and "
-				 "setup, consider `pg_autoctl drop node --destroy`");
-	}
-}
-
-
-/*
- * stop_postgres_and_remove_pgdata_and_config stops PostgreSQL and then removes
- * PGDATA, and then config and state files.
- */
-static void
-stop_postgres_and_remove_pgdata_and_config(ConfigFilePaths *pathnames,
-										   PostgresSetup *pgSetup)
-{
-	log_info("Stopping PostgreSQL at \"%s\"", pgSetup->pgdata);
-
-	if (!pg_ctl_stop(pgSetup->pg_ctl, pgSetup->pgdata))
-	{
-		log_error("Failed to stop PostgreSQL at \"%s\"", pgSetup->pgdata);
-		log_fatal("Skipping removal of directory \"%s\"", pgSetup->pgdata);
-		exit(EXIT_CODE_PGCTL);
-	}
-
-	/*
-	 * Only try to rm -rf PGDATA if we managed to stop PostgreSQL.
-	 */
-	if (directory_exists(pgSetup->pgdata))
-	{
-		log_info("Removing \"%s\"", pgSetup->pgdata);
-
-		if (!rmtree(pgSetup->pgdata, true))
-		{
-			log_error("Failed to remove directory \"%s\": %m", pgSetup->pgdata);
-			exit(EXIT_CODE_INTERNAL_ERROR);
-		}
-	}
-	else
-	{
-		log_warn("Skipping removal of \"%s\": directory does not exists",
-				 pgSetup->pgdata);
-	}
-
-	log_info("Removing \"%s\"", pathnames->config);
-
-	if (!unlink_file(pathnames->config))
-	{
-		/* errors have already been logged. */
-		exit(EXIT_CODE_BAD_CONFIG);
-	}
-}
-
-
 /*
  * logLevelToString returns the string to use to enable the same logLevel in a
  * sub-process.

src/bin/pg_autoctl/cli_common.c

@@ -177,7 +177,6 @@ void keeper_cli_destroy_node(int argc, char **argv);
 bool cli_getopt_ssl_flags(int ssl_flag, char *optarg, PostgresSetup *pgSetup);
 bool cli_getopt_accept_ssl_options(SSLCommandLineOptions newSSLOption,
 								   SSLCommandLineOptions currentSSLOptions);
-void cli_drop_local_node(KeeperConfig *config, bool dropAndDestroy);
 
 char * logLevelToString(int logLevel);