Feature/watch (#809)

* Implement pg_autoctl watch command.

The idea is that our users and customers could have an interactive dashboard
without needing to build one themselves from the watch(1) command and other
utilities.

* Add a --watch option to pg_autoctl show state|events.

* Add libncurses to the Dockerfile dependencies.

* Per review, show logs when failing to contact the monitor.

To enable that, we switch the terminal back to "cooked" mode where we can
read the logs on stderr. When the connection to the monitor could be
established again, we switch back to the "raw" mode with the previous
settings and continue displaying our dashboard there.

Co-authored-by: Jelte Fennema <github-tech@jeltef.nl>

Author: DimCitus

Dockerfile

@@ -20,6 +20,8 @@ RUN apt-get update \
     libxml2-dev \
     libxslt1-dev \
     libselinux1-dev \
+	libncurses-dev \
+    libncurses6 \
     make \
     openssl \
     pipenv \
@@ -84,7 +86,8 @@ RUN apt-get update \
     make \
     sudo \
     tmux \
-    watch \
+	watch \
+    libncurses6 \
     lsof \
     psutils \
     dnsutils \

docs/architecture-multi-standby.rst

@@ -85,6 +85,8 @@ following three replication settings:
   - Replication quorum
   - Candidate priority
 
+.. _number_sync_standbys:
+
 Number Sync Standbys
 ^^^^^^^^^^^^^^^^^^^^
 

docs/conf.py

@@ -417,6 +417,13 @@ def setup(app):
         [author],
         1,
     ),
+    (
+        "ref/pg_autoctl_watch",
+        "pg_autoctl watch",
+        "pg_autoctl watch",
+        [author],
+        1,
+    ),
     (
         "ref/pg_autoctl_stop",
         "pg_autoctl stop",

docs/faq.rst

@@ -7,6 +7,20 @@ and your question and its answer might make it to this FAQ.
 
 __ https://github.com/citusdata/pg_auto_failover/issues_
 
+I stopped the primary and no failover is happening for 20s to 30s, why?
+-----------------------------------------------------------------------
+
+In order to avoid spurious failovers when the network connectivity is not
+stable, pg_auto_failover implements a timeout of 20s before acting on a node
+that is known unavailable. This needs to be added to the delay between
+health checks and the retry policy.
+
+See the :ref:`configuration` part for more information about how to setup
+the different delays and timeouts that are involved in the decision making.
+
+See also :ref:`pg_autoctl watch` to have a dashboard that helps
+understanding the system and what's going on in the moment.
+
 The secondary is blocked in the CATCHING_UP state, what should I do?
 --------------------------------------------------------------------
 

docs/how-to.rst

@@ -96,6 +96,15 @@ formation with the following command::
 
   $ pg_autoctl show state
 
+The ``pg_autoctl show state`` commands outputs the current state of the
+system only once. Sometimes it would be nice to have an auto-updated display
+such as provided by common tools such as `watch(1)` or `top(1)` and the
+like. For that, the following commands are available (see also
+:ref:`pg_autoctl_watch`)::
+
+  $ pg_autoctl watch
+  $ pg_autoctl show state --watch
+
 To analyze what's been happening to get to the current state, it is possible
 to review the past events generated by the pg_auto_failover monitor with the
 following command::

docs/ref/configuration.rst

@@ -1,3 +1,5 @@
+.. _configuration:
+
 Configuring pg_auto_failover
 ============================
 

docs/ref/manual.rst

@@ -22,6 +22,7 @@ have their own manual page.
    pg_autoctl_perform
    pg_autoctl_do
    pg_autoctl_run
+   pg_autoctl_watch
    pg_autoctl_stop
    pg_autoctl_reload
    pg_autoctl_status

docs/ref/pg_autoctl.rst

@@ -20,6 +20,7 @@ pg_autoctl provides the following commands::
   + set      Set a pg_auto_failover node, or formation setting
   + perform  Perform an action orchestrated by the monitor
     run      Run the pg_autoctl service (monitor or keeper)
+    watch    Display a dashboard to watch monitor's events and state
     stop     signal the pg_autoctl service for it to stop
     reload   signal the pg_autoctl for it to reload its configuration
     status   Display the current status of the pg_autoctl service

docs/ref/pg_autoctl_show_events.rst

@@ -18,6 +18,7 @@ about state changes of the pg_auto_failover nodes managed by the monitor::
   --formation   formation to query, defaults to 'default'
   --group       group to query formation, defaults to all
   --count       how many events to fetch, defaults to 10
+  --watch       display an auto-updating dashboard
   --json        output data in the JSON format
 
 Options
@@ -46,6 +47,16 @@ Options
 
   By default only the last 10 events are printed.
 
+--watch
+
+  Take control of the terminal and display the current state of the system
+  and the last events from the monitor. The display is updated automatically
+  every 500 milliseconds (half a second) and reacts properly to window size
+  change.
+
+  Depending on the terminal window size, a different set of columns is
+  visible in the state part of the output. See :ref:`pg_autoctl_watch`.
+
 --json
 
   Output a JSON formated data instead of a table formatted list.

docs/ref/pg_autoctl_show_state.rst

@@ -18,6 +18,7 @@ registered to the pg_auto_failover monitor::
   --formation   formation to query, defaults to 'default'
   --group       group to query formation, defaults to all
   --local       show local data, do not connect to the monitor
+  --watch       display an auto-updating dashboard
   --json        output data in the JSON format
 
 Options
@@ -51,14 +52,24 @@ Options
 
   Print the local state information without connecting to the monitor.
 
+--watch
+
+  Take control of the terminal and display the current state of the system
+  and the last events from the monitor. The display is updated automatically
+  every 500 milliseconds (half a second) and reacts properly to window size
+  change.
+
+  Depending on the terminal window size, a different set of columns is
+  visible in the state part of the output. See :ref:`pg_autoctl_watch`.
+
 --json
 
   Output a JSON formated data instead of a table formatted list.
 
 Description
 -----------
 
-The ``pg_autoctl show state`` outputs includes the following columns:
+The ``pg_autoctl show state`` output includes the following columns:
 
   - Name