Skip to content

Commit 350775f

Browse files
committed
Implement Citus support in pg_auto_failover.
Squashed commit of the following: commit bd5cf30 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Wed Oct 5 17:07:45 2022 +0200 Reflow docs word wrapping to trigger a new CI build... commit 56cf731 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Wed Oct 5 17:05:08 2022 +0200 Update docs/citus.rst Co-authored-by: Jelte Fennema <github-tech@jeltef.nl> commit f1ff615 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Wed Oct 5 11:33:58 2022 +0200 Tests: install faster metadata_sync_interval for Citus coordinators. commit 37571fe Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Tue Oct 4 14:43:10 2022 +0200 Another round of Citus test fixes. - Fix the tests to be consistent about using the "citus" dbname - Fix the timeout when calling walt_until_metadata_sync in some tests commit 4eec076 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Tue Oct 4 14:06:24 2022 +0200 Update Citus tests for Citus 11.1. In Citus 11.1 we need to make sure the nodes metadata is in sync before some cluster wide operations such as DROP TABLE. This is best done by calling a citus test only function: wait_until_metadata_sync. The function needs to first be created/exposed at the SQL level. commit 948b820 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 19:23:37 2022 +0200 Improve workflow for linting / non-linting tests. commit 833cea9 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 19:10:01 2022 +0200 Skip building the docker image for TEST=linting. commit c6c4687 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 19:05:06 2022 +0200 Separate building the docker image and running tests in CI steps. commit d1c4dc5 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 18:42:58 2022 +0200 Increate tests timeout. commit 6dabc56 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 17:40:00 2022 +0200 Fix missed Python import line. commit dd066b2 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 17:26:20 2022 +0200 Fix the github action worklow file (remove LINTING env). commit 23735a8 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 17:26:04 2022 +0200 Fix Citus tests to use the "citus" database. commit c2f0426 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 17:10:17 2022 +0200 Update Citus version matrix and compatibility with PGVERSIONs. Also we need to drop support for Postgres 10, for which I failed to find a version of Citus that would be compatible. Fortunately, it's the time when we also add support for Postgres 15, Postgres 10 is soon to be unsupported anyway. commit 07b5195 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 16:38:26 2022 +0200 Fix rebase with master. commit d86a407 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 13:34:17 2022 +0200 Another try at fixing the Citus PGVERSION for citusdata/tools. commit 45f8842 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 13:33:08 2022 +0200 Docs improvements for Citus coverage. commit ea76b43 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 13:18:35 2022 +0200 Try and fix CITUSPGVERSION for install_citus script. commit 8c6122a Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 13:13:32 2022 +0200 Attempt to use install_citus from citusdata/tools. commit d09cb86 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Oct 3 13:10:01 2022 +0200 Update Python imports for Citus test files. commit e197496 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Fri Sep 30 12:11:22 2022 +0200 Fix monitor installcheck expected file, and automated naming. commit 9e5ffdc Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 29 19:00:34 2022 +0200 Review how to install the Postgres Citus package in CI. commit 50eafec Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Sep 26 13:40:24 2022 +0200 Refactor CLI environment variable support code. commit 5226fe6 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Sep 26 13:29:02 2022 +0200 Rework Citus Cluster Tutorial using docker-compose. commit 10ca9bd Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Fri Sep 23 17:56:55 2022 +0200 First pass review of the quickstart-cluster Citus HA tutorial. commit 3a89aaf Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Fri Sep 23 13:33:25 2022 +0200 Improve default node names for a Citus cluster. commit 3efb773 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 22 18:36:32 2022 +0200 Fix computing Citus nodes group/seq and names. When given W workers, we create N Nodes per worker, plus N coordinators. commit d58fa14 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 15 18:10:28 2022 +0200 Given Citus workers, allow for greater NODE_ARRAY_MAX_COUNT. commit ef7cf82 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 15 18:10:11 2022 +0200 Fix make compose again for standalone Postgres nodes. commit b29c274 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 15 17:32:37 2022 +0200 Add better support for Citus in the dev env (Dockerfile, make compose). commit e6d4609 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 15 17:23:16 2022 +0200 Add support for --pg-hba-lan for create coordinator and worker. commit ebcf0ce Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 15 17:22:49 2022 +0200 Fix docs, Citus is now included in the Open Source version. commit 7258f73 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 15 15:25:11 2022 +0200 When using Citus, the default dbname is now "citus". commit 90826c7 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 15 15:20:18 2022 +0200 Add support for quorum and priority to make compose. commit a8d138c Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 15 15:19:57 2022 +0200 Fix the FSM with missing transition that got lost in the merge. commit c83f213 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Thu Sep 15 12:43:50 2022 +0200 Attempt to fix github workflow. commit a71dd3d Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Aug 16 15:50:28 2021 +0200 Another round of improvements for the diagrams. commit cb835d4 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Fri Aug 13 18:14:26 2021 +0200 Improve all diagrams. Keep the concept of the first diagram with the kind of a circle with four nodes (application, primary, secondary, monitor) and build on-top of the same idea for the others diagrams. Every diagram builds by adding some level of complexity on-top of the previous one. By keeping a central theme it makes it obvious and much easier to re-use the understanding from the previous diagram in the next. commit 8775ae1 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Tue Aug 16 18:41:04 2022 +0200 Add an architecture diagram covering Citus formations. commit 34352a6 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Tue Aug 16 18:40:50 2022 +0200 Allow CI testing of the Citus support. commit 088bc64 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Tue Aug 10 13:38:37 2021 +0200 Implement make CITUS=1 cluster. commit 272f1eb Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Aug 9 18:20:12 2021 +0200 Implement Citus support for pg_auto_failover. commit 50cb713 Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Tue Aug 16 18:33:06 2022 +0200 Implement Citus support for pg_auto_failover. commit dfb044e Author: Dimitri Fontaine <dimitri@citusdata.com> Date: Mon Aug 9 16:29:30 2021 +0200 Add a PgInstanceKind guard in the Keeper FSM. This guard allows to choose a transition function specialized in a given node kind.
1 parent 27d5d9f commit 350775f

68 files changed

Lines changed: 9712 additions & 1505 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
1-
name: Build Package
1+
name: Run Tests
22

3-
env:
4-
TRAVIS: "true"
53
on:
64
push:
75
branches:
@@ -13,14 +11,13 @@ on:
1311
workflow_dispatch:
1412

1513
jobs:
16-
build_package:
14+
run_tests:
1715
name: Run test
1816
runs-on: ubuntu-latest
1917
strategy:
2018
fail-fast: false
2119
matrix:
2220
PGVERSION:
23-
- 10
2421
- 11
2522
- 12
2623
- 13
@@ -31,6 +28,7 @@ jobs:
3128
- single
3229
- monitor
3330
- ssl
31+
- citus
3432
include:
3533
- PGVERSION: 14
3634
TEST: tablespaces
@@ -48,6 +46,7 @@ jobs:
4846
echo "TRAVIS_BUILD_DIR=$(pwd)" >> $GITHUB_ENV
4947
5048
- name: Clone and install linting tools
49+
if: ${{ env.TEST == 'linting' }}
5150
run: |
5251
sudo apt-get install python3-pip
5352
pip3 install --user black
@@ -58,7 +57,18 @@ jobs:
5857
install_uncrustify
5958
rm -rf uncrustify*
6059
60+
- name: Check code formatting and banned function
61+
if: ${{ env.TEST == 'linting' }}
62+
run: |
63+
make lint
64+
65+
- name: Build Docker Test Image
66+
if: ${{ env.TEST != 'linting' }}
67+
run: |
68+
make build-test-image
69+
6170
- name: Run Test
71+
if: ${{ env.TEST != 'linting' }}
6272
timeout-minutes: 15
6373
run: |
6474
make ci-test

.travis.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ matrix:
2222
- env: PGVERSION=12 TEST=single
2323
- env: PGVERSION=13 TEST=single
2424
- env: PGVERSION=14 TEST=single
25+
- env: PGVERSION=11 CITUSVERSION=10.0 TEST=citus
26+
- env: PGVERSION=12 CITUSVERSION=10.1 TEST=citus
27+
- env: PGVERSION=13 CITUSVERSION=10.1 TEST=citus
2528
- env: PGVERSION=10 TEST=monitor
2629
- env: PGVERSION=11 TEST=monitor
2730
- env: PGVERSION=12 TEST=monitor
@@ -37,6 +40,7 @@ matrix:
3740
before_install:
3841
- git clone -b v0.7.18 --depth 1 https://github.com/citusdata/tools.git
3942
- sudo make -C tools install
43+
- 'if [ -z "$LINTING" ]; then curl https://install.citusdata.com/community/deb.sh | sudo bash; fi'
4044
- 'if [ -z "$LINTING" ]; then setup_apt; fi'
4145
- 'if [ -z "$LINTING" ]; then nuke_pg; fi'
4246
- python --version
@@ -56,6 +60,8 @@ install:
5660
- 'if [ -n "$LINTING" ]; then rm -rf uncrustify*; fi'
5761
- 'if [ -z "$LINTING" ]; then install_pg; fi'
5862
- 'if [ -z "$LINTING" ]; then install_custom_pg; fi'
63+
- 'if [ -z "$LINTING" ]; then sudo apt-get install postgresql-${PGVERSION}-citus-${CITUSVERSION}; fi'
64+
- PIPENV_PIPFILE="${TRAVIS_BUILD_DIR}"/tests/Pipfile pipenv install --system --deploy
5965
- env
6066
- 'if [ -z "$LINTING" ]; then pg_config; fi'
6167
- 'if [ -z "$LINTING" ]; then PATH=`pg_config --bindir`:$PATH which pg_ctl; fi'

Dockerfile

Lines changed: 43 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,14 @@
33
# target version of Postgres. In the Makefile, we use that to our advantage
44
# and tag test images such as pg_auto_failover_test:pg14.
55
#
6-
ARG PGVERSION=10
6+
ARG PGVERSION=14
77

88
#
99
# Define a base image with all our build dependencies.
1010
#
1111
# This base image contains all our target Postgres versions.
1212
#
13-
FROM debian:buster-slim as base
13+
FROM debian:bullseye-slim as base
1414

1515
ARG PGVERSION
1616

@@ -21,21 +21,28 @@ RUN apt-get update \
2121
curl \
2222
gnupg \
2323
git \
24+
gawk \
25+
flex \
26+
bison \
2427
iproute2 \
25-
libicu-dev \
26-
libkrb5-dev \
27-
libssl-dev \
28+
libcurl4-gnutls-dev \
29+
libicu-dev \
30+
libncurses-dev \
31+
libxml2-dev \
32+
zlib1g-dev \
2833
libedit-dev \
29-
libreadline-dev \
30-
libpam-dev \
31-
zlib1g-dev \
34+
libkrb5-dev \
3235
liblz4-dev \
33-
libxml2-dev \
34-
libxslt1-dev \
35-
libselinux1-dev \
36-
libncurses-dev \
3736
libncurses6 \
38-
make \
37+
libpam-dev \
38+
libreadline-dev \
39+
libselinux1-dev \
40+
libssl-dev \
41+
libxslt1-dev \
42+
libzstd-dev \
43+
uuid-dev \
44+
make \
45+
autoconf \
3946
openssl \
4047
pipenv \
4148
python3-nose \
@@ -53,7 +60,7 @@ RUN apt-get update \
5360
&& rm -rf /var/lib/apt/lists/*
5461

5562
RUN curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
56-
RUN echo "deb http://apt.postgresql.org/pub/repos/apt buster-pgdg main ${PGVERSION}" > /etc/apt/sources.list.d/pgdg.list
63+
RUN echo "deb http://apt.postgresql.org/pub/repos/apt bullseye-pgdg main ${PGVERSION}" > /etc/apt/sources.list.d/pgdg.list
5764

5865
# bypass initdb of a "main" cluster
5966
RUN echo 'create_main_cluster = false' | sudo tee -a /etc/postgresql-common/createcluster.conf
@@ -70,12 +77,25 @@ RUN adduser docker sudo
7077
RUN adduser docker postgres
7178
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
7279

80+
FROM base as citus
81+
82+
ARG PGVERSION
83+
ARG CITUSTAG=v11.1.2
84+
85+
ENV PG_CONFIG /usr/lib/postgresql/${PGVERSION}/bin/pg_config
86+
87+
RUN git clone -b ${CITUSTAG} --depth 1 https://github.com/citusdata/citus.git /usr/src/citus
88+
WORKDIR /usr/src/citus
89+
90+
RUN ./configure
91+
RUN make -s clean && make -s -j8 install
92+
7393
#
7494
# On-top of the base build-dependencies image, now we can build
7595
# pg_auto_failover for a given --build-arg PGVERSION target version of
7696
# Postgres.
7797
#
78-
FROM base as build
98+
FROM citus as build
7999

80100
ARG PGVERSION
81101

@@ -109,7 +129,7 @@ ENV PATH /usr/lib/postgresql/${PGVERSION}/bin:/usr/local/sbin:/usr/local/bin:/us
109129
#
110130
# And finally our "run" images with the bare minimum for run-time.
111131
#
112-
FROM debian:buster-slim as run
132+
FROM debian:bullseye-slim as run
113133

114134
ARG PGVERSION
115135

@@ -127,12 +147,14 @@ RUN apt-get update \
127147
psutils \
128148
dnsutils \
129149
bind9-host \
130-
postgresql-common \
150+
libcurl4-gnutls-dev \
151+
libzstd-dev \
152+
postgresql-common \
131153
libpq-dev \
132154
&& rm -rf /var/lib/apt/lists/*
133155

134156
RUN curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
135-
RUN echo "deb http://apt.postgresql.org/pub/repos/apt buster-pgdg main ${PGVERSION}" > /etc/apt/sources.list.d/pgdg.list
157+
RUN echo "deb http://apt.postgresql.org/pub/repos/apt bullseye-pgdg main ${PGVERSION}" > /etc/apt/sources.list.d/pgdg.list
136158

137159
# bypass initdb of a "main" cluster
138160
RUN echo 'create_main_cluster = false' | sudo tee -a /etc/postgresql-common/createcluster.conf
@@ -145,6 +167,9 @@ RUN adduser docker sudo
145167
RUN adduser docker postgres
146168
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
147169

170+
COPY --from=build /usr/lib/postgresql/${PGVERSION}/lib/citus.so /usr/lib/postgresql/${PGVERSION}/lib
171+
COPY --from=build /usr/share/postgresql/${PGVERSION}/extension/citus* /usr/share/postgresql/${PGVERSION}/extension/
172+
148173
COPY --from=build /usr/lib/postgresql/${PGVERSION}/lib/pgautofailover.so /usr/lib/postgresql/${PGVERSION}/lib
149174
COPY --from=build /usr/share/postgresql/${PGVERSION}/extension/pgautofailover* /usr/share/postgresql/${PGVERSION}/extension/
150175
COPY --from=build /usr/lib/postgresql/${PGVERSION}/bin/pg_autoctl /usr/local/bin

0 commit comments

Comments
 (0)