Skip to content

Nebula Graph v3.8.0 Restore Fails with E_RESTORE_FAILURE During Meta Restore #6142

Open
Open
Nebula Graph v3.8.0 Restore Fails with E_RESTORE_FAILURE During Meta Restore#6142
Labels
affects/nonePR/issue: this bug affects none version.severity/noneSeverity of bugtype/bugType: something is unexpected
@jkiran-emb-global

Description

@jkiran-emb-global
jkiran-emb-global
opened on Mar 25, 2026

br restore full fails during the meta restore phase with:

RestoreMeta failed: E_RESTORE_FAILURE

Trying to backup nebula from one cluster and restore on another cluster

  • OS: Kubernetes
  • Version: Nebula Graph v3.8.0
  • BR Version: 3.6.0

Steps to reproduce the behavior:

  1. Deploy Nebula Graph v3.8.0 cluster on Kubernetes (multi-meta, multi-storage setup).
  2. Take a full backup using br backup full to S3.
  3. Run restore command:
br restore full \
  --meta "nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559" \
  --s3.endpoint "https://s3.region.amazonaws.com" \
  --s3.access_key "<redacted>" \
  --s3.secret_key "<redacted>" \
  --s3.region "region" \
  --storage "s3://<redacted>/nebula" \
  --name "backup"

Logs:

{"level":"info","meta address":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.175Z"}
{"level":"info","meta address":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.178Z"}
{"level":"info","meta address":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.179Z"}
{"level":"info","meta address":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.181Z"}
{"level":"info","meta address":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.181Z"}
{"level":"info","meta address":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.184Z"}
{"level":"info","meta address":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.188Z"}
{"level":"info","meta address":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.190Z"}
{"level":"info","meta address":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:20:22.190Z"}
{"level":"info","meta address":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:20:22.192Z"}
{"host info":"map[nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:9669[GRAPH]: (data: , root: /usr/local/nebula) nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) | nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559[META]: (data: /usr/local/nebula/data/meta, root: /usr/local/nebula) nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: ) nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779[STORAGE]: (data: /usr/local/nebula/data/storage, root: /usr/local/nebula) | nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:8888[AGENT]: (data: , root: )]","level":"info","msg":"Get cluster topology from the nebula.","time":"2026-03-25T20:20:22.193Z"}
{"backup":"backup","level":"info","msg":"Check backup dir successfully.","time":"2026-03-25T20:20:22.259Z","uri":"s3://bucket-backup/nebula/backup"}
{"dir":"/usr/local/nebula","host":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:20:22.325Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:20:32.336Z"}
{"dir":"/usr/local/nebula","host":"nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"GRAPH","time":"2026-03-25T20:20:42.356Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:20:52.372Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:21:02.384Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:21:12.393Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:21:22.403Z"}
{"backup":"backup","level":"info","msg":"Stop cluster successfully.","time":"2026-03-25T20:21:32.415Z"}
{"addr":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:21:32.417Z"}
{"addr":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/cluster.id_old_1774470092","level":"info","msg":"Backup origin cluster.id path successfully.","origin not exist":true,"origin path":"/usr/local/nebula/cluster.id","time":"2026-03-25T20:21:32.418Z"}
{"addr":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:21:32.420Z"}
{"addr":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/cluster.id_old_1774470092","level":"info","msg":"Backup origin cluster.id path successfully.","origin not exist":true,"origin path":"/usr/local/nebula/cluster.id","time":"2026-03-25T20:21:32.421Z"}
{"addr":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:21:32.422Z"}
{"addr":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779","backup path":"/usr/local/nebula/cluster.id_old_1774470092","level":"info","msg":"Backup origin cluster.id path successfully.","origin not exist":true,"origin path":"/usr/local/nebula/cluster.id","time":"2026-03-25T20:21:32.424Z"}
{"addr":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:21:32.425Z"}
{"addr":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:21:32.426Z"}
{"addr":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:21:32.427Z"}
{"backup":"backup","level":"info","msg":"Backup origin cluster data successfully.","time":"2026-03-25T20:21:32.427Z"}
{"level":"info","msg":"Download meta data to cluster successfully.","time":"2026-03-25T20:21:33.749Z"}
{"addr":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779","external":"s3://bucket-backup/nebula/backup/data/nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779/data0/","level":"info","local":"/usr/local/nebula/data/storage/nebula","msg":"Download storage data successfully.","time":"2026-03-25T20:22:02.755Z"}
{"addr":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779","external":"s3://bucket-backup/nebula/backup/data/nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779/data0/","level":"info","local":"/usr/local/nebula/data/storage/nebula","msg":"Download storage data successfully.","time":"2026-03-25T20:22:30.930Z"}
{"addr":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779","external":"s3://bucket-backup/nebula/backup/data/nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779/data0/","level":"info","local":"/usr/local/nebula/data/storage/nebula","msg":"Download storage data successfully.","time":"2026-03-25T20:22:58.798Z"}
{"level":"info","msg":"Download storage data to cluster successfully.","time":"2026-03-25T20:22:58.798Z"}
{"addr":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start meta service successfully.","time":"2026-03-25T20:23:03.809Z"}
{"addr":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start meta service successfully.","time":"2026-03-25T20:23:08.818Z"}
{"addr":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start meta service successfully.","time":"2026-03-25T20:23:13.828Z"}
{"level":"info","msg":"Start meta service successfully.","time":"2026-03-25T20:23:23.835Z"}
{"level":"info","meta address":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Try to connect meta service.","time":"2026-03-25T20:23:23.835Z"}
{"level":"info","meta address":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","msg":"Connect meta server successfully.","time":"2026-03-25T20:23:23.838Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:23:23.883Z"}
{"dir":"/usr/local/nebula","host":"nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"GRAPH","time":"2026-03-25T20:23:33.893Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:23:43.901Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:23:53.910Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:24:03.919Z"}
{"dir":"/usr/local/nebula","host":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"STORAGE","time":"2026-03-25T20:24:13.928Z"}
{"dir":"/usr/local/nebula","host":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local","level":"info","msg":"Stop services.","role":"META","time":"2026-03-25T20:24:23.937Z"}
{"backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Moveback origin STORAGE data path successfully","name":"STORAGE[nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779]","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:24:33.983Z"}
{"backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Moveback origin STORAGE data path successfully","name":"STORAGE[nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779]","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:24:34.020Z"}
{"backup path":"/usr/local/nebula/data/storage/nebula_old_1774470092","level":"info","msg":"Moveback origin STORAGE data path successfully","name":"STORAGE[nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779]","origin path":"/usr/local/nebula/data/storage/nebula","time":"2026-03-25T20:24:34.058Z"}
{"backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Moveback origin META data path successfully","name":"META[nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559]","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:24:34.062Z"}
{"backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Moveback origin META data path successfully","name":"META[nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559]","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:24:34.064Z"}
{"backup path":"/usr/local/nebula/data/meta/nebula_old_1774470092","level":"info","msg":"Moveback origin META data path successfully","name":"META[nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559]","origin path":"/usr/local/nebula/data/meta/nebula","time":"2026-03-25T20:24:34.068Z"}
{"addr":"nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779","level":"info","msg":"Start STORAGE[nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local:9779] by agent successfully.","time":"2026-03-25T20:24:39.126Z"}
{"addr":"nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779","level":"info","msg":"Start STORAGE[nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local:9779] by agent successfully.","time":"2026-03-25T20:24:44.136Z"}
{"addr":"nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start META[nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559] by agent successfully.","time":"2026-03-25T20:24:49.147Z"}
{"addr":"nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start META[nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559] by agent successfully.","time":"2026-03-25T20:24:54.156Z"}
{"addr":"nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:9669","level":"info","msg":"Start GRAPH[nebula-graphd-0.nebula-graphd-headless.nebula.svc.cluster.local:9669] by agent successfully.","time":"2026-03-25T20:24:59.163Z"}
{"addr":"nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559","level":"info","msg":"Start META[nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559] by agent successfully.","time":"2026-03-25T20:25:04.174Z"}
{"addr":"nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779","level":"info","msg":"Start STORAGE[nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local:9779] by agent successfully.","time":"2026-03-25T20:25:09.183Z"}
Error: restore cluster meta failed: restore meta service nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559 failed: call nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559:RestoreMeta failed: E_RESTORE_FAILURE
Usage:
  br restore full [flags]

Flags:
  -h, --help   help for full

Global Flags:
      --concurrency int        Max concurrency for download data (default 5)
      --debug                  Output log in debug level or not
      --log string             Specify br detail log path (default "br.log")
      --meta string            Specify meta server
      --name string            Specify backup name
      --s3.access_key string   S3 Option: set access key id
      --s3.endpoint string     S3 Option: set the S3 endpoint URL, please specify the http or https scheme explicitly
      --s3.region string       S3 Option: set region or location to upload or download backup
      --s3.secret_key string   S3 Option: set secret key for access id
      --storage string         backup target url, format: <SCHEME>://<PATH>.
                                   <SCHEME>: a string indicating which backend type. optional: local, s3.
                                   now only s3-compatible is supported.
                                   example:
                                   for local - "local:///the/local/path/to/backup"
                                   for s3  - "s3://example/url/to/the/backup"

{"level":"fatal","msg":"restore cluster meta failed: restore meta service nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559 failed: call nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559:RestoreMeta failed: E_RESTORE_FAILURE","time":"2026-03-25T20:25:09.184Z"}

K8s Config to deploy cluster and trigger br from another k8s container.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nebula-metad
  namespace: nebula
  labels:
    app.kubernetes.io/cluster: nebula
    app.kubernetes.io/component: metad
    app.kubernetes.io/managed-by: nebula-operator
    app.kubernetes.io/name: nebula-graph
spec:
  podManagementPolicy: Parallel
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/cluster: nebula
      app.kubernetes.io/component: metad
      app.kubernetes.io/managed-by: nebula-operator
      app.kubernetes.io/name: nebula-graph
  serviceName: nebula-metad-headless
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/cluster: nebula
        app.kubernetes.io/component: metad
        app.kubernetes.io/managed-by: nebula-operator
        app.kubernetes.io/name: nebula-graph
    spec:
      serviceAccountName: nebula-sa
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/cluster: nebula
                    app.kubernetes.io/component: metad
                    app.kubernetes.io/name: nebula-graph
                topologyKey: kubernetes.io/hostname
      initContainers:
        - name: copy-agent
          image: vesoft/nebula-agent:latest
          imagePullPolicy: IfNotPresent
          command: ["cp", "/usr/local/bin/agent", "/shared/agent"]
          volumeMounts:
            - mountPath: /shared
              name: shared-bin
      containers:
        - name: metad
          image: vesoft/nebula-metad:v3.8.0
          imagePullPolicy: IfNotPresent
          # Start metad as a background daemon, then exec the agent as the main process.
          # The agent runs in the same container so it can access nebula binaries
          # and the custom nebula.service script for stop/start/status during BR restore.
          command:
            - /bin/sh
            - -c
            - |
              /usr/local/nebula/bin/nebula-metad \
                --flagfile=/usr/local/nebula/etc/nebula-metad.conf \
                --meta_server_addrs=nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559 \
                --local_ip=$(hostname).nebula-metad-headless.nebula.svc.cluster.local \
                --daemonize=true
              until bash -c "echo >/dev/tcp/localhost/9559" 2>/dev/null; do
                echo "waiting for metad thrift port..."; sleep 2
              done
              exec /shared/agent \
                --agent=$(hostname).nebula-metad-headless.nebula.svc.cluster.local:8888 \
                --debug --ratelimit=0 --hbs=10 \
                --meta=nebula-metad-headless.nebula.svc.cluster.local:9559
          ports:
            - containerPort: 8888
              name: grpc
          volumeMounts:
            - mountPath: /usr/local/nebula/data
              name: metad-data
              subPath: data
            - mountPath: /usr/local/nebula/etc/nebula-metad.conf
              name: nebula-metad-conf
              subPath: nebula-metad.conf
            - mountPath: /usr/local/nebula/scripts
              name: agent-scripts-vol
            - mountPath: /shared
              name: shared-bin
      volumes:
        - name: nebula-metad-conf
          configMap:
            defaultMode: 420
            name: nebula-metad
            items:
              - key: nebula-metad.conf
                path: nebula-metad.conf
        - name: agent-scripts-vol
          configMap:
            defaultMode: 0755
            name: nebula-agent-scripts
        - name: shared-bin
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: metad-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: csi-cinder-high-speed
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nebula-storaged
  namespace: nebula
  labels:
    app.kubernetes.io/cluster: nebula
    app.kubernetes.io/component: storaged
    app.kubernetes.io/managed-by: nebula-operator
    app.kubernetes.io/name: nebula-graph
spec:
  podManagementPolicy: Parallel
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/cluster: nebula
      app.kubernetes.io/component: storaged
      app.kubernetes.io/managed-by: nebula-operator
      app.kubernetes.io/name: nebula-graph
  serviceName: nebula-storaged-headless
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/cluster: nebula
        app.kubernetes.io/component: storaged
        app.kubernetes.io/managed-by: nebula-operator
        app.kubernetes.io/name: nebula-graph
    spec:
      serviceAccountName: nebula-sa
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/cluster: nebula
                    app.kubernetes.io/component: storaged
                    app.kubernetes.io/name: nebula-graph
                topologyKey: kubernetes.io/hostname
      initContainers:
        - name: copy-agent
          image: vesoft/nebula-agent:latest
          imagePullPolicy: IfNotPresent
          command: ["cp", "/usr/local/bin/agent", "/shared/agent"]
          volumeMounts:
            - mountPath: /shared
              name: shared-bin
      containers:
        - name: storaged
          image: vesoft/nebula-storaged:v3.8.0
          imagePullPolicy: IfNotPresent
          # Start storaged as a background daemon, then exec the agent as the main process.
          # The agent runs in the same container so it can access nebula binaries
          # and the custom nebula.service script for stop/start/status during BR restore.
          command:
            - /bin/sh
            - -c
            - |
              /usr/local/nebula/bin/nebula-storaged \
                --flagfile=/usr/local/nebula/etc/nebula-storaged.conf \
                --meta_server_addrs=nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559 \
                --local_ip=$(hostname).nebula-storaged-headless.nebula.svc.cluster.local \
                --daemonize=true
              until pgrep -f /usr/local/nebula/bin/nebula-storaged > /dev/null; do
                echo "waiting for storaged process..."; sleep 2
              done
              sleep 10
              exec /shared/agent \
                --agent=$(hostname).nebula-storaged-headless.nebula.svc.cluster.local:8888 \
                --debug --ratelimit=0 --hbs=10 \
                --meta=nebula-metad-headless.nebula.svc.cluster.local:9559
          ports:
            - containerPort: 8888
              name: grpc
          volumeMounts:
            - mountPath: /usr/local/nebula/data
              name: storaged-data
              subPath: data
            - mountPath: /usr/local/nebula/etc/nebula-storaged.conf
              name: nebula-storaged-conf
              subPath: nebula-storaged.conf
            - mountPath: /usr/local/nebula/scripts
              name: agent-scripts-vol
            - mountPath: /shared
              name: shared-bin
      volumes:
        - name: nebula-storaged-conf
          configMap:
            defaultMode: 420
            name: nebula-storaged
            items:
              - key: nebula-storaged.conf
                path: nebula-storaged.conf
        - name: agent-scripts-vol
          configMap:
            defaultMode: 0755
            name: nebula-agent-scripts
        - name: shared-bin
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: storaged-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: csi-cinder-high-speed
---
apiVersion: v1
kind: List
metadata:
  resourceVersion: ""
items:
- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: nebula-storaged
    namespace: nebula
  data:
    nebula-storaged.conf: |
      --daemonize=true
      --pid_file=pids/nebula-storaged.pid
      --local_config=true
      --log_dir=logs
      --minloglevel=0
      --v=0
      --logbufsecs=0
      --redirect_stdout=false
      --stdout_log_file=storaged-stdout.log
      --stderr_log_file=storaged-stderr.log
      --stderrthreshold=0
      --timestamp_in_logfile_name=true
      --meta_server_addrs=127.0.0.1:9559
      --local_ip=127.0.0.1
      --port=9779
      --ws_ip=0.0.0.0
      --ws_http_port=19779
      --heartbeat_interval_secs=10
      --raft_heartbeat_interval_secs=30
      --raft_rpc_timeout_ms=500
      --wal_ttl=14400
      --snapshot_send_files=true
      --data_path=data/storage
      --minimum_reserved_bytes=268435456
      --rocksdb_batch_size=4096
      --rocksdb_block_cache=4096
      --disable_page_cache=false
      --engine_type=rocksdb
      --rocksdb_compression=lz4
      --rocksdb_compression_per_level=
      --enable_rocksdb_statistics=false
      --rocksdb_stats_level=kExceptHistogramOrTimers
      --enable_rocksdb_prefix_filtering=true
      --enable_rocksdb_whole_key_filtering=false
      --rocksdb_db_options={}
      --rocksdb_column_family_options={"write_buffer_size":"67108864","max_write_buffer_number":"4","max_bytes_for_level_base":"268435456"}
      --rocksdb_block_based_table_options={"block_size":"8192"}
      --enable_storage_cache=false
      --storage_cache_capacity=0
      --storage_cache_entries_power=20
      --enable_vertex_pool=false
      --vertex_pool_capacity=50
      --vertex_item_ttl=300
      --enable_negative_pool=false
      --negative_pool_capacity=50
      --negative_item_ttl=300
      --query_concurrently=true
      --auto_remove_invalid_space=true
      --num_io_threads=16
      --num_max_connections=0
      --num_worker_threads=32
      --max_concurrent_subtasks=10
      --snapshot_part_rate_limit=10485760
      --snapshot_batch_size=1048576
      --rebuild_index_part_rate_limit=4194304
      --rebuild_index_batch_size=1048576
      --nv_cache_path=/tmp/cache
      --nv_cache_size=0
      --nv_dram_size=50
      --nv_bucket_power=20
      --nv_lock_power=10
      --ng_black_box_switch=false
      --ng_black_box_home=black_box
      --ng_black_box_dump_period_seconds=5
      --ng_black_box_file_lifetime_seconds=1800
      --memory_tracker_limit_ratio=0.8
      --memory_tracker_untracked_reserved_memory_mb=50
      --memory_tracker_detail_log=false
      --memory_tracker_detail_log_interval_ms=60000
      --memory_purge_enabled=true
      --memory_purge_interval_seconds=10
      --enable_ssl=false
      --enable_meta_ssl=false
      --cert_path=certs/server.crt
      --key_path=certs/server.key
      --ca_path=certs/ca.crt
      --ca_client_path=certs/ca.crt
      --password_path=certs/password
      --ssl_watch_path=certs
      --logtostderr=true

- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: nebula-metad
    namespace: nebula
  data:
    nebula-metad.conf: |
      --daemonize=true
      --pid_file=pids/nebula-metad.pid
      --log_dir=logs
      --minloglevel=0
      --v=0
      --logbufsecs=0
      --redirect_stdout=false
      --stdout_log_file=metad-stdout.log
      --stderr_log_file=metad-stderr.log
      --stderrthreshold=0
      --timestamp_in_logfile_name=true
      --meta_server_addrs=127.0.0.1:9559
      --local_ip=127.0.0.1
      --port=9559
      --ws_ip=0.0.0.0
      --ws_http_port=19559
      --ws_storage_http_port=19779
      --data_path=data/meta
      --minimum_reserved_bytes=268435456
      --default_parts_num=100
      --default_replica_factor=3
      --heartbeat_interval_secs=10
      --agent_heartbeat_interval_secs=60
      --rocksdb_wal_sync=true
      --ng_black_box_switch=false
      --ng_black_box_home=black_box
      --ng_black_box_dump_period_seconds=5
      --ng_black_box_file_lifetime_seconds=1800
      --enable_ssl=false
      --enable_meta_ssl=false
      --cert_path=certs/server.crt
      --key_path=certs/server.key
      --ca_path=certs/ca.crt
      --ca_client_path=certs/ca.crt
      --password_path=certs/password
      --ssl_watch_path=certs
      --logtostderr=true

- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: nebula-graphd
    namespace: nebula
  data:
    nebula-graphd.conf: |
      --daemonize=true
      --pid_file=pids/nebula-graphd.pid
      --enable_optimizer=true
      --default_charset=utf8
      --default_collate=utf8_bin
      --heartbeat_interval_secs=10
      --local_config=true
      --log_dir=logs
      --minloglevel=0
      --v=0
      --logbufsecs=0
      --redirect_stdout=false
      --stdout_log_file=graphd-stdout.log
      --stderr_log_file=graphd-stderr.log
      --stderrthreshold=0
      --timestamp_in_logfile_name=true
      --accept_partial_success=false
      --max_allowed_query_size=4194304
      --meta_server_addrs=127.0.0.1:9559
      --local_ip=127.0.0.1
      --listen_netdev=any
      --port=9669
      --reuse_port=false
      --listen_backlog=1024
      --client_idle_timeout_secs=28800
      --session_idle_timeout_secs=28800
      --num_accept_threads=1
      --num_netio_threads=0
      --num_max_connections=0
      --num_worker_threads=0
      --ws_ip=0.0.0.0
      --ws_http_port=19669
      --storage_client_timeout_ms=60000
      --enable_record_slow_query=true
      --slow_query_limit=100
      --slow_query_threshold_us=200000
      --ws_meta_http_port=19559
      --enable_authorize=true
      --auth_type=password
      --system_memory_high_watermark_ratio=0.8
      --enable_audit=false
      --audit_log_handler=file
      --audit_log_file=./logs/audit/audit.log
      --audit_log_strategy=synchronous
      --audit_log_max_buffer_size=1048576
      --audit_log_format=xml
      --audit_log_es_address=
      --audit_log_es_user=
      --audit_log_es_password=
      --audit_log_es_batch_size=1000
      --audit_log_exclude_spaces=
      --audit_log_categories=login,exit
      --enable_space_level_metrics=true
      --enable_experimental_feature=false
      --ng_black_box_switch=false
      --ng_black_box_home=black_box
      --ng_black_box_dump_period_seconds=5
      --ng_black_box_file_lifetime_seconds=1800
      --max_sessions_per_ip_per_user=300
      --memory_tracker_limit_ratio=0.8
      --memory_tracker_untracked_reserved_memory_mb=50
      --memory_tracker_detail_log=false
      --memory_tracker_detail_log_interval_ms=60000
      --memory_purge_enabled=true
      --memory_purge_interval_seconds=10
      --max_job_size=1
      --min_batch_size=8192
      --optimize_appendvertices=false
      --path_batch_size=10000
      --enable_http2_routing=false
      --stream_timeout_ms=30000
      --enable_ssl=false
      --enable_graph_ssl=false
      --enable_meta_ssl=false
      --cert_path=certs/server.crt
      --key_path=certs/server.key
      --ca_path=certs/ca.crt
      --ca_client_path=certs/ca.crt
      --password_path=certs/password
      --ssl_watch_path=certs
      --logtostderr=true

- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: nebula-agent-scripts
    namespace: nebula
  data:
    nebula.service: |
      #!/bin/bash
      ACTION=$1
      ROLE=$2
      DAEMON="nebula-$ROLE"
      DAEMON_BIN="/usr/local/nebula/bin/$DAEMON"

      if [ "$ACTION" == "status" ]; then
          if pgrep -f "$DAEMON_BIN" > /dev/null; then
              echo -e "\033[0;32m[INFO] \033[0m ${DAEMON}(): Running"
              exit 0
          else
              echo -e "\033[0;32m[INFO] \033[0m ${DAEMON}(): Exited"
              exit 0
          fi

      elif [ "$ACTION" == "stop" ]; then
          echo -e "\033[0;32m[INFO] \033[0m Stopping ${DAEMON}..."
          pkill -15 -f "$DAEMON_BIN" || true
          sleep 10
          rm -f /usr/local/nebula/pids/${DAEMON}.pid
          echo -e "\033[0;32m[INFO] \033[0m Done"
          exit 0

      elif [ "$ACTION" == "start" ]; then
          echo -e "\033[0;32m[INFO] \033[0m Starting ${DAEMON}..."
          
          if [ "$ROLE" == "metad" ]; then
              rm -f /usr/local/nebula/pids/${DAEMON}.pid
              /usr/local/nebula/bin/nebula-metad \
                --flagfile=/usr/local/nebula/etc/nebula-metad.conf \
                --meta_server_addrs=nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559 \
                --local_ip=$(hostname).nebula-metad-headless.nebula.svc.cluster.local \
                --daemonize=true
              until bash -c "echo >/dev/tcp/localhost/9559" 2>/dev/null; do
                  echo "waiting for metad to be ready..."; sleep 3
              done

          elif [ "$ROLE" == "storaged" ]; then
              rm -f /usr/local/nebula/pids/${DAEMON}.pid
              /usr/local/nebula/bin/nebula-storaged \
                --flagfile=/usr/local/nebula/etc/nebula-storaged.conf \
                --meta_server_addrs=nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559 \
                --local_ip=$(hostname).nebula-storaged-headless.nebula.svc.cluster.local \
                --daemonize=true
              until bash -c "echo >/dev/tcp/localhost/9779" 2>/dev/null; do
                  echo "waiting for storaged to be ready..."; sleep 3
              done

          elif [ "$ROLE" == "graphd" ]; then
              /usr/local/nebula/bin/nebula-graphd \
                --flagfile=/usr/local/nebula/etc/nebula-graphd.conf \
                --meta_server_addrs=nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559 \
                --local_ip=$(hostname).nebula-graphd-headless.nebula.svc.cluster.local \
                --daemonize=true
          fi

          sleep 5
          echo -e "\033[0;32m[INFO] \033[0m Done"
          exit 0
      fi
      exit 0

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects/nonePR/issue: this bug affects none version.severity/noneSeverity of bugtype/bugType: something is unexpected

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions