Metadata Operations and Maintenance

warning

Avoid using metadata_failure_recovery unless absolutely necessary. Using it may cause metadata truncation, loss, and split-brains. Use cautiously to prevent irreversible data damage from improper operations.

This document focuses on how to manage Doris metadata in a real production environment. It includes the proposed deployment of FE nodes, some commonly used operational methods, and common error resolution methods.

For the time being, read the Doris metadata design document to understand how Doris metadata works.

Important tips

Current metadata design is not backward compatible. That is, if the new version has a new metadata structure change (you can see whether there is a new VERSION in the FeMetaVersion.java file in the FE code), it is usually impossible to roll back to the old version after upgrading to the new version. Therefore, before upgrading FE, be sure to test metadata compatibility according to the operations in the Upgrade Document.

Metadata catalog structure

Let's assume that the path of meta_dir specified in fe.conf is path/to/doris-meta. In a normal Doris cluster, the directory structure of metadata should be as follows:

/path/to/doris-meta/
            |-- bdb/
            |   |-- 00000000.jdb
            |   |-- je.config.csv
            |   |-- je.info.0
            |   |-- je.info.0.lck
            |   |-- je.lck
            |   `-- je.stat.csv
            `-- image/
                |-- ROLE
                |-- VERSION
                `-- image.xxxx

bdb

We use bdbje as a distributed kV system to store metadata journal. This BDB directory is equivalent to the "data directory" of bdbje.

The .jdb suffix is the data file of bdbje. These data files will increase with the increasing number of metadata journals. When Doris regularly completes the image, the old log is deleted. So normally, the total size of these data files varies from several MB to several GB (depending on how Doris is used, such as import frequency). When the total size of the data file is larger than 10GB, you may need to wonder whether the image failed or the historical journals that failed to distribute the image could not be deleted.

je.info.0 is the running log of bdbje. The time in this log is UTC + 0 time zone. We may fix this in a later version. From this log, you can also see how some bdbje works.
image directory

The image directory is used to store metadata mirrors generated regularly by Doris. Usually, you will see a image.xxxxx mirror file. Where xxxxx is a number. This number indicates that the image contains all metadata journal before xxxx. And the generation time of this file (viewed through ls -al) is usually the generation time of the mirror.

You may also see a image.ckpt file. This is a metadata mirror being generated. The du -sh command should show that the file size is increasing, indicating that the mirror content is being written to the file. When the mirror is written, it automatically renames itself to a new image.xxxxx and replaces the old image file.

Only FE with a Master role will actively generate image files on a regular basis. After each generation, FE is pushed to other non-Master roles. When it is confirmed that all other FEs have received this image, Master FE deletes the metadata journal in bdbje. Therefore, if image generation fails or image push fails to other FEs, data in bdbje will accumulate.

ROLE file records the type of FE (FOLLOWER or OBSERVER), which is a text file.

VERSION file records the cluster ID of the Doris cluster and the token used to access authentication between nodes, which is also a text file.

ROLE file and VERSION file may only exist at the same time, or they may not exist at the same time (e.g. at the first startup).

Basic operations

Start single node FE

Single node FE is the most basic deployment mode. A complete Doris cluster requires at least one FE node. When there is only one FE node, the type of the node is Follower and the role is Master.

First start-up
1. Suppose the path of meta_dir specified in fe.conf is path/to/doris-meta.
2. Ensure that path/to/doris-meta already exists, that the permissions are correct and that the directory is empty.
3. Start directly through sh bin/start_fe.sh.
4. After booting, you should be able to see the following log in fe.log:
  - Palo FE starting...
  - image does not exist: /path/to/doris-meta/image/image.0
  - transfer from INIT to UNKNOWN
  - transfer from UNKNOWN to MASTER
  - the very first time to open bdb, dbname is 1
  - start fencing, epoch number is 1
  - finish replay in xxx msec
  - QE service start
  - thrift server started
  The above logs are not necessarily strictly in this order, but they are basically similar.
5. The first start-up of a single-node FE usually does not encounter problems. If you haven't seen the above logs, generally speaking, you haven't followed the document steps carefully, please read the relevant wiki carefully.
Restart
1. Stopped FE nodes can be restarted by using sh bin/start_fe.sh.
2. After restarting, you should be able to see the following log in fe.log:
  - Palo FE starting...
  - finished to get cluster id: xxxx, role: FOLLOWER and node name: xxxx
  - If no image has been generated before reboot, you will see:
  - image does not exist: /path/to/doris-meta/image/image.0
  - If an image is generated before the restart, you will see:
  - start load image from /path/to/doris-meta/image/image.xxx. is ckpt: false
  - finished load image in xxx ms
  - transfer from INIT to UNKNOWN
  - replayed journal id is xxxx, replay to journal id is yyyy
  - transfer from UNKNOWN to MASTER
  - finish replay in xxx msec
  - master finish replay journal, can write now.
  - begin to generate new image: image.xxxx
  - start save image to /path/to/doris-meta/image/image.ckpt. is ckpt: true
  - finished save image /path/to/doris-meta/image/image.ckpt in xxx ms. checksum is xxxx
  - push image.xxx to other nodes. totally xx nodes, push successed xx nodes
  - QE service start
  - thrift server started
  The above logs are not necessarily strictly in this order, but they are basically similar.
Common problems

For the deployment of single-node FE, start-stop usually does not encounter any problems. If you have any questions, please refer to the relevant Wiki and check your operation steps carefully.

Add FE

Adding FE processes is described in detail in the Elastic Expansion Documents and will not be repeated. Here are some points for attention, as well as common problems.

Notes
- Before adding a new FE, make sure that the current Master FE runs properly (connection is normal, JVM is normal, image generation is normal, bdbje data directory is too large, etc.)
- The first time you start a new FE, you must make sure that the --helper parameter is added to point to Master FE. There is no need to add --helper when restarting. (If --helper is specified, FE will directly ask the helper node for its role. If not, FE will try to obtain information from ROLE and VERSION files in the doris-meta/image/ directory.
- The first time you start a new FE, you must make sure that the meta_dir of the FE is created, has correct permissions and is empty.
- Starting a new FE and executing the ALTER SYSTEM ADD FOLLOWER/OBSERVER statement adds FE to metadata in a sequence that is not required. If a new FE is started first and no statement is executed, the current node is not added to the group. Please add it first. in the new FE log. When the statement is executed, it enters the normal process.
- Make sure that after the previous FE is added successfully, the next FE is added.
- Connect to MASTER FE and execute ALTER SYSTEM ADD FOLLOWER/OBSERVER stmt.
Common problems
1. this need is DETACHED
  
  When you first start a FE to be added, if the data in doris-meta/bdb on Master FE is large, you may see the words this node is DETACHED. in the FE log to be added. At this point, bdbje is copying data, and you can see that the bdb/ directory of FE to be added is growing. This process usually takes several minutes (depending on the amount of data in bdbje). Later, there may be some bdbje-related error stack information in fe. log. If QE service start and thrift server start are displayed in the final log, the start is usually successful. You can try to connect this FE via mysql-client. If these words do not appear, it may be the problem of bdbje replication log timeout. At this point, restarting the FE directly will usually solve the problem.
2. Failure to add due to various reasons
  - If OBSERVER is added, because OBSERVER-type FE does not participate in the majority of metadata writing, it can theoretically start and stop at will. Therefore, for the case of adding OBSERVER failure. The process of OBSERVER FE can be killed directly. After clearing the metadata directory of OBSERVER, add the process again.
  - If FOLLOWER is added, because FOLLOWER is mostly written by participating metadata. So it is possible that FOLLOWER has joined the bdbje electoral team. If there are only two FOLLOWER nodes (including MASTER), then stopping one FE may cause another FE to quit because it cannot write most of the time. At this point, we should first delete the newly added FOLLOWER node from the metadata through the ALTER SYSTEM DROP FOLLOWER command, then kill the FOLLOWER process, empty the metadata and re-add the process.

Delete FE

The corresponding type of FE can be deleted by the ALTER SYSTEM DROP FOLLOWER/OBSERVER command. The following points should be noted:

For OBSERVER type FE, direct DROP is enough, without risk.
For FOLLOWER type FE. First, you should make sure that you start deleting an odd number of FOLLOWERs (three or more).
1. If the FE of non-MASTER role is deleted, it is recommended to connect to MASTER FE, execute DROP command, and then kill the process.
2. If you want to delete MASTER FE, first confirm that there are odd FOLLOWER FE and it works properly. Then kill the MASTER FE process first. At this point, a FE will be elected MASTER. After confirming that the remaining FE is working properly, connect to the new MASTER FE and execute the DROP command to delete the old MASTER FE.

Advanced Operations

Failure recovery

FE may fail to start bdbje and synchronize between FEs for some reasons. Phenomena include the inability to write metadata, the absence of MASTER, and so on. At this point, we need to manually restore the FE. The general principle of manual recovery of FE is to start a new MASTER through metadata in the current meta_dir, and then add other FEs one by one. Please follow the following steps strictly:

First, stop all FE processes and all business access. Make sure that during metadata recovery, external access will not lead to other unexpected problems.(if not, this may cause split-brain problem)
Identify which FE node's metadata is up-to-date:
- First of all, be sure to back up all FE's meta_dir directories first.
- Usually, Master FE's metadata is up to date. You can see the suffix of image.xxxx file in the meta_dir/image directory. The larger the number, the newer the metadata.
- Usually, by comparing all FOLLOWER FE image files, you can find the latest metadata.
- After that, we use the FE node with the latest metadata to recover.
- If using metadata of OBSERVER node to recover will be more troublesome, it is recommended to choose FOLLOWER node as far as possible.
The following operations are performed on the FE nodes selected in step 2.
1. Modify fe.conf
  - If the node is an OBSERVER, first change the role=OBSERVER in the meta_dir/image/ROLE file to role=FOLLOWER. (Recovery from the OBSERVER node will be more cumbersome, first follow the steps here, followed by a separate description)
  - If fe.version < 2.0.2, add configuration in fe.conf: metadata_failure_recovery=true.
2. Run sh bin/start_fe.sh --metadata_failure_recovery --daemon to start the FE. (If you are recovering from an OBSERVER node, jump to the subsequent OBSERVER document after this step.)
3. If normal, the FE will start in the role of MASTER, similar to the description in the previous section Start a single node FE. You should see the words transfer from XXXX to MASTER in fe.log.
4. After the start-up is completed, connect to the FE first, and execute some query imports to check whether normal access is possible. If the operation is not normal, it may be wrong. It is recommended to read the above steps carefully and try again with the metadata previously backed up. If not, the problem may be more serious.
5. If successful, through the show frontends; command, you should see all the FEs you added before, and the current FE is master.
6. If FE version < 2.0.2, delete the metadata_failure_recovery=true configuration item in fe.conf, or set it to false, and restart the FE (Important).
tip
If you are recovering metadata from an OBSERVER node, after completing the above steps, you will find that the current FE role is OBSERVER, but IsMaster appears as true. This is because the "OBSERVER" seen here is recorded in Doris's metadata, but whether it is master or not, is recorded in bdbje's metadata. Because we recovered from an OBSERVER node, there was inconsistency. Please take the following steps to fix this problem (we will fix it in a later version):
1. First, all FE nodes except this "OBSERVER" are DROPed out.
2. A new FOLLOWER FE is added through the ADD FOLLOWER command, assuming that it is on hostA.
3. Start a new FE on hostA and join the cluster by helper.
4. After successful startup, you should see two FEs through the show frontends; statement, one is the previous OBSERVER, the other is the newly added FOLLOWER, and the OBSERVER is the master.
5. After confirming that the new FOLLOWER is working properly, the new FOLLOWER metadata is used to perform a failure recovery operation again.
6. The purpose of the above steps is to manufacture a metadata of FOLLOWER node artificially, and then use this metadata to restart fault recovery. This avoids inconsistencies in recovering metadata from OBSERVER.
The meaning of metadata_failure_recovery is to empty the metadata of bdbje. In this way, bdbje will not contact other FEs before, but start as a separate FE. This parameter needs to be set to true only when restoring startup. After recovery, it must be set to false. Otherwise, once restarted, the metadata of bdbje will be emptied again, which will make other FEs unable to work properly.
After the successful execution of step 3, we delete the previous FEs from the metadata by using the ALTER SYSTEM DROP FOLLOWER/OBSERVER command and add them again by adding new FEs.
If the above operation is normal, it will be restored.

FE type change

If you need to change the existing FOLLOWER/OBSERVER type FE to OBSERVER/FOLLOWER type, please delete FE in the way described above, and then add the corresponding type FE.

FE Migration

If you need to migrate one FE from the current node to another, there are several scenarios.

FOLLOWER, or OBSERVER migration for non-MASTER nodes

After adding a new FOLLOWER / OBSERVER directly, delete the old FOLLOWER / OBSERVER.
Single-node MASTER migration

When there is only one FE, refer to the Failure Recovery section. Copy the doris-meta directory of FE to the new node and start the new MASTER in Step 3 of the Failure Recovery section
A set of FOLLOWER migrates from one set of nodes to another set of new nodes

Deploy FE on the new node and add the new node first by adding FOLLOWER. The old nodes can be dropped by DROP one by one. In the process of DROP-by-DROP, MASTER automatically selects the new FOLLOWER node.

Replacement of FE port

FE currently has the following ports

Ed_log_port: bdbje's communication port
http_port: http port, also used to push image
rpc_port: thrift server port of Frontend
query_port: Mysql connection port
arrow_flight_sql_port: Arrow Flight SQL connection port

edit_log_port

If this port needs to be replaced, it needs to be restored with reference to the operations in the Failure Recovery section. Because the port has been persisted into bdbje's own metadata (also recorded in Doris's own metadata), it is necessary to clear bdbje's metadata by setting metadata_failure_recovery when Fe start.
http_port

All FE http_ports must be consistent. So if you want to modify this port, all FEs need to be modified and restarted. Modifying this port will be more complex in the case of multiple FOLLOWER deployments (involving laying eggs and laying hens...), so this operation is not recommended. If necessary, follow the operation in the Failure Recovery section directly.
rpc_port

After modifying the configuration, restart FE directly. Master FE informs BE of the new port through heartbeat. Only this port of Master FE will be used. However, it is still recommended that all FE ports be consistent.
query_port

After modifying the configuration, restart FE directly. This only affects mysql's connection target.
arrow_flight_sql_port

After modifying the configuration, restart FE directly. This only affects arrow flight sql server connection target.

Recover metadata from FE memory

In some extreme cases, the image file on the disk may be damaged, but the metadata in the memory is intact. At this point, we can dump the metadata from the memory and replace the image file on the disk to recover the metadata. the entire non-stop query service operation steps are as follows:

Stop all Load, Create, Alter operations.
Execute the following command to dump metadata from the Master FE memory: (hereafter called image_mem)

curl -u $root_user:$password http://$master_hostname:8030/dump

Replace the image file in the meta_dir/image directory on the OBSERVER FE node with the image_mem file, restart the OBSERVER FE node, and verify the integrity and correctness of the image_mem file. You can check whether the DB and Table metadata are normal on the FE Web page, whether there is an exception in fe.log, whether it is in a normal replayed jour.

Since 1.2.0, it is recommanded to use following method to verify the image_mem file:
```
sh start_fe.sh --image path_to_image_mem
```
Notice: path_to_image_mem is the path of image_mem.

If verify succeed, it will print: Load image success. Image file /absolute/path/to/image.xxxxxx is valid.

If verify failed, it will print: Load image failed. Image file /absolute/path/to/image.xxxxxx is invalid.
Replace the image file in the meta_dir/image directory on the FOLLOWER FE node with the image_mem file in turn, restart the FOLLOWER FE node, and confirm that the metadata and query services are normal.
Replace the image file in the meta_dir/image directory on the Master FE node with the image_mem file, restart the Master FE node, and then confirm that the FE Master switch is normal and The Master FE node can generate a new image file through checkpoint.
Recover all Load, Create, Alter operations.

Note: If the Image file is large, the entire process can take a long time, so during this time, make sure Master FE does not generate a new image file via checkpoint. When the image.ckpt file in the meta_dir/image directory on the Master FE node is observed to be as large as the image.xxx file, the image.ckpt file can be deleted directly.

View data in BDBJE

The metadata log of FE is stored in BDBJE in the form of Key-Value. In some abnormal situations, FE may not be started due to metadata errors. In this case, Doris provides a way to help users query the data stored in BDBJE to facilitate troubleshooting.

First, you need to add configuration in fe.conf: enable_bdbje_debug_mode=true, and then start FE through sh start_fe.sh --daemon.

At this time, FE will enter the debug mode, only start the http server and MySQL server, and open the BDBJE instance, but will not load any metadata and other subsequent startup processes.

This is, we can view the data stored in BDBJE by visiting the web page of FE, or after connecting to Doris through the MySQL client, through show proc "/bdbje";.

mysql> show proc "/bdbje";
+----------+---------------+---------+
| DbNames  | JournalNumber | Comment |
+----------+---------------+---------+
| 110589   | 4273          |         |
| epochDB  | 4             |         |
| metricDB | 430694        |         |
+----------+---------------+---------+

The first level directory will display all the database names in BDBJE and the number of entries in each database.

mysql> show proc "/bdbje/110589";
+-----------+
| JournalId |
+-----------+
| 1         |
| 2         |

...
| 114858    |
| 114859    |
| 114860    |
| 114861    |
+-----------+
4273 rows in set (0.06 sec)

Entering the second level, all the entry keys under the specified database will be listed.

mysql> show proc "/bdbje/110589/114861";
+-----------+--------------+---------------------------------------------+
| JournalId | OpType       | Data                                        |
+-----------+--------------+---------------------------------------------+
| 114861    | OP_HEARTBEAT | org.apache.doris.persist.HbPackage@6583d5fb |
+-----------+--------------+---------------------------------------------+
1 row in set (0.05 sec)

The third level can display the value information of the specified key.

Best Practices

The deployment recommendation of FE is described in the Installation and Deployment Document. Here are some supplements.

If you don't know the operation logic of FE metadata very well, or you don't have enough experience in the operation and maintenance of FE metadata, we strongly recommend that only one FOLLOWER-type FE be deployed as MASTER in practice, and the other FEs are OBSERVER, which can reduce many complex operation and maintenance problems. Don't worry too much about the failure of MASTER single point to write metadata. First, if you configure it properly, FE as a java process is very difficult to hang up. Secondly, if the MASTER disk is damaged (the probability is very low), we can also use the metadata on OBSERVER to recover manually through fault recovery.
The JVM of the FE process must ensure sufficient memory. We strongly recommend that FE's JVM memory should be at least 10GB and 32GB to 64GB. And deploy monitoring to monitor JVM memory usage. Because if OOM occurs in FE, metadata writing may fail, resulting in some failures that cannot recover!
FE nodes should have enough disk space to prevent the excessive metadata from causing insufficient disk space. At the same time, FE logs also take up more than a dozen gigabytes of disk space.

Important tips​

Metadata catalog structure​

Basic operations​

Start single node FE​

Add FE​

Delete FE​

Advanced Operations​

Failure recovery​

FE type change​

FE Migration​

Replacement of FE port​

Recover metadata from FE memory​

View data in BDBJE​

Best Practices​

Other common problems​