BladePipe
BladePipe is a real-time end-to-end data replication tool, moving data between 30+ databases, message queues, search engines, caching, real-time data warehouses, data lakes and more, with ultra-low latency less than 3 seconds. It features efficiency, stability and scalability, compatibility with diverse database engines, one-stop management, enhanced security, and complex data transformation.
Functions
BladePipe presents a visual management interface, allowing you to easily create DataJobs to realize schema migration, data migration, synchronization, verification and correction, etc. In addition, more refined and customized configurations are supported by setting parameters. Now BladePipe supports data movement from the following source DataSources to Doris:
| Source DataSource | Schema Migration | Data Migration | Data Sync | Verification & Correction |
|---|---|---|---|---|
| MySQL/MariaDB/AuroraMySQL | Yes | Yes | Yes | Yes |
| Oracle | Yes | Yes | Yes | Yes |
| PostgreSQL/AuroraPostgreSQL | Yes | Yes | Yes | Yes |
| SQL Server | Yes | Yes | Yes | Yes |
| Kafka | No | No | Yes | No |
| AutoMQ | No | No | Yes | No |
| TiDB | Yes | Yes | Yes | Yes |
| Hana | Yes | Yes | Yes | Yes |
| PolarDB-X | Yes | Yes | Yes | Yes |
For more functions and parameter settings, please refer to BladePipe Connections.
Installation
Follow the instructions in Install Worker (Docker) or Install Worker (Binary) to download and install a BladePipe Worker.
Example
Taking a MySQL instance as an example, the following part describes how to move data from MySQL to Doris.
Add DataSources
Log in to the BladePipe Cloud. Click DataSource > Add DataSource.
Select MySQL and Doris as the Type respectively, and fill in the setup form accordingly.
Click Test Connection. After successful connection, click Add DataSource to add the DataSource.
Create a DataJob
Click DataJob > Create DataJob.
Select the source and target DataSources, and click Test Connection to ensure the connection to the source and target DataSources are both successful.
Select Incremental for DataJob Type, together with the Full Data option.
Select the tables to be replicated.
Select the columns to be replicated.
Confirm the DataJob creation.
The DataJob runs automatically. BladePipe will automatically run the following DataTasks:
- Schema Migration: The schemas of the source tables will be migrated to the target instance.
- Full Data: All existing data of the source tables will be fully migrated to the target instance.
- Incremental: Ongoing data changes will be continuously synchronized to the target instance.
