Creating Your First Data Pipeline in Talend Cloud
Step 1: Open Pipeline Designer
Navigate to Pipeline Designer from the pull-down menu.
Click Add Pipeline to start creating a new data pipeline.
Step 2: Explore the Interface
Review the main interface:
o Canvas: The central area where you add components and processes for data
extraction, transformation, and loading (ETL).
o Preview Pane: At the bottom, shows live data previews at every step.
o Configuration Panel: On the right, used to set parameters and view feedback.
Step 3: Name the Pipeline and Select the Engine
Assign a name, such as Report Data Generation for Dashboard.
Choose a Cloud Engine to run the pipeline.
Step 4: Add the Source Data
Use the clean dataset created in the previous module as the source.
The data preview pane will show the dataset at the bottom of the screen.
Step 5: Add a Lookup Table
Add a tax rate lookup table to calculate state-wise tax rates for transactions:
o Click the plus button between the boxes to open the Add Processor screen.
o Choose the Join Processor to combine the sales data with the tax rates table.
o Configure the join by selecting the state column as the key from both tables.
Step 6: Preview the Joined Data
View the side-by-side comparison of the original and transformed data in the preview
pane.
Confirm that tax rate data has been successfully joined to each transaction.
Step 7: Add a Math Processor
Calculate the estimated tax for each transaction:
o Add a Math Processor.
o Select the amount field and apply the multiplication operator with the tax
column.
o Save the configuration.
The preview pane will now show the calculated tax column.
Step 8: Aggregate Data by State
To prepare for the dashboard, group transactions by state:
o Add an Aggregate Processor and set state as the group-by field.
o Set up operations such as:
Count: Total number of orders per state.
Max: Largest order amount per state.
Sum: Total tax collected per state.
Save the configuration and review the aggregated data in the preview pane.
Step 9: Address Data Issues
The preview highlights issues, such as invalid state codes like "DC" or incomplete
data.
Use Talend’s Data Preparation tools to fix these issues in the dataset.
Step 10: Write Data to the Destination
Write the final processed data to a MySQL database:
o Choose an existing database or configure a new one.
o Select the option to create a new table if it doesn’t exist.
Step 11: Run the Pipeline
Click Run to execute the pipeline.
Monitor real-time updates in the feedback pane, including logs and potential warnings
or errors.
The pipeline scales efficiently, even with large datasets.
Step 12: Review the Output
After the run completes, verify the final table in the database.
This table can now drive dashboards for the marketing team, providing insights into
customer data.
Conclusion
Congratulations! You have successfully created your first data pipeline.
The pipeline transforms raw sales data, enriches it with tax rates, and organizes it for
analytical use.
Next steps include experimenting with new data sources, optimizing workflows, and building
trust in your data processes.