Overview

A new Deduplication feature node has been introduced under the Data Preparation module to identify and remove duplicate records based on full rows or selected columns.

Feature Enhancements

• Added Deduplication option under the Data Preparation module.

• Users can remove duplicates using entire rows or selected columns.

• Multiple column selection supported for deduplication.

• Support added for Keep First, Keep Last, and Remove All duplicate handling strategies.

• Dataset validation added before execution.

• Record count updates after deduplication execution.

• All predecessor columns are retained in the output dataset.

Deduplication Module

Deduplication option available under Data Preparation.

Deduplication Configuration

Users can configure duplicate handling columns and strategies.

Benefits

• Improved data quality and consistency.

• Flexible duplicate handling options.

• Better preprocessing support for analytics workflows.

• Simplified duplicate record management.