Skip to main content
Docs
Feature Requests Onboarding Trust Center Status Page

Community

Connector Improvement: source side filtering for JIRA source

Answered
Vaze, Prasad User

Problem statement~

"Null values essentially represent wasted MAR charges in Jira tables ISSUE_MULTISELECT_HISTORY, and ISSUE_FIELD_HISTORY. We are essentially paying for rows that contain incomplete or unusable data being replicated from these JIRA tables into Delta Lake target. 

Investigation results

ISSUE_FIELD_HISTORY = 51 Million MAR with Null value entries. 31 Million MAR with valid values. 

select count(*) from edl_raw.api_jira_default.issue_field_history where value is null -- 530471286 rows

select count(*) from edl_raw.api_jira_default.issue_field_history where value is NOT null --
31552747 rows
 
ISSUE_MULTISELECT_HISTORY = 86 Million MAR with Null value entries. 40 Million MAR with valid values. 
select count(*) from edl_raw.api_jira_default.issue_multiselect_history where value is null -- 86452602 rows

select count(*) from edl_raw.api_jira_default.issue_multiselect_history where value is NOT null -- 4066336 rows

Expectations~

"It would be beneficial if there was an option in the Jira connector to block null value entries from loading into target delta tables ISSUE_MULTISELECT_HISTORY, and ISSUE_FIELD_HISTORY."

Please sign in to leave a comment.

Comments

2 comments

Date Votes
  • Official comment
    Frank Fivetranner

    Hi Prasad, 

    Can you help provide an example of null values in the history for a specific issue field? The purpose of the history table is to provide a timeline of the field history and that could include null values if the field was null for a certain period of time. 

    Thanks, 

    Frank

  • Mo Agarwal User

    Frank, the record can be end-dated when a value goes from not null to null. This way we don't need a record when the value is null.
    Also, pivoting a single record into a key-value pair for each column also massively inflates row counts. If we needed reconstruct the record at a point in time, we need to unpivot the current layout. I'd rather just have an SCD2 style complete record than the key-pair setup.