Skip to main content

Community

Connector Improvement: HVR : Resilience to ignore data difference between source and Target and process only existing data

Answered

Please sign in to leave a comment.

Comments

6 comments

  • Mark Van de Wiel User

    Tushar,

    HVR in regular mode is designed to get data in two or more systems in sync. The foundation is a table definition with a key to identify a row.

    Resilient processing is available to deal with discrepancies in the initial state of the data to get toward a synchronized data set:

    • If HVR wants to insert a row but another row with the key already exists then we update the row instead. Note that per the key definition for the table only one row can have the particular key value.
    • If HVR wants to update a row that does not exist then the row is created instead.
    • If HVR wants to delete a row and there is no row with such key then we ignore the delete.

    For replication to be correct it is important we have a key definition. If there is no key then HVR by default allows duplicate rows (the behavior of which you can influence with TableProperties NoDuplicateRows).

    It feels like you may have a very unique use case for which you have a particular behavior in mind. However, it is unclear to me how you want to combine a core concept like row identification (key) with existing data, and how HVR should process increments. Can you please clarify?

    Note soft delete and timekey mode are somewhat different; if you want the details on these then let me know.

    Thanks,
    Mark.

    Hi Mark 

     

    The issue comes when the data is mismatch between source and target,

    Main issue is there when an extra record at target where we need to keep historic data  or on target records are coming from various source and clubbed into one table.

    This request to have an additional option where "update affected 0 rows; less than the 1 row expected" like scenario we don't want that HVR to add/ modify / delete data and just if no rows there don't insert, we may not need that row on target,

  • Mark Van de Wiel User

    With data coming from multiple sources you would want to add a column on the target storing the origin (to keep rows unique).

    E.g. add ColumnProperties Name=source_system Extra IntegrateExpression={hvr_cap_loc} DataType=varchar Length=12 Key

    Subsequent to this you can have values from alternative sources with duplicate keys.

    Or if you want to follow historical changes then I recommend you add a column TimeKey to translate all operations into inserts.

    Thanks,
    Mark.

    Hi Mark 

    user don't want to manipulate with data .  Do not insert/update / delete any record which is not exists. Any Structure change means change in all business logic. We just need a functionality that do not modify data which is extra on target or missing on target. we just need a functionality that if data does not exists for update just update what is there, do not try to add / remove missing or extra data.

    We should leave sync of data call to user as data is their responsibility, they are comparing Sybase Replication Server functionality where if there is no or missing data depend upon PK it do that change only  

  • Mark Van de Wiel User

    Tushar,

    Like I said HVR was not designed for the scenario you describe. Honestly, in 11+ years at HVR you are the first person to ask for the capability. Given such lack of demand it is very hard to justify introducing a custom behavior that nobody else needs.

    Anyway let me discuss with engineering and see what an estimated effort would be. A small effort naturally would be easier to justify.

    Thanks,
    Mark.

    Hi Mark 

    Thansk Mark for taking ahead.

    You may be correct that not many is asking, but the complexity of data what Morgan process is raising this request. I have few user base who is looking for this. As it may be one organization but in that we have few requests where this functionality is needed. changing table structure and introducing new columns or introducing workaround is risk for human error during setup.

    for various business reason some data is extra added or removed from target and replication is causing data issue. Their main ask is to send data, nature of data is app responsibility. 

     

     

    Thanks

    Tushar