INFORMATION_SCHEMA discovery script should be SET based not RBAR
Which destination?:
Any PostgreSQL Destination (presumably this behavior is similar on any SQL database destination except for Oracle)
Additional details:
The following query is executed per table at every sync job.
/*Fivetran*/
SELECT
c.COLUMN_NAME
, c.DATA_TYPE
, c.NUMERIC_PRECISION
, c.NUMERIC_SCALE
, c.CHARACTER_MAXIMUM_LENGTH
, c.IS_NULLABLE
, k.COLUMN_NAME AS PRIMARY_KEY
FROM
INFORMATION_SCHEMA.COLUMNS c
LEFT JOIN INFORMATION_SCHEMA.TABLE_CONSTRAINTS t ON
c.TABLE_SCHEMA = t.TABLE_SCHEMA
AND c.TABLE_NAME = t.TABLE_NAME
AND t.CONSTRAINT_TYPE = $1
LEFT JOIN INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE k ON
c.TABLE_SCHEMA = k.TABLE_SCHEMA
AND c.TABLE_NAME = k.TABLE_NAME
AND c.COLUMN_NAME = k.COLUMN_NAME
AND t.CONSTRAINT_NAME = k.CONSTRAINT_NAME
WHERE
c.TABLE_SCHEMA = $2
AND c.TABLE_NAME = $3
It is RBAR by nature for no reason that I can understand. This should be set based and executed once at the start of the sync job instead. The results could be limited by providing a discreet list of schema names.
Effectively, grab all the data needed in one swoop (being sure to add the schema names and table names ) and then filter accordingly on the client side. Any modern programming language should be able to consume the results and compare/contrast against a control list with ease.
If there is concern about seeing more tables/columns than are necessary: In PostgreSQL, the results from the INFORMATION_SCHEMA are based on what the role executing the query can access. To the best of my knowledge this applies to most other database platforms with the exception of Oracle that does not even implement INFORMATION_SCHEMA.
It is possible that the role used in a Destination might have higher than necessary privileges or god-like powers ( i.e. superuser in PostgreSQL, sysadmin in SQL Server, etc.). A query without predicate conditions would return everything in such scenarios. However, there is a considerably higher risk in cases such as that and the customer should be alerted to it.
Please sign in to leave a comment.
Comments
0 comments