Question

My connector skips data (records or whole files) when ingesting, but the data is correctly formatted. Why is this?

Environment

File connectors

Answer

If have checked that your connector is formatted correctly, you must verify that you are specifying the delimiter used in your CSV files.

Make sure you explicitly state the delimiter instead of choosing infer.

Cause

When inferring delimiters, we can sometimes have a false positive if your file uses other characters that are also commonly used as CSV file delimiters such as ; , or |