Skip to content

Data Types

Jump to bottom
dandanlen edited this page Jan 7, 2020 · 1 revision

Supported data types

The following data types are supported by Diffix and must be considered by the explorer:

  integer | real | text | boolean | datetime | date | time

Continuous vs Categorical

Most data types can be considered either continuous or categorical based on context.

For example:

  • the type of a gender column might be text, but there may only be two possible values, MALE and FEMALE.
  • a month column may have type integer and categorical values 1-12

The distinction is not always clear: we will need to use some heuristics to determine which exploration approach to use for each column.

Numerical

Numerical columns (integer and real) can be analysed using a bucketing approach to extract histograms.

Text

For text columns, useful metrics may be word count, prefix, postfix, or substring counts, and other patterns for example email domains, postcodes, etc.

We can take inspiration from the ColumnPatternExplorer (https://github.com/diffix/ColumnPatternExplorer).

Date / Time / Datetime

These can be analysed similarly to numeric types with bucketing, but with the added complexity of date ranges...

Clone this wiki locally