Datawizz lets you configure endpoints to annonimize logs by removing personally identifiable information (PII) from the logs. This can be useful to ensure user privacy and comply with regulations like GDPR.
If there’s a high risk of users sending sensative PII to your AI system, you should consider enabling PII removal to ensure that this information is not stored in the logs. You can also select specific entity types to remove from logs to constrain PII removal scope.
Configuring PII Removal
To enable PII removal, turn on the Anonymize Logs option at the endpoint settings. You may also configure two additional options:
- Score Threshold: The minimum confidence score required to remove an entity from the log. If the entity has a confidence score below this threshold, it will not be removed - this is a number between 0 and 1, with 0 being the lowest confidence and 1 being the highest.
- Entities: a comma separated list of entity types to remove from the logs. If you leave this field blank, all entities will be removed. See supported entity types below.
List of supported entities
Global
| Entity Type | Description | Detection Method |
|---|
| CREDIT_CARD | A credit card number is between 12 to 19 digits. https://en.wikipedia.org/wiki/Payment_card_number | Pattern match and checksum |
| CRYPTO | A Crypto wallet number. Currently only Bitcoin address is supported | Pattern match, context and checksum |
| DATE_TIME | Absolute or relative dates or periods or times smaller than a day. | Pattern match and context |
| EMAIL_ADDRESS | An email address identifies an email box to which email messages are delivered | Pattern match, context and RFC-822 validation |
| IBAN_CODE | The International Bank Account Number (IBAN) is an internationally agreed system of identifying bank accounts across national borders to facilitate the communication and processing of cross border transactions with a reduced risk of transcription errors. | Pattern match, context and checksum |
| IP_ADDRESS | An Internet Protocol (IP) address (either IPv4 or IPv6). | Pattern match, context and checksum |
| NRP | A person’s Nationality, religious or political group. | Custom logic and context |
| LOCATION | Name of politically or geographically defined location (cities, provinces, countries, international regions, bodies of water, mountains | Custom logic and context |
| PERSON | A full person name, which can include first names, middle names or initials, and last names. | Custom logic and context |
| PHONE_NUMBER | A telephone number | Custom logic, pattern match and context |
| MEDICAL_LICENSE | Common medical license numbers. | Pattern match, context and checksum |
| URL | A URL (Uniform Resource Locator), unique identifier used to locate a resource on the Internet | Pattern match, context and top level url validation |
USA
| Entity Type | Description | Detection Method |
|---|
| US_BANK_NUMBER | A US bank account number is between 8 to 17 digits. | Pattern match and context |
| US_DRIVER_LICENSE | A US driver license according to https://ntsi.com/drivers-license-format/ | Pattern match and context |
| US_ITIN | US Individual Taxpayer Identification Number (ITIN). Nine digits that start with a “9” and contain a “7” or “8” as the 4 digit. | Pattern match and context |
| US_PASSPORT | A US passport number with 9 digits. | Pattern match and context |
| US_SSN | A US Social Security Number (SSN) with 9 digits. | Pattern match and context |
| Entity Type | Description | Detection Method |
|---|
| UK_NHS | A UK NHS number is 10 digits. | Pattern match, context and checksum |
| UK_NINO | UK National Insurance Number is a unique identifier used in the administration of National Insurance and tax. | Pattern match and context |
Spain
| Entity Type | Description | Detection Method |
|---|
| ES_NIF | A spanish NIF number (Personal tax ID) . | Pattern match, context and checksum |
| ES_NIE | A spanish NIE number (Foreigners ID card) . | Pattern match, context and checksum |
Italy
Poland
| Entity Type | Description | Detection Method |
|---|
| PL_PESEL | Polish PESEL number | Pattern match, context and checksum |
Singapore
| FieldType | Description | Detection Method |
|---|
| SG_NRIC_FIN | A National Registration Identification Card | Pattern match and context |
| SG_UEN | A Unique Entity Number (UEN) is a standard identification number for entities registered in Singapore. | Pattern match, context, and checksum |
Australia
| FieldType | Description | Detection Method |
|---|
| AU_ABN | The Australian Business Number (ABN) is a unique 11 digit identifier issued to all entities registered in the Australian Business Register (ABR). | Pattern match, context, and checksum |
| AU_ACN | An Australian Company Number is a unique nine-digit number issued by the Australian Securities and Investments Commission to every company registered under the Commonwealth Corporations Act 2001 as an identifier. | Pattern match, context, and checksum |
| AU_TFN | The tax file number (TFN) is a unique identifier issued by the Australian Taxation Office to each taxpaying entity | Pattern match, context, and checksum |
| AU_MEDICARE | Medicare number is a unique identifier issued by Australian Government that enables the cardholder to receive a rebates of medical expenses under Australia’s Medicare system | Pattern match, context, and checksum |
India
| FieldType | Description | Detection Method |
|---|
| IN_PAN | The Indian Permanent Account Number (PAN) is a unique 12 character alphanumeric identifier issued to all business and individual entities registered as Tax Payers. | Pattern match, context |
| IN_AADHAAR | Indian government issued unique 12 digit individual identity number | Pattern match, context, and checksum |
| IN_VEHICLE_REGISTRATION | Indian government issued transport (govt, personal, diplomatic, defence) vehicle registration number | Pattern match, context, and checksum |
| IN_VOTER | Indian Election Commission issued 10 digit alpha numeric voter id for all indian citizens (age 18 or above) | Pattern match, context |
| IN_PASSPORT | Indian Passport Number | Pattern match, Context |
Finland
| FieldType | Description | Detection Method |
|---|
| FI_PERSONAL_IDENTITY_CODE | The Finnish Personal Identity Code (Henkilötunnus) is a unique 11 character individual identity number. | Pattern match, context and custom logic. |
Credits
This feature is powered by Microsoft’s Presidio framework.
