Distribute frozen federation corpus by concept destination
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a070b5-4994-7271-bd8b-7c3dbcedec4b
This commit is contained in:
parent
3e3191549a
commit
372f671eef
81 changed files with 14146 additions and 99 deletions
|
|
@ -0,0 +1,109 @@
|
|||
# Deterministic vs Probabilistic Matching
|
||||
|
||||
## Source Type
|
||||
|
||||
Academic and industry practice. Entity resolution, record linkage, and
|
||||
duplicate detection literature plus operational MDM patterns.
|
||||
|
||||
## Domain
|
||||
|
||||
Entity resolution, record linkage, duplicate detection, and account matching
|
||||
confidence.
|
||||
|
||||
## Why This Source Matters
|
||||
|
||||
Entity resolution literature distinguishes deterministic keys from probabilistic
|
||||
matching — the foundation for weak vs. strong synonymity modeling.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Deterministic matching**: records match when agreed key fields are equal
|
||||
(exact email, government ID, OIDC `iss`+`sub`).
|
||||
- **Probabilistic matching**: match score from weighted field similarity
|
||||
(Fellegi-Sunter, Jaro-Winkler, ML classifiers).
|
||||
- **Record linkage**: identifying records across datasets referring to same
|
||||
entity.
|
||||
- **Blocking**: reduce comparison space by bucketing on partial keys.
|
||||
- **Match threshold**: score above which records are linked or flagged.
|
||||
- **False positive / false negative tradeoff**: precision vs. recall in linking.
|
||||
- **Golden record / survivor**: MDM pattern selecting canonical merged record.
|
||||
- **Non-destructive link**: associate records without merge (preferred in modern MDM).
|
||||
- **Human review queue**: ambiguous matches escalated for operator decision.
|
||||
- **Master data management (MDM)**: operational discipline around entity resolution.
|
||||
|
||||
## Relevant Terminology
|
||||
|
||||
| Term | Source meaning |
|
||||
| --- | --- |
|
||||
| Deterministic match | Equality on predefined key fields. |
|
||||
| Probabilistic match | Scored similarity above threshold. |
|
||||
| Record linkage | Cross-dataset entity correspondence. |
|
||||
| Blocking key | Partial key for candidate pair generation. |
|
||||
| Match score | Confidence metric for probable same entity. |
|
||||
| Duplicate | Records hypothesized to refer to same entity. |
|
||||
| Merge | Combine records into one (destructive). |
|
||||
| Link | Associate records preserving sources (non-destructive). |
|
||||
| Survivor record | Chosen primary after merge. |
|
||||
| Quarantine | Hold ambiguous matches for review. |
|
||||
|
||||
## Modeling Assumptions
|
||||
|
||||
- **Same entity is hypothesis until verified** in probabilistic approaches.
|
||||
- **Deterministic rules are domain-specific** (no universal golden key).
|
||||
- **Merge destroys provenance** unless carefully audited — increasingly avoided.
|
||||
- **Confidence is continuous or banded** (weak/medium/strong).
|
||||
- **Source system identity must be preserved** for compliance and undo.
|
||||
- **Human review is part of high-assurance linking.**
|
||||
- **Privacy regulations constrain** which fields can be matched.
|
||||
|
||||
## Identity-Canon Implications
|
||||
|
||||
- Deterministic match maps to **strong Synonymity Assertion** when keys are
|
||||
authoritative (S13).
|
||||
- Probabilistic match maps to **weak Synonymity Assertion** with confidence
|
||||
score and method (S12).
|
||||
- **Link without merge** is the canonical preferred pattern (**P7**).
|
||||
- **Match score** maps to confidence/strength on Synonymity Assertion.
|
||||
- **Blocking/method** maps to Evidence Source metadata.
|
||||
- **Quarantine** maps to Lifecycle State `proposed` on assertion.
|
||||
- **Golden record** is downstream MDM pattern; canon should not require merge.
|
||||
- **Human review** maps to Evidence Source (operator decision).
|
||||
|
||||
## Terminology Conflicts
|
||||
|
||||
- **Duplicate vs. Synonymity**: duplicates imply merge; synonymity allows coexistence.
|
||||
- **Match vs. Link**: industry uses interchangeably; canon distinguishes strength.
|
||||
- **Entity vs. Actor**: resolution literature says entity; canon prefers Actor target.
|
||||
- **Identity vs. Record**: matching is between records, not persons directly.
|
||||
- **Deterministic vs. Strong**: deterministic can still be wrong if key is shared
|
||||
(shared email).
|
||||
|
||||
## Candidate Canonical Mappings
|
||||
|
||||
| Entity resolution concept | Candidate canonical concept |
|
||||
| --- | --- |
|
||||
| Deterministic match | Strong Synonymity Assertion |
|
||||
| Probabilistic match | Weak Synonymity Assertion |
|
||||
| Match score | Confidence / strength metadata |
|
||||
| Link (non-destructive) | Synonymity Assertion |
|
||||
| Merge | Downstream anti-pattern (avoid) |
|
||||
| Blocking key | Evidence Source method |
|
||||
| Review queue | Lifecycle State `proposed` |
|
||||
| Source record ID | Identifier |
|
||||
| Golden record | Downstream projection only |
|
||||
| False positive handling | Revocation / supersession of assertion |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- What confidence bands (weak/medium/strong) should canon standardize?
|
||||
- Which deterministic keys are authoritative per source family (OIDC iss+sub,
|
||||
persistent SAML NameID, verified email)?
|
||||
- Should probabilistic matchers be required to store feature-level Evidence Source?
|
||||
- How should shared-attribute false positives (family email) be classified?
|
||||
|
||||
## References
|
||||
|
||||
- Fellegi-Sunter model (1969) — foundational probabilistic record linkage
|
||||
- Christen, "Data Matching" (2012) — entity resolution textbook
|
||||
- NIST SP 800-63A evidence requirements — https://pages.nist.gov/800-63-4/sp800-63A.html
|
||||
- MDM Institute duplicate management practices — industry reference
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
# GDPR Pseudonymization and Privacy
|
||||
|
||||
## Source Type
|
||||
|
||||
Regulatory guidance. EU GDPR (Regulation 2016/679) Article 4(5) and Recital 26;
|
||||
EDPB guidance on identifiability, anonymization, and data subject rights.
|
||||
|
||||
## Domain
|
||||
|
||||
Privacy regulation, pseudonymization, identifiability, data minimization, and
|
||||
lawful basis for identity processing.
|
||||
|
||||
## Why This Source Matters
|
||||
|
||||
GDPR pseudonymization and identifiability concepts affect how canonical models
|
||||
should represent privacy-limited links, scoped identifiers, and correlation risk.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Personal data**: information relating to identified or identifiable natural
|
||||
person.
|
||||
- **Identifiable person**: can be identified directly or indirectly by reasonable
|
||||
means.
|
||||
- **Pseudonymization (Art. 4(5))**: processing personal data so it cannot be
|
||||
attributed to a subject without additional information kept separately.
|
||||
- **Anonymization**: irreversible de-identification; data no longer personal.
|
||||
- **Data subject**: identified or identifiable natural person.
|
||||
- **Controller / Processor**: roles responsible for processing personal data.
|
||||
- **Purpose limitation**: data used for specified, explicit, legitimate purposes.
|
||||
- **Data minimization**: adequate, relevant, limited to necessary.
|
||||
- **Right of access / erasure**: data subject rights affecting linked records.
|
||||
- **Additional information**: key held separately to re-identify pseudonymous data.
|
||||
|
||||
## Relevant Terminology
|
||||
|
||||
| Term | Source meaning |
|
||||
| --- | --- |
|
||||
| Personal data | Data about identifiable natural person. |
|
||||
| Pseudonymization | Reversible de-identification with separate key. |
|
||||
| Anonymization | Irreversible; no longer personal data (if effective). |
|
||||
| Data subject | Natural person the data relates to. |
|
||||
| Identifiable | Reasonably linkable to person. |
|
||||
| Additional information | Re-identification key stored separately. |
|
||||
| Controller | Determines purposes and means of processing. |
|
||||
| Processing | Any operation on personal data. |
|
||||
| Erasure | Delete personal data (right to be forgotten). |
|
||||
| Profiling | Automated evaluation of personal aspects. |
|
||||
|
||||
## Modeling Assumptions
|
||||
|
||||
- **Pseudonymization is not anonymization**; data may remain personal.
|
||||
- **Separate storage of additional information** is required for pseudonymization.
|
||||
- **Scope and access control on keys** determine correlation risk.
|
||||
- **Linking pseudonymous records across purposes** may increase identifiability.
|
||||
- **Legal basis and purpose** govern whether linking is permissible.
|
||||
- **Erasure requests** may require breaking links or deleting assertions.
|
||||
- **Regulatory role (controller)** is organizational, not purely technical.
|
||||
|
||||
## Identity-Canon Implications
|
||||
|
||||
- **Pseudonymous Identifier** and **Scoped Identifier** map to pseudonymization
|
||||
techniques (pairwise sub, hashed email, internal IDs).
|
||||
- **Privacy-limited Synonymity Assertion** must record privacy classification
|
||||
and scope (S14).
|
||||
- **Additional information** (re-identification key) maps to separately secured
|
||||
**Evidence Source** or **Credential** with strict Scope access.
|
||||
- **Data subject** maps to **Natural Person** with privacy rights overlay
|
||||
(downstream policy, not canon legal advice).
|
||||
- **Erasure** maps to Lifecycle State transitions: revoke assertions, sever
|
||||
bindings, archive with legal exceptions noted downstream.
|
||||
- Pairwise OIDC, tenant-local subjects, and restricted persona links are
|
||||
technical pseudonymization patterns aligned with GDPR concepts.
|
||||
- Reinforces visibility of privacy constraints on relationships (**P8**, S14 checks).
|
||||
|
||||
## Terminology Conflicts
|
||||
|
||||
- **Pseudonym vs. Pseudonymization**: pseudonym is identifier; pseudonymization
|
||||
is processing technique.
|
||||
- **Anonymous vs. Pseudonymous**: often conflated in product marketing.
|
||||
- **Identity vs. Personal data**: not all identifiers are personal data in all
|
||||
contexts.
|
||||
- **Deletion vs. Revocation**: erasure may require more than assertion revocation.
|
||||
- **Subject**: GDPR data subject vs. OIDC/SAML subject.
|
||||
|
||||
## Candidate Canonical Mappings
|
||||
|
||||
| GDPR concept | Candidate canonical concept |
|
||||
| --- | --- |
|
||||
| Data subject | Natural Person (privacy overlay) |
|
||||
| Pseudonymization | Processing pattern on Identifier / Profile |
|
||||
| Pseudonymous identifier | Scoped Identifier / Pseudonymous Identifier |
|
||||
| Additional information | Separately secured Evidence Source / key |
|
||||
| Purpose limitation | Scope + policy metadata on processing |
|
||||
| Cross-system link | Synonymity Assertion (privacy classification required) |
|
||||
| Erasure request | Lifecycle State + assertion revocation |
|
||||
| Identifiability risk | Privacy classification on links |
|
||||
| Controller | Organization actor (downstream legal role) |
|
||||
| Anonymized dataset | Out of scope for personal identity linking |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should canon include a standard `privacy_classification` enum for assertions?
|
||||
- How should erasure of one account affect Synonymity Assertions touching other
|
||||
accounts (S02)?
|
||||
- Does pseudonymization key storage warrant a canonical secured Scope type?
|
||||
- Should identifiability review be documented as operator workflow in downstream
|
||||
recommendations only?
|
||||
|
||||
## References
|
||||
|
||||
- GDPR Article 4(5) pseudonymization — https://gdpr-info.eu/art-4-gdpr/
|
||||
- GDPR Recital 26 on identifiability — https://gdpr-info.eu/recitals-novo/26/
|
||||
- EDPB Guidelines on identifiability (various) — https://edpb.europa.eu/
|
||||
- ISO/IEC 20889 privacy enhancing data de-identification terminology
|
||||
|
|
@ -0,0 +1,106 @@
|
|||
# Synonymity Assertions
|
||||
|
||||
## Source Type
|
||||
|
||||
Concept synthesis from identity-canon ResearchSeed, entity resolution practice,
|
||||
federation account linking, and semantic web `sameAs` patterns.
|
||||
|
||||
## Domain
|
||||
|
||||
Identity linking, scoped equivalence, account linking, and non-destructive record
|
||||
association.
|
||||
|
||||
## Why This Source Matters
|
||||
|
||||
Synonymity assertions are the identity-canon-native model for linking records
|
||||
without merge — synthesizing federation binding, entity resolution, and
|
||||
semantic equivalence patterns.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Synonymity assertion**: scoped, evidenced claim that two or more identifiers,
|
||||
records, or actors refer to the same target for a stated purpose.
|
||||
- **Relation type**: `same_as`, `probably_same_as`, `linked_to`, `represents`,
|
||||
`controls`, `acts_for` (from ResearchSeed).
|
||||
- **Strength**: weak, medium, strong, authoritative bands.
|
||||
- **Scope**: namespace, tenant, relying party, or purpose boundary limiting
|
||||
assertion validity.
|
||||
- **Evidence**: verification event, issuer signature, operator review, import
|
||||
job output.
|
||||
- **Source system**: system that created or maintains the assertion.
|
||||
- **Lifecycle**: proposed, active, revoked, expired, superseded.
|
||||
- **Privacy classification**: controls visibility and correlation risk.
|
||||
- **Non-merge invariant**: linked records retain independent identity and provenance.
|
||||
- **Supersession chain**: new assertion replaces old when identifiers change.
|
||||
|
||||
## Relevant Terminology
|
||||
|
||||
| Term | Source meaning |
|
||||
| --- | --- |
|
||||
| Synonymity | Sameness or equivalence under conditions. |
|
||||
| Assertion | Explicit modeled statement, not implicit merge. |
|
||||
| same_as | High-confidence equivalence. |
|
||||
| probably_same_as | Probabilistic equivalence. |
|
||||
| linked_to | Operational convenience link. |
|
||||
| represents | One record represents another (controller, profile). |
|
||||
| Scope | Boundary limiting assertion meaning. |
|
||||
| Strength | Confidence band. |
|
||||
| Revocation | Assertion no longer valid. |
|
||||
| Supersession | New assertion replaces prior. |
|
||||
|
||||
## Modeling Assumptions
|
||||
|
||||
- **Equivalence is contextual**, not universal.
|
||||
- **Multiple assertions can coexist** for different scopes and purposes.
|
||||
- **Conflicting assertions possible**; require review workflow.
|
||||
- **Downstream systems consume assertions** according to their assurance needs.
|
||||
- **Privacy-limited assertions** must not leak across scopes (S14).
|
||||
- **Federation bindings are synonymity assertions** (`iss`+`sub` → local account).
|
||||
- **sameAs on web is weak by default** unless corroborated.
|
||||
|
||||
## Identity-Canon Implications
|
||||
|
||||
- Synonymity Assertion is a **first-class Relationship** class in canon.
|
||||
- Recommended fields align with ResearchSeed and ConceptualModel invariants.
|
||||
- OIDC RP binding, SAML persistent NameID mapping, SCIM `externalId`
|
||||
correlation, and entity-resolution matches all project into Synonymity
|
||||
Assertions with appropriate strength.
|
||||
- Supports all linking scenarios: S12 (weak), S13 (strong), S14 (privacy-limited).
|
||||
- **P7** is the governing principle; merge is downstream exception only.
|
||||
- Revocation from RISC/SSF events should update assertion Lifecycle State.
|
||||
|
||||
## Terminology Conflicts
|
||||
|
||||
- **Link vs. Merge**: products say "linked accounts" after merge.
|
||||
- **sameAs vs. same_as**: semantic web informal vs. canon typed relation.
|
||||
- **Account linking vs. Identity linking**: may target accounts or identifiers.
|
||||
- **Alias vs. Synonymity**: alias is presentation; synonymity is assertion.
|
||||
- **Duplicate resolution vs. Synonymity**: MDM duplicate implies survivor selection.
|
||||
|
||||
## Candidate Canonical Mappings
|
||||
|
||||
| Practice / source pattern | Candidate canonical concept |
|
||||
| --- | --- |
|
||||
| OIDC iss+sub → local user | Strong Synonymity Assertion (scoped) |
|
||||
| SAML persistent NameID map | Strong Synonymity Assertion |
|
||||
| Probabilistic duplicate score | Weak Synonymity Assertion (`probably_same_as`) |
|
||||
| Operator-verified link | Strong Synonymity Assertion (authoritative) |
|
||||
| Pairwise sub RP binding | Privacy-limited Synonymity Assertion |
|
||||
| SCIM externalId correlation | Identifier Binding / medium Synonymity |
|
||||
| schema.org sameAs | Weak Synonymity Assertion (caution) |
|
||||
| DID equivalentId | Method-dependent Synonymity |
|
||||
| VC subject DID binding | Strong Synonymity Assertion with cryptographic evidence |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should `linked_to` remain distinct from `same_as` for operational vs. semantic links?
|
||||
- What minimum field set is mandatory for all Synonymity Assertions (see OpenQuestions)?
|
||||
- How should conflicting assertions be represented (priority, review state)?
|
||||
- Should privacy classification be enum or policy reference?
|
||||
|
||||
## References
|
||||
|
||||
- identity-canon ResearchSeed.md — synonymity assertion fields
|
||||
- identity-canon ConceptualModel.md — identity linking model
|
||||
- OIDC account linking practice — https://openid.net/specs/openid-connect-core-1_0.html
|
||||
- W3C VC subject identification — https://www.w3.org/TR/vc-data-model-2.0/
|
||||
Loading…
Add table
Add a link
Reference in a new issue