The documented AIGOV5 paired RIS–CSV search selected the first four government-facing modules. The wider local collection also contains the Global Think Tanks module. Module membership is collection provenance, not proof of national-policy eligibility; issuer and government relationship are classified later.

EXECUTED describes a verified local result. GOVERNING RULE is a mandatory protocol control. PENDING OR DISTRIBUTED means evidence remains in an earlier stage folder or requires consolidation. In particular, the governing protocol prescribes independent iCloud reprocessing; the current public v7.10 folder alone does not prove that full reconciliation step.

306,644 RIS + 21,711 CSV - 21,450 paired = 306,905 union candidates
21,450 paired + 285,194 RIS-only + 254 CSV-only + 7 conflicts = 306,905
306,898 conflict-cleared - 11,306 Golden matches - 6 converged duplicates = 295,586 new
11,306 preserved Golden + 258,547 new AI/document records = 269,853 master

The 295,586 new candidates include 295,354 broad AI inclusions and 258,777 document-eligible records. Their intersection is 258,547, so these are overlapping classification axes and must not be subtracted as independent sequential losses.

The working design is local first, iCloud second. Downloads is the intake and staging area. The iCloud-backed data3 collection is frozen and processed independently only after the local master passes its gates. Path aliases, symbolic links, file-system identity, size and SHA-256 are compared before counting, so a local path and its iCloud backing path cannot be counted as two sources.

Every RIS, CSV, JSON, JSONL, database and archive is inventoried with path, type, size, modification time, checksum, record count and collection category. The occurrence key is (source_file_sha256, source_record_ordinal). Raw bytes or their hash, parsed-record hash and normalised-content fingerprint remain attached to that occurrence. A later canonical record never replaces this source layer.

The mandatory order is: freeze inputs; validate structure and contamination; parse and normalise; resolve identity and versions; enrich missing fields; classify AI relevance; classify document form; resolve institutions and countries; determine policy eligibility; freeze the local master; process and reconcile iCloud; then release a new integrated version. Every transition records its run, configuration, rules, code and environment versions, counts, timestamps, reviewer actions and checksums.

The analytical entity is a document version, not an export row. Identity evidence is evaluated in this order: Policy Commons artefact and file identifiers; canonical Policy Commons URL; DOI, ISBN, report number or official publisher URL; finally a reviewed metadata match using title, publisher, year, language and abstract compatibility. Artefact IDs and file IDs are separate namespaces.

An incompatible title or abstract, different language or edition, substantive revision, conflicting typed date, cross-namespace collision or redirect to a collection page blocks automatic merging. Fuzzy title similarity creates review candidates only. Translations, abridgements, annual editions and substantive revisions keep separate record_id values and may be connected through work_cluster_id.

The best structurally valid and internally consistent observation anchors the canonical record. Other observations can fill a missing field through a field-survivorship table, but a non-empty conflict is never overwritten silently. The table records the source occurrence, previous and selected values, rule, confidence and conflict state. This prevents a false composite record assembled from unrelated exports.

AI relevance, document form, institution type, policy eligibility and country attribution are separate variables. A government issuer does not make a dataset a policy document; a private issuer does not make an AI report Non-AI; publication place does not establish issuer jurisdiction.

For AI screening, original full text outranks original publisher metadata, which outranks provider summaries, keywords and search provenance. The evidence tiers are E0 no usable evidence, E1 title only, E2 title plus abstract or summary and E3 usable original full text. Missing metadata terms cannot support exclusion. The retained labels are core_ai_policy, substantive_ai_component, incidental_ai_mention, non_ai and unresolved; only a positive, independently auditable basis can support non_ai.

For document form, the normalised labels distinguish policy documents, scholarly articles, patents, government-linked research, hybrid or ambiguous records and unresolved genre. A DOI, university affiliation, the word paper, proceedings or workshop is never sufficient on its own. The decision turns on the document's policy function.

The system stores issuer, bibliographic publisher, archive or host, commissioner and adopting government as different relations. The institution authority table contains the canonical name, aliases, institution class, governmental relationship, country, evidence URL, validity period, confidence and verification date.

National and subnational governments, legislatures, regulators, statutory agencies, public research institutes, official committees and qualifying public policy institutes enter the national-policy group. International organisations, EU bodies, independent think tanks, foundations, universities, corporations, networks and repositories remain separate. A non-government body can enter a government-linked layer when formal commissioning, adoption, funding or co-issuance is demonstrated.

Country concepts are not collapsed. The release retains published_in_country, issuer_country, policy_jurisdiction, target_country and author-affiliation country. The analytical code CTRY follows verified issuer jurisdiction. Reserved non-country codes are XIG for intergovernmental organisations, XEU for EU institutions, XMN for multinational private issuers and XUN for unresolved cases. Multiple eligible issuers are represented in link tables, not fractionalised. HKG is aggregated to CHN only in the analytical view; the observed value is retained.

A release contains the canonical master, source provenance, collection routes, duplicate and work-cluster maps, institution authority, document-country links, quarantine, exclusions, unresolved records, stage counts, scripts, environment information and SHA-256 manifests. Core grains are enforced with primary and foreign-key checks so many-to-many joins cannot inflate analysis rows.

Required accounting identities include: raw observations equal accepted parses plus parse quarantine; accepted parses equal linked observations plus unresolved identities; linked observations equal canonical anchors plus duplicate links; canonical records equal accepted master records plus exclusions, unresolved classifications and conflict quarantine.

Physical deletion is never a cleaning shortcut. It requires a frozen source manifest, persisted and checksummed canonical or exclusion record, reconciled counts, reviewed dry run, exact-path deletion manifest, recoverable quarantine, immediate pre-deletion size and checksum verification, recorded approval, successful restore test and completed iCloud synchronisation. No physical source deletion was performed for the current release.

The v7.10 augmented master contains 162 stored columns. The complete dictionary lists every physical column, stored type and definition, including raw bibliographic fields, RIS–CSV linkage, collection provenance, AI screening, document form, institution classification, country evidence, live-page validation, correction history and final release fields.

Open the complete Policy Commons augmented master v7.10 data dictionary. The page is generated from the actual Parquet schema so a column cannot be omitted merely because it was added after the original 63-variable Golden Set.

RIS can omit page fields such as country and topics. Always collect the paired CSV and join only normalised Policy Commons and artefact identifiers. Preserve unmatched and conflicting keys. Never join by title, filename or row order. Deduplicate with stable IDs, canonical URLs, compatible title, publisher, year, language and content evidence. Keep editions, translations, annual editions and substantive revisions separate.

Metadata absence is not Non-AI because the provider can match indexed full text. Retain records from documented AI-oriented searches unless positive evidence establishes an alternative meaning. Exclude avian influenza, air, biological neural networks and similar collisions only when no independent computational-AI evidence remains. Historical terms such as expert systems, machine intelligence and intelligent systems require computational context. Record query provenance, evidence location and AI centrality separately.

Standalone vacancy notices, position descriptions, product or user manuals, theses, datasets, course syllabi, videos and images are excluded from the policy-document view. Government or public ownership does not rescue a non-policy format. Exceptions require an explicit policy function: legal or regulatory implementation guidance, official curriculum or examination specification, commissioned policy evidence, or a substantive mixed document.

Proceedings, workshops and agendas are not excluded by title. Retain official government inquiries, parliamentary hearings, regulatory or standards work, testimony, recommendations, action plans and implementation records. Exclude only logistical event records and purely scholarly proceedings. Human review of 136 records confirmed this rule.

Store published_in_country, issuer_country, policy jurisdiction and target country separately. Explicit issuer jurisdiction overrides a conflicting place, as in Government of Kosovo versus Published in Serbia. Public libraries and national commissions can be genuine issuers. EU, international organisations, repositories and headquarters locations are not national issuers without a supported national office, commission or country-specific mandate. HKG is mapped to CHN for analysis while raw values remain preserved.

A headquarters or branch location alone does not represent national policy interest. Ordinary business, product and promotional documents from multinational firms are excluded from the national analysis but retained in the source master. A private research unit is included only when the source demonstrates government commissioning, funding or a national research remit. Unclear cases remain manual_review.

Original pages were checked during rule development, but intermediate sampling rounds are omitted from this results-focused page. The final gate requires unique document-country keys, unit weights of 1, retained evidence and rule fields, versioned outputs and SHA-256 checksums. Inaccessible pages are never counted as verified. Detailed review manifests remain in the local audit package and the error register.

All 162 physical Parquet columns are stored as strings. Missingness checks therefore treat both null values and empty strings as missing. Module provenance is not yet exposed as a dedicated master column for every record; source manifests remain authoritative and a future record-module link table is recommended.