Collection and cleaning workflow / Data dictionary

Policy Commons augmented master v7.10 data dictionary

The column names, order and physical types are generated from the Parquet schema of policycommons_augmented_master_final_v7_10.parquet. Definitions combine manually curated descriptions for core fields with documented prefix-based descriptions for versioned audit fields. It describes all 162 stored columns. The master grain is one canonical document version.

Authoritative analytical table. Country-level analysis must use document_country_pairs_final_v7_10.csv and its unique (record_id, analysis_country) rows. Do not reconstruct the release by filtering CTRY, analysis_country_final or other master columns. Base, intermediate and final fields coexist to preserve correction history, and the inherited release_version value inside the master predates the v7.10 directory-level release identity.

Country attribution (28 variables)

Variable	Stored type	Definition
`CTRY`	large_string	Legacy or upstream single-country field. Do not use it as the current analytical country; use the separate final pair file.
`analysis_country`	large_string	Base whole-count analytical country assignment.
`analysis_country_count`	large_string	Number of distinct countries in the base analytical assignment.
`analysis_country_count_final`	large_string	Number of distinct analytical countries at the named release stage.
`analysis_country_count_v2`	large_string	Number of distinct analytical countries at the named release stage.
`analysis_country_count_v3`	large_string	Number of distinct analytical countries at the named release stage.
`analysis_country_final`	large_string	Whole-count analytical country assignment at the named release stage.
`analysis_country_v2`	large_string	Whole-count analytical country assignment at the named release stage.
`analysis_country_v3`	large_string	Whole-count analytical country assignment at the named release stage.
`country_decision_source_v2`	large_string	Evidence, rule, confidence or status used in country resolution.
`country_decision_source_v3`	large_string	Evidence, rule, confidence or status used in country resolution.
`country_method_final`	large_string	Evidence, rule, confidence or status used in country resolution.
`country_resolution_band`	large_string	Evidence, rule, confidence or status used in country resolution.
`country_resolution_status`	large_string	Evidence, rule, confidence or status used in country resolution.
`country_status`	large_string	Evidence, rule, confidence or status used in country resolution.
`issuer_country`	large_string	Country or countries attributed to the issuing institution in the base classification.
`issuer_country_count`	large_string	Issuer-country value or automated normalisation at the named release stage.
`issuer_country_final`	large_string	Issuer-country value or automated normalisation at the named release stage.
`issuer_country_v2`	large_string	Issuer-country value or automated normalisation at the named release stage.
`issuer_country_v3`	large_string	Issuer-country value or automated normalisation at the named release stage.
`provider_country_evidence`	large_string	Evidence, rule, confidence or status used in country resolution.
`publisher_rule_country`	large_string	Evidence, rule, confidence or status used in country resolution.
`publisher_rule_country_count`	large_string	Evidence, rule, confidence or status used in country resolution.
`target_country`	large_string	Country discussed, targeted or covered by the document; not issuer provenance.
`issuer_country_auto`	large_string	Issuer-country value or automated normalisation at the named release stage.
`issuer_country_auto_iso3`	large_string	Issuer-country value or automated normalisation at the named release stage.
`country_rule_id`	large_string	Evidence, rule, confidence or status used in country resolution.
`country_codebook_rule_v7_3`	large_string	Evidence, rule, confidence or status used in country resolution.

Bibliographic metadata (19 variables)

Variable	Stored type	Definition
`abstract`	large_string	Provider-supplied or provider-generated abstract or summary.
`affiliations`	large_string	Author or contributor affiliation values when available.
`authors`	large_string	Authors or organisational contributors retained from RIS.
`coilink_url`	large_string	COI resolver URL associated with the Policy Commons identifier.
`content_type`	large_string	Normalised content-type value when available.
`doi`	large_string	External DOI when supplied; distinct from the Policy Commons identifier.
`isbn`	large_string	ISBN supplied by RIS or CSV.
`issn`	large_string	ISSN supplied by RIS or CSV.
`keywords`	large_string	RIS keywords, kept separate from Policy Commons Topics.
`language`	large_string	Exported document language or languages.
`people`	large_string	Contributor values supplied by the CSV export.
`publication_type`	large_string	Normalised publication-type value when available.
`publisher`	large_string	Recorded publisher or issuing organisation string.
`ris_type`	large_string	Original RIS reference type from TY.
`title`	large_string	Canonical document title.
`type`	large_string	Provider CSV object class.
`url`	large_string	Canonical or best available Policy Commons artefact-page URL.
`year`	large_string	Best normalised publication year retained by the release.
`year_type`	large_string	Evidence class or derivation method for the year value.

Identity (7 variables)

Variable	Stored type	Definition
`abstract_hash`	large_string	Hash of the normalised abstract used for integrity and contamination checks.
`identity_status`	large_string	Canonicalisation, duplicate or identity-review status.
`join_artifact_id`	large_string	Numeric Policy Commons artefact identifier parsed from the artefact URL.
`join_coi`	large_string	Normalised persistent content identifier used for conservative RIS–CSV linkage.
`policy_commons_id`	large_string	Persistent Policy Commons content identifier exported in RIS.
`record_id`	large_string	Internal canonical document-version identifier.
`ris_record_id`	large_string	Identifier retained from the canonical RIS record.

AI relevance (17 variables)

Variable	Stored type	Definition
`ai_broad_included`	large_string	Versioned high-recall inclusion flag; metadata absence alone is not a negative decision.
`ai_decision`	large_string	Versioned AI-relevance classification or scope value.
`ai_decision_protocol_3_7`	large_string	Versioned AI-relevance classification or scope value.
`ai_decision_v6`	large_string	Versioned AI-relevance classification or scope value.
`ai_evidence_json`	large_string	AI evidence field retained by the evidence json step.
`ai_evidence_location`	large_string	AI evidence field retained by the evidence location step.
`ai_evidence_status`	large_string	AI evidence field retained by the evidence status step.
`ai_exclusion_reason`	large_string	Rule or evidence explanation for the corresponding AI decision.
`ai_include`	large_string	Versioned high-recall inclusion flag; metadata absence alone is not a negative decision.
`ai_include_protocol_3_7`	large_string	Versioned high-recall inclusion flag; metadata absence alone is not a negative decision.
`ai_include_v6`	large_string	Versioned high-recall inclusion flag; metadata absence alone is not a negative decision.
`ai_metadata_label`	large_string	Versioned AI-relevance classification or scope value.
`ai_metadata_reason`	large_string	Rule or evidence explanation for the corresponding AI decision.
`ai_reason`	large_string	Rule or evidence explanation for the corresponding AI decision.
`ai_relevance`	large_string	Versioned AI-relevance classification or scope value.
`ai_scope`	large_string	Versioned AI-relevance classification or scope value.
`ai_fulltext_review_status_v7`	large_string	Versioned AI-screening output preserved for audit.

Other audit field (4 variables)

Variable	Stored type	Definition
`analysis_eligible_rule_based`	large_string	Versioned field retained by the augmented-master pipeline for reproducibility and audit.
`evidence_scope`	large_string	Versioned field retained by the augmented-master pipeline for reproducibility and audit.
`automatic_country_confidence`	large_string	Versioned field retained by the augmented-master pipeline for reproducibility and audit.
`automatic_country_merged_at`	large_string	Versioned field retained by the augmented-master pipeline for reproducibility and audit.

Audit and sampling (3 variables)

Variable	Stored type	Definition
`audit_stratum`	large_string	Frozen sampling stratum assigned for audit.
`risk_band`	large_string	Risk stratum used for review and validation.
`year_band`	large_string	Year stratum used for diagnostics and stratified validation.

Release (7 variables)

Variable	Stored type	Definition
`augmentation_status`	large_string	Status assigned during Golden Set augmentation.
`golden_match_status`	large_string	Outcome of matching the local record to the Golden Set.
`golden_membership`	large_string	Legacy augmentation label. Golden rows can be blank; use master_origin to distinguish golden_v6 from local enrichment.
`golden_record_id`	large_string	Matched Golden Set record identifier when present.
`golden_set_version`	large_string	Golden Set version from which the record or schema originated.
`master_origin`	large_string	Whether the record originated in the Golden Set or local augmentation.
`release_version`	large_string	Version label of the augmented release.

Change audit (4 variables)

Variable	Stored type	Definition
`changed_ai_v1_to_v6`	large_string	Flag recording whether the value changed between named release versions.
`changed_country_v1_to_v2`	large_string	Flag recording whether the value changed between named release versions.
`changed_document_v1_to_v2`	large_string	Flag recording whether the value changed between named release versions.
`v1_error_corrected`	large_string	Flag recording whether the value changed between named release versions.

Classification (1 variables)

Variable	Stored type	Definition
`classification_status`	large_string	Overall classification completion or review state.

Provenance (10 variables)

Variable	Stored type	Definition
`collection_categories`	large_string	All collection categories linked to the canonical record.
`collection_provenance`	large_string	Search and collection history supporting inclusion.
`observation_count`	large_string	Number of observations linked to the canonical record.
`observation_id`	large_string	Stable identifier for the representative source observation.
`query`	large_string	Exact or representative search expression associated with the record.
`query_id`	large_string	Short identifier for the search contract.
`raw_fields_json`	large_string	Loss-preserving JSON representation of original repeated RIS fields.
`source_file`	large_string	Representative source file selected for the canonical record.
`source_observation_count`	large_string	Count of source observations represented by the row.
`source_ordinal`	large_string	Record position within the representative source file.

Country evidence (4 variables)

Variable	Stored type	Definition
`country`	large_string	Raw country value from the paired CSV or inherited Golden Set record.
`place_published`	large_string	Exported publication place; not automatically the issuer country.
`published_in_country_code`	large_string	Country code parsed from the provider Published in field.
`published_in_country_name`	large_string	Country name parsed from the provider Published in field.

Eligibility (5 variables)

Variable	Stored type	Definition
`country_analysis_eligible`	large_string	Versioned flag for inclusion in national country-level analysis.
`country_analysis_eligible_final`	large_string	Versioned flag for inclusion in national country-level analysis.
`country_analysis_eligible_v2`	large_string	Versioned flag for inclusion in national country-level analysis.
`country_analysis_eligible_v3`	large_string	Versioned flag for inclusion in national country-level analysis.
`year_in_scope`	large_string	Whether the year falls within the analysis period.

RIS–CSV linkage (4 variables)

Variable	Stored type	Definition
`csv_observation_count`	large_string	Number of CSV observations sharing the accepted linkage key.
`csv_source_category`	large_string	Collection category attached to the CSV source.
`csv_source_files`	large_string	CSV files contributing fields to the canonical record.
`pairing_status`	large_string	RIS–CSV linkage outcome, including paired and format-only states.

Document form (10 variables)

Variable	Stored type	Definition
`document_eligible`	large_string	Base eligibility flag for substantive document analysis.
`document_eligible_protocol_3_7`	large_string	Versioned document-genre, eligibility or supporting-reason field.
`document_eligible_v2`	large_string	Versioned document-genre, eligibility or supporting-reason field.
`document_genre`	large_string	Base normalised document genre.
`document_genre_protocol_3_7`	large_string	Versioned document-genre, eligibility or supporting-reason field.
`document_genre_v2`	large_string	Versioned document-genre, eligibility or supporting-reason field.
`format_status`	large_string	Availability and structural status of RIS, CSV or paired metadata.
`genre_reason`	large_string	Rule or evidence supporting the base genre decision.
`genre_reason_protocol_3_7`	large_string	Versioned document-genre, eligibility or supporting-reason field.
`genre_reason_v2`	large_string	Versioned document-genre, eligibility or supporting-reason field.

Review (4 variables)

Variable	Stored type	Definition
`human_validation_status`	large_string	Original-source or human-adjudication status.
`original_source_verified`	large_string	Whether the original page or document was directly checked.
`quality_reason`	large_string	Reason a record entered the quality-review queue.
`quality_review_required`	large_string	Flag indicating that a record needs quality review.

Institution (5 variables)

Variable	Stored type	Definition
`institution_type`	large_string	Base normalised class of the issuer or publisher.
`institution_type_final`	large_string	Versioned issuer or publisher institution classification or applied rule.
`institution_type_rule`	large_string	Versioned issuer or publisher institution classification or applied rule.
`institution_type_v2`	large_string	Versioned issuer or publisher institution classification or applied rule.
`publisher_key`	large_string	Normalised publisher key used for authority matching.

CSV enrichment (2 variables)

Variable	Stored type	Definition
`topics`	large_string	Policy Commons Topics supplied by the CSV export.
`topics_combined`	large_string	Order-preserving union of RIS keywords and CSV Topics.

Live-page verification (6 variables)

Variable	Stored type	Definition
`policycommons_live_country_value`	large_string	Value, field, URL or timestamp captured from direct Policy Commons page verification.
`policycommons_live_country_field`	large_string	Value, field, URL or timestamp captured from direct Policy Commons page verification.
`policycommons_live_country_verified_at`	large_string	Value, field, URL or timestamp captured from direct Policy Commons page verification.
`policycommons_live_country_url`	large_string	Value, field, URL or timestamp captured from direct Policy Commons page verification.
`live_validation_rule_id`	large_string	Rule or exclusion class recorded during live-page validation.
`live_validation_exclusion_class`	large_string	Rule or exclusion class recorded during live-page validation.

Precision cleaning (6 variables)

Variable	Stored type	Definition
`precision_exclusion_class`	large_string	Status, rule or exclusion class from a precision-cleaning round.
`precision_rule_id`	large_string	Status, rule or exclusion class from a precision-cleaning round.
`precision_round3_status`	large_string	Status, rule or exclusion class from a precision-cleaning round.
`precision_round3_rule`	large_string	Status, rule or exclusion class from a precision-cleaning round.
`precision_round4_status`	large_string	Status, rule or exclusion class from a precision-cleaning round.
`precision_round4_rule`	large_string	Status, rule or exclusion class from a precision-cleaning round.

Country history (8 variables)

Variable	Stored type	Definition
`analysis_country_before_precision_exclusion`	large_string	Country value preserved immediately before the named correction or exclusion stage.
`issuer_country_before_precision_exclusion`	large_string	Country value preserved immediately before the named correction or exclusion stage.
`analysis_country_before_live_validation`	large_string	Country value preserved immediately before the named correction or exclusion stage.
`issuer_country_before_live_validation`	large_string	Country value preserved immediately before the named correction or exclusion stage.
`analysis_country_before_round2`	large_string	Country value preserved immediately before the named correction or exclusion stage.
`issuer_country_before_round2`	large_string	Country value preserved immediately before the named correction or exclusion stage.
`analysis_country_before_hkg_codebook`	large_string	Country value preserved immediately before the named correction or exclusion stage.
`issuer_country_before_hkg_codebook`	large_string	Country value preserved immediately before the named correction or exclusion stage.

Country review (2 variables)

Variable	Stored type	Definition
`round2_rule`	large_string	Rule or classification outcome from the second live-review round.
`round2_class`	large_string	Rule or classification outcome from the second live-review round.

Policy eligibility (2 variables)

Variable	Stored type	Definition
`national_policy_interest_scope`	large_string	National policy-interest scope or the rule supporting that decision.
`national_policy_interest_rule`	large_string	National policy-interest scope or the rule supporting that decision.

Document-form review (4 variables)

Variable	Stored type	Definition
`document_form_review_v7_7`	large_string	Human-review outcome or rule from the document-form validation release.
`document_form_review_rule`	large_string	Human-review outcome or rule from the document-form validation release.
`human_form_review_v7_8`	large_string	Human-review outcome or rule from the document-form validation release.
`human_form_review_rule`	large_string	Human-review outcome or rule from the document-form validation release.

Generated directly from the v7.10 Parquet schema. Schema order is preserved within each semantic group.