The CAP Babel Machine

Introduction

The CAP Babel Machine Project uses the Comparative Agendas Project policy codebook's major topics to identify the policy areas of texts. (Please let us know if you have suitable training data for minor topics and we will be happy to add that task to the service as well.) The codes assigned are unequivocal, mutually exclusive, and cover all potential policy topics. The codebook distinguishes 21 major policy areas: macroeconomics (1), civil rights (2), health (3), agriculture (4), labour (5), education (6), environment (7), energy (8), immigration (9), transportation (10), law and crime (12), social welfare (13), housing (14), banking, finance and domestic commerce (15), defence (16), technology (17), foreign trade (18), international affairs (19), government operations (20), public lands (21) and culture (23).

We use 11 CAP data domains. Please find a comparative table with the domains of the CAP website below:

Latest (08/30/2023) data per domain:

Most of the language models that the CAP Babel Machine uses were fine-tuned on training data containing the label 'None' (code: 999) in addition to the 21 CAP major policy topics, indicating that the given text contains no relevant policy content.

We have language-specific models for the following languages: Catalan, Czech, Danish, Dutch, English, French, German, Hungarian, Italian, Norwegian, Polish, Portuguese, Slovak and Spanish , but we encourage you to also submit datasets not covered under this list, as results may be useful for additional languages due to the nature of large language models.

You can upload your datasets here for automated CAP-coding. If you wish to submit multiple datasets one after another, please wait 5-10 minutes between each of your submissions. There are two possibilities for upload: pre-coded datasets or non-coded datasets. The explanation of the form and the dataset requirement is available here.

The upload requires to fill the following form on metadata regarding the dataset. We kindly ask you to upload your dataset, and in case of a pre-coded dataset, if available, please attach the codebook used besides the dataset.

The non-coded datasets should contain an id and a text column. The column names must be in row 1. You are free to add supplementary variables to the dataset beyond the compulsory ones in the columns following them.

Pre-coded datasets must contain the following columns: id, year, major_topic, text. The column names must be in row 1. Uploading a pre-coded sample is optional, but it can help us with calculating performance metrics and fine-tuning the language model behind CAP Babel Machine. The detailed rules of validations are available here. The mandatory data format of major_topic is numeric. All textual CAP categories must be converted to the appropriate numeric code before uploading. Furthermore, records with no policy content should be coded with 999. You are free to add supplementary variables to the dataset beyond the compulsory ones in the columns following them. Automatic processing requires to follow these rules.

After you upload your dataset and your file is successfully processed, you will receive the CAP-coded dataset and a file (both in CSV format) that includes the three highest probability category predictions by the CAP Babel model and the corresponding probability (softmax) scores assigned to each label. Please be aware that interpreting softmax scores as absolute model confidences could lead to false assumptions about model performance. The model prediction results are deterministic, so metrics such as the reliability coefficient do not apply.

For an example dataset, please use the following file which you can download here.

If the files you would like to upload are larger than 1 GB, please reach out to us with the download link attached (such as Dropbox or Google Drive) using our contact form.

If you have any questions or feedback regarding the CAP Babel Machine, please let us know using our contact form. Please keep in mind that we can only get back to you on Hungarian business days.

Submit a dataset:

Processing Unit: GPU: Please use this for more than 1000 observations in individual files. CPU: Please use this for less than 1000 observations in individual files.

Name*

E-mail address*

Please use a valid e-mail address.

The name of the institution with which you are affiliated*

The country of the institution with which you are affiliated* -- Please Choose One -- Multiple Countries / Intercountry Institution Afghanistan Albania Algeria American Samoa Andorra Angola Anguilla Antarctica Antigua and Barbuda Argentina Armenia Aruba Australia Austria Azerbaijan Bahamas Bahrain Bangladesh Barbados Belarus Belgium Belize Benin Bermuda Bhutan Bolivia Bonaire Bosnia and Herzegovina Botswana Bouvet Island Brazil British Indian Ocean Territory Brunei Darussalam Bulgaria Burkina Faso Burundi Cabo Verde Cambodia Cameroon Canada Cayman Islands Central African Republic Chad Chile China Christmas Island Cocos (Keeling) Islands Colombia Comoros Congo The Democratic Republic of the Congo Cook Islands Costa Rica Croatia Cuba Curaçao Cyprus Czechia Côte d'Ivoire Denmark Djibouti Dominica Dominican Republic Ecuador Egypt El Salvador Equatorial Guinea Eritrea Estonia Eswatini Ethiopia Falkland Islands (Malvinas) Faroe Islands Fiji Finland France French Guiana French Polynesia French Southern Territories Gabon Gambia Georgia Germany Ghana Gibraltar Greece Greenland Grenada Guadeloupe Guam Guatemala Guernsey Guinea Guinea-Bissau Guyana Haiti Heard Island and McDonald Islands Holy See (Vatican City State) Honduras Hong Kong Hungary Iceland India Indonesia Iran Iraq Ireland Isle of Man Israel Italy Jamaica Japan Jersey Jordan Kazakhstan Kenya Kiribati Democratic People's Republic of Korea Republic of Korea Kuwait Kyrgyzstan Lao People's Democratic Republic Latvia Lebanon Lesotho Liberia Libya Liechtenstein Lithuania Luxembourg Macao Madagascar Malawi Malaysia Maldives Mali Malta Marshall Islands Martinique Mauritania Mauritius Mayotte Mexico Micronesia Moldova Monaco Mongolia Montenegro Montserrat Morocco Mozambique Myanmar Namibia Nauru Nepal Netherlands New Caledonia New Zealand Nicaragua Niger Nigeria Niue Norfolk Island North Macedonia Northern Mariana Islands Norway Oman Pakistan Palau Palestine Panama Papua New Guinea Paraguay Peru Philippines Pitcairn Poland Portugal Puerto Rico Qatar Romania Russian Federation Rwanda Réunion Saint Barthélemy Saint Helena Saint Kitts and Nevis Saint Lucia Saint Martin (French part) Saint Pierre and Miquelon Saint Vincent and the Grenadines Samoa San Marino Sao Tome and Principe Saudi Arabia Senegal Serbia Seychelles Sierra Leone Singapore Sint Maarten (Dutch part) Slovakia Slovenia Solomon Islands Somalia South Africa South Georgia and the South Sandwich Islands South Sudan Spain Sri Lanka Sudan Suriname Svalbard and Jan Mayen Sweden Switzerland Syrian Arab Republic Taiwan Tajikistan Tanzania Thailand Timor-Leste Togo Tokelau Tonga Trinidad and Tobago Tunisia Turkmenistan Turks and Caicos Islands Tuvalu Türkiye Uganda Ukraine United Arab Emirates United Kingdom United States United States Minor Outlying Islands Uruguay Uzbekistan Vanuatu Venezuela Viet Nam British Virgin Islands U.S. Virgin Islands Wallis and Futuna Western Sahara Yemen Zambia Zimbabwe Åland Islands

Please specify:

Dataset name*

Level of dataset* -- Please Choose One -- Supranational/International Sovereign state Substate/U.S. state

Dataset country* -- Please Choose One -- Multiple Countries / Intercountry Data Afghanistan Albania Algeria American Samoa Andorra Angola Anguilla Antarctica Antigua and Barbuda Argentina Armenia Aruba Australia Austria Azerbaijan Bahamas Bahrain Bangladesh Barbados Belarus Belgium Belize Benin Bermuda Bhutan Bolivia Bonaire Bosnia and Herzegovina Botswana Bouvet Island Brazil British Indian Ocean Territory Brunei Darussalam Bulgaria Burkina Faso Burundi Cabo Verde Cambodia Cameroon Canada Cayman Islands Central African Republic Chad Chile China Christmas Island Cocos (Keeling) Islands Colombia Comoros Congo The Democratic Republic of the Congo Cook Islands Costa Rica Croatia Cuba Curaçao Cyprus Czechia Côte d'Ivoire Denmark Djibouti Dominica Dominican Republic Ecuador Egypt El Salvador Equatorial Guinea Eritrea Estonia Eswatini Ethiopia Falkland Islands (Malvinas) Faroe Islands Fiji Finland France French Guiana French Polynesia French Southern Territories Gabon Gambia Georgia Germany Ghana Gibraltar Greece Greenland Grenada Guadeloupe Guam Guatemala Guernsey Guinea Guinea-Bissau Guyana Haiti Heard Island and McDonald Islands Holy See (Vatican City State) Honduras Hong Kong Hungary Iceland India Indonesia Iran Iraq Ireland Isle of Man Israel Italy Jamaica Japan Jersey Jordan Kazakhstan Kenya Kiribati Democratic People's Republic of Korea Republic of Korea Kuwait Kyrgyzstan Lao People's Democratic Republic Latvia Lebanon Lesotho Liberia Libya Liechtenstein Lithuania Luxembourg Macao Madagascar Malawi Malaysia Maldives Mali Malta Marshall Islands Martinique Mauritania Mauritius Mayotte Mexico Micronesia Moldova Monaco Mongolia Montenegro Montserrat Morocco Mozambique Myanmar Namibia Nauru Nepal Netherlands New Caledonia New Zealand Nicaragua Niger Nigeria Niue Norfolk Island North Macedonia Northern Mariana Islands Norway Oman Pakistan Palau Palestine Panama Papua New Guinea Paraguay Peru Philippines Pitcairn Poland Portugal Puerto Rico Qatar Romania Russian Federation Rwanda Réunion Saint Barthélemy Saint Helena Saint Kitts and Nevis Saint Lucia Saint Martin (French part) Saint Pierre and Miquelon Saint Vincent and the Grenadines Samoa San Marino Sao Tome and Principe Saudi Arabia Senegal Serbia Seychelles Sierra Leone Singapore Sint Maarten (Dutch part) Slovakia Slovenia Solomon Islands Somalia South Africa South Georgia and the South Sandwich Islands South Sudan Spain Sri Lanka Sudan Suriname Svalbard and Jan Mayen Sweden Switzerland Syrian Arab Republic Taiwan Tajikistan Tanzania Thailand Timor-Leste Togo Tokelau Tonga Trinidad and Tobago Tunisia Turkmenistan Turks and Caicos Islands Tuvalu Türkiye Uganda Ukraine United Arab Emirates United Kingdom United States United States Minor Outlying Islands Uruguay Uzbekistan Vanuatu Venezuela Viet Nam British Virgin Islands U.S. Virgin Islands Wallis and Futuna Western Sahara Yemen Zambia Zimbabwe Åland Islands British Antarctic Territory Burma, Socialist Republic of the Union of Byelorussian SSR Soviet Socialist Republic Canton and Enderbury Islands Czechoslovakia, Czechoslovak Socialist Republic Dahomey Dronning Maud Land East Timor France, Metropolitan French Afars and Issas French Southern and Antarctic Territories German Democratic Republic Gilbert and Ellice Islands Johnston Island Midway Islands Netherlands Antilles Neutral Zone New Hebrides Pacific Islands (trust territory) Panama Canal Zone Serbia and Montenegro Sikkim Southern Rhodesia US Miscellaneous Pacific Islands USSR, Union of Soviet Socialist Republics Upper Volta, Republic of Viet-Nam, Democratic Republic of Wake Island Yemen, Democratic, People's Democratic Republic of Yugoslavia, (Socialist) Federal Republic of Zaire, Republic of

Please specify:

US State*

-- Please Choose One or More --

Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming

Geographical unit*

You selected Supranational Level as geographical unit. Dataset Country must be Multiple Countries / Intercountry Data.

The level of dataset must be Supranational / International for Multiple Countries / Intercountry Data level.

Dataset language (language of data in text column)* -- Please Choose One -- Armenian Assamese Azerbaijani Basque Belarusian Bengali Bengali Romanize Bosnian Breton Bulgarian Burmese Burmese zawgyi font Cantonese Catalan Chinese (Simplified) Chinese (Traditional) Croatian Czech Danish Dutch English Esperanto Estonian Filipino Finnish French Galician Georgian German Greek Gujarati Hausa Hebrew Hindi Hindi Romanize Hungarian Icelandic Indonesian Irish Italian Japanese Javanese Kannada Kazakh Khmer Korean Kurdish (Kurmanji) Kyrgyz Lao Latin Latvian Lithuanian Macedonian Malagasy Malay Malayalam Mandarin Marathi Mongolian Nepali Norwegian Oriya Oromo Pashto Persian Polish Portuguese Punjabi Romanian Russian Sanskrit Scottish Gaelic Serbian Sindhi Sinhala Slovak Slovenian Somali Spanish Sundanese Swahili Swedish Tamil Tamil Romanize Telugu Telugu Romanize Thai Turkish Ukrainian Urdu Urdu Romanize Uyghur Uzbek Vietnamese Welsh Western Frisian Xhosa Yiddish

Please specify:

You selected a language for which limited or no training data is available. The prediction results, therefore, may be worse than for those with extensive training data: Catalan , Czech , Danish , Dutch , English , French , German , Hungarian , Italian , Norwegian , Polish , Portuguese , Slovak , Spanish .

Domain* -- Please Choose One -- Media Social media Parliamentary speech (oral questions, interpellations, bill debates, other plenary speeches, urgent question) Legislative documents (bills, laws, motions, legislative decrees, hearings, resolution, other) Executive speech Executive orders Party programs Judiciary Budget Public opinion Local government (agenda items, regulations)

You selected a language and domain combination for which limited or no training data is available. The prediction results, therefore, may be worse than for those with extensive training data.

Media Outlet

Social media platform*

-- Please Choose One or More --

Other Facebook LinkedIn Twitter VK

Social Media Platform Name*

Classification for policy content vs. no policy content*

Implement first a binary classification to see if there is any policy content in the unit of observation (makes sense e.g. for speeches which may not be strictly about policy).

Classify all units of observation for CAP codes regardless of whether they contain policy content or not (makes sense for policy-heavy documents such as budgets).

Unit of observation* -- Please Choose One -- Quasi sentence Sentence Paragraph Full text Budget item

Period (from)*

Period (to)*

Use case* -- Please Choose One -- Research Commercial use Personal use Other

Description

The non-coded datasets should contain an "id" and "text" column. The column names must be in row 1. You are free to add supplementary variables to the dataset beyond the compulsory ones in the columns following them. All datasets must be uploaded in a CSV file format with UTF-8 encoding.

• Dataset in .CSV format:
• Choose file

• Pre-coded dataset:
• Choose file

• Codebook:
• Choose file

I have read the instructions and rules.

By ticking the checkbox I declare that this data does not contain any personal data and I am responsible for the legitimacy of the origin of the data. The uploaded data is suitable for text mining and I voluntarily provided the uploaded file(s) to poltextLAB. I also grant permission to poltextLAB researchers to make the data visualisation of the data I uploaded publicly available. In addition, I agree that the uploaded data may be stored on servers within the European Union and be used for data analysis and training purposes without time limit.

By ticking this checkbox, I consent to the processing of the personal data I provide in connection with the file upload (e.g., name, email address) for the sole purposes of returning the processed files and identifying the ownership of the uploaded data. I understand that my personal data will be securely stored and used exclusively for these purposes, in compliance with the purpose limitation principle (Article 5(1)(b)) and the data minimisation principle (Article 5(1)(c)) of the GDPR. I am also aware of my rights under Article 17 (Right to Erasure (‘right to be forgotten’)) and Article 15 (Right of access by the data subject) of the GDPR, and I may request the deletion or access to my personal data at any time (see full GDPR Compliance Statement below).

Loading...

Troubleshooting

If you are experiencing problems with the upload form, or your submission returns an error message (particularly "Something unexpected happened during upload. Please try again later."), please try performing the following steps:

• If you use an adblocker browser extension, please turn it off for our site. Adblockers may interfere with legitimate functionality, such as the dropdowns on the upload form. (We do not serve ads on the site.)
• Try turning off your VPN.
• Try submitting your data from another browser, preferably with default settings.

If you are still receiving the "Something unexpected..." error message, please get in touch with us via our email address or the contact form. Try to add as much information as possible, e.g., what browser you are using, notable browser extensions, whether you are using a VPN or not, and exactly how you tried to submit the data (for example, you filled out everything but waited 10 minutes before pressing submit).

The research was supported by the Ministry of Innovation and Technology NRDI Office within the RRF-2.3.1-21-2022-00004 Artificial Intelligence National Laboratory project and received additional funding from the European Union's Horizon 2020 program under grant agreement no 101008468. We also thank the Babel Machine project and HUN-REN Cloud (Héder et al. 2022; https://science-cloud.hu) for their support. We used the machine learning service of the Slices RI infrastructure (https://www.slices-ri.eu/)

HOW TO CITE: If you use the Babel Machine for your work or research, please cite this paper:

Sebők, M., Máté, Á., Ring, O., Kovács, V., & Lehoczki, R. (2024). Leveraging Open Large Language Models for Multilingual Policy Topic Classification: The Babel Machine Approach. Social Science Computer Review, 0(0). https://doi.org/10.1177/08944393241259434

GDPR Compliance Statement

1. Nature of the Uploaded Data: The files uploaded by users to the tool do not contain personal data as defined in Article 4(1) of the GDPR, which specifies personal data as "any information relating to an identified or identifiable natural person ('data subject')".
2. Data Process: The files submitted to our tool are stored in a secure cloud environment to allow processing and generation of the output (the coded CSV file). Personal data provided in connection with the file upload—such as the submitter's name, email address, and similar details—are used exclusively for the purpose of sending the coded files back to the user and identifying the organisation of our users. This processing is conducted in compliance with the purpose limitation principle (Article 5(1)(b)) and the data minimisation principle (Article 5(1)(c)) of the GDPR. By submitting the files, the user consents to this data processing, which is strictly limited to returning the results and identifying the file owner. The personal data is stored securely and retained solely for these purposes. In accordance with Article 17 of the GDPR (Right to Erasure, or "Right to be Forgotten"), users may request the deletion of their personal data at any time. Such requests will be processed promptly, and all related personal data will be permanently deleted from our systems.
3. Training Purposes: We do not use personal data to train machine learning models or perform any other type of analysis. When submitting files, the submitter must declare that the uploaded CSV files do not contain any personal data, as stated in the consent agreement. This approach aligns with the purpose limitation principle (Article 5(1)(b)) of the GDPR, which requires data to be collected for "specified, explicit, and legitimate purposes" and not further processed in a manner incompatible with those purposes.
4. Google Cloud Platform Compliance: The files submitted to our tool are stored in a secure cloud environment provided by Google Cloud Platform, with configurations ensuring that all processing occurs on servers located within the European Union (EU). This guarantees compliance with GDPR requirements related to data residency and cross-border data transfers. The use of Google Cloud Platform as our processing environment ensures high levels of data security and compliance with GDPR, including the application of the Standard Contractual Clauses (SCCs) for any necessary data transfers. Google Cloud's infrastructure is certified under internationally recognised standards, such as ISO 27001, ISO 27017, and ISO 27018, further ensuring the security and confidentiality of uploaded data.