WangPinchen_EnJaTranslation_slide

GS-6-02 Prompting pre-trained Large Language Models for formality controlled En-Ja Translation June 7th, 2023 王品蓁(1), Edison Marrese-Taylor(1,2), 松尾豊(1) (1)University of Tokyo, (2)AIST 1

Table of Contents 1. Introduction 2. Proposed Approach 3. Empirical Study 4. Conclusion 2

Table of Contents 1. Introduction 2. Proposed Approach 3. Empirical Study 4. Conclusion 3

1-1 Motivation Why Formality Controll in Japanese? • Formality and honorifics are linguistic phenomena that exhibit significant variation across different languages and cultures. • Most of the major languages (English, Chinese, etc.) rely on complex syntactic structures, advanced phrasings to convey formality. Type Vocabulary Formality in General Grammar Vocabulary Relative Clause Informal Formal The result of this research is a mess, The result of this study turns out so it is not surprising that the boss confusing, and it is unsurprising that decided to stop investing. the board of directors decide to no longer invest in this research. We passed the Voting Right Act and the Civil Right Act. We will create a world and promise you all freedom and democracy. At the same time, we believe all citizens will have a fair opportunity to live and work. Voting Right Act was passed, and Civil Right Act was signed. We will build a society where freedom and democracy will be embraced, and fair opportunity will be guaranteed. 4

1-2 Hurdle Why Formality Controlled Machine Translation? • Modern translation system (Google Translate, DeepL) can produce reliable translation between English and Japanese but struggle with applying accurate formality forms. • Lack of consistent evaluation method makes it difficult to verify performance of formality control of an MT model. → human assessment, rule-based simple models Objective • Propose a novel approach to evaluate the ability of En-Ja MT models to perform formality control, using a robust Transformer-based classifier that leverage state-of-the-art large language models (LLMs). • Present empirical study evaluating the zero-shot ability of several LLMs, including ChatGPT, to perform formality control by means of prompting. 5

1-3 Experimental Design LLM Informal output Classifier Polite Formal Evaluation Matrix Prompt source Formality Level BLEU score 6

Table of Contents 1. Introduction 2. Proposed Approach 3. Empirical Study 4. Conclusion 7

2-1 Formality Classes 常体（敬語でない語句で構成され ている文体） 丁寧語（です・ます形） 尊敬語、謙譲語、美化語（フォー マルの場合で使った方が良い言葉 8

2-2 Datasets CoCoA-MT Existing Dataset KeiCO ✅ IWSLT formality control for spoken language translation 2022 dataset Japanese formality corpus New Corpora Daily ✅ textbook, newspaper, novels, tweets KoKai ✅ meeting minutes from House of Representatives and Councilors of Japan (1947-2022) Reannotated ReCoCoA-MT ✅ flaws of incomplete and broken sentences, lack of clear semantic meaning reannotation of CoCoA-MT 9

2-3 Models & Performance Liu & Kobayashi’s 0.73 Performance • When tested on the KeiCO dataset, our classifier performs the best. Transformer BERT 0.81 BASELINE Rule-Based Ｍodel • When tested on CoCoA-MT, 3-way classifiers are adjusted to binary classifiers by combining the polite and formal class. 0.62 tested on KeiCO tested on CoCoA-MT 10

Table of Contents 1. Introduction 2. Proposed Approach 3. Empirical Study 4. Conclusion 11

3-1 Empirical Setup Dataset • CoCoA-MT_test → 594 examples s’_informal s_en 聞いたことがないね。 こうした偶然は、私にとって常に興味深いよ。 I've never heard that. Those types of coincidences are always interesting to me. s’_formal 聞いたことがありません。 こうした偶然は、私にとって常に興味深いです。 Models M2M100 (418M) M2M100 (1.2B) NLLB200 GPT-3 ChatGPT (GPT-3.5) Machine Translation Models 12

3-2 Prompt Setup M2M100 (418M) M2M100 (1.2B) s’_informal [ p s_en ] MT s’_formal NLLB200 s’_informal GPT-3 Translate English to ChatGPT (GPT-3.5) p Japanese: s_en LLM s’_formal 13

3-3 Performance CoCoA-MT released ”matching accuracy” 14

3-3 ChatGPT – Recognize Flaws or incomplete sentences Translate English to informal Japanese: my cousins down south, they are always looking out, North Carolina, they come to New York, everybody loves New York, their family want to see I'm sorry, there seems to be some part of the sentence missing. Could you please provide me with the complete sentence in English so I can translate it accurately to informal Japanese? Translate English to informal Japanese: Oh yeah, it’s true I guess. And you know, it surprises me that surprises that Berlin are, you know, any of Jessy’s until now I’m sorry, but the sentence you provided doesn’t make much sense in English, so it’s difficult for me to translate it accurately. Could you please clarify or provide more context? 15

3-4 ChatGPT – Explain Translation Translate English to formal Japanese: plan and be spontaneous and a little bit of everything, not everything planned for me that’s just not fun. But my funniest vacation 計画的でありつつも、spontaneity(自発性)があり、いろいろなことを少しずつ やりたいです。すべてのことが計画されていると、それは楽しくありません。 しかし、私が最も楽しかった休暇は... (Note: The word ‘funniest’ could have different meanings, so I translated it as ‘most enjoyable.’) 16

Table of Contents 1. Introduction 2. Proposed Approach 3. Results 4. Conclusion 17

4-1 Conclusion Conclusion • Propose new approach to evaluate the En-Ja MT models’ formality control ability using a transformerbased classifier trained on multiple datasets. • Offer a reliable and accurate evaluation matrix for formality-controlled MT systems. • Demonstrates the ability of LLMs to generate sequences with varying levels of formality through well-designed prompts. Future Tasks? • LLM の呪文詠唱 → ちゃんとする長いpromptを使う • LLM の無詠唱呪文 → zero-shot. promptを無視する。（詠唱無視すると、魔法の効果は通常に比べ、1/2よ り少ない威力になることがある。）