Generative AI Study Group_15thSesssion_20240116

3.7K Views

January 24, 24

スライド概要

2024年生成系AIの展望と考察

こんにちわ! GASG 主催の杉山です。 https://www.ai-tech-c.jp/generative-ai-study-group-gasg/ 第15回となります。

さて、今回2024年のGen-AIを展望する新春企画です。 2023年大きく発展飛躍したGen-AIが今年どのように変化していくか。 様々な角度から今年一年のトレンドを読み解いていきたいと思います。 ぜひお楽しみに!!

profile-image

Generative Ai Study Group Master

シェア

またはPlayer版

埋め込む »CMSなどでJSが使えない場合

関連スライド

各ページのテキスト
1.

Generative AI Study Group

2.

Agenda

3.

Agenda

4.

Introduction • 今日のパネラー紹介 • 勉強会貢献メンバー

5.

Introduction How make AI do it is all you need!!

6.

Introduction Yes, AI can!!

7.

Introduction • 公式X • https://twitter.com/kunihirosks

8.

Introduction • 前回の振り返り • Web更新 • https://www.ai-tech-c.jp/generative-ai-study-group-gasg/ • アンケート結果 • 今回のアンケート

9.

Introduction •進捗 • テーマとゲスト急募!! • 新企画 • セキュリティ • アルファブロガー hoge様

10.

Introduction • 参加者名簿 • 個別にご連絡をさせていただくため活用させていただく場合があ りますので、是非。 • Form • https://forms.gle/eRmeGtfkQMkDrs7K9

11.

Introduction •AITeCに是非ご入会をご検討ください •驚きの安さ!! •なんと年間10万円ポッキリ!! •安い! 楽しい! 凄い! AITeC!!

12.

Introduction • AITeC会員限定 • Mattermost • • 情報 / 企画 / ビジョン • 厳選とれたて新鮮Info from bookmarks • 僕が一番ChatGPTをうまくつかえるんだぁー!!大会 • ChatGPT相談室 • 疑似人格 • AIを使った社会実験 • Generative Agents • 新AIパラダイム時代の生き方論 • Deparure of Computable Fundamentalism ABCI利用 • スーパーエンジニアが親切にサポート (予定

13.

Introduction • AITeCオープンイベント o【ABCIでLLMを動かそう!セミナー 第2回】 機械学 習・ディープラーニングとは - 人工知能技術コンソー シアム | Doorkeeper https://aitconsortium.doorkeeper.jp/events/168119

14.

Agenda

15.

Theme • Title • NISTEPによる生成AIとDXのすがた • Presenter • 盛岡広域振興局 佐藤 清忠氏 • Contents • https://drive.google.com/file/d/1kU6PVDymrcyABr59ag90Nb4VpywZ0WB/view?usp=sharing

16.

Theme • Title • 2024年生成系AIの展望と考察 ▪ Mainly NLP topics • Presenter • 杉山邦洋 @Generative Ai Study Group .host • Contents • Tech trend, Use case, Issue

17.

Theme •「マルチモーダルAI」「小規模言語モデ ル」2024年の生成AI重要トレンド(Forbes JAPAN) - Yahoo!ニュース https://news.yahoo.co.jp/articles/633019417533 9a101c2f6e8ad36b0c4109943939?page=1

18.

Tech trend

19.

Tech trend •Small model •Beyond Transformer •Related tech

20.

Tech trend • Small model • Open LLM Leaderboard - a Hugging Face Space by HuggingFaceH4 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard • Points ▪ コンピューティング量 ▪ 実行メモリ量 ▪ 推論速度 ▪ 特化型カスタマイゼーション

21.

Tech trend • Small model • Pickup ▪ TinyLlama • jzhang38/TinyLlama:The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens. https://github.com/jzhang38/TinyLlama • 【軽量かつ高速なLLM】TinyLlamaについてまとめてみた #LLM - Qiita https://qiita.com/sergicalsix/items/7cd7665ab90b9f3b343c • LLAMAと完全互換のアーキテクチャおよびトークンナイザー • 1.1Bパラメータモデルは4bit量子化でおよそ550MB RAM上で動作

22.

Tech trend • Small model • Pickup ▪ Phi-2 • Phi-2: The surprising power of small language models - Microsoft Research https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-powerof-small-language-models/ • [2306.11644] Textbooks Are All You Need https://arxiv.org/abs/2306.11644 • 2.7Bパラメータモデル • 高品質な学習データセットでモデル品質を確保

23.

Tech trend • Small model • Pickup ▪ Orca 2 • Orca - Microsoft Research https://www.microsoft.com/en-us/research/project/orca/ • Microsoft's Orca 2 LLM Outperforms Models That Are 10x Larger https://www.infoq.com/news/2023/12/microsoft-orca-2-llm/ • 7B, 13Bパラメータモデル • LLAMA-2のFinetunedモデル • Reasoningが含まれる合成データセットで訓練

24.

Tech trend • Small model • Pickup ▪ DeciLM • Deci/DeciLM-7B · Hugging Face https://huggingface.co/Deci/DeciLM-7B • [2305.13245] GQA:Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints https://arxiv.org/abs/2305.13245 • What is Grouped Query Attention (GQA)? — Klu https://klu.ai/glossary/grouped-queryattention • GQAの仕組みを採用し軽量で高性能 • GQAは例えばLLaMA2 70Bでも使われている

25.

Tech trend 二次的複雑性 (Quadratic complexity O(n^2)) http://tinyurl.com/yqs8pwec • Beyond Transformer • Points ▪ Transformerの課題解決の試み • 推論速度 • 実行メモリ量 • シーケンス長 • コンピューティング量

26.

Tech trend • Beyond Transformer • Pickup ▪ MoE (Mixture of Experts) Reference: http://tinyurl.com/ylxsvomj

27.

Tech trend • Beyond Transformer • Pickup ▪ MoE (Mixture of Experts) • 論文解説 Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (MoE) - ディープラーニングブログ https://deeplearning.hatenablog.com/entry/moe • Mixture of expertsの簡単な実装をしてみる https://zenn.dev/if001/articles/40917524959913 • MixtralSparseMoeBlockを読む https://zenn.dev/if001/articles/fcea9fe9f1bdb1

28.

Tech trend • Beyond Transformer • Pickup ▪ RWKV (Reinventing RNNs for the Transformer Era) • RWKVについて解説 | AGIRobots Blog https://developers.agirobots.com/jp/rwkv/ • RWKVを論文と実装から読み解く https://zenn.dev/jow/articles/f66d6403b9a509 • RNNでTransformer並みの性能を実現するRWKVがやばい https://zenn.dev/hikettei/articles/5d6c1318998411 • これは衝撃!1.5Bで超高性能LLM!RWKV-5-World-v2|shi3z https://note.com/shi3zblog/n/nfc8dd1abf494

29.

Tech trend • Beyond Transformer • Pickup ▪ Mamba • state-spaces/mamba https://github.com/state-spaces/mamba • [2312.00752] Mamba: Linear-Time Sequence Modeling with Selective State Spaces https://arxiv.org/abs/2312.00752 • Mamba: Redefining Sequence Modeling and Outforming Transformers Architecture - Unite.AI https://www.unite.ai/mambaredefining-sequence-modeling-and-outforming-transformers-architecture/ • Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Arxiv Dives | by Oxen | Dec, 2023 | Medium https://medium.com/@oxenai/mamba-linear-time-sequence-modeling-with-selective-state-spaces-arxiv-dives-cf96518d7ec4 ▪ StripedHyena • Architectures for longer sequences and efficient inference: StripedHyena | hessian.AI https://hessian.ai/architectures-for-longersequences-and-efficient-inference-stripedhyena/ • [2302.10866] Hyena Hierarchy:Towards Larger Convolutional Language Models https://arxiv.org/abs/2302.10866

30.

Tech trend • Beyond Transformer • Pickup ▪ MoE-Mamba • [2401.04081] MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts https://arxiv.org/abs/2401.04081 ▪ Monarch Mixer • [2310.12109] Monarch Mixer: A Simple Sub-Quadratic GEMMBased Architecture https://arxiv.org/abs/2310.12109

31.

Tech trend •Related tech •RAG (Retrieval-Augmented Generation) •Agent •Synthetic data (合成データ) •Distributed

32.

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪第6回で取り上げました ▪進化するRAGアーキテクチャ ▪[2312.10997] Retrieval-Augmented Generation for Large Language Models:A Survey https://arxiv.org/abs/2312.10997

33.

Reference: https://arxiv.org/pdf/2312.10997.pdf Figure 6: RAG compared with other model optimization methods Theme • Tech Trend • Related tech ▪ RAG (Retrieval-Augmented Generation)

34.

Reference: https://arxiv.org/pdf/2312.10997.pdf Figure 2: A representative instance of the RAG process applied to question answering Theme • Tech Trend • Related tech ▪ RAG (Retrieval-Augmented Generation)

35.

Reference: https://arxiv.org/pdf/2312.10997.pdf Figure 3: Comparison between the three paradigms of RAG Theme • Tech Trend • Related tech ▪ RAG (Retrieval-Augmented Generation)

36.

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪Advanced • Optimizing data indexing • Pre retrieval process • Post retrieval process

37.

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪ Modular • 多様な機能モジュール • ニーズに適したPipeline

38.

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪ A Cheat Sheet and Some Recipes For Building Advanced RAG | by Andrei | Jan, 2024 | LlamaIndex Blog https://blog.llamaindex.ai/a-cheat-sheet-and-somerecipes-for-building-advanced-rag-803a9d94c41b

39.

Tech trend • Related tech • RAG (Retrieval-Augmented Generation) ▪ Scaling context window ▪ Robustness • Hallucination ▪ Hybrid (RAG+FT) ▪ Expanding LLM role ▪ Scaling law • Embedding model ▪ Production ready • 精度, 再現性, セキュリティ(アクセスコントロール) ▪ Multi modal • Image, Audio and video, Code

40.

Tech trend • Related tech • Agent ▪ 第11回で取り上げました Reference: https://medium.com/scisharp/understand-the-llm-agent-orchestration-043ebfaead1f

41.

Tech trend • Related tech • Agent ▪ 2024 AI Agent ▪ https://e2b.dev/blog/ai-agents-in-2024 ▪ 評価フレームワーク • THUDM/AgentBench:A Comprehensive Benchmark to Evaluate LLMs as Agents https://github.com/THUDM/AgentBench • AutoGPT/benchmark at master · Significant-Gravitas/AutoGPT https://github.com/Significant-Gravitas/AutoGPT/tree/master/benchmark • Benchmarking Agent Tool Use https://blog.langchain.dev/benchmarking-agent-tool-use

42.

Tech trend • Related tech • 合成データ ▪ Synthetic data:Anthropic’s CAI, scaling, OpenAI’s Superalignment, tips, and open-source examples https://www.interconnects.ai/p/llm-synthetic-data ▪ [2305.15041] Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Science https://arxiv.org/abs/2305.15041 ▪ [2310.07849] Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations https://arxiv.org/abs/2310.07849 ▪ [2401.00368] Improving Text Embeddings with Large Language Models https://arxiv.org/abs/2401.00368 ▪ [2312.17742] Learning Vision from Models Rivals Learning Vision from Data https://arxiv.org/abs/2312.17742

43.

Tech trend • Related tech • Distributed ▪ Petals • Petals – Run LLMs at home, BitTorrent-style https://petals.dev/ • bigscience-workshop/petals: Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading https://github.com/bigscience-workshop/petals • [2209.01188] Petals: Collaborative Inference and Fine-tuning of Large Models https://arxiv.org/abs/2209.01188 • [2312.08361] Distributed Inference and Fine-tuning of Large Language Models Over The Internet https://arxiv.org/abs/2312.08361 ▪ AI Horde • AI Horde https://stablehorde.net/ • Haidra-Org/AI-Horde: A crowdsourced distributed cluster for AI art and text generation https://github.com/Haidra-Org/AI-Horde?tab=readme-ov-file

44.

Use case

45.

Use case • 企業における生成AIの未来:ChatGPTを越えてその先へ | ガートナー https://www.gartner.co.jp/ja/articles/beyond-chatgptthe-future-of-generative-ai-for-enterprises • Top 100+ Generative AI Applications / Use Cases in 2024 https://research.aimultiple.com/generative-ai-applications/ • 2024 AI Predictions | NVIDIA Blog https://blogs.nvidia.com/blog/2024-ai-predictions/

46.

Use case • Device ▪ Order Ai Pin Now https://hu.ma.ne/ ▪ rabbit — home https://www.rabbit.tech/ ▪ Avi Schiffmann’s Tab AI necklace raised $1.9 million to replace God https://www.fastcompany.com/91007630/avi-schiffmanns-tab-ai-necklace-hasraised-1-9-million-to-replace-god ▪ OpenAI、「AI版iPhone」をアップル元デザイナーと開発か 孫正義に 出資打診 | Forbes JAPAN 公式サイト(フォーブス ジャパン) https://forbesjapan.com/articles/detail/66348

47.

Issue

48.

Issue •Security •Data contamination •Socialization

49.

Issue • Security • Overview ▪ 8 Generative AI Security Risks That You Should Know https://www.globalsign.com/en/blog/8-generative-ai-security-risks ▪ Safety and security risks of generative artificial intelligence to 2025 (Annex B) - GOV.UK https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/safety-and-security-risks-of-generativeartificial-intelligence-to-2025-annex-b • Prompt hack • GPTs のプロンプトリーキング対策|ぬこぬこ https://note.com/schroneko/n/n6d6c2e645119 • Prompt Hacking | Learn Prompting:Your Guide to Communicating with AI https://learnprompting.org/docs/category/-prompt-hacking • Solution ▪ Introducing Purple Llama for Safe and Responsible AI Development | Meta https://about.fb.com/news/2023/12/purple-llama-saferesponsible-ai-development/ ▪ New generative AI-powered SaaS security expert from AppOmni | VentureBeat https://venturebeat.com/security/new-generative-aipowered-saas-security-expert-from-appomni

50.

Issue • Data contamination • Why data contamination is a big issue for LLMs - TechTalks https://bdtechtalks.com/2023/07/17/llm-data-contamination/ • [2312.16337] Task Contamination: Language Models May Not Be Few-Shot Anymore https://arxiv.org/abs/2312.16337 • Socialization • Sotopia https://www.sotopia.world/ • [2310.11667] SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents https://arxiv.org/abs/2310.11667

51.

Agenda

52.

Study contents • Coming Sooooooooooon!!

53.

Agenda

54.

News • みんなで生成系AIブックマーク • 新企画です!! 皆様の気になった生成系AIの情報をその場で募集して ディスカッションします。 • 募集はこちら https://forms.gle/kKVsb9jDwMTm2Rkk6 から事前投稿も受 け付けています。 • リスト • https://docs.google.com/spreadsheets/d/1bjJfAhfEvNMSuxAszCiSo0qdE89s p86ooAyBdRVuDO0/edit?usp=sharing

55.

Agenda

56.

Next • アンケートの結果とコメント • 次回 • みんなで生成系AIブックマーク" 継続 • 分析してみる? • Theme引き続き募集 • Study contents • セキュリティ • Fine tuning • ChatGPT Plugins • オープンソース各種 • Langchain,LlamaIndex, 他 • Business向けコンテンツ • New era personal computer

57.

EOF https://www.linkedin.com/in/kunihiro-sugiyama-49b0372a/ https://www.ai-tech-c.jp/generative-ai-study-group-gasg/