Cloud Foundry Summit 2017

>100 Views

June 21, 17

#Cloud Foundry #Yahoo! JAPAN #DevOps #CI/CD #大規模システム

スライド概要

Yahoo! JAPANが、どのようにCloud Foundryを使っているのか？
実際いどのように運用をしているか？をご紹介します。
https://www.cloudfoundry.org/event/summit-silicon-valley-2017/

Yahoo!デベロッパーネットワーク

@ydnjp

スライド一覧

2023年10月からSpeaker Deckに移行しました。最新情報はこちらをご覧ください。 https://speakerdeck.com/lycorptech_jp

またはPlayer版

埋め込む »CMSなどでJSが使えない場合

（ダウンロード不可）

関連スライド

深層学習による自然言語処理入門: word2vecからBERT, GPT-3まで

Yahoo!デベロッパーネットワーク 166.1K

ゼロから始める転移学習

Yahoo!デベロッパーネットワーク 86.6K

ヤフーにおける WebAuthn と Passkey の UX の紹介と考察 #idcon #fidcon

idcon fidcon

Yahoo!デベロッパーネットワーク 74.2K

OpenID Connectとネイティブアプリを取り巻く仕様と動向 Yahoo! JAPANの取り組み #openid #openid_tokyo

openid openid_tokyo

Yahoo!デベロッパーネットワーク 58.1K

運用業務とスクラムは本当に組み合わせにくいのか︖運用業務が大半を占めるプロダクト開発での試行錯誤

devsumi

Yahoo!デベロッパーネットワーク 36K

ヤフーのオンプレ機械学習基盤AIPFについて #ml_kubernetes

ml_kubernetes

Yahoo!デベロッパーネットワーク 29.3K

各ページのテキスト

Outline Yahoo Japan Corporation (SoftBank Group) Businesses: Internet Advertising - e-Commerce - Member services - Others Headquarters: Tokyo Japan Founded: January 31, 1996 # of Employees: 5,826 (As of March 31, 2017) 7 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

10.

11.

12.

13.

14.

15.

16.

17.

Architecture Differs by Web Service Small-scale web services Large-scale web services API Gate way e.g. Travel tips 17 Search logic list logic cart logic e.g. Yahoo! Auction Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

18.

19.

Solutions Enroll CF Coach in each web services around 20 staff / 15 services Coaches role: Promote cloud design methods that suit for each web services 19 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

20.

Role map Core Team CF Coach For Shopping Shopping engineer 20 CF Coach For Auction Shopping engineer Auction engineer Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. CF Coach For Media Media engineer …

21.

22.

23.

List Necessary Functions Service A Service B MySQL Oracle Service C Service D Service E ● ● ● ● Service F Service G ● ● ● KVS ObjectStrage ● C/C++ PHP ● ● ● Node.js ● Java ● advertisement ● beacon ● 23 ● ● ● ● ● ● ● ● Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. ● ● ● ● ●

24.

Challenges we encountered Functions that can’t be used in the cloud because of complicated dependency Internal security polices are not suited for cloud environment Most of our web services were stateful design 24 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

25.

How we started We selected one web service, and started by preparing the necessary functions for that service Resolve issues each time they occur 25 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

26.

27.

Criteria for the web service 1. Simplicity • Service with limited functions and external PF that can be used 2. Actively developed • Web services that actively developed so that the effectiveness of introducing CF can be measured 27 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

28.

First target : CS tool n Characteristics • • • • Language: PHP Framework： cakephp Uses REST API MySQL HTTPS (our auction service) n Server Configuration • • • 28 ATS ATS CS tool (apache) CS tool (apache) HTTP API Constructed with few servers in OpenStack environment WebServer: apache Apache Traffic Server (ATS) : Reverse Proxy Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. MySQL MySQL

29.

Partial release using ATS (entry points) Partially diverted entry point to CF apps using ATS: • So that CF or OpenStack can be switched in entry points HTTPS ATS ATS Some entry points CS tool (apache) CS tool (apache) HTTP API 29 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. CS tool (CF)

30.

Lessons learned from the first target • How to Implement in Production • Development method based on OSS • How to make service stateless on CF 30 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

31.

32.

33.

Hello CF Summit 2017! Yusuke Kondo or @konfoo Responsible to... • operating Cloud Foundry & Concourse on IaaS • increasing engineersʼ productivity by providing tools and best practices around CI/CD Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

34.

35.

36.

Current Status 36 (As of Jun. 9, 2017) dev production Cluster # 1 1 Cell # per Cluster 40 30 Org # 136 38 App Instance # approx. 2,000 approx. 400 Rps at peak time N/A approx. 2,000 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

37.

Future Plan 37 (As of Jun. 9, 2017) dev production Cluster # 1 1 => 6 Cell # per Cluster 40 30 => 100 Org # 136 38 App Instance # approx. 2,000 approx. 400 Rps at peak time N/A approx. 2,000 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

38.

39.

40.

Integration with Existent Platforms • Cookie off-loading Route Service • On-demand MySQL (OpenStack Trove API) • 40 Distributed pubsub service (Pulsar) Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

41.

42.

Issues we faced Platform ACL is based on IP address or hostname => Requesting for exceptional permission for accessing via IP Range with a limited term. => Migrating from Host-based ACL to Role-Based ACL in the long term 42 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

43.

44.

45.

User-side Logs and Metrics PCF Cluster-1 VM App App VM Splunk App App App Loggregat or NoVMaction isVMneeded for App developers APP APP APP 45 APP APP APP Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. Monitoring PF

46.

What we prepared Firehose Nozzle and Relay Server • Nozzle filters and formats the App logs streamed by Firehose • Relay Server forwards the log stream to specific index 46 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

47.

48.

Operator-side logs and Metrics Splunk • Platform logs such as CF component syslog Prometheus • • 48 Bosh metrics, VM metrics, Firehose metrics Emitting alerts to our smartphone Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

49.

50.

51.

Integration with package monitoring tool Vulnerable Package Monitoring Tool Application Source Code Dependent packages Runtime Buildpack version Track the buildpack version which the App are staged with and report outdated apps. 51 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

52.

Integration with package monitoring tool Application Source Code Dependent packages Scan whole source code Scan package version Runtime 52 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

53.

Integration with Concourse We use Concourse for 53 • deploying new Cloud Foundry release • updating buildpacks • syncing employee accounts with UAA • backup database to object storage • ... Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

54.

55.

We are still on the way to change mind Changing your organization mind is the most essential part. • Educate not only users, but also platform division where you belong. • Work closely with your security paranoid team. Involve them to update the policy 55 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

56.