Cloud Foundry Summit 2017

>100 Views

June 21, 17

スライド概要

Yahoo! JAPANが、どのようにCloud Foundryを使っているのか?
実際いどのように運用をしているか?をご紹介します。
https://www.cloudfoundry.org/event/summit-silicon-valley-2017/

profile-image

2023年10月からSpeaker Deckに移行しました。最新情報はこちらをご覧ください。 https://speakerdeck.com/lycorptech_jp

シェア

またはPlayer版

埋め込む »CMSなどでJSが使えない場合

(ダウンロード不可)

関連スライド

各ページのテキスト
1.

The Road to "JYU-BAI" - Adopting Cloud Foundry at Yahoo! JAPAN - 2017年6月20日 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

2.

About me Yasuhiko Kubono Software Engineer Manager Yahoo! JAPAN 2 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

3.

• Introducing Cloud Foundry into our services -Yasuhiko Kubono • How do we Actually Operate -Yusuke Kondo 3 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

4.

Introducing Cloud Foundry into our services Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

5.

Agenda • • • • 5 About Yahoo! JAPAN Why we use Cloud Foundry? Introducing Cloud Foundry into our services Case study Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

6.

About Yahoo! JAPAN Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

7.

Outline Yahoo Japan Corporation (SoftBank Group) Businesses: Internet Advertising - e-Commerce - Member services - Others Headquarters: Tokyo Japan Founded: January 31, 1996 # of Employees: 5,826 (As of March 31, 2017) 7 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

8.

# of Engineers & Designers 2,500 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

9.

Web Services More than 100 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

10.

Total requests 39,89M 67,4B Active User IDs Page Views 1Month ※2017年1-3月の平均 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

11.

Why we use Cloud Foundry? Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

12.

Why we use Cloud Foundry? Speed up development time JYU-BAI increase productivity by 10 times 12 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

13.

Adoption Plan Here 2017 Full-scale implementation 2018 Expand implementation 2016 Initial introduction to a few services 13 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

14.

Introducing Cloud Foundry into our Services Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

15.

Web Services More than 100 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

16.

Programming Languages C, Perl, C++, PHP, Node.js, Java... 16 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

17.

Architecture Differs by Web Service Small-scale web services Large-scale web services API Gate way e.g. Travel tips 17 Search logic list logic cart logic e.g. Yahoo! Auction Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

18.

Obstacles Same architecture does not fit in each web services 18 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

19.

Solutions Enroll CF Coach in each web services around 20 staff / 15 services Coaches role: Promote cloud design methods that suit for each web services 19 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

20.

Role map Core Team CF Coach For Shopping Shopping engineer 20 CF Coach For Auction Shopping engineer Auction engineer Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. CF Coach For Media Media engineer …

21.

Case study Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

22.

Where we started from Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

23.

List Necessary Functions Service A Service B MySQL Oracle Service C Service D Service E ● ● ● ● Service F Service G ● ● ● KVS ObjectStrage ● C/C++ PHP ● ● ● Node.js ● Java ● advertisement ● beacon ● 23 ● ● ● ● ● ● ● ● Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. ● ● ● ● ●

24.

Challenges we encountered Functions that can’t be used in the cloud because of complicated dependency Internal security polices are not suited for cloud environment Most of our web services were stateful design 24 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

25.

How we started We selected one web service, and started by preparing the necessary functions for that service Resolve issues each time they occur 25 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

26.

So, which web service did we start with? Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

27.

Criteria for the web service 1. Simplicity • Service with limited functions and external PF that can be used 2. Actively developed • Web services that actively developed so that the effectiveness of introducing CF can be measured 27 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

28.

First target : CS tool n Characteristics • • • • Language: PHP Framework: cakephp Uses REST API MySQL HTTPS (our auction service) n Server Configuration • • • 28 ATS ATS CS tool (apache) CS tool (apache) HTTP API Constructed with few servers in OpenStack environment WebServer: apache Apache Traffic Server (ATS) : Reverse Proxy Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. MySQL MySQL

29.

Partial release using ATS (entry points) Partially diverted entry point to CF apps using ATS: • So that CF or OpenStack can be switched in entry points HTTPS ATS ATS Some entry points CS tool (apache) CS tool (apache) HTTP API 29 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. CS tool (CF)

30.

Lessons learned from the first target • How to Implement in Production • Development method based on OSS • How to make service stateless on CF 30 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

31.

Adopting & Expanding to other services Decide target Knowhow accumulated Solve issues 31 Investigate issues Adopting knowledge Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

32.

Next Presentation, How do we Actually Operate Photo by: Aflo Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

33.

Hello CF Summit 2017! Yusuke Kondo or @konfoo Responsible to... • operating Cloud Foundry & Concourse on IaaS • increasing engineersʼ productivity by providing tools and best practices around CI/CD Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

34.

Overview of Yahoo! JAPAN proprietary Infrastructure More than four DCs in Japan More than 90,000 VM running on OpenStack 34 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

35.

Cluster Spec 35 dev production Load Balancer Software Hardware x2 IaaS Openstack Openstack Hypervisor # 40 40 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

36.

Current Status 36 (As of Jun. 9, 2017) dev production Cluster # 1 1 Cell # per Cluster 40 30 Org # 136 38 App Instance # approx. 2,000 approx. 400 Rps at peak time N/A approx. 2,000 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

37.

Future Plan 37 (As of Jun. 9, 2017) dev production Cluster # 1 1 => 6 Cell # per Cluster 40 30 => 100 Org # 136 38 App Instance # approx. 2,000 approx. 400 Rps at peak time N/A approx. 2,000 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

38.

Integration with Backend Services Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

39.

Existent Platforms Cache Service MQ Service FaaS App Role Based ACL RDB Key Value Store 39 Object Storag e Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

40.

Integration with Existent Platforms • Cookie off-loading Route Service • On-demand MySQL (OpenStack Trove API) • 40 Distributed pubsub service (Pulsar) Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

41.

Marketplace Dashboard Goal: 41 Providing all PFs in CF Marketplace Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

42.

Issues we faced Platform ACL is based on IP address or hostname => Requesting for exceptional permission for accessing via IP Range with a limited term. => Migrating from Host-based ACL to Role-Based ACL in the long term 42 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

43.

Integration with Logs and Metrics Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

44.

What we already have In-house Monitoring & Alerting PF based on Apache Kafka, Hbase, Splunk, an enterprise log analytics platform 44 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

45.

User-side Logs and Metrics PCF Cluster-1 VM App App VM Splunk App App App Loggregat or NoVMaction isVMneeded for App developers APP APP APP 45 APP APP APP Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. Monitoring PF

46.

What we prepared Firehose Nozzle and Relay Server • Nozzle filters and formats the App logs streamed by Firehose • Relay Server forwards the log stream to specific index 46 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

47.

Issue we faced High log traffic. 900 lines per sec ! (as of Jun. 8, 2017) => Provided users with CF friendly logger 47 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

48.

Operator-side logs and Metrics Splunk • Platform logs such as CF component syslog Prometheus • • 48 Bosh metrics, VM metrics, Firehose metrics Emitting alerts to our smartphone Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

49.

49 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

50.

Integration with other System Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

51.

Integration with package monitoring tool Vulnerable Package Monitoring Tool Application Source Code Dependent packages Runtime Buildpack version Track the buildpack version which the App are staged with and report outdated apps. 51 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

52.

Integration with package monitoring tool Application Source Code Dependent packages Scan whole source code Scan package version Runtime 52 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

53.

Integration with Concourse We use Concourse for 53 • deploying new Cloud Foundry release • updating buildpacks • syncing employee accounts with UAA • backup database to object storage • ... Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

54.

Lessons learned Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

55.

We are still on the way to change mind Changing your organization mind is the most essential part. • Educate not only users, but also platform division where you belong. • Work closely with your security paranoid team. Involve them to update the policy 55 Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.

56.

Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved.