Pulsar - Yahoo! JAPAN Meetup

PULSAR Distributed pub/sub platform github.com/yahoo/pulsar Matteo Merli — [email protected] Yahoo Japan — Meetup #3 — 11/04/2016 YAHOO!

What is Pulsar? • Hosted multi-tenant pub/sub messaging platform • Simple messaging model • Horizontally scalable - Topics, Message throughput • Ordering, durability & delivery guarantees • Geo-replication • Easy to operate (Add capacity, replace machines) • Few numbers for production usage: › 1.5 years — 1.4 M topics — 100 B msg/day — Zero data loss › Average publish latency < 5ms, 99pct 15ms › 80+ application onboarded — Self-serve provisioning › Presence in 8 data centers 2 Pulsar YAHOO!

Common use cases • Application integration › Server-to-server control, status propagation, notifications • Persistent queue › Stream processing, buffering, feed ingestion, tasks dispatcher • Message bus for large scale data stores › Durable log › Replication within and across geo-locations 3 Pulsar YAHOO!

Main features • REST / Java / Command line administrative APIs › Provision users / grant permissions › Users self-administration › Metrics for topics / brokers usage • Multi tenancy › Authentication / Authorization › Storage quota management › Tenant isolation policies › Message TTL › Backlog and subscriptions management tools • Message retention and replay › Rollback to redeliver already acknowledged messages 4 Pulsar YAHOO!

Why build a new system? • No existing solution to satisfy requirements › Multi tenant — 1M topics — Low latency — Durability — Geo replication • Kafka doesn’t scale well with many topics: › Storage model based on individual directory per topic partition › Enabling durability kills the performance • Ability to manage large backlogs • Operations are not very convenient › eg: replacing a server, manual commands to copy the data and involves clients › clients access to ZK clusters not desirable • No scalable support to keep consumer position 5 Pulsar YAHOO!

Messaging Model Producer-X Producer-Y Topic-T Exclusive Subscription-A Shared Subscription-B Consumer-A1 Consumer-B1 Consumer-B2 Consumer-B3 Consumer-A1 receives all messages published on T; B1, B2, B3 receive one third each 6 Pulsar YAHOO!

Client API
Producer
PulsarClient client = PulsarClient.create(
"http://broker.usw.example.com:8080");
Producer producer = client.createProducer(
"persistent://my-prop/us-west/my-ns/my-topic");
// Handles retries in case of failure
producer.send("my-message".getBytes());
// Async version:
producer.sendAsync("my-message".getBytes())
.thenAccept(msgId -> {
// Message was persisted
});
Consumer
PulsarClient client = PulsarClient.create(
"http://broker.usw.example.com:8080");
Consumer consumer = client.subscribe(
"persistent://my-prop/us-west/my-ns/my-topic",
"my-subscription-name");
while (true) {
// Wait for a message
Message msg = consumer.receive();
// Process message ...
// Acknowledge the message so that
// it can be deleted by broker
consumer.acknowledge(msg);
}
7 Pulsar YAHOO!

Main client library features • Sync / Async operations • Partitioned topics • Transparent batching of messages • Compression • End-to-end checksum • TLS encryption • Individual and cumulative acknowledgment • Client side stats 8 Pulsar YAHOO!

Architecture / 1 Producer Consumer Broker 1 Broker 2 Broker 3 Bookie 1 Bookie 2 Bookie 3 Bookie 4 Bookie 5 ZK Pulsar Cluster Broker • クライアントはBrokerと のみ接続する • 永続的な状態を持たない Bookie • Apache BookKeeper の ストレージノード • 分散型 write-ahead ログ • 各マシンが多くのtopicの データを保存する 9 Pulsar YAHOO!

Architecture / 2 Producer Consumer Broker 1 Broker 2 Broker 3 Bookie 1 Bookie 2 Bookie 3 Bookie 4 Bookie 5 ZK Pulsar Cluster Brokerとストレージ (Bookie) のレイヤを分割 • BrokerとBookieは独立 して追加できる • トラフィックはBroker を通じて高速に分散す る • 新しいBookieは高速に 立ち上がる 10 Pulsar YAHOO!

Architecture / 3 Producer App Pulsar lib Consumer App Pulsar lib Service discovery Dispatcher Load Balancer Managed Ledger BK Client Global replicators Cache Bookie ZK Replication Global ZK Pulsar Cluster Client library • service discoveryを通 じて接続するBrokerを 見つける • Broker に直接接続する • 接続が確立すると認証/ 認可の処理が実行され る • 遅延再接続 11 Pulsar YAHOO!

Architecture / 4 Producer App Pulsar lib Consumer App Pulsar lib Service discovery Dispatcher Load Balancer Managed Ledger BK Client Global replicators Cache Bookie ZK Replication Global ZK Pulsar Cluster Dispatcher • End-to-endの非同期 メッセージ処理 • メッセージをProducer, Bookie, Consumerにコ ピーすることなく中継 する • 参照カウンタを使った バッファをプール Managed Ledger • topic ストレージの抽象 表現 • 最近のメッセージを キャッシュする 12 Pulsar YAHOO!

BookKeeper • レプリケーテッド・ログサービス • 一貫性と永続性を提供 • ノードの障害時にはレプリケーションが実行される • なぜPulsarにとってよい選択なのか？ › シーケンシャルなデータに対して効率のよいストレージ › 各トピックについて、次々と多数のLedgerを作成 › 入出力がBookie間で分散される › 書き込み/読み込みの独立 › 永続性とスピードに応じて、書き込みの方法を調節可能 13 Pulsar YAHOO!

BookKeeper - Storage Writes Reads Bookie Journal Ledger Storage Write Cache Read-Ahead Cache RocksDB - Active Ledgers idx RocksDB - Entry Locations idx Entry Log Entry Log • 一つのBookieは数千のLedgerを 記録・提供可能 • WriteとReadのパスが分離され ている: › 読み込みが書き込みに影響を与え ない › 書き込みはインメモリの書き込み キャッシュに追加され, Journalに commitされる › 書き込みキャッシュはバックグラウ ンドでディスクにフラッシュされる • シーケンシャルな読み込みを可 能にするため, エントリをソート する 14 Pulsar YAHOO!

Performance — Single topic throughput and latency Throughput and 99pct publish latency — 1 Topic — 1 Producer 1,800,000 Latency (ms) 6 5 4 3 2 1 0 1,000 10,000 100,000 1,000,000 10,000,000 Throughput (msg/s) 10 Bytes 100 Bytes 1KB 15 Pulsar YAHOO!

Final Remarks • Check out the code and docs at github.com/yahoo/pulsar • Give feedback or ask for more details on mailing lists: • Pulsar-Users • Pulsar-Dev