Parallel image processing with Lambda
-- Views
April 22, 16
スライド概要
See also http://blog.serverworks.co.jp/tech/2016/03/09/lambda-piv-en/
Cloud Computing, Fluid Dynamics, Mechanical Engineering | PhD | Consulting Director at SUPWAT
関連スライド
各ページのテキスト
Parallel image processing with Lambda Lambdaで並列画像処理 Takahisa Shiratori Dept. Tech. 2 Serverworks Co., Ltd. JAWS-UG HPC, 22nd April, 2016
About Me Takahisa Shiratori 白鳥 貴久 Serverworks Co., Ltd. Experience Mar. 2016 – Present Dept. Tech. 2 @ Serverworks Apr. 2015 – Feb. 2016 Job rotation as newcomer training @ Serverworks Apr. 2012 – Mar. 2015 Ph.D student in Fluid Mechanics
About Serverworks Co., Ltd. Cloud integrator specialized in AWS Premier Consulting Partner 46 in the world 5 in Japan Advanced in backbone systems Our customers
Fluid Mechanics 流体力学 http://www.mdte.jp/laboratory/l_train.html http://guncanon.web.fc2.com/text/kuriki-01.htm https://www.sbd.jp/case/category/industry/electronics.shtml Investigate how fluids (liquids, gases etc) flow 流体（液体, 気体など）がどのように流れるのかを探る
PIV in Fluid Mechanics PIV (Particle Image Velocimetry, 粒子画像流速測定法) PIV
How PIV derive flows?
How PIV derive flows?
How PIV derive flows?
How PIV derive flows? Similar or Not similar…? N N → Evaluate with Cross Correlation f (X i , Y j ) − f m g (X i + X , Y j + Y ) − g m i =1 j =1 R fg = (X , Y ) = f (X , Y ) − f g (X + X , Y + Y ) − g N N i =1 j =1 2 i j m N i =1 j =1 Disadvantage of PIV Solution • • Long calculation time ex) 6000 images, 1024*1024 pixel N 2 i j Parallel image processing with AWS Lambda m
Why Parallel processing? Serial Processing Parallel Processing 1st Image & 2nd Image 2nd Image & 3rd Image 3rd Image & 4th Image ・・・ Time 1/(N-1) Calculation time N: The number of image
What’s & Why AWS Lambda? S3 API Gateway cron etc… Requests Advantages Easy setup Pricing per 100ms Loose coupling Lambda function
Architecture for SINGLE processing 1 2 Preparation Saving PIV algorithm as Lambda function Procedure 5 3 1. Saving two image files on S3 2. Saving an input CSV file on S3 3. Lambda function is invoked. 4 PIV algorithm on Lambda 4. The flow distribution is written out as an output CSV file. 5. The output CSV file is saved on S3.
Architecture for PARALLEL processing 1 2 5 4 6 3 7 Procedure 1. Saving all image files on S3 5. Child-CSV files are saved on S3. 2. Saving parent-CSV on S3 6. Each child-CSV files invoke Lambda function. 3. Lambda function is invoked. 7. Output CSV files are saved on S3. 4. Child-CSV files are generated.
Good results Error vectors will be removed in the post-processing.
Calculation Performance Single Memory Size Timeout Calculation time Max Memory Used Price [MB] [sec] [sec] [MB] [USD] Requests 112 28 0.280 1st & 2nd 1536 300 235 22 0.587 Parallel 2nd & 3rd 234 23 0.585 Interaction? Interaction? 3rd & 4th 111 28 0.278
Summary Image processing was easily parallelized with AWS Lambda. Motion of particles, humans, vehicles etc will be quantified in shorter calculation time.
Thank you for your kind attention! Contact Us Serverworks Co., Ltd. [email protected] +81-(0)3-5579-8029 日本語 http://www.serverworks.co.jp English http://www.serverworks.co.jp/en 17