Cloud Computing in PHP With the Amazon Web Services

1. Cloud Computing in PHP With the Amazon Web Services Jeff Barr, Senior Web Services Evangelist jbarr@amazon.com @jeffbarr on Twitter

2. Introduction • Based in Seattle • Career path: • Startups • Microsoft • Consultant to VCs and startups • Amazon Web Services • Lead AWS Blogger • Author of “Host Your Website in the Cloud”

3. Session Outline • Amazon / Cloud / AWS Intro • AWS Overview • Programmable Infrastructure • S3, EC2, SDB, SQS • Programming AWS in PHP • AWS SDK for PHP • Scalable Image Processing • Visualizing Infrastructure • Dynamic Infrastructure • Wrapup / Q & A

4. AMAZON’S THREE BUSINESSES Consumer (Retail)Business SellerBusiness Developers &IT Professionals Tens of millions of active customer accounts Seven countries: US, UK, Germany, Japan, France, Canada, China Sell on Amazon websites Use Amazon technology for your own retail website Leverage Amazon’s massive fulfillment center network On-demand infrastructure for hosting web-scale solutions Hundreds of thousands of registered customers

5. What is Cloud Computing? First, think of your electricity service… Power is available to you on-demand, you pay only for what you use… …and you plug into a vast electrical grid managed by professionals to get you the lowest cost, most reliable power with much greater efficiency and safety than you could probably do on your own.

6. Introducing Amazon Web Services AWS provides flexible, scalable, secure, and cost-effective IT infrastructure for businesses of all sizes around the world. Compute power and storage is available to you on-demand, you pay only for the resources you use… …running on scalable, reliable, and secure infrastructure operated by Amazon Web Services, based on the knowledge gleaned from over a decade of building efficient and dependable infrastructure for Amazon.com.

7. Amazon Web Services Application Monitoring Amazon CloudWatch Management AWS Management Console Tools AWS Toolkit for Eclipse Java, PHP, Ruby, Python, .Net Developer Centers Isolated Network Virtual Private Cloud Parallel Processing Amazon Elastic MapReduce Content Delivery Amazon CloudFront Messaging Amazon Simple Queue Service (SQS) Amazon Simple Notification Service (SNS) Payments Amazon Flexible Payments Service (FPS) On-Demand Workforce Amazon Mechanical Turk • Compute • Amazon Elastic Compute Cloud (EC2) • Elastic Load Balancing • Auto Scaling • Storage • Amazon Simple Storage Service (S3) • AWS Import/Export Database Amazon RDS Amazon SimpleDB Third-Party Offerings Identity and Access Management Metering and Billing

8. Programmable Infrastructure • All functionality accessed by APIs • Amazon and third-party libraries • Command-line tools • AWS Management Console • Third-party Tools

9. Code to Launch an EC2 Instance // Run an instance $EC2 = new AmazonEC2(); $Options = array('KeyName' => "Jeff's Keys", 'InstanceType' => "m1.small"); $Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);

10. Amazon Simple Storage Service (S3) • Scalable data storage in-the-cloud • Highly available and durable • Pay-as-you-go pricing: • Storage: tiered $0.15/GB to $0.055/GB • Data Transfer Out: tiered $0.15/GB to $0.080/GB • Data Transfer In:Free until June 30, 2010 • Requests: nominal charges • Big and busy: • 152 billion objects • 150K requests/second Prices and facts are current as of October 2010.

11. Amazon Elastic Compute Cloud (EC2) • Amazon EC2: on-demand compute power • Obtain and boot new server instances in minutes • Quickly scale capacity up or down • Key features: • Support for Linux, Windows, and OpenSolaris • Supports all major web and application platforms • Deploy across Availability Zones for reliability • Elastic IPs provide greater flexibility • Persistent storage with Amazon Elastic Block Store • Monitoring (CloudWatch), Load Balancing, Auto-Scaling

12. Keypairs: CreateKeyPair DescribeKeyPairs DeleteKeyPair Security Groups: CreateSecurityGroup DescribeSecurityGroups DeleteSecurityGroup AuthorizeSecurityGroupIngress RevokeSecurityGroupIngress Block Storage Volumes: CreateVolume DeleteVolume DescribeVolumes AttachVolume DetachVolume CreateSnapshot DescribeSnapshots DeleteSnapshot EC2 Programming Interface (API) • Images: • RegisterImage • DescribeImages • DeregisterImage • ModifyImageAttribute • DescribeImageAttribute • ResetImageAttribute • Instances: • RunInstances • DescribeInstances • TerminateInstances • StopInstances • GetConsoleOutput • RebootInstances • IP Addresses: • AllocateAddress • ReleaseAddress • AssociateAddress • DisassociateAddress • DescribeAddresses • VPC: • CreateCustomerGateway • DeleteCustomerGateway • DescribeCustomerGateways • AssociateDhcpOptions • CreateDhcpOptions • DeleteDhcpOptions • DescribeDhcpOptions • CreateSubnet • DeleteSubnet • DescribeSubnets • CreateVpc • DeleteVpc • DescribeVpcs • CreateVpnConnection • DeleteVpnConnection • DescribeVpnConnections • AttachVpnGateway • CreateVpnGateway • DeleteVpnGateway • DescribeVpnGateways • DetachVpnGateway

13. Amazon EC2 Instance Specs Prices and facts are current as of October 2010.

14. Amazon SimpleDB • Simple, scalable storage solution for structured data • Provides core database functionality for data storage and querying • No schema, no data modeling, no DBA • SQL queries Store: PUT (item, 123), (description, Sweater), (color, Blue), (color, Red) Query: SELECT * FROM Inventory WHERE material='Leather'

15. Amazon Simple Queue Service • Reliable, highly scalable, hosted queue for messaging • Build automated workflows for all applications • Coordinate multiple Amazon EC2 instances Consumer Producer Queue Consumer Producer Producer

16. Getting Started With AWS Create Developer Account Retrieve Private and Public Keys Enter Payment Information Build & Deploy Application Sign Up for Desired Services Monitor and Scale Application

17. Moving Right Along! "Enough with the talk, show us some code, Jeff!"

18. Programming AWS With PHP We’ll use the free AWS SDK for PHP libraries from http://aws.amazon.com/sdkforphp

19. SDK Basics • Download and unpack • Add directory to PHP’s include_path • Add AWS keys to config.inc.php define('AWS_KEY', 'J35NTGFCQOIUY3OMNSQQ'); define('AWS_SECRET_KEY', '99pizu2vVOK11rk9UAgWVj7PBGzWwertqJlgLV0c'); • Include one file: require_once('sdk.class.php');

20. SDK XML / PHP SimpleXML • All methods return a SimpleXML Object: [body] => SimpleXMLElement Object ( [Name] => sitepoint-aws-cloud-book [Prefix] => SimpleXMLElement Object () [Marker] => SimpleXMLElement Object () [MaxKeys] => 1000 [IsTruncated] => false [Contents] => Array ( [0] => SimpleXMLElement Object ( [Key] => images/2008_shiller_housing_projection.jpg • Access as $Res->body->Contents[0]->Key

21. Example 1 – Scalable Processing Pipeline

22. Building a Scalable Image Processing Pipeline • Fetch a web page, store in Amazon S3 • Parse page and extract links to images • Fetch first 16 images on page, store in Amazon S3 • Render images as composite image

23. SQS-Based Application Architecture S3 URL Queue Fetch Images Fetch & Store Page Render Queue S3 S3 Parse Queue Render Images & Pages Parse Page Image Queue

25. Architecture Attributes • Simple • Each stage is easy to understand • AWS reduces low-level coding • Scalable • Add more processes • Add more instances • Add more stages • Storage any amount of data • Fault Tolerant • Messages remain in queues until processed • Messages reappear in queues if process dies • Asynchronous • Each stage runs at its own speed • Build, test,run one stage at a time

26. Define Queues // Queues define('URL_QUEUE', 'c_url'); define('PARSE_QUEUE', 'c_parse'); define('IMAGE_QUEUE', 'c_image'); define('RENDER_QUEUE', 'c_render'); define('FEED_QUEUE', 'c_feed');

27. Create Queues // Create the SQS access object $SQS = new AmazonSQS(); for ($i = 1; $i < count($argv); $i++) { $Queue = $argv[$i]; $Res = $SQS->create_queue($Queue); if ($Res->isOK()) { print("Created queue '${Queue}'\n"); } }

28. Create Queues & Check Status $ ./create_queues.php c_urlc_parsec_imagec_renderc_feed $ ./crawl_queue_status.php c_urlc_parsec_imagec_render ----- ------- ------- -------- 0 0 0 0

29. Locate a Queue's URL // Create the SQS access object $SQS = new AmazonSQS(); // Locate the queue $QueueURL = FindQueueURL($SQS, URL_QUEUE);

30. Post Message to Queue // Create message $HistItem = array('Posted by ' . $argv[0] . ' at ' . date('c')); $Message = json_encode(array('Action' => 'FetchPage', 'Origin' => $argv[0], 'Data' => $argv[$i], 'History' => $HistItem)); // Post message $Res = $SQS->send_message($QueueURL, $Message); if ($Res->isOK()) { print("Posted '${Message}' to QueueURL '${QueueURL}'\n"); }

31. Receive Message From Queue function PullMessage($SQS, $QueueURL) { while (true) { $Res = $SQS->receive_message($QueueURL); if ($Res->isOk()) { if (IsSet($Res->body->ReceiveMessageResult->Message)) { } else { sleep(1); } } }

32. Return Received Message $Message = $Res->body->ReceiveMessageResult->Message; $MessageBody = $Message->Body; $MessageDetail = json_decode($MessageBody, true); $ReceiptHandle = $Message->ReceiptHandle; return array('QueueURL' => $QueueURL, 'Timestamp' => date('c'), 'Message' => $Message, 'MessageBody' => $MessageBody, 'MessageDetail' => $MessageDetail, 'ReceiptHandle' => $ReceiptHandle);

33. Sample Processing Stage // Pull, process, post while (true) { // Pull the message from the queue $Message = PullMessage($SQS, $QueueURL); if ($Message != null) { // Extract message detail $MessageDetail = $Message['MessageDetail']; $ReceiptHandle = $Message['ReceiptHandle']; $PageURL = $MessageDetail['Data']; … } }

34. Detailed Processing // Fetch the page print("Processing URL '${PageURL}':\n"); $HTML = file_get_contents($PageURL); print(" Retrieved " . strlen($HTML) . " bytes of HTML\n"); // Store the page in S3 $Key = 'page_' . md5($PageURL) . '.html'; if (UploadObject($S3, BOOK_BUCKET, $Key, $HTML, S3_ACL_PUBLIC)) { // Get URL in S3 $S3URL = $S3->get_object_url(BOOK_BUCKET, $Key); print(" Uploaded page to S3 as '${Key}'\n");

35. Detailed Processing // Form message to pass page along to parser $Origin = $MessageDetail['Origin']; $History = $MessageDetail['History']; $History[] = 'Fetched by ' . $argv[0] . ' at ' . date('c'); $Message = json_encode(array('Action' => 'ParsePage', 'Origin' => $Origin, 'Data' => $S3URL, 'PageURL' => $PageURL, 'History' => $History)); // Pass the page along to the parser $Res = $SQS->send_message($QueueParse, $Message);

36. Render Images foreach ($ImageKeys as $ImageKey) { // Fetch the image print(" Fetch image '${ImageKey}'\n"); $Image = $S3->get_object(BOOK_BUCKET, $ImageKey); // Convert it to GD format $ImageBits = ImageCreateFromString($Image->body); // Copy it to proper spot in the destination print(" Render image at ${NextX}, ${NextY}\n"); ImageCopy($ImageOut, $ImageBits, $NextX, $NextY, 0, 0, ImageSx($ImageBits), ImageSy($ImageBits)); // Update position for next image $NextX += THUMB_SIZE + GAP_SIZE; if (($NextX + THUMB_SIZE) > $OutX) { $NextX = BORDER_LEFT; $NextY += THUMB_SIZE + GAP_SIZE; } }

37. Example 2 – Create Infrastructure Graph

38. Processing Flow Get AWS Metadata Build Object Model Render Object Model as DOT Graph Render Dot Graph as PDF

39. Get AWS Metadata // Create the service access objects $Service_EC2 = new AmazonEC2); $Service_S3 = new AmazonS3); $Service_SDB = new AmazonSDB); $Service_CF = new AmazonCloudFrontetKey); // Fetch information about all of the EC2 objects $ResElasticIP = $Service_EC2->describe_addresses(); $ResAvailabilityZones = $Service_EC2->describe_availability_zones(); $ResInstances = $Service_EC2->describe_instances(); $ResVolumes = $Service_EC2->describe_volumes(); $ResSnapshots = $Service_EC2->describe_snapshots();

40. Build Object Model // EC2 Instances foreach ($ResInstances->body->reservationSet->item as $ItemSet) { foreach ($ItemSet->instancesSet->item as $Item) { $InstanceId = (string) $Item->instanceId; $ImageId = (string) $Item->imageId; $State = (string) $Item->instanceState->name; $InstanceType = (string) $Item->instanceType; $AvailabilityZone = (string) $Item->placement->availabilityZone; $LaunchTime = (string) $Item->launchTime; $Region->AddInstance(new Instance($AvailabilityZone, $InstanceId, $ImageId, $State, $InstanceType, $LaunchTime)); } }

41. Sample Object (EC2 Instance) class Instance { var $State; var $ImageId; var $InstanceId; var $LaunchTime; var $InstanceType; var $AvailabilityZone; function __construct(…) {} public function GetAvailabilityZone() {} private function GetLabel() {} function Render() {} function RenderEdges() {} }

42. Render Object Model as DOT Graph function WriteNode($FP, $IndentLevel, $Node, $Label, $NodeType) { fwrite($FP, Indent($IndentLevel) . Quote($Node) . ' [' . 'label=' . Quote($Label) . ', ' . GetNodeStyle($NodeType) . "];\n"); }

43. DOT Text graph aws { // Region subgraph "cluster_us-east-1" { label="Region us-east-1"; style=filled; fillcolor=yellow; color=black; // Elastic IP Addresses subgraph "cluster_us-east-1_ips" { label="Elastic IP Addresses"; style=filled; fillcolor=darksalmon; color=black; "75.101.154.199" [label="Elastic IP 75.101.154.199\nec2-75-101-154-199.compute-1.amazonaws.com", color=black, style=filled, fillcolor=wheat, shape=rect]; "us-east-1_phantom_ips" [label="", shape=point, style=invis]; }

44. Render Dot Text to PDF $ dot –Tpdf aws_meta.dot > ~jeff/public_html/aws_meta.pdf

45. Finished Infrastructure Graph

46. Example 3 – Dynamically Instantiate Infrastructure 10 GB EBS Volume EC2 Instance Elastic IP Address 100 GB EBS Volume

47. Run an EC2 Instance // Create the EC2 access object $EC2 = new AmazonEC2(AWS_PUBLIC_KEY, AWS_SECRET_KEY); // Run an instance $Options = array('KeyName' => "Jeff's Keys", 'InstanceType' => "m1.small"); $Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);

48. Get Instance Info // Get the Id and Availability Zone of the instance $Instances = $Res->body->instancesSet; $InstanceId = $Instances->item->instanceId; $AvailabilityZone = $Instances->item->placement ->availabilityZone; print("Launched instance ${InstanceId} " . "in availability zone ${AvailabilityZone}.\n");

49. Allocate Elastic IP Address // Allocate an Elastic IP address $Res = $EC2->allocate_address(); if (!$Res->isOK()) { exit("Could not allocate public IP address.\n"); } // Get the allocated Elastic IP address $PublicIP = $Res->body->publicIp; print("Assigned IP address ${PublicIP}.\n");

50. Attach IP Address to Instance // Associate the Elastic IP address with the instance $Res = $EC2->associate_address($InstanceId, $PublicIP); if (!$Res->IsOK()) { exit("Could not associate IP address ${PublicIP} " . "with instance ${InstanceId}.\n"); } print("Associated IP address ${PublicIP} " . "with instance ${InstanceId}.\n");