diff --git a/zk/AWS_CLI.md b/zk/AWS_CLI.md new file mode 100644 index 0000000..e290e53 --- /dev/null +++ b/zk/AWS_CLI.md @@ -0,0 +1,24 @@ +--- +tags: [AWS] +--- + +# SAM frequent commands + +### Retrieve current user + +``` +aws-sts get-caller-identity +``` + +### List users + +``` +aws configure list +aws configure list-profiles +``` + +### View profile data + +``` +vim ./aws/credentials +``` diff --git a/zk/Local_AWS_development_with_SAM.md b/zk/Local_AWS_development_with_SAM.md new file mode 100644 index 0000000..fb9c3b8 --- /dev/null +++ b/zk/Local_AWS_development_with_SAM.md @@ -0,0 +1,131 @@ +--- +tags: [AWS, docker] +--- + +# Local AWS development with SAM + +We can run a local instance of our SAM stack for a given application without +sending requests to the cloud. This is implemented through Docker. + +The SAM CLI handles all the Docker-related tasks, such as pulling the required +Lambda runtime images, creating containers, mounting your code and dependencies, +and running your Lambda functions inside those containers. + +## Basic set up + +Enter your project directory. + +First build your SAM application: + +```sh +sam build +``` + +We then run: + +```sh +sam local start-api +``` + +If you have an API Gateway endpoint that you want to call over the local server. +You will be able to call it after executing the above. + +This will be indicated by: + +![](/img/local-sam-docker.png) + +If we want to invoke the function directly we use: + +```sh + sam local invoke [FunctionName] +``` + +## Using environment variables + +If you have an API key or database credentials, you are going to typically want +to use different values dependent on environment. + +Even if the values are the same accross environments, it's a good idea to not +call a secret when working locally since this request is billable. + +In the example below I show how to set up environment variables for an API key +locally and in production. + +## Create secret + +Go to AWS SecretsManager and add the API key as a secret. This will be sourced +in production. + +## Create local ENV file + +> These must be in JSON to work with SAM: + +### Local env + +```json +// local-env.json +{ + "FunctionName": { + "API_KEY": "xxx-yyy-xxx", + "NODE_ENV": "development" + } +} +``` + +We save these to the root of the given function's directory not at the global +repo level. + +> Be sure to add this to `.gitignore` so that it does not become public + +### Update `template.yaml` + +Every environment variable you intend to use, must exist in the `template.yaml`, +otherwise it will not be sourced at runtime: + +```yaml +... +Resources: + Properties: + Environment: + Variables: + SECRET_ARN: "arn:aws:secretsmanager:eu-west-2:885135949562:secret:wakatime-api-key-X9oF3v", + NODE_ENV: production + API_KEY: +... +``` + +> We go ahead and populate the values for production. But we leave the variable +> we use in development blank, since we don't want it committed and we will +> source it at the SAM invocation. It still needs to exist though. + +### Pass in the environment variable at invocation: + +```sh +sam local start-api --env-vars /home/thomas/repos/lambdas/wakatime-api/get-coding-stats/local-env.json +``` + +In production, the variables required will be automatically sourced from the +`template.yaml` + +### Create handler within the Lambda itself + +You are obviously going to need to distinguish between the different deployments +when the Lambda executes. Here is an example helper function: + +```ts +import * as AWS from "aws-sdk"; + +const secretsManager = new AWS.SecretsManager(); + +async function getApiKey(): Promise { + if (process.env.NODE_ENV === "production") { + const response = await secretsManager + .getSecretValue({ SecretId: process.env.SECRET_ARN as string }) + .promise(); + const secretValues = JSON.parse(response.SecretString as string); + return secretValues.API_KEY; + } else { + return process.env.API_KEY as string; + } +} +``` diff --git a/zk/SAM.md b/zk/SAM.md new file mode 100644 index 0000000..cf5ec50 --- /dev/null +++ b/zk/SAM.md @@ -0,0 +1,294 @@ +--- +tags: [AWS] +--- + +# AWS SAM + +SAM stands for **serverless application model**. It is a framework developed by +AWS to simplify the process of building, deploying and managing serverless +applications. It provides a concise syntax for defining the components of a +serverless application, such as +[Lambda functions](zk/Lambda_programming_model.md), +[API gateway](/zk/AWS_API_Gateway.md) and database tables. + +The SAM infrastructure is defined in a YAML file which is then deployed to AWS. +SAM syntax gets transformed into CloudFormation during the deployment process. +(CloudFormation is a broader and more robust AWS tool for large, highly +scaleable infrastructures). + +## Key features of SAM + +- Single deployment configuration +- Integration with development tools +- Local testing and debugging +- Built on AWS CloudFormation + +## Main technologies required + +### Docker + +Whilst SAM can be used to create a deployable file for AWS it can also be run as +a container for local development with Docker. + +### AWS CLI + +This is installed using Python and allows you to interact directly with AWS via +the command-line. + +### AWS SAM CLI + +See +[https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) + +## Setting up credentials for the AWS CLI + +You require an access key for the given +[IAM user](zk/AWS_User_management_and_roles.md#iam). You should create an IAM +account specific to the project with bounded permissions. + +``` +aws configure +AWS Access Key ID [None]: AK******* +AWS Secret Access Key [None]: ukp****** +Default region name [None]: +Default output format [None]: +``` + +This information can be found in the Security Credentials section of the given +[IAM](zk/AWS_User_management_and_roles.md#iam) user: + +![](/img/access-key-aws.png) + +### Switching between credentials + +You should set up a different IAM user for each project. + +You can do this with: + +```sh +aws configure --profile +``` + +This will then ask you to add the credentials for the user. + +You can switch between different credentials for the user as follows: + +```sh +AWS_PROFILE= sam build + +``` + +## Starting a SAM project + +First create a directory for your project which will serve as the repository: + +```sh +mkdir aws-sam-learning +cd aws-sam-learning +``` + +Then we can use the `sam` cli to bootstrap the project: + +```sh +sam init --runtime nodejs16.x +``` + +We can just click through and accept the basic HelloWorld Lambda. + +This will create the Lambda as well as an API Gateway trigger URL. + +### `template.yaml` + +This is autogenerated and details the main constituents of the project. There +are lots of fields but the most important are the following: + +```yaml +HelloWorldFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: hello-world/ + Handler: app.lambdaHandler + Runtime: nodejs16.x + Architectures: + - x86_64 + Events: + HelloWorld: + Type: Api + Properties: + Path: /hello + Method: get +``` + +This details the location of the [handler function](/Lambda_handler_function.md) +which is contained at the path `hello-world/app.js`: + +```js +exports.lambdaHandler = async (event, context) => { + try { + // const ret = await axios(url); + response = { + statusCode: 200, + body: JSON.stringify({ + message: "hello world", + // location: ret.data.trim() + }), + }; + } catch (err) { + console.log(err); + return err; + } + + return response; +}; +``` + +It also lists the `get` event that we can use to call API Gateway and trigger +the Lambda. + +The full template is below: + +![](/img/sam-template-yaml.png) + +## Adding our own code + +We will create our own function and API Gateway trigger. + +We will place our function after the existing `HelloWorldFunction` + +```yaml +ClockFunction: + Type: AWS::Serverless::Function + Properties: + CodeUri: clock/ + Handler: handler.clock + Runtime: nodejs16.x + Events: + ClockApi: + Type: Api + Properties: + Path: /clock + Method: get +``` + +We can test the syntax with: + +```sh +sam validate +``` + +Just like with `HelloWorld`, we will create a directory for this function: +`clock` and we will initialise it as an `npm` project. + +```sh +mkdir clock +cd clock +npm init +``` + +We will use `handler.js` as our root, handler function. + +We have said in the template file that our `Handler: handler.clock`, therefore +the main function in the `handler` module should be `clock`: + +```js +const moment = require("moment"); + +exports.clock = async (event) => { + console.log("Clock function run"); + const message = moment().format(); + const response = { + statusCode: 200, + body: JSON.stringify(message), + }; + return response; +}; +``` + +The directory structure is as follows: + +![](/_img/sam-directory.png) + +When we call the API Gateway path `/clock` with `GET`, our function will be +triggered. + +## Deploying the project + +We will now deploy our project to AWS from the local environment. + +The process is as follows: + +1. Build +2. Package +3. Deploy + +### Build + +We need to install the runtime dependencies for the function. We do this by +running `sam build`. This ignores test files and development dependencies and +installs the project dependencies and source files to a temporary subdirectory. + +![](/img/sam-build.png) + +The build directory is `.aws-sam/build/`. There will be a subdirectory for each +of our files. + +### Package + +As noted, CloudFront handles the deployment of the application. It can only +receive one file as an input. The packaging process consists in creating that +single file. + +The packaging proces will first archive all of the project artefacts into a zip +file and then upload that to [S3](zk/AWS_S3.md). A reference to this S3 entity +is then provided to CloudFormation. + +![](/img/s3-package-again.svg) + +The command is as follows: + +```sh +sam package + --template-file template.yaml + --output-template-file pkg.yml + --region eu-west-1 +``` + +This will automatically create a hashed bucket name for you in S3 (I have tried +to add my own naming but it doesn't comply.) + +### Local development with Docker + +In order to work with your application locally without actually sending requests +to AWS and using credit, you can run a local instance. + +See [Local AWS Development with SAM](zk/Local_AWS_development_with_SAM.md). + +### Deploy + +Once you have packaged the app you can deploy with `sam deploy --guided`. This +will talk you through the defaults and will deploy the package to +CloudFormation. In CloudFormation each individual project is called a **stack**. + +If we then go to Cloud Formation we will see the deployed application. + +![](/img/cloud-formation-stack.png) + +## Call the endpoint + +If we now go to the Lambda console, we will see our function listed, and the API +Gateway endpoint under `triggers`: + +![](/img/gateway-trigger.png) + +We can then call this from Postman to check everything is working as it should: + +![](/img/postman-aws-output.png) + +## Clean up and erase the stack + +We can delete the stack and remove all the resources we have created with a +single CLI method: + +```sh +aws cloudformation delete-stack --stack-name --region +```