Rainsbrook

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Amazon Web Services CLI ======


===== Initial install =====

AWS cli tool is written in python, and as python3 is the most recent, this is what will be installed. The awscli tool is installed through pip3. Consider installing this in a virtual environment ([[python:python#virtual_environment|Python Virtual Env]] )

<code>
# yum install python3

... edited...

Install  1 Package (+3 Dependent packages)

Total download size: 11 M
Installed size: 51 M
Is this ok [y/d/N]: y
Downloading packages:
(1/4): python3-3.7.0-0.20.rc1.amzn2.0.1.x86_64.rpm                                     |  64 kB  00:00:01     
(2/4): python3-pip-9.0.3-1.amzn2.0.1.no                                                | 1.9 MB  00:00:03     
(3/4): python3-setuptools-38.4.0-3.amzn2.0.6.noarch.rpm                                | 617 kB  00:00:01     
(4/4): python3-libs-3.7.0-0.20.rc1.amzn2.0.1.x86_64.rpm                                | 8.0 MB  00:00:14     
------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        487 kB/s |  11 MB  00:00:22    

# pip3 install awscli
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting awscli
  Downloading https://files.pythonhosted.org/packages/f8/ab/ab7b15a7a5524f47bb39279a59a7afdb1237162159ba7ff15cab28c96915/awscli-1.16.15-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 981kB/s 
...edited...    
</code>



AWS linux 2 does have a awscli tool in the linux repo, based on python2, but it is not as recent as the pip installed one, this is the python3 based version, check if you need py3 or py2 before installing:-

<code>
# aws --version
aws-cli/1.14.8 Python/2.7.14 Linux/4.14.47-64.38.amzn2.x86_64 botocore/1.8.12
[root@amazonlinux02 ~]#
</code>


Compare to the pip3 installed version:-
<code>
# /usr/local/bin/aws --version
aws-cli/1.16.15 Python/3.7.0rc1 Linux/4.14.47-64.38.amzn2.x86_64 botocore/1.12.5
[root@amazonlinux02 ~]#
</code>



===== Setting up profiles =====

The ''aws configure'' command will set up a default profile, if you need to use different accounts or keys, these can be set in a named profile, the two files, ''config'' and ''credentials'' in ''~/.aws/'' control this (needless to say, these are made up keys and secrets!):-

config
<code>
[default]
output = text


[profile admin1]
role_arn = arn:aws:iam::0123456781234:role/role_admin
source_profile = default
region = eu-west-1


[profile profile2]
region = eu-west-2
source_profile = default
output = text
</code>

credentials
<code>
[default]
aws_access_key_id = QWERTYUIOPASDFGHKEYQ
aws_secret_access_key = HaMPb65IFf0bVoEiLSKEJtuCUo3490nWlrJBES9n


[profile2]
aws_access_key_id = QWERTYUIOPASDFGHKEYA
aws_secret_access_key = wifisUMegS9pY_tpOnQpSY0YJYSiqgeKneMWqqIa
</code>


FIXME

aws configure list

aws iam get-user

iam sts get-caller-identity

===== Setting up roles =====

Roles allow proviledge escalation for a user to perform specific tasks. For this to be used in the cli, an extra section is added to the ''.aws/config'' file:-

<code>
[default]
output = json
region = eu-west-1


[profile sandbox]
role_arn = arn:aws:iam::12345678:role/roles-sandboxadmin
source_profile = default
region = eu-west-1
</code>

When a cli command is run, the ''--profile sandbox'' is added to awitch to that role:-

<code>
server:~/.aws$ aws ec2 describe-instances --filters "Name=instance-type,Values=t2.micro" --query Reservations[].Instances[].InstanceId --profile sandbox
[
    "i-0b99cb98dfabcdefg",
    "i-04888fe41agfedcba"
]
server:~/.aws$
</code>


==== Errors ====

<code>
$ aws ec2 describe-instances --profile nonprod_admin

An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.
$
</code>

This was solved by updating the ''aws_access_key_id'' and ''aws_secret_access_key'' in ~/.aws/credentials.
As it said "The security token included in the request is invalid." Obvious in hindsight.
===== Using roles and profiles with Boto3 =====

''boto3'' is Amazon's python library to interface with the aws cli commands.

A client needs to be set up, and for local cli usage, this needs to be linked with a profile as set above.

<code python>
#!/usr/bin/env python

import boto3

profile = 'nonprod_admin'

# Create ec2 client
session = boto3.session.Session(profile_name=profile)
ec2 = session.client('ec2')

# Create SQS client
session = boto3.session.Session(profile_name=profile)
sqs = session.client('sqs')

</code>

This client (ec2, sqs etc) can be used to set or retreive information as the user in the profile:-

<code python>
AMIResponse = ec2.describe_images(Filters=[{'Name': 'name', 'Values': [Regex]}, ], Owners=['self'])
</code>

===== AWS CodeCommit =====

It's worth pointing out that Code Commit repos are tied to a particular AWS account, so if you operate in a multiple account environment, you will need to ensure your code is stored in the right repo, eg if you have a prod and staging/dev environment, maybe codecommit may not be the easiest repo choice.

See:- [[https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html]].

AWS CodeCommit is a git compatible repository. It uses the git command locally, but if you are using roles, there is a restriction on using only https, not ssh to communticate to the remote repo. Also, there is a tie in with the aws command line which is why CodeCommit is here and not under [[rb:git-cheatsheet|Git Cheatsheet]].

AWS's IAM requires ''HTTPS Git credentials for AWS CodeCommit'' to be created from the ''Security Credentials'' tab under ''Users''. Click Generate and make a note of the values. These will become your git credentials to be used on the cli.
Also, you will be **required** to configure a Credential Helper, the name of this sounds like it is optional, but it isn't. 

IAM periodically resets the password used with the git credentials (above) and the Credential Helper is used to call out to IAM to get the updates password which is then used in the git command.

<code>
git config --global credential.helper '!aws codecommit credential-helper $@'
git config --global credential.UseHttpPath true
</code>

Profiles can be defined per repository by using ''--local'' instead of ''--global''. 


==== Roles with CodeCommit ====

In the .gitconfig file, the commands above add the helper line, but to use it with roles, it needs the ''--profile'' adding:-

<code>
$ more  /home/user/.gitconfig 
[credential]
	helper = !aws --profile sandbox codecommit credential-helper $@
	UseHttpPath = true
$
</code>

My understanding is that ''git'' feeds a string of arguments to the credential-helper ($@) and consumes the string returned to forward on to CodeCommit as the user password.  As it is an ''aws'' command it can take the ''--profile'' option. Without that, the helper will try to return the IAM users credentials not the role's credentials, git will present these and it will pushes and pulls will fail with ''fatal: unable to access <xx-repo> : The requested URL returned error: 403''


If you get an error similar to ''The requested URL returned error: 403'', you may need to export your AWS credentials:-

  $ export AWS_PROFILE=shared-services

 

==== Creating a new repo ====

This is shown with the role option ''--profile sandbox''


<code>
$ aws codecommit create-repository --repository-name CIS-Hardening --repository-description "Repo for ansible code to harden aws Linux2 image." --profile sandbox
{
    "repositoryMetadata": {
        "repositoryDescription": "Repo for ansible code to harden aws Linux2 image.",
        "cloneUrlSsh": "ssh://git-codecommit.eu-west-2.amazonaws.com/v1/repos/CIS-AWS_Linux2",
        "repositoryId": "91fb1234-833e-4705-8e1c-xxxxxxxx",
        "lastModifiedDate": 1537199788.236,
        "accountId": "1234567890",
        "repositoryName": "CIS-Hardening",
        "Arn": "arn:aws:codecommit:eu-west-2:987654321:CIS-Hardening",
        "cloneUrlHttp": "https://git-codecommit.eu-west-2.amazonaws.com/v1/repos/CIS-AWS_Linux2",
        "creationDate": 1537199788.236
    }
}
$
$ aws codecommit list-repositories --profile sandbox
{
    "repositories": [
        {
            "repositoryName": "AMI_PackerDefinitions-AWS_Linux2",
            "repositoryId": "12345678-a3af-4d07-8319-20f112345678"
        },
        {
            "repositoryName": "CIS-Hardening",
            "repositoryId": "87654321-14cd-495a-8b4f-121987654321"
        }
    ]
}

</code>


See the [[rb:git-cheatsheet|Git cheatsheet]] for adding files etc to the repo.


==== 403 error with git push ====

This is an error encountered on a Mac:-

<code>
me (ajs/cpe-1806) $ git push --set-upstream origin ajs/cpe-1806
fatal: unable to access 'https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myrepo/': The requested URL returned error: 403
me (ajs/cpe-1806) $ 
</code>

It's caused (probably) by the "Helper" having rotated the access keys at the AWS end and the Keychain Access app caching the original credentials which are no longer valid. 

Solution is to search for and delete CodeCommit credentials in Keychain Access. 

<code>
me (ajs/cpe-1806) $ git push --set-upstream origin ajs/cpe-1806
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 4 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 988 bytes | 988.00 KiB/s, done.
Total 6 (delta 4), reused 0 (delta 0)
To https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myrepo
 * [new branch]      ajs/cpe-1806 -> ajs/cpe-1806
Branch 'ajs/cpe-1806' set up to track remote branch 'ajs/cpe-1806' from 'origin'.
me (ajs/cpe-1806) $
</code>


==== Get repos ====

Used ''jq'' to process output, this requires json, so the export forces json format. ''-r'' gives a raw output, ie, no quotes round variables. 

get_repos.sh:- 
<code bash>
#!/bin/bash
# List repos from AWS codecommit

export AWS_DEFAULT_OUTPUT="json"

aws codecommit list-repositories | jq -r '.repositories | .[].repositoryName'
</code>

<code>
credential-age-check
pyside6-stuff
</code>


===== AWS NTP Time =====


Use 169.254.169.123 with ntp or chrony.

  server 169.254.169.123 prefer iburst


==== Force timesync with chrony ====


If you get this message:-

<code>
$ ./get_repos.sh

An error occurred (InvalidSignatureException) when calling the ListRepositories operation: Signature expired: 20231207T141711Z is now earlier than 20231207T152038Z (20231207T153538Z - 15 min.)
</code>

your time may be out by more than AWS allows, use this with chrony to force a reset. ''ntpdate'' will force a resync with the ntpd package. You will need to stop the ''ntpd'' daemon first and then restart it after.

<code>
root@ubuntu20:~# chronyd -q 'server time.domain.com  iburst'
2023-10-26T15:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 -DEBUG)
2023-10-26T15:32:22Z Initial frequency 5.379 ppm
2023-10-26T15:32:26Z System clock wrong by 1931.782000 seconds (step)
2023-10-26T16:04:38Z chronyd exiting
root@ubuntu20:~#

</code>

===== Getting info from within a running instance =====

The 169.254.169.254 address allows access to metadata about an instance from within THAT instance, eg:-

<code>
[root@ip-172-31-21-109 ~]# curl http://169.254.169.254/latest/meta-data/ami-id
ami-0f1229ec7823be3db
[root@ip-172-31-21-109 ~]# 

[root@ip-172-31-21-109 ~]# curl http://169.254.169.254/latest/meta-data/public-keys/
0=AndrewAWS
[root@ip-172-31-21-109 ~]#

[root@ip-172-31-21-109 ~]# curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/06:b7:e8:98:98:0a/public-hostname/
ec2-34-244-253-26.eu-west-1.compute.amazonaws.com
[root@ip-172-31-21-109 ~]#
</code>


===== AWS cli =====

<code>
[ec2-user@ip-10-96-10-231 ~]$ aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`].Value|[0], IamInstanceProfile.Arn]' --output table
-----------------------------------------------------------------------------------------------------------------------------------
|                                                        DescribeInstances                                                        |
+---------------------+---------------------------------+-------------------------------------------------------------------------+
|  i-0ec2f28f95c0b4396|  MadLib API Tier - AutoScaled   |  arn:aws:iam::399862743030:instance-profile/MadLib-APIrole              |
|  i-0fd0f2f4e072463b0|  MadLib Save Tier - AutoScaled  |  arn:aws:iam::399862743030:instance-profile/MadLib-Saverole             |
|  i-0ac39407f3b79e43b|  MadLib API Tier - AutoScaled   |  arn:aws:iam::399862743030:instance-profile/MadLib-APIrole              |
|  i-0eba4f6906abf1833|  MadLib Web Tier - AutoScaled   |  arn:aws:iam::399862743030:instance-profile/MadLib-AppRole              |
|  i-0b558db478ac2bdbc|  CommandHost                    |  arn:aws:iam::399862743030:instance-profile/CommandHostInstanceProfile  |
|  i-09a53d2758f4d749d|  MadLib Web Tier - AutoScaled   |  arn:aws:iam::399862743030:instance-profile/MadLib-AppRole              |
|  i-03804db70790dc0ed|  MadLib Save Tier - AutoScaled  |  arn:aws:iam::399862743030:instance-profile/MadLib-Saverole             |
+---------------------+---------------------------------+-------------------------------------------------------------------------+
[ec2-user@ip-10-96-10-231 ~]$ 

[ec2-user@ip-10-96-10-231 ~]$ aws ec2 describe-instances --filter "Name=tag:Name,Values=MadLib Save*" --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`].Value|[0], IamInstanceProfile.Arn]' --output table
------------------------------------------------------------------------------------------------------------------------
|                                                   DescribeInstances                                                  |
+---------------------+---------------------------------+--------------------------------------------------------------+
|  i-0fd0f2f4e072463b0|  MadLib Save Tier - AutoScaled  |  arn:aws:iam::399862743030:instance-profile/MadLib-Saverole  |
|  i-03804db70790dc0ed|  MadLib Save Tier - AutoScaled  |  arn:aws:iam::399862743030:instance-profile/MadLib-Saverole  |
+---------------------+---------------------------------+--------------------------------------------------------------+
[ec2-user@ip-10-96-10-231 ~]$ 
</code>


<code>
[ec2-user@ip-10-96-10-231 ~]$ aws ec2 describe-instances --filter "Name=tag:Name,Values=MadLib Web*" --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' --output text
arn:aws:iam::399862743030:instance-profile/MadLib-AppRole
[ec2-user@ip-10-96-10-231 ~]$


[ec2-user@ip-10-96-10-231 ~]$ appROLEARN=$(aws ec2 describe-instances --filter "Name=tag:Name,Values=MadLib Web*" --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' --output text)
[ec2-user@ip-10-96-10-231 ~]$ 

[ec2-user@ip-10-96-10-231 ~]$ echo ${appROLEARN}
arn:aws:iam::399862743030:instance-profile/MadLib-AppRole
[ec2-user@ip-10-96-10-231 ~]$ 

[ec2-user@ip-10-96-10-231 ~]$ aws iam list-instance-profiles --query "InstanceProfiles[?Arn=='$appROLEARN']"
[
    {
        "InstanceProfileId": "AIPAJJGZDTBTYGJDSLFVM", 
        "Roles": [
            {
                "AssumeRolePolicyDocument": {
                    "Version": "2008-10-17", 
                    "Statement": [
                        {
                            "Action": "sts:AssumeRole", 
                            "Effect": "Allow", 
                            "Principal": {
                                "Service": "ec2.amazonaws.com"
                            }
                        }
                    ]
                }, 
                "RoleId": "AROAJ7OSABF7346MV6RIY", 
                "CreateDate": "2018-10-09T08:06:50Z", 
                "RoleName": "qls-1577787-84859361afe35637-AppLayerWebSi-AppRole-5CZ1MUJYE8Y4", 
                "Path": "/", 
                "Arn": "arn:aws:iam::399862743030:role/qls-1577787-84859361afe35637-AppLayerWebSi-AppRole-5CZ1MUJYE8Y4"
            }
        ], 
        "CreateDate": "2018-10-09T08:07:07Z", 
        "InstanceProfileName": "MadLib-AppRole", 
        "Path": "/", 
        "Arn": "arn:aws:iam::399862743030:instance-profile/MadLib-AppRole"
    }
]
[ec2-user@ip-10-96-10-231 ~]$ 


[ec2-user@ip-10-96-10-231 ~]$ aws iam list-instance-profiles --query "InstanceProfiles[?Arn=='$appROLEARN'].Roles[0].RoleName"
[
    "qls-1577787-84859361afe35637-AppLayerWebSi-AppRole-5CZ1MUJYE8Y4"
]
[ec2-user@ip-10-96-10-231 ~]$


[ec2-user@ip-10-96-10-231 ~]$ appROLENAME=$(aws iam list-instance-profiles --query "InstanceProfiles[?Arn=='$appROLEARN'].Roles[0].RoleName" --output text)
[ec2-user@ip-10-96-10-231 ~]$ aws iam list-role-policies --role-name ${appROLENAME}
{
    "PolicyNames": [
        "MabLib-App-Policy"
    ]
}
[ec2-user@ip-10-96-10-231 ~]$ appPOLNAME=$(aws iam list-role-policies --role-name ${appROLENAME} --query PolicyNames[] --output text)
[ec2-user@ip-10-96-10-231 ~]$ 
[ec2-user@ip-10-96-10-231 ~]$ aws iam get-role-policy --role-name ${appROLENAME} --policy-name ${appPOLNAME}
{
    "RoleName": "qls-1577787-84859361afe35637-AppLayerWebSi-AppRole-5CZ1MUJYE8Y4", 
    "PolicyDocument": {
        "Statement": [
            {
                "Action": [
                    "s3:List*", 
                    "s3:Get*"
                ], 
                "Resource": "*", 
                "Effect": "Allow"
            }
        ]
    }, 
    "PolicyName": "MabLib-App-Policy"
}
[ec2-user@ip-10-96-10-231 ~]$ 
</code>



<code>
[ec2-user@ip-10-96-10-231 ~]$ aws deploy list-applications
{
    "applications": [
        "qls-1577787-84859361afe35637-AppLayerWebSite-1P4CE84PXN67F-MadLibsSite-1AG3943MCP2N9", 
        "qls-1577787-84859361afe35637-AppStackAPI-FO024805JDNG-MadLibsAPI-1Q30CGWVEPZDA", 
        "qls-1577787-84859361afe35637-AppStackSave-1O3LTSI3CAKLB-MadLibsSave-XM6SBRZK607M"
    ]
}
[ec2-user@ip-10-96-10-231 ~]$ aws deploy list-deployments
{
    "deployments": [
        "d-W13R99NVV", 
        "d-Y639UTFVV", 
        "d-EAQ1SUMVV"
    ]
}
[ec2-user@ip-10-96-10-231 ~]$ DEPLOYARRAY=$(aws deploy list-deployments --output text)
[ec2-user@ip-10-96-10-231 ~]$ IFS=' ' read -r -a DEPLOYID <<< $DEPLOYARRAY
[ec2-user@ip-10-96-10-231 ~]$ echo "${DEPLOYID[1]}"
d-W13R99NVV
[ec2-user@ip-10-96-10-231 ~]$ echo "${DEPLOYID[3]}"
d-Y639UTFVV
[ec2-user@ip-10-96-10-231 ~]$ echo "${DEPLOYID[5]}"
d-EAQ1SUMVV
[ec2-user@ip-10-96-10-231 ~]$ 
[ec2-user@ip-10-96-10-231 ~]$ aws deploy list-deployment-instances --deployment-id ${DEPLOYID[1]}
{
    "instancesList": [
        "i-09a53d2758f4d749d", 
        "i-0eba4f6906abf1833"
    ]
}
[ec2-user@ip-10-96-10-231 ~]$ aws ec2 describe-instances --filter "Name=tag:Name,Values=MadLib*" --query 'Reservations[].Instances[].[InstanceId, Tags[?Key==`Name`].Value | [0]]' --output table
----------------------------------------------------------
|                    DescribeInstances                   |
+----------------------+---------------------------------+
|  i-0ec2f28f95c0b4396 |  MadLib API Tier - AutoScaled   |
|  i-0fd0f2f4e072463b0 |  MadLib Save Tier - AutoScaled  |
|  i-0ac39407f3b79e43b |  MadLib API Tier - AutoScaled   |
|  i-0eba4f6906abf1833 |  MadLib Web Tier - AutoScaled   |
|  i-09a53d2758f4d749d |  MadLib Web Tier - AutoScaled   |
|  i-03804db70790dc0ed |  MadLib Save Tier - AutoScaled  |
+----------------------+---------------------------------+
[ec2-user@ip-10-96-10-231 ~]$


[ec2-user@ip-10-96-10-231 ~]$ aws deploy get-deployment --deployment-id ${DEPLOYID[1]}
{
    "deploymentInfo": {
        "applicationName": "qls-1577787-84859361afe35637-AppLayerWebSite-1P4CE84PXN67F-MadLibsSite-1AG3943MCP2N9", 
        "status": "Succeeded", 
        "deploymentOverview": {
            "Skipped": 0, 
            "Succeeded": 2, 
            "Failed": 0, 
            "Ready": 0, 
            "InProgress": 0, 
            "Pending": 0
        }, 
        "description": "[CFN-DSHWMLJA] Deploying App MadLibs-Site Version-1.0\n", 
        "deploymentConfigName": "MadLibs-Site", 
        "creator": "user", 
        "fileExistsBehavior": "DISALLOW", 
        "deploymentId": "d-W13R99NVV", 
        "deploymentStatusMessages": [], 
        "ignoreApplicationStopFailures": true, 
        "autoRollbackConfiguration": {
            "enabled": false
        }, 
        "deploymentStyle": {
            "deploymentType": "IN_PLACE", 
            "deploymentOption": "WITHOUT_TRAFFIC_CONTROL"
        }, 
        "updateOutdatedInstancesOnly": false, 
        "instanceTerminationWaitTimeStarted": false, 
        "computePlatform": "Server", 
        "deploymentGroupName": "WebAppDeplyGroup", 
        "createTime": 1539072614.847, 
        "completeTime": 1539072703.42, 
        "revision": {
            "revisionType": "S3", 
            "s3Location": {
                "bundleType": "zip", 
                "bucket": "us-east-1-tcprod", 
                "key": "courses/AWS-200-DOP/v2.1.5/lab-1-CLI/scripts/MadLibs-WebSite-Package.zip"
            }
        }
    }
}
[ec2-user@ip-10-96-10-231 ~]$ 

</code>


===== Embedding a BASH variable in JSON =====

<code bash>
STANDARD='arn:aws:securityhub:::ruleset/nist-800-53/v/5.0.0'

aws securityhub  batch-update-standards-control-associations \
    --standards-control-association-updates \
    --profile OrgDeployRole \
    '[
    {
    "SecurityControlId": "IAM.9",
    "StandardsArn": "'"${STANDARD}"'",
    "AssociationStatus": "DISABLED",
    "UpdatedReason": "Not applicable to environment"
    }
    ]'


</code>

==== Explanation of "StandardsArn": "'"${STANDARD}"'" ====

JSON keys and values need to be quoted, so that's the outside double quotes, STANDARD cannot be expnded by the aws command so it is single quoted, but lastly as BASH sees it, it is double quoted so BASH can expand the variable "${var}" to the value it represents.

I think.... Not 100% sure on the single quotes usage