[[aws:aws-cli]]
 
Trace: • aws-cli

Table of Contents

Amazon Web Services CLI

Initial install

AWS cli tool is written in python, and as python3 is the most recent, this is what will be installed. The awscli tool is installed through pip3. Consider installing this in a virtual environment (Python Virtual Env ) 

# yum install python3

... edited...

Install  1 Package (+3 Dependent packages)

Total download size: 11 M
Installed size: 51 M
Is this ok [y/d/N]: y
Downloading packages:
(1/4): python3-3.7.0-0.20.rc1.amzn2.0.1.x86_64.rpm                                     |  64 kB  00:00:01     
(2/4): python3-pip-9.0.3-1.amzn2.0.1.no                                                | 1.9 MB  00:00:03     
(3/4): python3-setuptools-38.4.0-3.amzn2.0.6.noarch.rpm                                | 617 kB  00:00:01     
(4/4): python3-libs-3.7.0-0.20.rc1.amzn2.0.1.x86_64.rpm                                | 8.0 MB  00:00:14     
------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        487 kB/s |  11 MB  00:00:22    

# pip3 install awscli
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting awscli
  Downloading https://files.pythonhosted.org/packages/f8/ab/ab7b15a7a5524f47bb39279a59a7afdb1237162159ba7ff15cab28c96915/awscli-1.16.15-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 981kB/s 
...edited...

AWS linux 2 does have a awscli tool in the linux repo, based on python2, but it is not as recent as the pip installed one, this is the python3 based version, check if you need py3 or py2 before installing:- 

# aws --version
aws-cli/1.14.8 Python/2.7.14 Linux/4.14.47-64.38.amzn2.x86_64 botocore/1.8.12
[root@amazonlinux02 ~]#

Compare to the pip3 installed version:- 

# /usr/local/bin/aws --version
aws-cli/1.16.15 Python/3.7.0rc1 Linux/4.14.47-64.38.amzn2.x86_64 botocore/1.12.5
[root@amazonlinux02 ~]#

Setting up profiles

The aws configure command will set up a default profile, if you need to use different accounts or keys, these can be set in a named profile, the two files, config and credentials in ~/.aws/ control this (needless to say, these are made up keys and secrets!):-

config 

[default]
output = text


[profile admin1]
role_arn = arn:aws:iam::0123456781234:role/role_admin
source_profile = default
region = eu-west-1


[profile profile2]
region = eu-west-2
source_profile = default
output = text

credentials 

[default]
aws_access_key_id = QWERTYUIOPASDFGHKEYQ
aws_secret_access_key = HaMPb65IFf0bVoEiLSKEJtuCUo3490nWlrJBES9n


[profile2]
aws_access_key_id = QWERTYUIOPASDFGHKEYA
aws_secret_access_key = wifisUMegS9pY_tpOnQpSY0YJYSiqgeKneMWqqIa

FIXME

aws configure list

aws iam get-user

iam sts get-caller-identity

Setting up roles

Roles allow proviledge escalation for a user to perform specific tasks. For this to be used in the cli, an extra section is added to the .aws/config file:- 

[default]
output = json
region = eu-west-1


[profile sandbox]
role_arn = arn:aws:iam::12345678:role/roles-sandboxadmin
source_profile = default
region = eu-west-1

When a cli command is run, the –profile sandbox is added to awitch to that role:- 

server:~/.aws$ aws ec2 describe-instances --filters "Name=instance-type,Values=t2.micro" --query Reservations[].Instances[].InstanceId --profile sandbox
[
    "i-0b99cb98dfabcdefg",
    "i-04888fe41agfedcba"
]
server:~/.aws$

Errors

$ aws ec2 describe-instances --profile nonprod_admin

An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.
$

This was solved by updating the aws_access_key_id and aws_secret_access_key in ~/.aws/credentials. As it said “The security token included in the request is invalid.” Obvious in hindsight.

Using roles and profiles with Boto3

boto3 is Amazon's python library to interface with the aws cli commands.

A client needs to be set up, and for local cli usage, this needs to be linked with a profile as set above. 

#!/usr/bin/env python
 
import boto3
 
profile = 'nonprod_admin'
 
# Create ec2 client
session = boto3.session.Session(profile_name=profile)
ec2 = session.client('ec2')
 
# Create SQS client
session = boto3.session.Session(profile_name=profile)
sqs = session.client('sqs')

This client (ec2, sqs etc) can be used to set or retreive information as the user in the profile:- 

AMIResponse = ec2.describe_images(Filters=[{'Name': 'name', 'Values': [Regex]}, ], Owners=['self'])

AWS CodeCommit

It's worth pointing out that Code Commit repos are tied to a particular AWS account, so if you operate in a multiple account environment, you will need to ensure your code is stored in the right repo, eg if you have a prod and staging/dev environment, maybe codecommit may not be the easiest repo choice.

See:- https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html.

AWS CodeCommit is a git compatible repository. It uses the git command locally, but if you are using roles, there is a restriction on using only https, not ssh to communticate to the remote repo. Also, there is a tie in with the aws command line which is why CodeCommit is here and not under Git Cheatsheet.

AWS's IAM requires HTTPS Git credentials for AWS CodeCommit to be created from the Security Credentials tab under Users. Click Generate and make a note of the values. These will become your git credentials to be used on the cli. Also, you will be required to configure a Credential Helper, the name of this sounds like it is optional, but it isn't.

IAM periodically resets the password used with the git credentials (above) and the Credential Helper is used to call out to IAM to get the updates password which is then used in the git command. 

git config --global credential.helper '!aws codecommit credential-helper $@'
git config --global credential.UseHttpPath true

Profiles can be defined per repository by using –local instead of –global.

Roles with CodeCommit

In the .gitconfig file, the commands above add the helper line, but to use it with roles, it needs the –profile adding:- 

$ more  /home/user/.gitconfig 
[credential]
	helper = !aws --profile sandbox codecommit credential-helper $@
	UseHttpPath = true
$

My understanding is that git feeds a string of arguments to the credential-helper ($@) and consumes the string returned to forward on to CodeCommit as the user password. As it is an aws command it can take the –profile option. Without that, the helper will try to return the IAM users credentials not the role's credentials, git will present these and it will pushes and pulls will fail with fatal: unable to access <xx-repo> : The requested URL returned error: 403

If you get an error similar to The requested URL returned error: 403, you may need to export your AWS credentials:- 

$ export AWS_PROFILE=shared-services

Creating a new repo

This is shown with the role option –profile sandbox 

$ aws codecommit create-repository --repository-name CIS-Hardening --repository-description "Repo for ansible code to harden aws Linux2 image." --profile sandbox
{
    "repositoryMetadata": {
        "repositoryDescription": "Repo for ansible code to harden aws Linux2 image.",
        "cloneUrlSsh": "ssh://git-codecommit.eu-west-2.amazonaws.com/v1/repos/CIS-AWS_Linux2",
        "repositoryId": "91fb1234-833e-4705-8e1c-xxxxxxxx",
        "lastModifiedDate": 1537199788.236,
        "accountId": "1234567890",
        "repositoryName": "CIS-Hardening",
        "Arn": "arn:aws:codecommit:eu-west-2:987654321:CIS-Hardening",
        "cloneUrlHttp": "https://git-codecommit.eu-west-2.amazonaws.com/v1/repos/CIS-AWS_Linux2",
        "creationDate": 1537199788.236
    }
}
$
$ aws codecommit list-repositories --profile sandbox
{
    "repositories": [
        {
            "repositoryName": "AMI_PackerDefinitions-AWS_Linux2",
            "repositoryId": "12345678-a3af-4d07-8319-20f112345678"
        },
        {
            "repositoryName": "CIS-Hardening",
            "repositoryId": "87654321-14cd-495a-8b4f-121987654321"
        }
    ]
}

See the Git cheatsheet for adding files etc to the repo.

403 error with git push

This is an error encountered on a Mac:- 

me (ajs/cpe-1806) $ git push --set-upstream origin ajs/cpe-1806
fatal: unable to access 'https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myrepo/': The requested URL returned error: 403
me (ajs/cpe-1806) $

It's caused (probably) by the “Helper” having rotated the access keys at the AWS end and the Keychain Access app caching the original credentials which are no longer valid.

Solution is to search for and delete CodeCommit credentials in Keychain Access. 

me (ajs/cpe-1806) $ git push --set-upstream origin ajs/cpe-1806
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 4 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 988 bytes | 988.00 KiB/s, done.
Total 6 (delta 4), reused 0 (delta 0)
To https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myrepo
 * [new branch]      ajs/cpe-1806 -> ajs/cpe-1806
Branch 'ajs/cpe-1806' set up to track remote branch 'ajs/cpe-1806' from 'origin'.
me (ajs/cpe-1806) $

AWS NTP Time

Use 169.254.169.123 with ntp or chrony. 

server 169.254.169.123 prefer iburst

Force timesync with chrony

If you get this message:- 

$ ./get_repos.sh

An error occurred (InvalidSignatureException) when calling the ListRepositories operation: Signature expired: 20231207T141711Z is now earlier than 20231207T152038Z (20231207T153538Z - 15 min.)

your time may be out by more than AWS allows, use this with chrony to force a reset. ntpdate will force a resync with the ntpd package. You will need to stop the ntpd daemon first and then restart it after. 

root@ubuntu20:~# chronyd -q 'server time.domain.com  iburst'
2023-10-26T15:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 -DEBUG)
2023-10-26T15:32:22Z Initial frequency 5.379 ppm
2023-10-26T15:32:26Z System clock wrong by 1931.782000 seconds (step)
2023-10-26T16:04:38Z chronyd exiting
root@ubuntu20:~#

Getting info from within a running instance

The 169.254.169.254 address allows access to metadata about an instance from within THAT instance, eg:- 

[root@ip-172-31-21-109 ~]# curl http://169.254.169.254/latest/meta-data/ami-id
ami-0f1229ec7823be3db
[root@ip-172-31-21-109 ~]# 

[root@ip-172-31-21-109 ~]# curl http://169.254.169.254/latest/meta-data/public-keys/
0=AndrewAWS
[root@ip-172-31-21-109 ~]#

[root@ip-172-31-21-109 ~]# curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/06:b7:e8:98:98:0a/public-hostname/
ec2-34-244-253-26.eu-west-1.compute.amazonaws.com
[root@ip-172-31-21-109 ~]#

AWS cli

[ec2-user@ip-10-96-10-231 ~]$ aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`].Value|[0], IamInstanceProfile.Arn]' --output table
-----------------------------------------------------------------------------------------------------------------------------------
|                                                        DescribeInstances                                                        |
+---------------------+---------------------------------+-------------------------------------------------------------------------+
|  i-0ec2f28f95c0b4396|  MadLib API Tier - AutoScaled   |  arn:aws:iam::399862743030:instance-profile/MadLib-APIrole              |
|  i-0fd0f2f4e072463b0|  MadLib Save Tier - AutoScaled  |  arn:aws:iam::399862743030:instance-profile/MadLib-Saverole             |
|  i-0ac39407f3b79e43b|  MadLib API Tier - AutoScaled   |  arn:aws:iam::399862743030:instance-profile/MadLib-APIrole              |
|  i-0eba4f6906abf1833|  MadLib Web Tier - AutoScaled   |  arn:aws:iam::399862743030:instance-profile/MadLib-AppRole              |
|  i-0b558db478ac2bdbc|  CommandHost                    |  arn:aws:iam::399862743030:instance-profile/CommandHostInstanceProfile  |
|  i-09a53d2758f4d749d|  MadLib Web Tier - AutoScaled   |  arn:aws:iam::399862743030:instance-profile/MadLib-AppRole              |
|  i-03804db70790dc0ed|  MadLib Save Tier - AutoScaled  |  arn:aws:iam::399862743030:instance-profile/MadLib-Saverole             |
+---------------------+---------------------------------+-------------------------------------------------------------------------+
[ec2-user@ip-10-96-10-231 ~]$ 

[ec2-user@ip-10-96-10-231 ~]$ aws ec2 describe-instances --filter "Name=tag:Name,Values=MadLib Save*" --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`].Value|[0], IamInstanceProfile.Arn]' --output table
------------------------------------------------------------------------------------------------------------------------
|                                                   DescribeInstances                                                  |
+---------------------+---------------------------------+--------------------------------------------------------------+
|  i-0fd0f2f4e072463b0|  MadLib Save Tier - AutoScaled  |  arn:aws:iam::399862743030:instance-profile/MadLib-Saverole  |
|  i-03804db70790dc0ed|  MadLib Save Tier - AutoScaled  |  arn:aws:iam::399862743030:instance-profile/MadLib-Saverole  |
+---------------------+---------------------------------+--------------------------------------------------------------+
[ec2-user@ip-10-96-10-231 ~]$ 
[ec2-user@ip-10-96-10-231 ~]$ aws ec2 describe-instances --filter "Name=tag:Name,Values=MadLib Web*" --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' --output text
arn:aws:iam::399862743030:instance-profile/MadLib-AppRole
[ec2-user@ip-10-96-10-231 ~]$


[ec2-user@ip-10-96-10-231 ~]$ appROLEARN=$(aws ec2 describe-instances --filter "Name=tag:Name,Values=MadLib Web*" --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' --output text)
[ec2-user@ip-10-96-10-231 ~]$ 

[ec2-user@ip-10-96-10-231 ~]$ echo ${appROLEARN}
arn:aws:iam::399862743030:instance-profile/MadLib-AppRole
[ec2-user@ip-10-96-10-231 ~]$ 

[ec2-user@ip-10-96-10-231 ~]$ aws iam list-instance-profiles --query "InstanceProfiles[?Arn=='$appROLEARN']"
[
    {
        "InstanceProfileId": "AIPAJJGZDTBTYGJDSLFVM", 
        "Roles": [
            {
                "AssumeRolePolicyDocument": {
                    "Version": "2008-10-17", 
                    "Statement": [
                        {
                            "Action": "sts:AssumeRole", 
                            "Effect": "Allow", 
                            "Principal": {
                                "Service": "ec2.amazonaws.com"
                            }
                        }
                    ]
                }, 
                "RoleId": "AROAJ7OSABF7346MV6RIY", 
                "CreateDate": "2018-10-09T08:06:50Z", 
                "RoleName": "qls-1577787-84859361afe35637-AppLayerWebSi-AppRole-5CZ1MUJYE8Y4", 
                "Path": "/", 
                "Arn": "arn:aws:iam::399862743030:role/qls-1577787-84859361afe35637-AppLayerWebSi-AppRole-5CZ1MUJYE8Y4"
            }
        ], 
        "CreateDate": "2018-10-09T08:07:07Z", 
        "InstanceProfileName": "MadLib-AppRole", 
        "Path": "/", 
        "Arn": "arn:aws:iam::399862743030:instance-profile/MadLib-AppRole"
    }
]
[ec2-user@ip-10-96-10-231 ~]$ 


[ec2-user@ip-10-96-10-231 ~]$ aws iam list-instance-profiles --query "InstanceProfiles[?Arn=='$appROLEARN'].Roles[0].RoleName"
[
    "qls-1577787-84859361afe35637-AppLayerWebSi-AppRole-5CZ1MUJYE8Y4"
]
[ec2-user@ip-10-96-10-231 ~]$


[ec2-user@ip-10-96-10-231 ~]$ appROLENAME=$(aws iam list-instance-profiles --query "InstanceProfiles[?Arn=='$appROLEARN'].Roles[0].RoleName" --output text)
[ec2-user@ip-10-96-10-231 ~]$ aws iam list-role-policies --role-name ${appROLENAME}
{
    "PolicyNames": [
        "MabLib-App-Policy"
    ]
}
[ec2-user@ip-10-96-10-231 ~]$ appPOLNAME=$(aws iam list-role-policies --role-name ${appROLENAME} --query PolicyNames[] --output text)
[ec2-user@ip-10-96-10-231 ~]$ 
[ec2-user@ip-10-96-10-231 ~]$ aws iam get-role-policy --role-name ${appROLENAME} --policy-name ${appPOLNAME}
{
    "RoleName": "qls-1577787-84859361afe35637-AppLayerWebSi-AppRole-5CZ1MUJYE8Y4", 
    "PolicyDocument": {
        "Statement": [
            {
                "Action": [
                    "s3:List*", 
                    "s3:Get*"
                ], 
                "Resource": "*", 
                "Effect": "Allow"
            }
        ]
    }, 
    "PolicyName": "MabLib-App-Policy"
}
[ec2-user@ip-10-96-10-231 ~]$ 
[ec2-user@ip-10-96-10-231 ~]$ aws deploy list-applications
{
    "applications": [
        "qls-1577787-84859361afe35637-AppLayerWebSite-1P4CE84PXN67F-MadLibsSite-1AG3943MCP2N9", 
        "qls-1577787-84859361afe35637-AppStackAPI-FO024805JDNG-MadLibsAPI-1Q30CGWVEPZDA", 
        "qls-1577787-84859361afe35637-AppStackSave-1O3LTSI3CAKLB-MadLibsSave-XM6SBRZK607M"
    ]
}
[ec2-user@ip-10-96-10-231 ~]$ aws deploy list-deployments
{
    "deployments": [
        "d-W13R99NVV", 
        "d-Y639UTFVV", 
        "d-EAQ1SUMVV"
    ]
}
[ec2-user@ip-10-96-10-231 ~]$ DEPLOYARRAY=$(aws deploy list-deployments --output text)
[ec2-user@ip-10-96-10-231 ~]$ IFS=' ' read -r -a DEPLOYID <<< $DEPLOYARRAY
[ec2-user@ip-10-96-10-231 ~]$ echo "${DEPLOYID[1]}"
d-W13R99NVV
[ec2-user@ip-10-96-10-231 ~]$ echo "${DEPLOYID[3]}"
d-Y639UTFVV
[ec2-user@ip-10-96-10-231 ~]$ echo "${DEPLOYID[5]}"
d-EAQ1SUMVV
[ec2-user@ip-10-96-10-231 ~]$ 
[ec2-user@ip-10-96-10-231 ~]$ aws deploy list-deployment-instances --deployment-id ${DEPLOYID[1]}
{
    "instancesList": [
        "i-09a53d2758f4d749d", 
        "i-0eba4f6906abf1833"
    ]
}
[ec2-user@ip-10-96-10-231 ~]$ aws ec2 describe-instances --filter "Name=tag:Name,Values=MadLib*" --query 'Reservations[].Instances[].[InstanceId, Tags[?Key==`Name`].Value | [0]]' --output table
----------------------------------------------------------
|                    DescribeInstances                   |
+----------------------+---------------------------------+
|  i-0ec2f28f95c0b4396 |  MadLib API Tier - AutoScaled   |
|  i-0fd0f2f4e072463b0 |  MadLib Save Tier - AutoScaled  |
|  i-0ac39407f3b79e43b |  MadLib API Tier - AutoScaled   |
|  i-0eba4f6906abf1833 |  MadLib Web Tier - AutoScaled   |
|  i-09a53d2758f4d749d |  MadLib Web Tier - AutoScaled   |
|  i-03804db70790dc0ed |  MadLib Save Tier - AutoScaled  |
+----------------------+---------------------------------+
[ec2-user@ip-10-96-10-231 ~]$


[ec2-user@ip-10-96-10-231 ~]$ aws deploy get-deployment --deployment-id ${DEPLOYID[1]}
{
    "deploymentInfo": {
        "applicationName": "qls-1577787-84859361afe35637-AppLayerWebSite-1P4CE84PXN67F-MadLibsSite-1AG3943MCP2N9", 
        "status": "Succeeded", 
        "deploymentOverview": {
            "Skipped": 0, 
            "Succeeded": 2, 
            "Failed": 0, 
            "Ready": 0, 
            "InProgress": 0, 
            "Pending": 0
        }, 
        "description": "[CFN-DSHWMLJA] Deploying App MadLibs-Site Version-1.0\n", 
        "deploymentConfigName": "MadLibs-Site", 
        "creator": "user", 
        "fileExistsBehavior": "DISALLOW", 
        "deploymentId": "d-W13R99NVV", 
        "deploymentStatusMessages": [], 
        "ignoreApplicationStopFailures": true, 
        "autoRollbackConfiguration": {
            "enabled": false
        }, 
        "deploymentStyle": {
            "deploymentType": "IN_PLACE", 
            "deploymentOption": "WITHOUT_TRAFFIC_CONTROL"
        }, 
        "updateOutdatedInstancesOnly": false, 
        "instanceTerminationWaitTimeStarted": false, 
        "computePlatform": "Server", 
        "deploymentGroupName": "WebAppDeplyGroup", 
        "createTime": 1539072614.847, 
        "completeTime": 1539072703.42, 
        "revision": {
            "revisionType": "S3", 
            "s3Location": {
                "bundleType": "zip", 
                "bucket": "us-east-1-tcprod", 
                "key": "courses/AWS-200-DOP/v2.1.5/lab-1-CLI/scripts/MadLibs-WebSite-Package.zip"
            }
        }
    }
}
[ec2-user@ip-10-96-10-231 ~]$
 
aws/aws-cli.txt · Last modified: 07/12/2023 15:39 by andrew