Rainsbrook

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Amazon Web Services CLI ======


===== Initial install =====

AWS cli tool is written in python, and as python3 is the most recent, this is what will be installed. The awscli tool is installed through pip3. Consider installing this in a virtual environment ([[python:python#virtual_environment|Python Virtual Env]] )

<code>
# yum install python3

... edited...

Install  1 Package (+3 Dependent packages)

Total download size: 11 M
Installed size: 51 M
Is this ok [y/d/N]: y
Downloading packages:
(1/4): python3-3.7.0-0.20.rc1.amzn2.0.1.x86_64.rpm                                     |  64 kB  00:00:01     
(2/4): python3-pip-9.0.3-1.amzn2.0.1.no                                                | 1.9 MB  00:00:03     
(3/4): python3-setuptools-38.4.0-3.amzn2.0.6.noarch.rpm                                | 617 kB  00:00:01     
(4/4): python3-libs-3.7.0-0.20.rc1.amzn2.0.1.x86_64.rpm                                | 8.0 MB  00:00:14     
------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        487 kB/s |  11 MB  00:00:22    

# pip3 install awscli
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting awscli
  Downloading https://files.pythonhosted.org/packages/f8/ab/ab7b15a7a5524f47bb39279a59a7afdb1237162159ba7ff15cab28c96915/awscli-1.16.15-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 981kB/s 
...edited...    
</code>



AWS linux 2 does have a awscli tool in the linux repo, based on python2, but it is not as recent as the pip installed one, this is the python3 based version, check if you need py3 or py2 before installing:-

<code>
# aws --version
aws-cli/1.14.8 Python/2.7.14 Linux/4.14.47-64.38.amzn2.x86_64 botocore/1.8.12
[root@amazonlinux02 ~]#
</code>


Compare to the pip3 installed version:-
<code>
# /usr/local/bin/aws --version
aws-cli/1.16.15 Python/3.7.0rc1 Linux/4.14.47-64.38.amzn2.x86_64 botocore/1.12.5
[root@amazonlinux02 ~]#
</code>



===== Setting up profiles =====

The ''aws configure'' command will set up a default profile, if you need to use different accounts or keys, these can be set in a named profile, the two files, ''config'' and ''credentials'' in ''~/.aws/'' control this (needless to say, these are made up keys and secrets!):-

config
<code>
[default]
output = text


[profile admin1]
role_arn = arn:aws:iam::0123456781234:role/role_admin
source_profile = default
region = eu-west-1


[profile profile2]
region = eu-west-2
source_profile = default
output = text
</code>

credentials
<code>
[default]
aws_access_key_id = QWERTYUIOPASDFGHKEYQ
aws_secret_access_key = HaMPb65IFf0bVoEiLSKEJtuCUo3490nWlrJBES9n


[profile2]
aws_access_key_id = QWERTYUIOPASDFGHKEYA
aws_secret_access_key = wifisUMegS9pY_tpOnQpSY0YJYSiqgeKneMWqqIa
</code>


FIXME

==== aws configure sso ====

Single Sign On configuration.


==== aws configure list ====

<code>
$ aws configure list 
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************3AGN   container-role    
secret_key     ****************k3sa   container-role    
    region                eu-west-9              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

</code>


==== aws iam get-user ====


==== get-caller-identity ====

Returns details about the IAM user or role whose credentials are used to call the operation. No permissions are required to perform this operation.

This can be used to test AWS CLI setup and network connectivity.

<code>
$ aws sts get-caller-identity
{
    "UserId": "AROAQ8AB123CD4321ABCD:me1234",
    "Account": "123443210986",
    "Arn": "arn:aws:sts::123443210986:assumed-role/AWSReservedSSO_Access_9d3a256a5a2001321/me1234"
}
$
</code>


===== Setting up roles =====

Roles allow proviledge escalation for a user to perform specific tasks. For this to be used in the cli, an extra section is added to the ''.aws/config'' file:-

<code>
[default]
output = json
region = eu-west-1


[profile sandbox]
role_arn = arn:aws:iam::12345678:role/roles-sandboxadmin
source_profile = default
region = eu-west-1
</code>

When a cli command is run, the ''--profile sandbox'' is added to awitch to that role:-

<code>
server:~/.aws$ aws ec2 describe-instances --filters "Name=instance-type,Values=t2.micro" --query Reservations[].Instances[].InstanceId --profile sandbox
[
    "i-0b99cb98dfabcdefg",
    "i-04888fe41agfedcba"
]
server:~/.aws$
</code>


==== Errors ====

<code>
$ aws ec2 describe-instances --profile nonprod_admin

An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.
$
</code>

This was solved by updating the ''aws_access_key_id'' and ''aws_secret_access_key'' in ~/.aws/credentials.
As it said "The security token included in the request is invalid." Obvious in hindsight.
===== Using roles and profiles with Boto3 =====

''boto3'' is Amazon's python library to interface with the aws cli commands.

A client needs to be set up, and for local cli usage, this needs to be linked with a profile as set above.

<code python>
#!/usr/bin/env python

import boto3

profile = 'nonprod_admin'

# Create ec2 client
session = boto3.session.Session(profile_name=profile)
ec2 = session.client('ec2')

# Create SQS client
session = boto3.session.Session(profile_name=profile)
sqs = session.client('sqs')

</code>

This client (ec2, sqs etc) can be used to set or retreive information as the user in the profile:-

<code python>
AMIResponse = ec2.describe_images(Filters=[{'Name': 'name', 'Values': [Regex]}, ], Owners=['self'])
</code>

===== AWS CodeCommit =====

It's worth pointing out that Code Commit repos are tied to a particular AWS account, so if you operate in a multiple account environment, you will need to ensure your code is stored in the right repo, eg if you have a prod and staging/dev environment, maybe codecommit may not be the easiest repo choice.

See:- [[https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html]].

AWS CodeCommit is a git compatible repository. It uses the git command locally, but if you are using roles, there is a restriction on using only https, not ssh to communticate to the remote repo. Also, there is a tie in with the aws command line which is why CodeCommit is here and not under [[rb:git-cheatsheet|Git Cheatsheet]].

AWS's IAM requires ''HTTPS Git credentials for AWS CodeCommit'' to be created from the ''Security Credentials'' tab under ''Users''. Click Generate and make a note of the values. These will become your git credentials to be used on the cli.
Also, you will be **required** to configure a Credential Helper, the name of this sounds like it is optional, but it isn't. 

IAM periodically resets the password used with the git credentials (above) and the Credential Helper is used to call out to IAM to get the updates password which is then used in the git command.

<code>
git config --global credential.helper '!aws codecommit credential-helper $@'
git config --global credential.UseHttpPath true
</code>

Profiles can be defined per repository by using ''--local'' instead of ''--global''. 


==== Roles with CodeCommit ====

In the .gitconfig file, the commands above add the helper line, but to use it with roles, it needs the ''--profile'' adding:-

<code>
$ more  /home/user/.gitconfig 
[credential]
	helper = !aws --profile sandbox codecommit credential-helper $@
	UseHttpPath = true
$
</code>

My understanding is that ''git'' feeds a string of arguments to the credential-helper ($@) and consumes the string returned to forward on to CodeCommit as the user password.  As it is an ''aws'' command it can take the ''--profile'' option. Without that, the helper will try to return the IAM users credentials not the role's credentials, git will present these and it will pushes and pulls will fail with ''fatal: unable to access <xx-repo> : The requested URL returned error: 403''


If you get an error similar to ''The requested URL returned error: 403'', you may need to export your AWS credentials:-

  $ export AWS_PROFILE=shared-services

 

==== Creating a new repo ====

This is shown with the role option ''--profile sandbox''


<code>
$ aws codecommit create-repository --repository-name CIS-Hardening --repository-description "Repo for ansible code to harden aws Linux2 image." --profile sandbox
{
    "repositoryMetadata": {
        "repositoryDescription": "Repo for ansible code to harden aws Linux2 image.",
        "cloneUrlSsh": "ssh://git-codecommit.eu-west-2.amazonaws.com/v1/repos/CIS-AWS_Linux2",
        "repositoryId": "91fb1234-833e-4705-8e1c-xxxxxxxx",
        "lastModifiedDate": 1537199788.236,
        "accountId": "1234567890",
        "repositoryName": "CIS-Hardening",
        "Arn": "arn:aws:codecommit:eu-west-2:987654321:CIS-Hardening",
        "cloneUrlHttp": "https://git-codecommit.eu-west-2.amazonaws.com/v1/repos/CIS-AWS_Linux2",
        "creationDate": 1537199788.236
    }
}
$
$ aws codecommit list-repositories --profile sandbox
{
    "repositories": [
        {
            "repositoryName": "AMI_PackerDefinitions-AWS_Linux2",
            "repositoryId": "12345678-a3af-4d07-8319-20f112345678"
        },
        {
            "repositoryName": "CIS-Hardening",
            "repositoryId": "87654321-14cd-495a-8b4f-121987654321"
        }
    ]
}

</code>


See the [[rb:git-cheatsheet|Git cheatsheet]] for adding files etc to the repo.


==== 403 error with git push ====

This is an error encountered on a Mac:-

<code>
me (ajs/cpe-1806) $ git push --set-upstream origin ajs/cpe-1806
fatal: unable to access 'https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myrepo/': The requested URL returned error: 403
me (ajs/cpe-1806) $ 
</code>

It's caused (probably) by the "Helper" having rotated the access keys at the AWS end and the Keychain Access app caching the original credentials which are no longer valid. 

Solution is to search for and delete CodeCommit credentials in Keychain Access. 

<code>
me (ajs/cpe-1806) $ git push --set-upstream origin ajs/cpe-1806
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 4 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 988 bytes | 988.00 KiB/s, done.
Total 6 (delta 4), reused 0 (delta 0)
To https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myrepo
 * [new branch]      ajs/cpe-1806 -> ajs/cpe-1806
Branch 'ajs/cpe-1806' set up to track remote branch 'ajs/cpe-1806' from 'origin'.
me (ajs/cpe-1806) $
</code>


==== Get repos ====

Used ''jq'' to process output, this requires json, so the export forces json format. ''-r'' gives a raw output, ie, no quotes round variables. 

get_repos.sh:- 
<code bash>
#!/bin/bash
# List repos from AWS codecommit

export AWS_DEFAULT_OUTPUT="json"

aws codecommit list-repositories | jq -r '.repositories | .[].repositoryName'
</code>

<code>
credential-age-check
pyside6-stuff
</code>




===== Embedding a BASH variable in JSON =====

<code bash>
STANDARD='arn:aws:securityhub:::ruleset/nist-800-53/v/5.0.0'

aws securityhub  batch-update-standards-control-associations \
    --standards-control-association-updates \
    --profile OrgDeployRole \
    '[
    {
    "SecurityControlId": "IAM.9",
    "StandardsArn": "'"${STANDARD}"'",
    "AssociationStatus": "DISABLED",
    "UpdatedReason": "Not applicable to environment"
    }
    ]'


</code>

==== Explanation of variable substitution====

<code>
"StandardsArn": "'"${STANDARD}"'" 
</code>

JSON keys and values need to be quoted, so that's the outside double quotes, STANDARD cannot be expanded by the aws command so it is single quoted, but lastly as BASH sees it, it is double quoted so BASH can expand the variable '' "${STANDARD}" '' to the value it represents.

I think.... Not 100% sure on the single quotes usage