AWS Simple Systems Manager (SSM or Systems Manager) is an AWS service for bulk management of EC2 instances, and on-premises servers too. Like many AWS services though, it is accessed via the internet. This means that:

a) You need some way to access the service
b) The control data is going out over the internet

Getting around statement a. is pretty easy, either have a direct internet connection or you can configure the SSM agent to use a proxy.  For statement b. though, you can use an interface endpoint to have a PrivateLink connection from your VPC to AWS. This blog’s focus is on configuring that PrivateLink connection. The process is actually very easy, but there are a few items that can trip you up if you aren’t prepared.

Prerequisites

There is really only one prerequisite when configuring the SSM endpoints, they need to accept HTTPS/TCP 443 from your address range. Generally, this is the CIDR range for your VPC. If you don’t already have a Security Group configured for this, create a new one.

A CloudFormation script can be used to configure the SSM endpoints and a dedicated Security Group. Having a separate Security Group just makes things tidier if a clean-up/deletion is required. The CloudFormation allows for consistent reuse and easy upgrade.

Required Endpoints

As stated in the AWS documentation, the following endpoints are needed, depending on your requirements.

  • amazonaws..ssm: The endpoint for the Systems Manager service.
  • amazonaws..ec2messages: Systems Manager uses this endpoint to make calls from SSM Agent to the Systems Manager service.
  • amazonaws..ssmmessages: This endpoint is required only if you are connecting to your instances through a secure data channel using Session Manager.
  • amazonaws..ec2: If you’re using Systems Manager to create VSS-enabled snapshots, you need to ensure that you have an endpoint to the EC2 service. Without the EC2 endpoint defined, a call to enumerate attached EBS volumes fails, which causes the Systems Manager command to fail.

The first two are needed, and ssm messages is also recommended. Session Manager is a great tool for accessing your environments from the AWS Console. The final endpoint depends on what you use SSM for.

Creating the Endpoints

Now that we have our Security Group and we know what endpoints we need, there are two other questions we need to have answered … what subnets do we want to connect via, and do we need private DNS? Creating an interface endpoint is almost like a pseudo device with one or more Elastic Network Interfaces configured. Those ENIs need to be associated with one or more subnets. Each subnet needs to be in a different AZ. Having multiple subnets allows for redundancy. Only one endpoint is needed per VPC, so long as the routing tables allow the subnets to talk to those associated with the endpoint.

From the console, this is really easy and set in one page. I’ll break it down here.

First, select Create Endpoint button to take you to the configure page. Then select the “AWS services” radio button.


Next, select the service. You can only select one service at a time, so you’ll need to redo this process for each endpoint. SSM is needed, along with EC2Messages & SSMMessages.


After selecting the service, select the VPC. It will make some suggestions, but you can alter that from the dropdown.


The next section after choosing the subnets is to enable Private DNS. The default has this enabled. I’ll discuss this shortly.


Finally, select the security group. It will default to “default”, so remember to uncheck that and then check the security group you want. You also have the option to create a new SG.


Following is the AWS documentation on how to create an interface endpoint: https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint

Creating with Cloudformation

Since creating SSM endpoints is fairly common and actually comprises three endpoint services, I have a Cloudformation script to do this. The script takes VPC ID, VPC CIDR and three subnet IDs as inputs. It then creates a common security group with port 443 open to the CIDR range and the three endpoints. In YAML format, the syntax for creating a VPC endpoint is:

Type: AWS::EC2::VPCEndpoint

 Properties:

   PolicyDocument: Json #Note: This is only for gateway endpoints, e.g. S3.

   PrivateDnsEnabled: Boolean #Remember this for Private DNS

   RouteTableIds:

     - String

   SecurityGroupIds:

     - String

   ServiceName: String

   SubnetIds:

     - String

   VpcEndpointType: String

   VpcId: String 

The AWS documentation is: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpcendpoint.html

Private DNS

When creating a VPC interface endpoint, several new DNS entries are created. There is a general VPC Endpoint (-.ssm.ap-souteast-2.vpce.amazonaws.com) plus entries for each subnet (--ap-southeast-2X.ssm.ap…).


Along with that, if private DNS is enabled, the IP for the default service, e.g. ssm.ap-southeast-2.amazonaws.com, is set to the same IPs as the vpce-XXX-YY.ssm… entry.

If Private DNS is not enabled, you will need to configure the SSM Agent to use the vpce endpoint created, and not the default. To do this, follow these steps:

  • Make a copy of /etc/amazon/ssm/amazon-ssm-agent.json.template to /etc/amazon/ssm/amazon-ssm-agent.json
  • Edit amazon-ssm-agent.json:
    • Go to “Ssm”: { section
    • Edit “Endpoint”: “”, to reference the vpce-xxx-yyy.ssm address
  • Restart SSM Agent

The AWS documentation describes Private DNS here: https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#vpce-private-dns

SSM Agent

If an instance is launched prior to SSM endpoints being created, the agent will have the default internet accessible SSM address. If the agent is not configured to use a proxy, it should be a simple task of restarting the SSM agent for it to recognise the endpoint.

In the case where SSM agent has been configured to use a proxy, this needs to be disabled first. After that, restart the SSM agent and it should be all good.

Pricing

VPC Endpoints are not free, but they are fairly cheap. It costs roughly US$8.75/mth in Sydney region for each endpoint, so around US$26/mth if all three endpoints are used. There is also a very small charge for data, but SSM doesn’t use a lot. The following link contains the current pricing for PrivateLink connections: https://aws.amazon.com/privatelink/pricing/

Wrap Up

SSM is a great way to manage your environment in a bulk way. To add an extra layer of security or just allow access to non-internet facing environments, SSM Endpoints are the way to go. They are simple to set up and fairly cheap. Just remember to configure Private DNS and restart the agent.