Merge branch 'debug' into 'master'

Mount EFS volume in the same AZ as the ECS instance

See merge request interop/iics_secure_agent!10

README.md

 # IICS Secure Agent Docker Image
+This document covers following topics:
+1. How ro run IICS Secure Agent using Docker image.
+2. IICS Secure Agent deployed(including Terraform configuration) in AWS(interop). 
 
 ## Description
-This project was forked from [jbrazda/ic-sagent-docker|https://github.com/jbrazda/ic-sagent-docker] with the intention of containerizing the IICS Secure Agent to run in Amazon Web Services.  As of early January 2019, there is no official Docker image for IICS Secure Agent.
+This project was forked from [jbrazda/ic-sagent-docker](https://github.com/jbrazda/ic-sagent-docker) with the intention of containerizing the IICS Secure Agent to run in Amazon Web Services.  As of early January 2019, there is no official Docker image for IICS Secure Agent.
 
 ## Requirements
 * Docker.
@@ -106,20 +109,27 @@ Minimum Hardware Specs ([reference](https://network.informatica.com/servlet/Jive
 ## Enterprise Integration Secure Agent
 Secure Agent containers are deployed in following tiers in interop/AWS account. This deployment is automated through Terraform (see below section on Terraform).
 
-| environment| tier        |
-| -----------|-------------|
-| test      | us-east-1(N. Virginia)|
-| prod      | us-east-2(Ohio)|
+| environment| tier        |runtime name(default) in IICS |
+| -----------|-------------|-------------------------|
+| test      | us-east-1(N. Virginia)|test.ei.secureagent.doit.wisc.edu |
+| prod      | us-east-2(Ohio)|prod.ei.secureagent.doit.wisc.edu|
 
 
-## CI/CD and Terraform
-* Infrastructure is available as code in terraform. 
+### CI/CD and Terraform
+* Infrastructure is available as code in terraform for `AWS` provider, and [.gitlab-ci.yml](.gitlab-ci.yml) pipeline push the Secure Agent 
+image into [AWS ECR](https://aws.amazon.com/ecr/). 
 * Before disposing and create a new instance of Informatica Secure Agent, existing live connections to various targets(for e.g. databases) 
-should be reviewed and terminated. Due to this reason this is not part of CI/CD pipeline and can be executed as per necessity.
+should be reviewed and terminated. Due to this reason this is not part of CI/CD pipeline and can be executed as per necessity. Also, back up any logs and configurations
+before next run using Terraform, see below section on EFS mounting.
 * Various configurations can be overridden using  `-var=`, see [variables.tf](./terraform/variables.tf) for available parameters.
 See Terraform doc on [variables](https://www.terraform.io/docs/configuration/variables.html) to see how to pass command line arguments.
-* Secure Agent Docker images are tagged using the value, `git-${CI_COMMIT_SHA:0:8}`, and push to ECR registry in `us-east-1`. The same 
+* Secure Agent Docker images are tagged using the value, `git-${CI_COMMIT_SHA:0:8}` and push to ECR registry in `us-east-1` in CD pipeline ([.gitlab-ci.yml](.gitlab-ci.yml)). The same 
 image can be used for production set up.
+* Note that if this Terraform set up need to use with a different AWS account then the image will have to upload to [AWS ECR](https://aws.amazon.com/ecr/) under that account(so that
+can be passed as a Terraform variable for `image_name`, see below for an example).
+* Once the Agent is deployed it can be seen under `Administrator`->`Runtime Environments` in IICS(the group name can be configured 
+in `infaagent.ini` using `InfaAgent.GroupName=aws-interop`).
+* Terraform state files are stored in shared S3 buckets `test-interop-terraform-state` and `prod-interop-terraform-state`.
 
 ### Credentials in Terraform
 * It's recommended to define all variables values in a `*.tfvars` file and pass that to terraform using `-var-file` argument.
@@ -127,13 +137,39 @@ image can be used for production set up.
 
 ```shell script
 $ cd terraform
-$ terraform init
+```
+#### Test Environment 
+```shell script
+$ terraform init -backend-config="bucket=test-interop-terraform-state" \
+                 -backend-config="key=ei.secureagent.tfstate" \
+                 -backend-config="region=us-east-1"  
+```
+#### Production Environment 
+```shell script
+$ terraform init -backend-config="bucket=prod-interop-terraform-state" \
+                 -backend-config="key=ei.secureagent.tfstate" \
+                 -backend-config="region=us-east-2"  
+```
+```shell script
 $ terraform validate
-$ terraform plan -out agent.tfplan \ 
-    -var="aws_shared_cred_file=<aws-credentials-file>" \
-    -var="aws_profile=<aws-profile>" \
-    -var="informatica_username=<informatica-username>" \
-    -var="informatica_password=<informatica-password>" \
-    -var="image_name=265723766240.dkr.ecr.us-east-1.amazonaws.com/enterprise-integrations/iics_secure_agent:<tag>"
+
+# create `secure_agent.tfvars` with required variables. 
+$ cat secure_agent.tfvars
+informatica_username="<informatica-username>"
+informatica_password="<informatica-password>"
+image_name="265723766240.dkr.ecr.us-east-1.amazonaws.com/enterprise-integrations/iics_secure_agent:git-8f6f0d24"
+
+$ terraform plan -out agent.tfplan -var-file=<path>/<to>/secure_agent.tfvars
 $ terraform apply "agent.tfplan"
-```
+```
\ No newline at end of file
+
+### Memory and CPU for Secure Agent
+* By default, Secure Agent is deployed on to a `t2.large` instance and container is given an 5GB of memory. `instance_type` 
+and `container_memory` variables can be used to adjust these values. See [variables.tf](./terraform/variables.tf) for details.
+
+### EFS Mounting, Accessing Secure Agent Configurations, Logs and Additional Debugging
+* Above terraform configuration creates a EFS file system that is used by the container to persist Secure Agent configurations 
+and logs files.
+* In order to access Secure Agent log files and configurations files use the instructions documented in 
+`Settings` in created EFS file system under `File system access` in AWS console.
+* If there's a requirement to log into EC2 instance that runs the container, [AWS Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) can be [used](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-quick-setup.html). 
\ No newline at end of file

conf/infaagent.ini

 InfaAgent.UseToken=true
-InfaAgent.GroupName=aws
+InfaAgent.GroupName=aws-interop
 #
 #Tue Jan 14 01:16:27 PST 2020
 InfaAgent.MasterUrl=https\://dm-us.informaticacloud.com
\ No newline at end of file

terraform/autoscaling-group.tf

@@ -6,7 +6,9 @@ resource "aws_autoscaling_group" "secure-agent-autoscaling-group" {
   max_size = 1
   min_size = 1
 
-  vpc_zone_identifier = data.aws_subnet_ids.subnets.ids
+  # make sure deployed EC2 instance is in the same AZ as the EFS
+  # see https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-cmd-dns-name.html
+  vpc_zone_identifier = [sort(data.aws_subnet_ids.subnets.ids)[0]]
   health_check_type = "EC2"
   launch_configuration = aws_launch_configuration.secure-agent-launch-config.name
 }
@@ -17,7 +19,7 @@ data "aws_ami" "ecs-optimized" {
 
   filter {
     name   = "name"
-    values = ["amzn2-ami-hvm-*"]
+    values = ["*-amazon-ecs-optimized"]
   }
 }
 
@@ -31,9 +33,7 @@ resource "aws_launch_configuration" "secure-agent-launch-config" {
   user_data = <<EOF
               #!/bin/bash
               echo ECS_CLUSTER=${var.ecs_cluster_name} >> /etc/ecs/ecs.config
-              yum install -y ecs-init
-              service docker start
-              start ecs
+              sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
               EOF
   instance_type = var.instance_type
   lifecycle {

terraform/backend.tf

+terraform {
+  # s3 parameters are passed in command line.
+  backend "s3" {}
+}
\ No newline at end of file

terraform/ecs.tf

 data "template_file" "container" {
   template = file("./templates/container-definitions.tpl")
   vars     = {
-    container_name          = var.container_name
-    image_name              = var.image_name
-    app_port1               = var.container_app_port[0]
-    app_port2               = var.container_app_port[1]
-    app_port3               = var.container_app_port[2]
-    informatica_username    = var.informatica_username
-    informatica_password    = var.informatica_password
-    volume1                 = var.secure_agnet_container_volumes[0]
-    volume2                 = var.secure_agnet_container_volumes[1]
-    volume3                 = var.secure_agnet_container_volumes[2]
-    volume4                 = var.secure_agnet_container_volumes[3]
-    secure_agent_efs_volume = var.secure_agent_efs_volume
+    container_name            = var.container_name
+    image_name                = var.image_name
+    container_memory          = var.container_memory
+    container_hostname        = var.container_hostname
+    app_port1                 = var.container_app_port[0]
+    app_port2                 = var.container_app_port[1]
+    app_port3                 = var.container_app_port[2]
+    informatica_username      = var.informatica_username
+    informatica_password      = var.informatica_password
+    secure_agent_configs_path = var.secure_agent_configs_path
   }
 }
 
@@ -28,11 +26,11 @@ resource "aws_ecs_task_definition" "task" {
     "EC2"]
   container_definitions    = data.template_file.container.rendered
   volume {
-    name      = var.secure_agent_efs_volume
-    host_path = ""
+    name = "agent-configs-path"
     docker_volume_configuration {
       autoprovision = true
       scope         = "shared"
+      driver        = "local"
 
       driver_opts = {
         "type"   = "nfs"
@@ -51,13 +49,6 @@ resource "aws_ecs_service" "service" {
   desired_count   = 1
   # secure agent configs and logs are persisted into an EFS volume.
   launch_type     = "EC2"
-
-  network_configuration {
-    security_groups  = [
-      data.aws_security_group.sec-group.id]
-    subnets          = data.aws_subnet_ids.subnets.ids
-    assign_public_ip = false
-  }
 }
 
 resource "aws_ecs_cluster" "cluster" {

terraform/iam.tf

@@ -25,6 +25,6 @@ resource "aws_iam_role_policy_attachment" "ecs-agent" {
 
 # allow instance profile to assume this role
 resource "aws_iam_instance_profile" "ecs-instance-profile" {
-  name = "secure-agent-ecs-instance-profile"
+  name = var.iam_instance_profile
   role = aws_iam_role.ecs-agent.name
 }
\ No newline at end of file

terraform/provider.tf

 provider "aws" {
   region                  = var.aws_region
-  shared_credentials_file = var.aws_shared_cred_file
   profile                 = var.aws_profile
 }
\ No newline at end of file

terraform/security.tf

@@ -4,8 +4,8 @@ data "aws_security_group" "sec-group" {
 }
 
 // open port 2049 for NFSv4
-resource "aws_security_group" "secure-agent-fs-" {
-  name   = "secure-agent-efs-sg"
+resource "aws_security_group" "secure-agent-efs-sg" {
+  name   = var.secure_agent_efs_sg
   vpc_id = data.aws_vpc.vpc.id
 
   // NFS
@@ -17,6 +17,16 @@ resource "aws_security_group" "secure-agent-fs-" {
     protocol        = "tcp"
   }
 
+  # allow SSH connections from configured security group,
+  # for debugging purposes(can set up a bastion host).
+  ingress {
+    security_groups = [
+    data.aws_security_group.sec-group.id]
+    from_port = 22
+    to_port   = 22
+    protocol  = "tcp"
+  }
+
   egress {
     security_groups = [
       data.aws_security_group.sec-group.id]

terraform/templates/container-definitions.tpl

@@ -2,7 +2,8 @@
   {
     "name": "${container_name}",
     "image": "${image_name}",
-    "memory": 4096,
+    "memory": ${container_memory},
+    "hostname": "${container_hostname}",
     "portMappings": [
       {
         "containerPort": ${app_port1},
@@ -29,20 +30,8 @@
     ],
     "mountPoints": [
         {
-            "containerPath": "${volume1}",
-            "sourceVolume": "${secure_agent_efs_volume}"
-        },
-        {
-            "containerPath": "${volume2}",
-            "sourceVolume": "${secure_agent_efs_volume}"
-        },
-        {
-            "containerPath": "${volume3}",
-            "sourceVolume": "${secure_agent_efs_volume}"
-        },
-        {
-            "containerPath": "${volume4}",
-            "sourceVolume": "${secure_agent_efs_volume}"
+            "containerPath": "${secure_agent_configs_path}",
+            "sourceVolume": "agent-configs-path"
         }
     ]
   }

terraform/variables.tf

 variable "informatica_username" {}
 variable "informatica_password" {}
 
-variable "aws_shared_cred_file" {}
 variable "aws_profile" {
   default = "default"
 }
@@ -52,8 +51,19 @@ variable "container_name" {
   default = "iics-secure-agent-test"
 }
 variable "container_network_mode" {
-  default = "awsvpc"
+  # in order to pass a container hostname bridge network mode is used
+  # instead of awsvpc mode.
+  default = "bridge"
 }
+
+variable "container_hostname" {
+  default = "test.ei.secureagent.doit.wisc.edu"
+}
+
+variable "container_memory" {
+  default = 5120 # 5GB
+}
+
 variable "container_app_port" {
   type    = list(string)
   default = [
@@ -69,13 +79,12 @@ variable "container_count" {
 }
 
 # see Secure Agent system requirements, before changing instance type, see the
-# supported instance types for launch configuration.
+# supported instance types for launch configuration. See `container_memory` above for
+# configuring memory for container process.
 variable "instance_type" {
-  default = "t2.large"
+  default = "t2.large" # 2 vCPU, 8 GB
 }
 
-# ecs
-
 variable "ecs_iam_role" {
   default = "iics-secure-agent-iam-role"
 }
@@ -116,19 +125,8 @@ variable "efs_tags" {
   }
 }
 
-variable "secure_agnet_container_volumes" {
-  type = list(string)
-  default = [
-    # see Dockerfile for these default values.
-    "/home/agent/infaagent/apps/agentcore/infaagent.log",
-    "/home/agent/infaagent/apps/agentcore/agentcore.log",
-    "/home/agent/infaagent/apps/agentcore/logs",
-    "/home/agent/infaagent/apps/agentcore/data"
-  ]
-}
-
-variable "secure_agent_efs_volume" {
-  default = "agent"
+variable "secure_agent_configs_path" {
+  default = "/home/agent/infaagent/apps/agentcore/"
 }
 
 variable "secure_agent_sg_tags" {
@@ -136,4 +134,12 @@ variable "secure_agent_sg_tags" {
     Name = "secure-agent-efs-sg"
     tier = "test"
   }
+}
+
+variable "iam_instance_profile" {
+  default = "secure-agent-ecs-instance-profile"
+}
+
+variable "secure_agent_efs_sg" {
+  default = "secure-agent-efs-sg"
 }
\ No newline at end of file