README.md

Table Of Contents

• User Guide
• IICS Secure Agent Docker Image
  • Description
  • Requirements
  • The Image
  • Containers
    • Environment Variables
    • Externalized Configurations
    • Starting
    • Monitoring
  • Hardware Requirements
  • Known Issues
  • Enterprise Integration Secure Agent
    • Architecture Diagram
    • CI/CD and Terraform
    • Informatica User
    • Credentials in Terraform
      • Test Environment
      • Production Environment
    • Memory and CPU for Secure Agent
    • EFS Mounting, Accessing Secure Agent Configurations, Logs and Additional Debugging

User Guide

User guide for AWS hosted Secure Agent is available here.

IICS Secure Agent Docker Image

This document covers following topics:

1. How ro run IICS Secure Agent using Docker image.
2. IICS Secure Agent deployed(including Terraform configuration) in AWS(interop).

Description

This project was forked from jbrazda/ic-sagent-docker with the intention of containerizing the IICS Secure Agent to run in Amazon Web Services. As of early January 2019, there is no official Docker image for IICS Secure Agent.

Requirements

• Docker.
• An IICS user account and password for your organization that has appropriate rights to create and manage secure agents. This user will be used to login to the Informatica APIs to register the agent. At a minimum, the user should be assigned a role that has Create, Read, and Update privileges for the "Secure Agent" and "Secure Agent Group" asset types.
• Your Informatica POD and REGION, which can be inferred from https://${POD}.${REGION}.informaticacloud.com/. Defaults are set to POD=usw3 and REGION=dm-us, but can be overriden with Docker environment variables.

The Image

The image is based on Ubuntu 18.04. It downloads and installs necessary dependencies, and then downloads the installer file from the correct Informatica URL based on your POD and REGION.

1. Clone this repository.
2. docker build . -t iics_secure_agent:<tag> - optionally pass --build-arg parameters for POD and REGION if the defaults are not correct. POD and REGION are also set as environment variables, because containers will use these values for communicating with the Informatica API.

Containers

The container executes a bash script called run_agent.sh upon start. It needs INFORMATICA_USER and INFORMATICA_PASSWORD environment variables set during runtime in order to communicate with the Informatica API to check and register the Secure Agent.

Environment Variables

• INFORMATICA_USER (required) - User that can run the agent and access Informatica APIs
• INFORMATICA_PASSWORD (required) - Password for above credential. Used for API access only.
• JSON_LOG (optional) - If this is set, then the log output will be in JSON format.

Externalized Configurations

Secure Agent's configurations can be externalized using Docker volumes. Following Secure Agent's directories and files can be externalized for containers. Note that after initial container startup, these configurations can be used to start subsequent containers.

• agentcore.log - Secure Agent default runtime log file.
• infaagent.log - Secure Agent start up log file.
• logs - contains Secure Agent runtime log files and lock files.
• data - contains the configuration data for mapping and integration components.
• conf - contains Secure Agent runtime configuration files.
  1. infaagent.ini - contains the registration information for the secure agent(this will be populated by Secure Agent during start up).
  2. proxy.ini - proxy configurations for Secure Agent, if running behind a proxy server.

Following ports(among others) in Secure Agents can be mapped to host for externalized control.

• 7080 - Process Engine shutdown port.
• 7443 - Process Engine https port.
• 5432 - Process Engine Postgres DB port.

See below section for examples on how to use volume and port mapping.

Starting

• Setting the hostname will provide the associated name in the IICS website.
• Note: since anybody who has access to see the processes can view the values INFORMATICA_USER and INFORMATICA_PASSWORD, it's recommenced to configure them in a Docker .env file.

$ touch agentcore.log infaagent.log 
$ docker run -d \
    -h <hostname> \
    --env INFORMATICA_USER=xxxxx \
    --env INFORMATICA_PASSWORD=xxxx \
    -v $(pwd)/infaagent.log:/home/agent/infaagent/apps/agentcore/infaagent.log  \
    -v $(pwd)/agentcore.log:/home/agent/infaagent/apps/agentcore/agentcore.log  \
    -v $(pwd)/logs:/home/agent/infaagent/apps/agentcore/logs  \
    -v $(pwd)/data:/home/agent/infaagent/apps/agentcore/data \
    -p 7080:7080 \
    -p 7443:7443 \
    -p 5432:5432 \ 
    --name <container_name> iics_secure_agent:<tag>

• Note that if conf directory needs mounting, a minimum configuration for starting the Secure Agent is required, see conf for example.

$ touch agentcore.log infaagent.log 
$ docker run -d \
    -h <hostname> \
    --env INFORMATICA_USER=xxxxx \
    --env INFORMATICA_PASSWORD=xxxx \
    -v $(pwd)/infaagent.log:/home/agent/infaagent/apps/agentcore/infaagent.log  \
    -v $(pwd)/agentcore.log:/home/agent/infaagent/apps/agentcore/agentcore.log  \
    -v $(pwd)/logs:/home/agent/infaagent/apps/agentcore/logs  \
    -v $(pwd)/data:/home/agent/infaagent/apps/agentcore/data \
    -v $(pwd)/conf:/home/agent/infaagent/apps/agentcore/conf \
    -p 7080:7080 \
    -p 7443:7443 \
    -p 5432:5432 \ 
    --name <container_name> iics_secure_agent:<tag>

Monitoring

If volume mapping is not used use docke exec to attach to the running container.

docker exec -it <container_name> less agentCore.log

Hardware Requirements

Minimum Hardware Specs (reference):

• Windows version 7 (32-bit or 64-bit) or Windows XP Pro SP2/SP3 (32-bit).
• Linux (32-bit or 64-bit).
• Red Hat Enterprise Linux version 5.3 or above.
• SUSE Linux Enterprise version 10 and above.
• 2-4 CPU recommended.
• 2 GB RAM recommended dedicated for the Agent (Recent customers have used 8GB).
• 1 GB Hard disk space.
• 1 GB Network connectivity (faster the better as this will most likely the bottleneck).

Known Issues

• Doesn't seem to be able to run in host network mode because it won't be able to talk to internal ports. It would probably work if you expose those ports.

Enterprise Integration Secure Agent

Secure Agent containers are deployed in following tiers in interop/AWS account. This deployment is automated through Terraform (see below section on Terraform).

environment	tier	runtime name(default) in IICS
test	us-east-1(N. Virginia)	ei.secureagent.doit.wisc.edu
prod	us-east-2(Ohio)	ei.secureagent.doit.wisc.edu

Architecture Diagram

The diagram source can be found here.

CI/CD and Terraform

• Infrastructure is available as code in terraform for AWS provider, and .gitlab-ci.yml pipeline push the Secure Agent image into AWS ECR.
• Before disposing and create a new instance of Informatica Secure Agent, existing live connections to various targets(for e.g. databases) should be reviewed and terminated. Due to this reason this is not part of CI/CD pipeline and can be executed as per necessity. Also, back up any logs and configurations before next run using Terraform, see below section on EFS mounting.
• Various configurations can be overridden using -var=, see variables.tf for available parameters. See Terraform doc on variables to see how to pass command line arguments.
• Secure Agent Docker images are tagged using the value, git-${CI_COMMIT_SHA:0:8} and push to ECR registry in us-east-1 in CD pipeline (.gitlab-ci.yml). The same image can be used for production set up.
• Note that if this Terraform set up need to use with a different AWS account then the image will have to upload to AWS ECR under that account(so that can be passed as a Terraform variable for image_name, see below for an example).
• Once the Agent is deployed it can be seen under Administrator->Runtime Environments in IICS(the group name can be configured in infaagent.ini using InfaAgent.GroupName=aws-interop).
• Terraform state files are stored in shared S3 buckets test-interop-terraform-state and prod-interop-terraform-state.

Informatica User

Credentials for Informatica User (for test and prod instances) are stored in parameter store (in regions us-east-1 and us-east-2 respectively) and expected to be available with the following names:

• username - /iics/cicd_username.
• password - /iics/cicd_password.

Credentials in Terraform

• It's recommended to define all variables values in a *.tfvars file and pass that to terraform using -var-file argument. This will avoid having any sensitive parameters in bash history.

$ cd terraform

Test Environment

$ terraform init -backend-config="bucket=test-interop-terraform-state" \
                 -backend-config="key=ei.secureagent.tfstate" \
                 -backend-config="region=us-east-1"  

Production Environment

$ terraform init -backend-config="bucket=prod-interop-terraform-state" \
                 -backend-config="key=ei.secureagent.tfstate" \
                 -backend-config="region=us-east-2"  

$ terraform validate

# create `secure_agent.tfvars` with required variables. 
$ cat secure_agent.tfvars
image_name="265723766240.dkr.ecr.us-east-1.amazonaws.com/enterprise-integrations/iics_secure_agent:git-8f6f0d24"

# sample `secure_agent.tfvars` for production set up.
$ cat prod.secure_agent.tvfars
image_name="265723766240.dkr.ecr.us-east-1.amazonaws.com/enterprise-integrations/iics_secure_agent:git-8f6f0d24"
aws_region="us-east-2"
vpc_tags={"Name":"prod-tier","tier":"prod"}
private_subnets_filter={"Name":"prod-private-*"}
security_group={"Name":"internal","tier":"prod"}
container_name="iics-secure-agent-prod"
ecs_iam_role="iics-secure-agent-iam-role-prod"
ecs_execution_role="iics-secure-agent-ecs-execution-role-prod"
ecs_cluster_tags={"Name":"iics-agent-cluster","tier":"prod"}
ecs_task_tags={"Name":"iics-secure-agent","tier":"prod"}
efs_tags={"Name":"iics-secure-agent","tier":"prod"}
secure_agent_sg_tags={"Name":"secure-agent-efs-sg","tier":"prod"}
iam_instance_profile="secure-agent-ecs-instance-profile-prod"
iics_secret_access_policy="iics-secret-access-policy-prod"

$ terraform plan -out agent.tfplan -var-file=<path>/<to>/secure_agent.tfvars
$ terraform apply "agent.tfplan"

Memory and CPU for Secure Agent

• By default, Secure Agent is deployed on to a t2.xlarge instance and container is given an 15GB of memory. instance_type and container_memory variables can be used to adjust these values. See variables.tf for details.

EFS Mounting, Accessing Secure Agent Configurations, Logs and Additional Debugging

• Above terraform configuration creates a EFS file system that is used by the container to persist Secure Agent configurations and logs files.
• One way to access Secure Agent's log files and configurations files is to mount the EFS file system use by Secure Agent. EFS file system is accessible from any instance within test and prod tiers. Start an EC2 instance in test-tier VPC (or in prod VPC - prod-tier ) and this instance can be used to mount the EFS file system (be sure to that this EC2 instance can be accessed from SSH). Then use the instructions documented in Settings in Secure Agent's EFS file system(iics-secure-agent) under File system access in AWS console.
• If there's a requirement to log into EC2 instance that runs the container, AWS Session Manager can be used.