diff --git a/docs/tutorials/connectors/FTP_and_SFTP/FTP_connector.md b/docs/tutorials/connectors/FTP_and_SFTP/FTP_connector.md index 7736ed30bfc096b723825fdd2eba7c23de8d6bf6..c03fe01274abb3905c917990b82a8cd328448f99 100644 --- a/docs/tutorials/connectors/FTP_and_SFTP/FTP_connector.md +++ b/docs/tutorials/connectors/FTP_and_SFTP/FTP_connector.md @@ -35,7 +35,11 @@ If using this connector as a source, there needs to be a file ready to move in t 1) In the Data Integration section of IICS, browse to your Project Folder, then select "New..." in the upper right of your browser. 2) Select Mappings > Mapping, then click the "Create" button. 3) Create a name for your Mapping. -4) In the Source tile, +4) In the Source tile, in Properties > Source, select the connector we created in the section above. + - Select Source Type: Single Object + - Object: Click on the "Select..." button and then select your file you would like to start your Integration with. + +If using this FTP/SFTP connector as a target, you can select "Create New at Runtime" as the Object. If you do so, you will need to create the Object Name with the file extention included. # References diff --git a/docs/tutorials/connectors/FTP_and_SFTP/SSH_key_file_processor.md b/docs/tutorials/connectors/FTP_and_SFTP/SSH_key_file_processor.md index d90fe403f52f6893077fb428c77aef3b3a428cbd..3dd0ef2da1edbf9a01968854fa0f48471de511e3 100644 --- a/docs/tutorials/connectors/FTP_and_SFTP/SSH_key_file_processor.md +++ b/docs/tutorials/connectors/FTP_and_SFTP/SSH_key_file_processor.md @@ -1,13 +1,81 @@ -# SSH Keypair authentication SFTP mapping using File Processor Connector +# SSH Keypair authentication SFTP mapping using File Processor and Flat File Connectors This style of file syncing for an SFTP source or target is used if SSH keypair authentication is required. This can be with or without a passphrase on the keypair. Either the Integration Team can create the SSH keypair for you on the Secure Agent, or you can provide private key to the Integration Team to place on the Secure Agent. # Requirements -# Connector +* A FTP or SFTP server's settings + * Hostname + * User credentials + * Port number + * Directory path on the FTP/SFTP server for delivery, if not to the root directory +* SSH Keypair. You will either need to request this from [The Integration Team](mailto:integration-platform@doit.wisc.edu) or provide the Private Key to The Integration Team if you have created it yourself. +* A local directory on the Secure Agent you are creating the mapping on. If your team does not have a folder dedicated for your work, you can reach out to the [Integration Team](mailto:integration-platform@doit.wisc.edu) to request one. This mapping style will not be appropriate for the /tmp folder. + +# Connectors + +Integrations that use SSH keypairs need to use the File Processor connector, and in turn that requires the transformation part of the integration to drop or receive a file from the Secure Agent with a Flat File connection. + +### Flat File + +1) In the Administrator section of IICS, select Connections > New Connection. +2) Input these settings: + * Connection Name: Please follow our [naming guidelines](/docs/best-practices/naming.md). + * Type: Flat File + * Runtime Environment: The Secure Agent you typically use. + * Directory: This is folder path provided to you by the Integration Team after you've requested a team folder from them. Typically, this will look like **/home/agent/infaagent/apps/agentcore/partner-integrations/YourTeamFolder** +3) Save your Connection. + + + +### File Processor + +1) In the Administrator section of IICS, select Connections > New Connection. +2) Input these settings: + * Connection Name: Please follow our [naming guidelines](/docs/best-practices/naming.md). + * Type: FileProcessor + * Runtime Environment: The Secure Agent you typically use. + * Source File Directory: This is folder path provided to you by the Integration Team after you've requested a team folder from them. Typically, this will look like **/home/agent/infaagent/apps/agentcore/partner-integrations/YourTeamFolder** + * Target File Directory: This will be the folder you're syncing to on the SFTP server. + * Select File and File Pattern: You can choose to gather ALL files and sync them, files with specific patterns, or a file with a specific name. + * PassKey1: If using a pass phrase on your SSH key, it should be entered here. +3) Save your Connection. + + # Mapping +This style of integration uses two parts: + +* Transform your data. +* Transport your data between the Secure Agent and the SFTP server. + +The order of these parts depents on if you're picking up or dropping off your data to the SFTP server. The reason we need to do this in two parts is because the File Processor Connection syncs a file or all of the contents within a directory between the Secure Agent and the SFTP server. That file needs to exist before it can be processed. + +Typically, the data transformation looks something like using a database view as a source, transform the data, deliver to the Flat File location on the Secure Agent. + +The File Processor mapping for syncing your file to or from the SFTP server is a bit more complicated. + +1) Browse to your Project Folder and select "New..." to create a Mapping. +2) In the Source tile, under Properties > Source, select your File Processor connection. +3) Select Single Object for the Source Type. +4) For Object, select SFTP_PUT if you are using the SFTP server as your target. You can select SFTP_GET if you're retrieving files from the SFTP server. In both cases, the File Processor is your Source for this mapping. Another thing to note is that this Object **IS NOT** the file you have data in, which would be the normal setting for any other mapping. +5) Expand "Query Options" under the Source section and configure the filters: + * SFTP_User_ID: The user for the SFTP server. + * SFTP_Hostname: The SFTP server's hostname. + * SFTP_Port: This should be 22 unless otherwise directed by the SFTP server administrator. + * SFTP_SSH2_KeyFile: The full file path of the location of the SSH key, without the file extension (unless using a pass phrase). The Integration Team will work with you to give you this file path. + * SFTP_PassPhrase: If you are using a pass phrase with your SSH keys, enter "PassKey1" here, **NOT THE ACTUAL PASSPHRASE**. Only in this instance will you want to add the .pvk file extension to the SFTP_SSH2_KeyFile entry above. + + + + +6) In the Target of this mapping you will want to add your Flat File connection with Target Type of Single Object. +7) You can Create New at Runtime for this, unless there already exists a file of the name you choose (the second run of this mapping will fail unless you alter this between jobs). This file you create is a log of each job of this mapping, NOT data that you're processing. It will tell you how many files were synced, what time, etc. +8) Save your mapping and test it. + +This type of integration is very finicky and there are many pitfalls. If you run into issues, please feel free to reach out to the [Integration Team](mailto:integration-platform@doit.wisc.edu) for assistance with questions, troubleshooting, or recommendations for adjusting this documentation. + # References * [IICS KB Article on the File Processor Connector](https://docs.informatica.com/integration-cloud/data-integration-connectors/current-version/file-processor-connector/introduction-to-file-processor-connector/file-processor-connector-overview.html) \ No newline at end of file diff --git a/images/FileProcessor.png b/images/FileProcessor.png new file mode 100644 index 0000000000000000000000000000000000000000..861d0112900c65704ce547b458a210080d6efa75 Binary files /dev/null and b/images/FileProcessor.png differ diff --git a/images/FlatFile.png b/images/FlatFile.png new file mode 100644 index 0000000000000000000000000000000000000000..c57b2055fd8e5c0cf4224ab05db1e03f5688daf9 Binary files /dev/null and b/images/FlatFile.png differ diff --git a/images/SFTPFilters.png b/images/SFTPFilters.png new file mode 100644 index 0000000000000000000000000000000000000000..9432d5444f4a03478f7b23d99478be88a3ed7036 Binary files /dev/null and b/images/SFTPFilters.png differ diff --git a/images/SFTPSource.png b/images/SFTPSource.png new file mode 100644 index 0000000000000000000000000000000000000000..03528170dff828c5f7117b54e96bcc12e48c0950 Binary files /dev/null and b/images/SFTPSource.png differ