# Creating your First Integration This tutorial covers the full process that a University of Madison-Wisconsin integrator will use to create their own end-to-end data integration, gaining access to several sources and targets and facilitating an automated data transfer using IICS. ## Table of Contents - [Gaining Access to IICS](#gaining-access-to-iics) - [Gaining Access to PersonAPI](#gaining-access-to-personapi) - [Creating a Box Folder](#creating-a-box-folder) - [Using IICS to Create Connectors](#using-iics-to-create-connectors) - [Creating The Box Connector](#creating-the-box-connector) - [Creating the PersonAPI Connector](#creating-the-personapi-connector) - [Creating the Mapping](#creating-the-mapping) ## Gaining Access to IICS You will be unable to use any practical functionality in IICS until they have an IICS account with permissions applied to it. Acquiring this account is dependent on contacting the IICS administrators. This process is outlined on our [Onboarding Page](/docs/on-boarding-to-iics.md). If you are following along with this tutorial, you most likely will have already started the process of accessing the IICS service. The steps of this process will be hosted on this page as a quick reference, however. 1. Create a Manifest Group, and add all related personell who will be using IICS (including yourself). - The Madison-Wisconsin KnowledgeBase has [Documentation on Creating Manifest Groups](https://kb.wisc.edu/page.php?id=25878) and [Documentation on Managing Group Members](https://kb.wisc.edu/page.php?id=25882). - Manifest Groups must be created within a folder. The KnowledgeBase has [Documentation on Folders](https://kb.wisc.edu/page.php?id=27783). - As an alternative to this first step, if your team already has a Manifest Group, that may be used. This will be likely if members of your team have worked with IIS before. - Contact the leader of your team about such a manifest group as they would likely have information on it, in the case it exists. - If this Manifest Group is already used to provision access to IICS, you may skip to step 4. Your team lead would likely also know about this. 2. Grant our team at uw:org:ais:ais-admins Read and View permissions to your group. - We will be unable to view your group or apply the automated IICS account provisioning unless you do this. - There is [Documentation on Managing Manifest Group Permissions](https://kb.wisc.edu/25880). 3. Contact our Team so that we can apply the correct permissions to your Manifest Group. - IICS automatically sets up accounts for any users in Manifest Groups with specified permissions, but those permissions must be set up by us. - There may be some delay on this step, although the turnaround should be no more than 1-2 business days. Our team will contact you once this step has been completed. - Note that gaining access to IICS involves filling out a formal request. Information on this process can be found in our [Onboarding Docs](/docs/on-boarding-to-iics.md). 5. Log in to IICS. This can easily be done from [Our Log-In Documentation](/docs//logging-in.md). - IICS has two orgs associated with it - test and prod. It is heavily reccomended that you log in to both, as to make sure your account exists within the system. - If you intend to change org on the same browser, it is heavily reccomended that you manually log out by clicking the "log out" button on the top right of the homepage or under the profile tab of the top bar on any other page. - If you fail to log out, IICS will fail to change organizations when logging in using a different link. 6. Contact our Team so that we can apply correct permissions to you. - If you are working alongside multiple team members it is reccommended that all team members are logged in before you contact us, so we can provide permissions for everyone in one round of communication. ## Gaining Access to PersonAPI The primary components of any integration are a Data Source (where the information you are retrieving comes from) and a Target (where you are putting the information you retrieved). For this Integration we will be using the Mock Person API as the data source. It is highly reccommended that you pursue accessing this data source while waiting to gain access to IICS, as to minimize the amount of time spent making no progress. 1. Sign in to the API portal and register your Application. - Information on this process can be found at the [Person API Docs](https://developer.wisc.edu/get-started). - The information relevant to this specific tutorial can be found in all steps up to and including "Register an Application". - When creating your Application, Enable the Mock Person API. You will not need any other APIs for this specific tutorial. - When the Application is created, it will also be provisioned an API key. Make sure to note down the Key and Secret from this, as they will both be used multiple times in this tutorial. - They will most notably be used in the IICS connector to the Mock PersonAPI (which will allow you to transfer data from the Mock API into IICS). - You must have a Wisconsin NetID in order to use the PersonAPI service. 2. Sign up for Postman to test your Application. - Postman is a free service that allows a user to make calls to APIs from an easy-to understand interface. Information and downloads can be found at the [Postman Homepage](https://www.postman.com/). - Note that Postman can also be used directly from a web browser, in case downloading the app is unwanted or undesireable. - If you are having trouble understanding Postman, you can pursue their training options. They have [A Tutorial on Sending your First Request](https://learning.postman.com/docs/getting-started/first-steps/sending-the-first-request/), as well as [A More Generalized Overview of All of Their Training Services](https://learning.postman.com/docs/introduction/overview/). 3. Set up the API call and credentials within Postman. - You will be sending a GET request to the url https://mock.api.wisc.edu/people. - Once the basic request has been set up, click the tab below the request bar that says "authentication" and scroll down to where it says "configure new token". - Name the token as you please. - Set the grant type to "Client Credentials". - The Access Token URL is https://api.wisc.edu/oauth/token. - The Client ID is the "Key" from the API Key of your Application. - The Client Secret is the "Secret" from the API Key of your Application. - Leave the Scope Blank. - Client Authentication should be "Send client credentials in body". - Once the token has been configured, press "Get New Access Token". - If you are testing in multiple sessions, you will need to Get a New Access Token multiple times, as the Access Token expires after a few hours. - Note that Postman will save the information you filled out, so you will only need to press the "Get a New Access Token" button. - To test the API, press the blue "Send" button near the URL input bar. - A successful run will return a list of several people, each with unique associated information (most identifiably, a name). - A 401 error will represent an error with validation - either an expired token, or invalid information used in the creation of the token. A successful test should look something like this:  ## Creating a Box Folder Box is an online file storage solution, and is free for University of Madison-Wisconsin users. More information can be found at [The UW-Madison Information Technology Department's Box Page](https://it.wisc.edu/services/box/). This segment of the tutorial is also able to be completed without direct access to IICS. 1. Sign up for a Box account with your UW-Madison netid - UW Madison's offerings for Box can be found at http://uwmadison.box.com/ - If you already have a Box account set up in UW Madison, this step is unnecessary. 2. Create a new Folder in your Box account. - When logging in, there will be a button labelled "New+" at the top right of the screen. Clicking this will bring up a dropdown, in which you can create a New Folder. - More specific documentation on creating a folder can be found in [Box's Folder and File Creation Tutorial](https://support.box.com/hc/en-us/articles/360043696394-Create-New-Files-And-Folders) 3. Save the unique Folder ID of the folder you just created. - The URL to a UW Madison Box folder will look like https://uwmadison.app.box.com/folder/id with "id" being replaced by a long string of numbers. This string of numbers is the Folder ID. Note this down somewhere, as it will be used in creating the IICS connector (which will allow you to transfer information from IICS to your Box folder). ## Using IICS to Create Connectors At this point you will have set up everything you need to access all relevant data for this tutorial. It is at this point where you will need to start directly accessing IICS. If you still don't have access to IICS, please return to the [First Section of this Tutorial](gaining-access-to-iics). 1. Log In to the IICS Test Organization. - Access this documentation's [Log In Page](/docs/logging-in.md) for more information on Logging In. - Log in specifically to the Test Org The Production Org is used to host and run live integrations, and keeping it clean is paramount. 2. Navigate from the main screen to the "Administrator" window. - The name of the window is a bit of a misnomer; Various functions in the tab are usable by regular users, and unrelated to IICS administration. - If you have already navigated to a different window, click the line of text in the top-left of the screen. This will bring up a similar window to the main screen, allowing you to navigate to any tab in IICS. 3. Navigate from the main page of the Administrator window to the Connections page. - The icon for the Connection page will have a plug on it. Once you are at this section, you will be able to create the Connections needed for this tutorial. Connections are the main asset used in IICS to determine how data is accessed from and pushed to various locations. You will be creating two Connections; a Connection to the Mock Person API to use as a data source, and a Connection to your Box folder to use as a target for your data. ### Creating The Box Connector 1. Click the green "New Connection" button in the top right. This will bring up a window with various fields to fill in. 2. Name the Connection. The owner and purpose of this connection should be easily identifiable. - Something along the lines of "User FirstIntegrationTut Box" would fit these criteria, communicating which integration the Connection will be for as well as what data source or target it is referring to. 3. Set the "Type" field to "Box (Informatica)". This will indicate which service Informatica is interfacing with; In this instance, Box. - Selecting this option will cr4eate many more fields that can potentially be filled in. This may look overwhelming, but only the fields indicated in the tutorial will actually be relevant in connecting to the Box folder. 4. Click the "Runtime Enviornment" dropdown menu and select ei.secureagent.doit.wisc.edu. - This is the Secure Agent - The machine which the Integration will use to run the IICS Process you will create. ei.secureagent.doit.wisc.edu is available for all Integrators to use. 5. Click the "Get Token" button next to the "OAuth Access Token" field. This will bring up a new window. Follow the instructions in that window. - The window will prompt you to log in to Box, then request Read and Write permissions. This will give IICS all necessary permissions to push files to your Box folder. 6. In the "Box File or Folder ID" Field, enter the unique Folder ID of your Box folder. - You were previously instructed to save this field in the section on [Creating a Box Folder](#creating-a-box-folder). If you do not have it, return to this section and acquire it. 7. Save the connection by pressing the blue "Save" button in the top right. No other fields displayed are relevant to your current use-case. A correctly set-up Box connector should look like this:  ### Creating the PersonAPI Connector 1. Navigate back to the Connection page. You can do this by clicking the "X" button in the top right, or by clicking the Connections Page icon on the left sidebar. 2. Once again click the green "New Connection" button in the top right. This will allow you to create a new connection in a similar manner to the previous one. 3. Name this new connection, in a smiliar fashion to the previous connection. - In the case of the previous example of a name, you would name this Connection "User FirstIntegrationTut MockPersonAPI". 4. Set the type of the connection to "REST v2 (Informatica)". - This will indicate that the connection is one that connects to a REST API. - The fields created by selecting each option can differ wildly depending on what information is needed. In this instance, only fields for Runtime Enviornment and Authentication will be displayed initially. 5. Click the "Runtime Enviornment" dropdown menu and select ei.secureagent.doit.wisc.edu. 6. Click the "Authentication" dropdown menu and select "OAuth 2.0-Client Credentials". - This will display many more fields for you to potentially fill. As before, only a fraction of these will be relevant to the current use case - Many of these fields will contain information previously used when testing the Mock PersonAPI in Postman. Navigate to [Gaining Access to PersonAPI](#gaining-access-to-personapi) if you need information on how to access any of this information. 7. For the "Access Token URL" field, input https://api.wisc.edu/oauth/token. - This is the specific URL that the Mock PersonAPI uses to authenticate its requests. 8. For "Client ID" and "Client Secret", input the "Key" and "Secret" from the Api Key of the Application you set up, respectively. (TODO refer to application more so people catch on to this) 9. Set "Client Authentication" to "Basic Access Header". TODO CONFIRM THIS 10. Click the "Generate Access Token" Button next to the "Access Token" Field. If all previous credentials are set up correctly, this will fill the field automatically. 11. In the "Swagger File Path" field place the value https://git.doit.wisc.edu/interop/iics/external-iics-docs/-/raw/master/docs/tutorials/PersonAPI/swagger.json - Swagger Files are used to determine what actions an API can take, and what values can be returned by an api. 12. Save the connection by pressing the blue "Save" button in the top right. No other fields displayed are relevant to your current use-case. A correctly set-up Mock PersonAPI connector should look like this:  ## Creating the Mapping The process up to this point was in preparation of creating a Mapping, an IICS asset that directly enables the transfer of data from source to target. This type of asset is contained under the umbrella of Informatica's "Cloud Data Integration" service. 1. Navigate to Cloud Data Integration - The top left will have a drop-down labelled as "Administrator". Clicking this will bring up a window containing all IICS services avilable. Locate "Data Integration" and click it. 2. Create a Mapping - The Data Integration page has a dashboard with many pieces of information. Fo the purposes of this tutorial, most of the information on here can be safely ignored. - On the left sidebar (in a similar location to the "Connections" tab within the Administrator window) there is a button labelled "New" marked with a green plus icon. This button will allow you to create new assets. - Upon clicking the "New" button, a window labelled "New Asset" will pop up. - On the left sidebar of this window, click "Mappings", located just below the default tab of "Tasks". - In the main window, select "Mapping". - At the bottom right, click the blue button labelled "Create". 3. Change the Mapping Information - DO NOT CLICK ON ANY ICONS YET. - A mapping is made up of many Transformations (specific actions taken on the data, represented by the boxes connected together by arrows). A list of potential Transformations you can use is located on the left side, although none of these will be used in this tutorial. - All information about the currently selected Transformation (or if no Transformation is currently selected, the Mapping as a whole) is located at the bottom of the screen. - If there is no currently selected transformation, you will be able to edit the properties of the mapping. - Set the name to something that identifies this mapping as both a tutorial mapping, and as YOUR tutorial mapping. To follow the previous examples of naming from this tutorial, "User FirstIntegrationTut Mapping" would suffice. - Click the "Browse" button next to "Location". This will bring up a menu with many folders, sorted alphabetically. Locate "IntegrationTutorials" and click it. This will make sure the Mapping is located alongside the tutorials done by other users. - If the current information on this bottom screen regards a Transformation you will need to deselect whatever Transformation you have selected. This can be done easily by deleting the Transformation, using the trash can icon near the top right. - Once this is done follow the steps from above to change the information. 4. Edit the Source Transformation - In the main view of the mapping, click on the "Source" Transformation. - Turn your attention to the bottom panel. It will now display information regarding the Source Transformation. - On the left bar of the bottom panel, click on the "Source" tab. - Next to the "Connection" property, click the dropdown menu. Select the PersonAPI Connector you previously created. - Underneath the "Connection" property there will now be a property labelled "Operation". Press the "Select" button and select "get_people". - On the left bar, click on the "Field Mapping" tab. - TODO THE REST OF THIS 5. Edit the Target Transformation - In the main view of the mapping, click on the "Target" Transformation. - On the left bar of the bottom panel, click on the "Target" tab. - Next to the "Connection" property, click the dropdown menu. Select the Box Connector you previously created. - Underneath the "Connection" property will be several other fields. Leave "Target Type" and "Operation" alone. - Press the "Select" button next to "Object". This will bring up a new window. - Change "Target Object" from "Existing" to "Create New at Runtime" - Type in the name of the file as you would like it named in your Box folder. When the mapping runs, it will create a new .csv file with the same name as this setting. - Press the blue "Ok" button. 6. Save and Test your Mapping NAVIGATE TO DATA INTEGRATION CREATE MAPPING click create go from tasks to mappings click the first option - "mapping" CHANGE NAME CHANGE LOCATION IntegrationTutorials EDIT SOURCE Click "Source" transformation Click "Source" box on bottom left Change the Connection to the previously created PersonAPI Connection Change the operation to get_people go to field mapping TODO STOP HAVING THIS BREAK TO ALL HELL EDIT TARGET Click "Target" transformation Click "Incoming Fields" on bottom left -look at the pretty incoming fields oooo Click "Target" on bottom left CHANGE THE CONNECTION TO THE BOX CONNECTION target type should be single object TODO ASK JARED TO HELP SELECT OBJECT operation should always be insert SAVE AND RUN THE MAPPING CHECK THE FILE VS. WHAT YOU GOT FROM THE PERSONAPI