Newer
Older
SAM CARPENTER
committed
# Creating your First Integration
This tutorial covers the full process that a University of Madison-Wisconsin integrator will use to create their own end-to-end data integration, gaining access to several sources and targets and facilitating an automated data transfer using IICS.
SAM CARPENTER
committed
## Table of Contents
- [Gaining Access to IICS](#gaining-access-to-iics)
- [Gaining Access to PersonAPI](#gaining-access-to-personapi)
- [Creating a Box Folder](#creating-a-box-folder)
- [Using IICS to Create Connectors](#using-iics-to-create-connectors)
- [Creating The Box Connector](#creating-the-box-connector)
- [Creating the PersonAPI Connector](#creating-the-personapi-connector)
- [Creating the Mapping](#creating-the-mapping)
SAM CARPENTER
committed
## Gaining Access to IICS
You will be unable to use any practical functionality in IICS until they have an IICS account with permissions applied to it. Acquiring this account is dependent on contacting the IICS administrators. This process is outlined on our [Onboarding Page](/docs/on-boarding-to-iics.md).
If you are following along with this tutorial, you most likely will have already started the process of accessing the IICS service. The steps of this process will be hosted on this page as a quick reference, however.
1. Create a Manifest Group, and add all related personnel who will be using IICS (including yourself).
- The Madison-Wisconsin KnowledgeBase has [Documentation on Creating Manifest Groups](https://kb.wisc.edu/page.php?id=25878) and [Documentation on Managing Group Members](https://kb.wisc.edu/page.php?id=25882).
- Manifest Groups must be created within a folder. The KnowledgeBase has [Documentation on Folders](https://kb.wisc.edu/page.php?id=27783).
- As an alternative to this first step, if your team already has a Manifest Group, that may be used. This will be likely if members of your team have worked with IICS before.
- Contact the leader of your team about such a manifest group as they would likely have information on it, in the case it exists.
- If this Manifest Group is already used to provision access to IICS, you may skip to step 4. Your team lead would likely also know about this.
2. Grant our team at uw:org:ais:ais-admins Read and View permissions to your group.
- We will be unable to view your group or apply the automated IICS account provisioning unless you do this.
- There is [Documentation on Managing Manifest Group Permissions](https://kb.wisc.edu/25880).
3. Contact our Team so that we can apply the correct permissions to your Manifest Group.
- IICS automatically sets up accounts for any users in Manifest Groups with specified permissions, but those permissions must be set up by us.
- There may be some delay on this step, although the turnaround should be no more than 1-2 business days. Our team will contact you once this step has been completed.
- Note that gaining access to IICS involves filling out a formal request. Information on this process can be found in our [Onboarding Docs](/docs/on-boarding-to-iics.md).
5. Log in to IICS. This can easily be done from [Our Log-In Documentation](/docs/logging-in.md).
- IICS has two orgs associated with it - test and prod. It is heavily recommended that you log in to both, as to make sure your account exists within the system.
- If you intend to change org on the same browser, it is heavily recommended that you manually log out by clicking the "log out" button on the top right of the homepage or under the profile tab of the top bar on any other page.
- If you fail to log out, IICS will fail to change organizations when logging in using a different link.
6. Contact our Team so that we can apply correct permissions to you.
- If you are working alongside multiple team members it is recommended that all team members are logged in before you contact us, so we can provide permissions for everyone in one round of communication.
## Gaining Access to PersonAPI
The primary components of any integration are a Data Source (where the information you are retrieving comes from) and a Target (where you are putting the information you retrieved). For this Integration we will be using the Mock Person API as the data source.
SAM CARPENTER
committed
It is highly recommended that you pursue accessing this data source while waiting to gain access to IICS, as to minimize the amount of time spent making no progress.
SAM CARPENTER
committed
1. Sign in to the API portal and register your Application.
- Information on this process can be found at the [Person API Docs](https://developer.wisc.edu/get-started).
- The information relevant to this specific tutorial can be found in all steps up to and including "Register an Application".
- When creating your Application, Enable the Mock Person API. You will not need any other APIs for this specific tutorial.
- When the Application is created, it will also be provisioned an API key. Make sure to note down the Key and Secret from this, as they will both be used multiple times in this tutorial.
- They will most notably be used in the IICS connector to the Mock PersonAPI (which will allow you to transfer data from the Mock API into IICS).
- You must have a Wisconsin NetID in order to use the PersonAPI service.
2. Sign up for Postman to test your Application.
- Postman is a free service that allows a user to make calls to APIs from an easy-to understand interface. Information and downloads can be found at the [Postman Homepage](https://www.postman.com/).
- Note that Postman can also be used directly from a web browser, in case downloading the app is unwanted or undesirable.
- If you are having trouble understanding Postman, you can pursue their training options. They have [A Tutorial on Sending your First Request](https://learning.postman.com/docs/getting-started/first-steps/sending-the-first-request/), as well as [A More Generalized Overview of All of Their Training Services](https://learning.postman.com/docs/introduction/overview/).
3. Set up the API call and credentials within Postman.
- You will be sending a GET request to the url https://mock.api.wisc.edu/people.
- Once the basic request has been set up, click the tab below the request bar that says "authentication" and scroll down to where it says "configure new token".
- Name the token as you please.
- Set the grant type to "Client Credentials".
- The Access Token URL is https://api.wisc.edu/oauth/token.
- The Client ID is the "Key" from the API Key of your Application.
- The Client Secret is the "Secret" from the API Key of your Application.
- Leave the Scope Blank.
- Client Authentication should be "Send client credentials in body".
- Once the token has been configured, press "Get New Access Token".
- If you are testing in multiple sessions, you will need to Get a New Access Token multiple times, as the Access Token expires after a few hours.
- Note that Postman will save the information you filled out, so you will only need to press the "Get a New Access Token" button.
- To test the API, press the blue "Send" button near the URL input bar.
- A successful run will return a list of several people, each with unique associated information (most identifiably, a name).
- A 401 error will represent an error with validation - either an expired token, or invalid information used in the creation of the token.
A successful test should look something like this:
## Creating a Box Folder
Box is an online file storage solution, and is free for University of Madison-Wisconsin users. More information can be found at [The UW-Madison Information Technology Department's Box Page](https://it.wisc.edu/services/box/).
This segment of the tutorial is also able to be completed without direct access to IICS.
1. Sign up for a Box account with your UW-Madison netid
- UW-Madison's offerings for Box can be found at http://uwmadison.box.com/
- If you already have a Box account set up in UW-Madison, this step is unnecessary.
2. Create a new Folder in your Box account.
- When logging in, there will be a button labelled "New+" at the top right of the screen. Clicking this will bring up a dropdown, in which you can create a New Folder.
- More specific documentation on creating a folder can be found in [Box's Folder and File Creation Tutorial](https://support.box.com/hc/en-us/articles/360043696394-Create-New-Files-And-Folders)
3. Save the unique Folder ID of the folder you just created.
- The URL to a UW-Madison Box folder will look like https://uwmadison.app.box.com/folder/id with "id" being replaced by a long string of numbers. This string of numbers is the Folder ID. Note this down somewhere, as it will be used in creating the IICS connector (which will allow you to transfer information from IICS to your Box folder).
## Using IICS to Create Connectors
At this point you will have set up everything you need to access all relevant data for this tutorial. It is at this point where you will need to start directly accessing IICS. If you still don't have access to IICS, please return to the [First Section of this Tutorial](gaining-access-to-iics).
1. Log In to the IICS Test Organization.
- Access this documentation's [Log In Page](/docs/logging-in.md) for more information on Logging In.
- Log in specifically to the Test Org The Production Org is used to host and run live integrations, and keeping it clean is paramount.
2. Navigate from the main screen to the "Administrator" window.
- The name of the window is a bit of a misnomer; Various functions in the tab are usable by regular users, and unrelated to IICS administration.
- If you have already navigated to a different window, click the line of text in the top-left of the screen. This will bring up a similar window to the main screen, allowing you to navigate to any tab in IICS.
3. Navigate from the main page of the Administrator window to the Connections page.
- The icon for the Connection page will have a plug on it.
- 
Once you have reached this page, you will be able to create the Connections needed for this tutorial. Connections are the main asset used in IICS to determine how data is accessed from and pushed to various locations. You will be creating two Connections; a Connection to the Mock Person API to use as a data source, and a Connection to your Box folder to use as a target for your data.
### Creating The Box Connector
1. Click the green "New Connection" button in the top right. This will bring up a window with various fields to fill in.
2. Name the Connection. The owner and purpose of this connection should be easily identifiable.
- Something along the lines of "User FirstIntegrationTut Box" would fit these criteria, communicating which integration the Connection will be for as well as what data source or target it is referring to.
3. Set the "Type" field to "Box (Informatica)". This will indicate which service Informatica is interfacing with; In this instance, Box.
- Selecting this option will create many more fields that can potentially be filled in. This may look overwhelming, but only the fields indicated in the tutorial will actually be relevant in connecting to the Box folder.
4. Click the "Runtime Environment" dropdown menu and select ei.secureagent.doit.wisc.edu.
- This is the Secure Agent - The machine which the Integration will use to run the IICS Process you will create. ei.secureagent.doit.wisc.edu is available for all Integrators to use.
5. Click the "Get Token" button next to the "OAuth Access Token" field. This will bring up a new window. Follow the instructions in that window.
- The window will prompt you to log in to Box, then request Read and Write permissions. This will give IICS all necessary permissions to push files to your Box folder.
6. In the "Box File or Folder ID" Field, enter the unique Folder ID of your Box folder.
- You were previously instructed to save this field in the section on [Creating a Box Folder](#creating-a-box-folder). If you do not have it, return to this section and acquire it.
7. Save the connection by pressing the blue "Save" button in the top right. No other fields displayed are relevant to your current use-case.
A correctly set-up Box connector should look like this:
### Creating the PersonAPI Connector
1. Navigate back to the Connection page. You can do this by clicking the "X" button in the top right, or by clicking the Connections Page icon on the left sidebar.
2. Once again click the green "New Connection" button in the top right. This will allow you to create a new connection in a similar manner to the previous one.
3. Name this new connection, in a similiar fashion to the previous connection.
- In the case of the previous example of a name, you would name this Connection "User FirstIntegrationTut MockPersonAPI".
4. Set the type of the connection to "REST v2 (Informatica)".
- This will indicate that the connection is one that connects to a REST API.
- The fields created by selecting each option can differ wildly depending on what information is needed. In this instance, only fields for Runtime Environment and Authentication will be displayed initially.
5. Click the "Runtime Environment" dropdown menu and select ei.secureagent.doit.wisc.edu.
6. Click the "Authentication" dropdown menu and select "OAuth 2.0-Client Credentials".
- This will display many more fields for you to potentially fill. As before, only a fraction of these will be relevant to the current use case
- Many of these fields will contain information previously used when testing the Mock PersonAPI in Postman. Navigate to [Gaining Access to PersonAPI](#gaining-access-to-personapi) if you need information on how to access any of this information.
7. For the "Access Token URL" field, input https://api.wisc.edu/oauth/token.
- This is the specific URL that the Mock PersonAPI uses to authenticate its requests.
8. For "Client ID" and "Client Secret", input the "Key" and "Secret" from the Api Key of the Application you set up, respectively.
- This is the same set of values you used while testing the Mock Person API in Postman.
9. Set "Client Authentication" to "Send Client Credentials In Body".
10. Click the "Generate Access Token" Button next to the "Access Token" Field. If all previous credentials are set up correctly, this will fill the field automatically.
11. In the "Swagger File Path" field place the value https://git.doit.wisc.edu/interop/iics/iics-swagger-file-public-repository/-/raw/main/get+people_2_.json
- Swagger Files are used to determine what actions an API can take, and what values can be returned by an api.
12. Save the connection by pressing the blue "Save" button in the top right. No other fields displayed are relevant to your current use-case.
A correctly set-up Mock PersonAPI connector should look like this:
## Creating the Mapping
The process up to this point was in preparation of creating a Mapping, an IICS asset that directly enables the transfer of data from source to target. This type of asset is contained under the umbrella of Informatica's "Cloud Data Integration" service.
- The top left will have a drop-down labelled as "Administrator". Clicking this will bring up a window containing all IICS services available. Locate "Data Integration" and click it.
- 
- The Data Integration page has a dashboard with many pieces of information. For the purposes of this tutorial, most of the information on here can be safely ignored.
- On the left sidebar (in a similar location to the "Connections" tab within the Administrator window) there is a button labelled "New" marked with a green plus icon. This button will allow you to create new assets.
- 
- Upon clicking the "New" button, a window labelled "New Asset" will pop up.
- On the left sidebar of this window, click "Mappings", located just below the default tab of "Tasks".
- In the main window, select "Mapping".
- At the bottom right, click the blue button labelled "Create".
- DO NOT CLICK ON ANY ICONS YET.
- A mapping is made up of many Transformations (specific actions taken on the data, represented by the boxes connected together by arrows). A list of potential Transformations you can use is located on the left side, although none of these will be used in this tutorial.
- All information about the currently selected Transformation (or if no Transformation is currently selected, the Mapping as a whole) is located at the bottom of the screen.
- If there is no currently selected transformation, you will be able to edit the properties of the mapping.
- Set the name to something that identifies this mapping as both a tutorial mapping, and as YOUR tutorial mapping. To follow the previous examples of naming from this tutorial, "User FirstIntegrationTut Mapping" would suffice.
- Click the "Browse" button next to "Location". This will bring up a menu with many folders, sorted alphabetically. Locate "IntegrationTutorials" and click it. This will make sure the Mapping is located alongside the tutorials done by other users.
- If the current information on this bottom screen regards a Transformation you will need to deselect whatever Transformation you have selected. This can be done easily by deleting the Transformation, using the trash can icon near the top right.
- If you have already set the mapping information, disregard this.
- Once this is done follow the steps from above to change the information.
- Afterwards, you can drag a copy of whatever transformation you deleted from the left bar to the center.
- In the main view of the mapping, click on the "Source" Transformation.
- Turn your attention to the bottom panel. It will now display information regarding the Source Transformation.
- If you ever need to view information in this panel more closely, there are several buttons on the top right of this panel that allow you to quickly resize it.
- 
- On the left bar of the bottom panel, click on the "Source" tab.
- Next to the "Connection" property, click the dropdown menu. Select the PersonAPI Connector you previously created.
- Underneath the "Connection" property there will now be a property labelled "Operation". Press the "Select" button and select "get_people".
- Scroll down, there will be a header labelled "Request Options". Click the arrow next to this header and click the blue "Configure" text next to "Request Message".
- 
- Enter the text {}
- On the left bar, click on the "Field Mapping" tab.
- On the left side of the window, labelled "Response Structure", a single element named "root" will be available. Pressing the adjacent arrow will allow you to view more nodes of a tree of various fields.
- Alternatively, you can click the button with a plus on it right underneath the text "Response Structure". Hovering over this button will reveal that it is the "Expand All" button.
- Once a field named "data" is available, click the box next to it, and select "Map all descendants".
5. Connect the Source and the Target Transformations.
- In the main mapping view, an arrow will be visible jutting off of the side of the Source transformation. Click on this arrow, and drag it to the purple dot on the Target transformation.
- 
- A window will appear, asking you to select an output group. Select "data".
- In the main view of the mapping, click on the "Target" Transformation.
- On the left bar of the bottom panel, click on the "Target" tab.
- Next to the "Connection" property, click the dropdown menu. Select the Box Connector you previously created.
- Underneath the "Connection" property will be several other fields. Leave "Target Type" and "Operation" alone.
- Press the "Select" button next to "Object". This will bring up a new window.
- Change "Target Object" from "Existing" to "Create New at Runtime"
- Type in the name of the file as you would like it named in your Box folder. When the mapping runs, it will create a new .csv file with the same name as this setting.
- Note that you do not need to include .csv in the name. The Box connector will automatically generate the new file with a .csv extension.
- Towards the top left of the screen, there will be two buttons labelled "Save" and "Run". Click the "Save" button.
- Once the mapping has been saved, click the "Run" button.
- A dialogue window will show up. Click the "Run" button at the bottom right of this dialogue.
- A green announcement will appear at the top of the page, prompting the user to check their "My Jobs" page. Either click the blue text in this announcement, or click the "My Jobs" tab in the bar at the left of your browser window.
- The my jobs page will show various tasks that you have either recently run or are currently running. The information on the page will not automatically refresh, but the page itself has a refresh button on it.
- Once the job status is shown as "Success", you can navigate to your Box folder. There should be a new file there, with a file name based on what you set up in the Target transformation.
- The information inside this file should be similar to what you saw when testing the PersonAPI.
After confirming that the data has been transferred to your Box Folder successfully, you will be finished with this first integration. The basic principles surrounding this process should allow you to do data transfers between various sources and targets.