Skip to content
Snippets Groups Projects
Commit 1bad2cdb authored by Nuwan Rajika Kumarasiri's avatar Nuwan Rajika Kumarasiri
Browse files

Add Box connector tutorial - INPLATFORM-229

parent a1906748
No related branches found
No related tags found
No related merge requests found
Showing
with 127 additions and 10 deletions
# Using AWS S3 connector as a flat file connector Alternative
It is inevitable that certain integrations will involve using flat files for import or export. However, when applicable, we recommend using AWS S3 as an alternative option.
AWS S3 is an excellent option for when you do not have easy access to the Secure Agent filesystem.
Both AWS S3 and Box are excellent options for when you do not have easy access to the Secure Agent filesystem.
## Issues with flat file connector
......@@ -34,21 +31,36 @@ Once the verification is successful, the next transformation can be designed and
This method can also be used for testing changes that need to be made on an ETL process.
Parts that need to be changed can be duplicated and verified in an S3 bucket.
## WiscAlerts example
### WiscAlerts example
The [WiscAlerts example](../tutorials/wiscalerts/wiscalerts.md) describes a successful use of AWS S3 connector as a flat file connector alternative.
The Integration Platform team has implemented the integration using AWS S3 instead of relying on flat files.
The Integration Platform team has implemented the integration using AWS S3 instead of relying on file system access for flat files.
In this integration, the team has created an AWS S3 connection as a target location for the different files that were required for the integration.
The team then created Mapping tasks that gathers information from Oracle Data sources using an Oracle connector and puts it in AWS S3 as a csv file.
The WiscAlerts example linked above describes the integration in finer detail and outlines the benefits and gaps of using AWS S3.
## Using Box as an Alternative
Box is a service that can be used to store files in a secure manner. See this [guide](https://kb.wisc.edu/33229) on how to
get started on using Box. IICS provides a Box connector that can be used as an alternative to using flat files.
There is a [tutorial](../tutorials/box/box.md), on how to set up Box connector for moving flat files around.
## When to use AWS S3 vs Box
While both AWS S3 and Box are excellent alternatives instead of relying on file system access for flat file operations,
following are few things you can consider when choosing between AWS S3 vs Box.
* Box is suitable for sensitive information (however it's not suitable for restricted data, see Box [terms of use](https://kb.wisc.edu/page.php?id=33978)).
* Box has a very small learning curve compared to AWS S3 and can be set up quickly. On the other hand AWS
S3 (and how it fits in AWS system) has a learning curve.
* If you are already familiar with AWS S3 and already using it, then AWS S3 may be a better alternative.
* If you are using other AWS services, then AWS S3 will best fit in with those.
## Training and Experimentation
For training and experimentation, flat files are an easy and effective way to learn IICS. When possible, you should use your local machine during training.
If using a hosted Secure Agent for training purposes, then we recommend that you use AWS S3.
If using a hosted Secure Agent for training purposes, then we recommend that you use IICS Box connector.
## Recommendation for using flat file connector
If there are limitations to effectively using AWS S3 instead of flat files. We recommend that you host a Secure Agent on a machine that you can manage.
\ No newline at end of file
If there are limitations to effectively using AWS S3 or Box instead of flat files, we recommend that you host a Secure Agent on a machine that you manage.
\ No newline at end of file
......@@ -22,7 +22,7 @@ See also Informatica [documentation](https://docs.informatica.com/integration-cl
## How to create a Manifest group for your team for access control?
[manifest-group]: #how-to-create-a-manifest-group-for-your-team-for-access-control
DoIT EI uses Manifest groups to give teams access to the UW Madison IICS environment.
DoIT EI uses Manifest groups to give teams access to the UW-Madison IICS environment.
**Provisioning a team**:
......@@ -39,7 +39,7 @@ DoIT EI uses Manifest groups to give teams access to the UW Madison IICS environ
DoIT EI will notify you when we have provided necessary access to the Manifest group. We will also provide you with an IICS User Group name which you will use to restrict access to your team's assets in IICS:
[Use user groups to control access](best-practices.md#use-user-groups-to-control-access)
* [ ] Each member of the Manifest group will need to log in to each IICS environment (e.g. test and production) using their UW Madison NetId, which will create an IICS user account for them automatically.
* [ ] Each member of the Manifest group will need to log in to each IICS environment (e.g. test and production) using their UW-Madison NetId, which will create an IICS user account for them automatically.
* [ ] Please let DoIT EI know when all group members have logged in once. We will then assign the IICS user accounts to the IICS User Group that was provided to you.
......
......@@ -2,3 +2,4 @@
Various integrations developed using IICS.
* [WiscAlerts Integrations](./tutorials/wiscalerts/wiscalerts.md)
* [Using Box Connector For Flat Files](./tutorials/box/box.md)
\ No newline at end of file
# Using Box Connector For Flat Files
[Box](https://uwmadison.app.box.com/) is a secured storage service offered by UW-Madison that can be used to store documents securely.
This tutorial walks through setting up the Box connector for moving a file from a source folder to a destination folder in Box.
The below diagram describes the scenario. In this scenario, `employee.csv` CSV file from the source folder is copied
to a target CSV file called, `employee.csv` inside a target folder. Note that any other type of connection (such as S3, database etc.) can be
used interchangeably with the Box connector.
![Using Box connector to move a file.](images/BoxConnectorScenario.svg)
The diagram source can be found [here](https://app.lucidchart.com/documents/edit/77f36467-4d01-4ed8-802a-c2ec3d97738e/0_0?beaconFlowId=26DFD7EF970567C0).
## Box Set Up
* Create two [Box](https://uwmadison.app.box.com/) folders, one as `source` folder and the other as the `target` folder.
* `source` folder contains the file `employee.csv`.
![Source folder.](images/source.png)
* `employee.csv` file looks like below.
```
FN, LN, Address, City, State, Zip
John,Doe,120 jefferson st.,Riverside, NJ, 08075
Jack,McGinnis,220 hobo Av.,Phila, PA,09119
Stephen,Tyler,"7452 Terrace ""At the Plaza"" road",SomeTown,SD, 91234
```
* `target` folder contains the file `employee.csv`. It just contains the header row. The Synchronization task will
populate the `employee.csv` file from source folder.
![!Target folder.](images/target.png)
* Content looks like below (only contains the header of the CSV).
```
FN, LN, Address, City, State, Zip
```
* `employee.csv` in target folder contains only header so that IICS can map source file header to target file header.
## IICS Set Up
### Box Source Connector
* Login to IICS.
* Choose Administrator in the top menu dropdown.
* Choose Connections in the left menu.
* Click on the New Connection button towards the top right of the window. This connection will fetch the `employee.csv` file
from `source` folder (see scenario diagram).
![!Add new connection.](images/connections.png)
* Select `Box` connector type for a Box connection. Fill `Connection Details`.
* For `OAuth Access Token:` click the `Get Token` button, and a pop will be appeared for you to authenticate with Box.
* For `URI Request Parameters:` provide `query=employee.csv`. This helps Box connector to narrow down the search for given file.
See reference at the end for more details.
* Inside Box select `employee.csv` file in each of `source` and `target` folder to get `Box File ID` value.
* After configuring a connection, save it and then click on the Test Connection button to verify that your connection works.
![!Add new Box source connection.](images/source-connection.png)
### Box Target Connector
* Repeat above steps and add a second `Box` connection. This connection will update the `employee.csv` file in `target`
folder (see scenario diagram).
* As noted before this also can be a replaced by any other type of connection (e.g. database, AWS S3 etc).
![!Add new Box target connection.](images/target-connection.png)
### Synchronization Task
* Create a Synchronization Task in Data Integration. Provide a name and select `insert` as the task operation.
![!Synchronization task name.](images/syn-task-1.png)
* Provide source information. Select source connection that was defined previously. Select `Single` as the source type.
Select `employee` as the `Source Object`.
![!Synchronization task source.](images/syn-task-2.png)
* Provide target information. Select target connection that was defined previously. Select `employee` as `Target Object`.
![!Synchronization task target.](images/syn-task-3.png)
* Keep the default values for `Data Filters`.
![!Synchronization task filters.](images/syn-task-4.png)
* Keep the default values for `Field Mappings`.
![!Synchronization task mapping.](images/syn-task-5.png)
* Keep the default values for `Schedule`.
![!Synchronization task schedule.](images/syn-task-6.png)
* Use the `Run` button to execute the Synchronization Task.
After successfully executing the task, `employee.csv` file inside `target` folder will contain the content from `employee.csv` file from source folder.
## Summary
This tutorial walks through how to set up Box connector to copy content from Box source CSV file to a Box target CSV file.
## References
* UW-Madison Box KB documents
* https://kb.wisc.edu/box/page.php?id=80045
* https://kb.wisc.edu/26278
* [Data classification](https://www.wisconsin.edu/uw-policies/uw-system-administrative-policies/information-security-data-classification-and-protection/information-security-data-classification/)
* [IICS Box connector documentation](https://docs.informatica.com/es_es/integration-cloud/cloud-data-integration-connectors/current-version/box-connector-guide/introduction-to-box-connector.html)
* [Box search API](https://developer.box.com/reference/get-search/)
\ No newline at end of file
This diff is collapsed.
docs/tutorials/box/images/connections.png

223 KiB

docs/tutorials/box/images/source-connection.png

200 KiB

docs/tutorials/box/images/source.png

102 KiB

docs/tutorials/box/images/syn-task-1.png

158 KiB

docs/tutorials/box/images/syn-task-2.png

185 KiB

docs/tutorials/box/images/syn-task-3.png

143 KiB

docs/tutorials/box/images/syn-task-4.png

127 KiB

docs/tutorials/box/images/syn-task-5.png

206 KiB

docs/tutorials/box/images/syn-task-6.png

400 KiB

docs/tutorials/box/images/target-connection.png

204 KiB

docs/tutorials/box/images/target.png

106 KiB

0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment