diff --git a/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CAI.md b/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CAI.md new file mode 100644 index 0000000000000000000000000000000000000000..7292dfb9cdc3b00607452ecf418574d06a24765a --- /dev/null +++ b/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CAI.md @@ -0,0 +1,37 @@ +# CAOS with Cloud Application Integration + +[CAOS](https://kb.wisc.edu/registrar/page.php?id=11753) (Curricular and Academic Web Services) is a SOAP API, organized primarily by curricular data. It also sits behind a firewall and is only accessible if your IP has been added to allow list. Our Informatica Secure Agents have been added to allow list, however the Informatica CAI service lives in the cloud, hosted by Informatica and cannot be. While this doesn't prevent us from consuming CAOS data from CAI, it adds a few steps to our process. + +Another thing to consider is that setting up an integration to [consume CAOS data is much easier in CDI](docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md). If you are receiving real-time updates with Webhooks or another messaging service through CAI, you can add a call to a CDI Taskflow or Mapping Task within the CAI Process. In order to do this, you will need to add the [CAI Anonymous User](https://docs.informatica.com/cloud-common-services/administrator/current-version/user-administration/users/application-integration-anonymous-user.html) to the permissions on your CDI objects. + +**We recommend that this CAI route be explored only by experienced IICS users or with close consultation with the [Integration Team](mailto:integration-platform@doit.wisc.edu).** + +## Cloud Application Integration + +The steps for setting up a Process in CAI are as follows: + +1) Create a Service Connector. + - For CAOS, you will need to download the .wsdl file instead of refer to the URL, due to the URL only being accessible if your IP is added to allow list. +2) Create an App Connection and refer it to the Service Connector created in step 1. +3) Create a Process and add a step within it that uses the App Connection that refers to the Service Connector. + + + +For more in-depth understanding of this process, please read through section 8 of the [Canvas Tutorial](https://canvas.wisc.edu/courses/378847) we created for general instruction on CAI. + +### Specific for CAOS data requests + +1) As stated above, the .wsdl will need to be in the form of a file instead of URL to create the Service Connector. +2) The Start Properties of your Process will need to have the "Run On" setting set to the Secure Agent server you typically work with. + + + +Once you have gotten this far, in order to extract the data and assign it to values for writing to a file, you will need to be familiar with XQuery. This is the language that the Expression Editor recognizes in the Assignment steps. Below is an example of taking a CAOS GetStudents response and assigning data elements. These assignments are later plugged into a File Writer step to write to a .csv. + + + +## References + +* [Canvas Course](https://canvas.wisc.edu/courses/378847) for general IICS training. Section 8 is focused on CAI. +* [CAOS User Guide](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/caos-user-guide.md) +* [Legacy CAOS Wiki](https://git.doit.wisc.edu/road/caos/legacy-wiki), migrated to preserve the old Atlassian Wiki pages, but is not as user-friendly as the User Guide. \ No newline at end of file diff --git a/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md b/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md new file mode 100644 index 0000000000000000000000000000000000000000..23e5432beb96cdb8d5d2dd0c94a241446e1e1d67 --- /dev/null +++ b/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md @@ -0,0 +1,83 @@ +# Consuming CAOS data with Cloud Data Integration + +[CAOS](https://kb.wisc.edu/registrar/page.php?id=11753) (Curricular and Academic Web Services) is a SOAP API, organized primarily by curricular data. We can retrieve a student's curricular data if we have specific student identifiers, or we can retrieve curricular data like Programs, Classes, and Subjects. It is easiest to approach consuming CAOS data with a clear idea of what elements you are seeking and the identifiers required for requesting it. + +It is important to keep in mind while making your SOAP API mapping that IICS cannot use a SOAP API as a starting Source. It's a mid-stream Web Services source only. This is understandable because we need to provide the SOAP API information in order to retrieve data (for example, giving a person's PVI or NetID to retrieve their graduation date). + +## Prerequesites + +* [Authorization](https://kb.wisc.edu/19278) and credentials for using CAOS. +* [SoapUI](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/setting-up-caos-soap-ui.md) set up for assisting with navigating CAOS to figure out the data elements you'd like and where they live within the xml responses you receive. + +## Exploring SoapUI and the CAOS XML Tree + +Once you have SoapUI installed and have CAOS added to it, we can explore the request options and their requirements for the API in a human-readable format. This is a convenient and organized structure that we highly recommend. Below we have an example of how to view the requests and requirements needed to receive a student's data from the TEST CAOS environment. + +First, we formulate a GetStudentEmplidsEnrolledInTerm. For this request, we need to provide a [term code](https://businessservices.wisc.edu/making-payments/charge-to-a-students-account/term-codes/). We are making this request to grab a random test student's emplid for a future request. + + + +From here, we can formulate a GetStudents request. In the image below, you can see that we can formulate this in many different ways. If we want specific students, we can use one of their four identifiers (emplid, pvi, netid, or campusid). If we would like to get all students from a specific program, we would need the program or plan code as well as the term code. In addition to this, we can include a [Student Data Option](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/caos-user-guide.md#getstudents-options) to provide specific information that's not provided with the base return. + + + +Once we have made a request and received a response, we can then see how the data is organized for future extraction. We will need to know where the data is located in the xml tree in order to extract it with IICS. It's also easier for us to conceptualize what we're extracting when we see it in this delivery structure. For instance, is a Program Name a full word or an acronym? Is it what we were hoping for or what systems downstream can digest? + + + +## Creating a CDI Mapping + +To consume data from a SOAP API we will need a WSConsumer connector that references the CAOS wsdl file as well as the SOAP endpoint. You will need to create this from the Administrator section of IICS, and select Connections from the left pane. + +CAOS Test wsdl: https://test.integration.services.wisc.edu/wsdl/CAOS_1.8.wsdl + +CAOS Prod wsdl: https://integration.services.wisc.edu/wsdl/CAOS_1.8.wsdl + +Test CAOS SOAP Endpoint URL: https://test.integration.services.wisc.edu/soap/CAOS_1.8 + +Prod CAOS SOAP Endpoint URL: https://integration.services.wisc.edu/soap/CAOS_1.8 + + + +Next, we need to wrap our WSConsumer connector with a Business Service. Business Services are located in the Data Integration section of IICS. Select "New..." in the upper left, then Components and Business Services. + + + +We will also need to select which operation(s) we will use this Business Service for. In this tutorial we have selected GetStudents. + + + +Since GetStudents requires information from us to send a request, we need to further configure the operation. Click the "configure..." link next to the operation and a window will pop up. Here we can select which identifier we will provide in the mapping. + + + +For our scenario, we will be requesting student data with a list of emplids. To set up our mapping, we have created a .csv in an AWS S3 bucket to use as a source. We used the GetStudentEmplidsEnrolledInTerm request with SoapUI to generate a list of emplids in term 1254. The .csv contains several random emplids from the response from the TEST CAOS API. + +Our mapping looks like this: + + + +The Source tile is our .csv with our gathered emplids. + +The second tile is an optional Expression that has defined a string to plug in the Student Data Options. Alternatively, you could have a column in your source .csv that says "ACADEMIC_OBJECTIVE" for all records. If you do not need any data from this option, you can forgo this step. If there is a different Student Data Option you need, you can replace "ACADEMIC_OBJECTIVE" with it. + + + +The third tile is the heart of this integration. This is a Web Services Transformation, set to our Business Service and its Operation we configured earlier. To configure the Request Mapping section, it is a simple drag and drop of the data element you want to plug into the required emplid field. This data element is pulled from our .csv in the Source. IICS sends one request for each record in the .csv to the CAOS API. The studentDataOptions is not required by CAOS and therefore does not need to be mapped. + + + + + +In the Response Mapping section of the Web Services transformation, we can see the entire schema for a GetStudents call with every one of the Student Data Options. Select the data elements you need and that you know are returned with your request option. If you need multiple Student Data Options, you will need multiple Web Services Transformation tiles to make additional requests to CAOS because only one option can be sent per request within IICS (but multiple options are possible with SoapUI). + + + +In the screenshot above, you will notice in the Output Fields two groups: FaultGroup and student. One contains the data elements we selected from the xml tree and the other contains data in the case we receive an error from the API. IICS requires we capture each in separate target files, therefore we have two targets in our mapping. We have these configured to deliver to the same S3 bucket as our Source, creating new files at runtime and named uniquely. If you see three Output groups, you may want to change "Relational" to "Denormalized" so that all of your returned data is sent to the same target. + + + +## References + +* [CAOS User Guide](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/caos-user-guide.md) +* [Legacy CAOS Wiki](https://git.doit.wisc.edu/road/caos/legacy-wiki) diff --git a/images/Aca_Obj.png b/images/Aca_Obj.png new file mode 100644 index 0000000000000000000000000000000000000000..5cf799f2ff71e847e1e7cc6b35dc7fb76ff61e3b Binary files /dev/null and b/images/Aca_Obj.png differ diff --git a/images/BusinessCaos.png b/images/BusinessCaos.png new file mode 100644 index 0000000000000000000000000000000000000000..d8eb82a0e5593e357944d412f6afaeddde9e7656 Binary files /dev/null and b/images/BusinessCaos.png differ diff --git a/images/BusinessServ.png b/images/BusinessServ.png new file mode 100644 index 0000000000000000000000000000000000000000..94317cd0a1adca5552fe4b1d4f243e7e735af8d6 Binary files /dev/null and b/images/BusinessServ.png differ diff --git a/images/CAI_Process.png b/images/CAI_Process.png new file mode 100644 index 0000000000000000000000000000000000000000..237f4a8b006b33921c127ec12cf4618701e32346 Binary files /dev/null and b/images/CAI_Process.png differ diff --git a/images/CAOSRestV2.png b/images/CAOSRestV2.png new file mode 100644 index 0000000000000000000000000000000000000000..711eece16eae9a91143efc8b0d7de741596fc70c Binary files /dev/null and b/images/CAOSRestV2.png differ diff --git a/images/CAOS_IICS_Tree.png b/images/CAOS_IICS_Tree.png new file mode 100644 index 0000000000000000000000000000000000000000..59d05bdd757baa00c474a9d2a2ea881f551c4673 Binary files /dev/null and b/images/CAOS_IICS_Tree.png differ diff --git a/images/CAOS_Mapping.png b/images/CAOS_Mapping.png new file mode 100644 index 0000000000000000000000000000000000000000..e37edf7fab98e8a3b565c647a3eb13dea886885f Binary files /dev/null and b/images/CAOS_Mapping.png differ diff --git a/images/CAOS_emplid.png b/images/CAOS_emplid.png new file mode 100644 index 0000000000000000000000000000000000000000..0a1a8ef576c01ed5a64237c8bc92e21e5aa4ac0a Binary files /dev/null and b/images/CAOS_emplid.png differ diff --git a/images/ExpressionCAI.png b/images/ExpressionCAI.png new file mode 100644 index 0000000000000000000000000000000000000000..54392c172ef2af56262cfc95e1657ad86a4171cc Binary files /dev/null and b/images/ExpressionCAI.png differ diff --git a/images/GetStudentEmplIDs.png b/images/GetStudentEmplIDs.png new file mode 100644 index 0000000000000000000000000000000000000000..872fb7e4685486b12c2f23a41af6fb5c1edae361 Binary files /dev/null and b/images/GetStudentEmplIDs.png differ diff --git a/images/GetStudentsFormula.png b/images/GetStudentsFormula.png new file mode 100644 index 0000000000000000000000000000000000000000..98fb1fdba41eda2729cb20f1bfed33603c670633 Binary files /dev/null and b/images/GetStudentsFormula.png differ diff --git a/images/RunOnSA.png b/images/RunOnSA.png new file mode 100644 index 0000000000000000000000000000000000000000..20f8b1c5eafcf34d430d6136494728c043e60074 Binary files /dev/null and b/images/RunOnSA.png differ diff --git a/images/StudentDataExample.png b/images/StudentDataExample.png new file mode 100644 index 0000000000000000000000000000000000000000..7ccec1e4fc45769147c2b37c421e856905cae86e Binary files /dev/null and b/images/StudentDataExample.png differ diff --git a/images/configureWebServ.png b/images/configureWebServ.png new file mode 100644 index 0000000000000000000000000000000000000000..065a284ccffc3fe4b2c6e6b67f1d18509815bd68 Binary files /dev/null and b/images/configureWebServ.png differ diff --git a/images/mapEmplid.png b/images/mapEmplid.png new file mode 100644 index 0000000000000000000000000000000000000000..301a18f1586bfdde7c8619b93e3519011d89d113 Binary files /dev/null and b/images/mapEmplid.png differ diff --git a/images/twotargets.png b/images/twotargets.png new file mode 100644 index 0000000000000000000000000000000000000000..a18e3cf71df5c3e557f37c80aae9c2cf18ccfb9c Binary files /dev/null and b/images/twotargets.png differ