diff --git a/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md b/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md index 52a4f02335e0594fa7f67964d45be59fa68a25ce..06458ef9f50a533765040bd21e98a1310bf32331 100644 --- a/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md +++ b/docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md @@ -1,8 +1,81 @@ -# UTTER CAOS +# Consuming CAOS data with Cloud Data Integration -[CAOS](https://kb.wisc.edu/registrar/page.php?id=11753) is a SOAP API, organized primarily by curricular data. We can retrieve person data if we have specific person identifiers, or we can retrieve curricular data like Programs, Classes, and Subjects. It is easiest to approach consuming CAOS data with a clear idea of what you need and identifiers for requesting it. +[CAOS](https://kb.wisc.edu/registrar/page.php?id=11753) (Curricular and Academic Web Services) is a SOAP API, organized primarily by curricular data. We can retrieve person data if we have specific person identifiers, or we can retrieve curricular data like Programs, Classes, and Subjects. It is easiest to approach consuming CAOS data with a clear idea of what you need and identifiers for requesting it. ## Prerequesites * [Authorization](https://kb.wisc.edu/19278) and credentials for using CAOS. -* [SoapUI](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/setting-up-caos-soap-ui.md) set up for assisting with navigating CAOS to figure out the data elements you'd like and where they live within the responses you receive. \ No newline at end of file +* [SoapUI](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/setting-up-caos-soap-ui.md) set up for assisting with navigating CAOS to figure out the data elements you'd like and where they live within the responses you receive. + +## Exploring SoapUI and the CAOS XML Tree + +Once you have SoapUI installed and have CAOS added to it, we can explore the request options and their requirements for the API in a human-readable format. This is a convenient and organized structure that we highly recommend. Below we have an example of how to view the requests and requirements needed to receive a student's data from the TEST CAOS environment. We did not pick a specific student, but we want to show the response structure. + +First, we formulate a GetStudentEmplidsEnrolledInTerm. For this request, we need to provide a [term code](https://businessservices.wisc.edu/making-payments/charge-to-a-students-account/term-codes/). + + + +From here, we can formulate a GetStudents request. In the image below, you can see that we can formulate this in many different ways. If we want specific students, we can use one of their four identifiers (emplid, pvi, netid, or campusid). If we would like to get all students from a specific program, we would need the program code as well as the term code. In addition to this, we can include a [Student Data Option](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/caos-user-guide.md#getstudents-options) to provide specific information that's not provided with the base return. + + + +Once we have made a request and received a response, we can then see how the data is organized for future extraction. We will need to know where the data is located in the xml tree in order to extract it with IICS. It's also easier for us to conceptualize what we're extracting when we see it in this delivery structure. For instance, is a Program Name a full word or an acronym? Is it what we were hoping for or what systems downstream can digest? + + + +## Creating a CDI Mapping + +To consume data from a SOAP API we will need a WSConsumer connector that references the CAOS wsdl file as well as the SOAP endpoint. You will need to create this from the Administrator section of IICS, and select Connections from the left pane. + +[CAOS Test wsdl](https://test.integration.services.wisc.edu/wsdl/CAOS_1.8.wsdl) + +[CAOS Prod wsdl](https://integration.services.wisc.edu/wsdl/CAOS_1.8.wsdl) + +Test CAOS SOAP Endpoint URL: https://test.integration.services.wisc.edu/soap/CAOS_1.8 + +Prod CAOS SOAP Endpoint URL: https://integration.services.wisc.edu/soap/CAOS_1.8 + + + +Next, we need to wrap our WSConsumer connector with a Business Service. Business Services are located in the Data Integration section of IICS. Select "New..." in the upper left, then Components and Business Services. + + + +We will also need to select which operation(s) we will use this Business Service for. In this tutorial we have selected GetStudents. + + + +Since GetStudents requires information from us to send a request, we need to further configure the operation. Click the "configure..." link next to the operation and a window will pop up. Here we can select which identifier we will provide in the mapping. + + + +For our scenario, we will be requesting student data with a list of emplids. To set up our mapping, we have created a .csv in an AWS S3 bucket to use as a source. We used the GetStudentEmplidsEnrolledInTerm request with SoapUI to generate a list of emplids in term 1254. The .csv contains 9 random emplids from the response from the TEST CAOS API. + +Our mapping looks like this: + + + +The Source tile is our .csv with our gathered emplids. + +The second tile is an optional Expression that has defined a string to plug in the Student Data Options. Alternatively, you could have a column in your source .csv that says "ACADEMIC_OBJECTIVE" for all records. If you do not need any data from this option, you can forgo this step. If there is a different Student Data Option you need, you can replace "ACADEMIC_OBJECTIVE" with it. + + + +The third tile is the heart of this integration. This is a Web Services Transformation, set to our Business Service and its Operation we configured earlier. To configure the Request Mapping section, it is a simple drag and drop of the data element you want to plug into the required emplid field. This data element is pulled from our .csv in the Source. IICS sends one request for each record in the .csv to the CAOS API. The studentDataOptions is not required by CAOS and therefore does not need to be mapped. + + + + + +In the Response Mapping section of the Web Services transformation, we can see the entire schema for a GetStudents call with every one of the Student Data Options. Select the data elements you need and that you know are returned with your request option. If you need multiple Student Data Options, you will need multiple Web Services Transformation tiles to make additional requests to CAOS because only one option can be sent per request. + + + +In the screenshot above, you will notice in the Output Fields two groups: FaultGroup and student. One contains the data elements we selected from the xml tree and the other contains data in the case we receive an error from the API. IICS requires we capture each in separate target files, therefore we have two targets in our mapping. In our mapping we have these configured to deliver to the same S3 bucket as our Source, creating new files at runtime and named uniquely. If you see three Output groups, you may want to change "Relational" to "Denormalized" so that all of your returned data is sent to the same target. + + + +## References + +* [CAOS User Guide](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/caos-user-guide.md) +* [Legacy CAOS Wiki](https://git.doit.wisc.edu/road/caos/legacy-wiki) diff --git a/images/Aca_Obj.png b/images/Aca_Obj.png new file mode 100644 index 0000000000000000000000000000000000000000..5cf799f2ff71e847e1e7cc6b35dc7fb76ff61e3b Binary files /dev/null and b/images/Aca_Obj.png differ diff --git a/images/BusinessCaos.png b/images/BusinessCaos.png new file mode 100644 index 0000000000000000000000000000000000000000..d8eb82a0e5593e357944d412f6afaeddde9e7656 Binary files /dev/null and b/images/BusinessCaos.png differ diff --git a/images/BusinessServ.png b/images/BusinessServ.png new file mode 100644 index 0000000000000000000000000000000000000000..94317cd0a1adca5552fe4b1d4f243e7e735af8d6 Binary files /dev/null and b/images/BusinessServ.png differ diff --git a/images/CAOSRestV2.png b/images/CAOSRestV2.png new file mode 100644 index 0000000000000000000000000000000000000000..711eece16eae9a91143efc8b0d7de741596fc70c Binary files /dev/null and b/images/CAOSRestV2.png differ diff --git a/images/CAOS_IICS_Tree.png b/images/CAOS_IICS_Tree.png new file mode 100644 index 0000000000000000000000000000000000000000..59d05bdd757baa00c474a9d2a2ea881f551c4673 Binary files /dev/null and b/images/CAOS_IICS_Tree.png differ diff --git a/images/CAOS_Mapping.png b/images/CAOS_Mapping.png new file mode 100644 index 0000000000000000000000000000000000000000..e37edf7fab98e8a3b565c647a3eb13dea886885f Binary files /dev/null and b/images/CAOS_Mapping.png differ diff --git a/images/CAOS_emplid.png b/images/CAOS_emplid.png new file mode 100644 index 0000000000000000000000000000000000000000..0a1a8ef576c01ed5a64237c8bc92e21e5aa4ac0a Binary files /dev/null and b/images/CAOS_emplid.png differ diff --git a/images/GetStudentEmplIDs.png b/images/GetStudentEmplIDs.png new file mode 100644 index 0000000000000000000000000000000000000000..872fb7e4685486b12c2f23a41af6fb5c1edae361 Binary files /dev/null and b/images/GetStudentEmplIDs.png differ diff --git a/images/GetStudentsFormula.png b/images/GetStudentsFormula.png new file mode 100644 index 0000000000000000000000000000000000000000..98fb1fdba41eda2729cb20f1bfed33603c670633 Binary files /dev/null and b/images/GetStudentsFormula.png differ diff --git a/images/StudentDataExample.png b/images/StudentDataExample.png new file mode 100644 index 0000000000000000000000000000000000000000..7ccec1e4fc45769147c2b37c421e856905cae86e Binary files /dev/null and b/images/StudentDataExample.png differ diff --git a/images/configureWebServ.png b/images/configureWebServ.png new file mode 100644 index 0000000000000000000000000000000000000000..065a284ccffc3fe4b2c6e6b67f1d18509815bd68 Binary files /dev/null and b/images/configureWebServ.png differ diff --git a/images/mapEmplid.png b/images/mapEmplid.png new file mode 100644 index 0000000000000000000000000000000000000000..301a18f1586bfdde7c8619b93e3519011d89d113 Binary files /dev/null and b/images/mapEmplid.png differ diff --git a/images/twotargets.png b/images/twotargets.png new file mode 100644 index 0000000000000000000000000000000000000000..a18e3cf71df5c3e557f37c80aae9c2cf18ccfb9c Binary files /dev/null and b/images/twotargets.png differ