Merge branch 'CAOS_tutorial' into 'master'

CAOS_consumption_Tutorials

See merge request !83

docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CAI.md

+# CAOS with Cloud Application Integration
+
+[CAOS](https://kb.wisc.edu/registrar/page.php?id=11753) (Curricular and Academic Web Services) is a SOAP API, organized primarily by curricular data. It also sits behind a firewall and is only accessible if your IP has been added to allow list. Our Informatica Secure Agents have been added to allow list, however the Informatica CAI service lives in the cloud, hosted by Informatica and cannot be. While this doesn't prevent us from consuming CAOS data from CAI, it adds a few steps to our process.
+
+Another thing to consider is that setting up an integration to [consume CAOS data is much easier in CDI](docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md). If you are receiving real-time updates with Webhooks or another messaging service through CAI, you can add a call to a CDI Taskflow or Mapping Task within the CAI Process. In order to do this, you will need to add the [CAI Anonymous User](https://docs.informatica.com/cloud-common-services/administrator/current-version/user-administration/users/application-integration-anonymous-user.html) to the permissions on your CDI objects.
+
+**We recommend that this CAI route be explored only by experienced IICS users or with close consultation with the [Integration Team](mailto:integration-platform@doit.wisc.edu).**
+
+## Cloud Application Integration
+
+The steps for setting up a Process in CAI are as follows:
+
+1) Create a Service Connector.
+    - For CAOS, you will need to download the .wsdl file instead of refer to the URL, due to the URL only being accessible if your IP is added to allow list.
+2) Create an App Connection and refer it to the Service Connector created in step 1.
+3) Create a Process and add a step within it that uses the App Connection that refers to the Service Connector.
+
+![CAI_Process](/images/CAI_Process.png)
+
+For more in-depth understanding of this process, please read through section 8 of the [Canvas Tutorial](https://canvas.wisc.edu/courses/378847) we created for general instruction on CAI.
+
+### Specific for CAOS data requests
+
+1) As stated above, the .wsdl will need to be in the form of a file instead of URL to create the Service Connector.
+2) The Start Properties of your Process will need to have the "Run On" setting set to the Secure Agent server you typically work with.
+
+![RunOnSA](/images/RunOnSA.png)
+
+Once you have gotten this far, in order to extract the data and assign it to values for writing to a file, you will need to be familiar with XQuery. This is the language that the Expression Editor recognizes in the Assignment steps. Below is an example of taking a CAOS GetStudents response and assigning data elements. These assignments are later plugged into a File Writer step to write to a .csv.
+
+![ExpressionCAI](/images/ExpressionCAI.png)
+
+## References
+
+* [Canvas Course](https://canvas.wisc.edu/courses/378847) for general IICS training. Section 8 is focused on CAI.
+* [CAOS User Guide](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/caos-user-guide.md)
+* [Legacy CAOS Wiki](https://git.doit.wisc.edu/road/caos/legacy-wiki), migrated to preserve the old Atlassian Wiki pages, but is not as user-friendly as the User Guide.
\ No newline at end of file

docs/tutorials/API_Related/CAOS_SOAP_API/CAOS_with_CDI.md

+# Consuming CAOS data with Cloud Data Integration
+
+[CAOS](https://kb.wisc.edu/registrar/page.php?id=11753) (Curricular and Academic Web Services) is a SOAP API, organized primarily by curricular data. We can retrieve a student's curricular data if we have specific student identifiers, or we can retrieve curricular data like Programs, Classes, and Subjects. It is easiest to approach consuming CAOS data with a clear idea of what elements you are seeking and the identifiers required for requesting it.
+
+It is important to keep in mind while making your SOAP API mapping that IICS cannot use a SOAP API as a starting Source. It's a mid-stream Web Services source only. This is understandable because we need to provide the SOAP API information in order to retrieve data (for example, giving a person's PVI or NetID to retrieve their graduation date).
+
+## Prerequesites 
+
+* [Authorization](https://kb.wisc.edu/19278) and credentials for using CAOS.
+* [SoapUI](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/setting-up-caos-soap-ui.md) set up for assisting with navigating CAOS to figure out the data elements you'd like and where they live within the xml responses you receive.
+
+## Exploring SoapUI and the CAOS XML Tree
+
+Once you have SoapUI installed and have CAOS added to it, we can explore the request options and their requirements for the API in a human-readable format. This is a convenient and organized structure that we highly recommend. Below we have an example of how to view the requests and requirements needed to receive a student's data from the TEST CAOS environment. 
+
+First, we formulate a GetStudentEmplidsEnrolledInTerm. For this request, we need to provide a [term code](https://businessservices.wisc.edu/making-payments/charge-to-a-students-account/term-codes/). We are making this request to grab a random test student's emplid for a future request. 
+
+![GetStudentEmplIDs](images/GetStudentEmplIDs.png)
+
+From here, we can formulate a GetStudents request. In the image below, you can see that we can formulate this in many different ways. If we want specific students, we can use one of their four identifiers (emplid, pvi, netid, or campusid). If we would like to get all students from a specific program, we would need the program or plan code as well as the term code. In addition to this, we can include a [Student Data Option](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/caos-user-guide.md#getstudents-options) to provide specific information that's not provided with the base return.
+
+![GetStudentsFormula](images/GetStudentsFormula.png)
+
+Once we have made a request and received a response, we can then see how the data is organized for future extraction. We will need to know where the data is located in the xml tree in order to extract it with IICS. It's also easier for us to conceptualize what we're extracting when we see it in this delivery structure. For instance, is a Program Name a full word or an acronym? Is it what we were hoping for or what systems downstream can digest?
+
+![StudentDataExample](images/StudentDataExample.png)
+
+## Creating a CDI Mapping
+
+To consume data from a SOAP API we will need a WSConsumer connector that references the CAOS wsdl file as well as the SOAP endpoint. You will need to create this from the Administrator section of IICS, and select Connections from the left pane.
+
+CAOS Test wsdl:  https://test.integration.services.wisc.edu/wsdl/CAOS_1.8.wsdl
+
+CAOS Prod wsdl:  https://integration.services.wisc.edu/wsdl/CAOS_1.8.wsdl
+
+Test CAOS SOAP Endpoint URL:  https://test.integration.services.wisc.edu/soap/CAOS_1.8
+
+Prod CAOS SOAP Endpoint URL:  https://integration.services.wisc.edu/soap/CAOS_1.8
+
+![REST_V2_CAOS](images/CAOSRestV2.png)
+
+Next, we need to wrap our WSConsumer connector with a Business Service. Business Services are located in the Data Integration section of IICS. Select "New..." in the upper left, then Components and Business Services.
+
+![BusServ](images/BusinessServ.png)
+
+We will also need to select which operation(s) we will use this Business Service for. In this tutorial we have selected GetStudents. 
+
+![CAOS_BS](images/BusinessCaos.png)
+
+Since GetStudents requires information from us to send a request, we need to further configure the operation. Click the "configure..." link next to the operation and a window will pop up. Here we can select which identifier we will provide in the mapping.
+
+![CAOS_emplid](images/CAOS_emplid.png)
+
+For our scenario, we will be requesting student data with a list of emplids. To set up our mapping, we have created a .csv in an AWS S3 bucket to use as a source. We used the GetStudentEmplidsEnrolledInTerm request with SoapUI to generate a list of emplids in term 1254. The .csv contains several random emplids from the response from the TEST CAOS API.
+
+Our mapping looks like this:
+
+![CAOS_Mapping](images/CAOS_Mapping.png)
+
+The Source tile is our .csv with our gathered emplids. 
+
+The second tile is an optional Expression that has defined a string to plug in the Student Data Options. Alternatively, you could have a column in your source .csv that says "ACADEMIC_OBJECTIVE" for all records. If you do not need any data from this option, you can forgo this step. If there is a different Student Data Option you need, you can replace "ACADEMIC_OBJECTIVE" with it. 
+
+![Aca_Obj](images/Aca_Obj.png)
+
+The third tile is the heart of this integration. This is a Web Services Transformation, set to our Business Service and its Operation we configured earlier. To configure the Request Mapping section, it is a simple drag and drop of the data element you want to plug into the required emplid field. This data element is pulled from our .csv in the Source. IICS sends one request for each record in the .csv to the CAOS API. The studentDataOptions is not required by CAOS and therefore does not need to be mapped.
+
+![configureWebServ](images/configureWebServ.png)
+
+![mapEmplid](images/mapEmplid.png)
+
+In the Response Mapping section of the Web Services transformation, we can see the entire schema for a GetStudents call with every one of the Student Data Options. Select the data elements you need and that you know are returned with your request option. If you need multiple Student Data Options, you will need multiple Web Services Transformation tiles to make additional requests to CAOS because only one option can be sent per request within IICS (but multiple options are possible with SoapUI). 
+
+![CAOS_Tree](images/CAOS_IICS_Tree.png)
+
+In the screenshot above, you will notice in the Output Fields two groups: FaultGroup and student. One contains the data elements we selected from the xml tree and the other contains data in the case we receive an error from the API. IICS requires we capture each in separate target files, therefore we have two targets in our mapping. We have these configured to deliver to the same S3 bucket as our Source, creating new files at runtime and named uniquely. If you see three Output groups, you may want to change "Relational" to "Denormalized" so that all of your returned data is sent to the same target.
+
+![TwoTargets](images/twotargets.png)
+
+## References
+
+* [CAOS User Guide](https://git.doit.wisc.edu/road/documentation/-/blob/main/caos/caos-user-guide.md)
+* [Legacy CAOS Wiki](https://git.doit.wisc.edu/road/caos/legacy-wiki)