API Implementation and Management Gu
API Implementation and Management Guidelines
Overview.....................................................................................................................................................4
Implementation...........................................................................................................................................5
Build........................................................................................................................................................5
Anypoint Studio...................................................................................................................................5
Template Implementation...................................................................................................................5
Best Practices and Guidelines..............................................................................................................5
Enterprise Integration Patterns with Mule..........................................................................................5
Readability and Maintainability...........................................................................................................6
Project Structure.................................................................................................................................6
HTTPS Listener.....................................................................................................................................7
Logging................................................................................................................................................8
Exception Handling..............................................................................................................................9
DataWeave........................................................................................................................................10
Object Store.......................................................................................................................................10
Test........................................................................................................................................................11
Unit Tests...........................................................................................................................................11
Integration Tests................................................................................................................................11
Performance Tuning and Load Testing..............................................................................................11
Management.............................................................................................................................................11
Security..................................................................................................................................................11
Transport Security.............................................................................................................................11
API Security........................................................................................................................................12
Data Security.....................................................................................................................................12
API Policies........................................................................................................................................12
Anypoint Audit Logs...........................................................................................................................13
Anypoint Access Management..........................................................................................................13
API Access Management...................................................................................................................13
Deployment...........................................................................................................................................14
CICD...................................................................................................................................................14
API Manager – Asset Version.................................................................................................................15
Anypoint Monitoring and Alerting.........................................................................................................16
5/24/2022 2
API Implementation and Management Guidelines
Built-in Dashboards...........................................................................................................................16
Custom Dashboards...........................................................................................................................16
Log Search.........................................................................................................................................16
API Manager Alerts............................................................................................................................16
Runtime Manager Alerts....................................................................................................................16
Monitoring Alerts..............................................................................................................................16
Reports..............................................................................................................................................17
Troubleshooting....................................................................................................................................17
5/24/2022 3
API Implementation and Management Guidelines
Overview
Each MuleSoft API includes three stages in its lifecycle: design, implementation, and management.
Figure 1 - MuleSoft API lifecycle
This document covers the implementation and management of a Mulesoft application.
5/24/2022 4
API Implementation and Management Guidelines
Implementation
Build
Anypoint Studio
Implementation of Mulesoft integrations is currently done using Anypoint Studio, an Eclipse-based IDE.
Installation and configuration instructions
Template Implementation
1. In Anypoint Studio, Select New Project from Template
2. Select the sample-api or sample-int template
3. Click Open
4. Follow the template instructions in exchange to modify the sample for your API and begin
implementation.
Best Practices and Guidelines
Enterprise Integration Patterns with Mule
Mule reduces the effort required when building integrations by implementing the patterns that you use
to design solutions. You can then simply configure and use these same patterns in Mule.1
Readability and Maintainability
When implementing Mule flows, keep in mind readability and maintainability. Use meaningful names
for flows and connectors rather than using the default names.
1
[Link]
mule#mapping-enterprise-integration-patterns-into-mule-objects
5/24/2022 5
API Implementation and Management Guidelines
Figure 2 - Example flow with meaningful names vs default
Project Structure
Parent POM and dependency management
A parent POM is used to manage the project dependencies and centrally manage the mule maven
plugins. Each project should define its dependencies, but the version for the dependencies should come
from the parent POM.
Parent POM is defined in the api-mulesoft-common project in github and can be installed in your local
maven repo for local development. The parent POM dependencies are reviewed and updated quarterly.
Project Configuration Files
Naming convention for all mule configuration files is the api/app name followed by type of configuration
([Link]. [Link], and [Link]).
Global Config
All global elements should be kept in a single global configuration file.
The global error handler should also be included with this file. All error message transformations should
return error messages in the standard error format (as defined in the raml specification).
No global elements should be included with any of the other configuration files. Only exception to this
is the router config, which should remain with the interface config so updating the api spec and
scaffolding works as expected.
5/24/2022 6
API Implementation and Management Guidelines
Interface Config
APIs will include an interface configuration file that includes the apikit/router configuration and https
listener for the API. The interface flows should only include flow refs to implementation flows and not
include any other implementation code.
Scaffolding-generated console flow and error handlers should be removed. Console is not needed with
the mocking service already available in Design Center and Exchange. The global error handler is
included in the global config.
Implementation Config(s)
All Mulesoft apps will include at least one implementation configuration file. This is where the
implementation flows are built.
For larger integrations with multiple implementation files, it makes sense to include the resource or
other logical name for the implementation files. ([Link]). Folders can also be
used to group implementation configurations logically.
For common components used across multiple implementation flows, a common implementation
configuration can be configured ([Link]).
Property Files
Create configuration files that will contain all environment properties that are needed for the app and
could potentially change per environment. Naming convention for these property files is
configuration_{env}.yaml.
Ensure the values contained in these property files will change by environment. Some examples include
autodiscovery id, client credentials, and hostnames.
Client secrets, passwords, API keys, and any other sensitive value must be encrypted in this file using the
[Link] for the specified environment. Any encrypted properties should also have their key listed in
the [Link] file as secure properties.
Any properties that do not vary by environment can be defined in a separate [Link] file.
HTTPS Listener
The listener path should be defined as /*
The header is defined to return x-correlation-id for both successful and error responses
The global config for the listener should have TLS keystore configured and the port set to $
{[Link]}
5/24/2022 7
API Implementation and Management Guidelines
Logging
Be sure any logging configured will not log sensitive or PII data.
Logger format
All loggers should begin with the following:
[Environment: ${[Link]}] [Application: #[[Link]]] [Flow: #[[Link]] [CorrelationId:
#[correlationId]] –
Logger categories
Logger categories can be used for larger integrations to help isolate logs by process or resource.
Flow loggers
Triggered flows should include starting and completed DEBUG loggers to help trace the total time the
process takes and whether it was successful or not.
This includes flows triggered by http requests for APIs as well as integration flows triggered by other
events.
Bookend loggers
Every call made externally from a Mulesoft flow should have a DEBUG logger before and after the call to
provide visibility into the time taken outside Mulesoft compared to the overall response time.
Error loggers
Only errors that need to be addressed/fixed should be logged at the ERROR level. If a service desk ticket
should be raised and worked as an ERROR, then ERROR level logging is appropriate. Otherwise, it should
be DEBUG. Client-side errors that are returned in the response to the clients are a good example of
errors that should not be logged at the ERROR level.
Non-prod message logging
In non-prod environments, there is an automated policy applied to all APIs to log the messages. This can
be enabled by following these steps:
1. Open Runtime Manager for the api in which you want to log the messages
2. Click on Settings and select the Logging tab
3. Add the following package name: [Link]-8569-<autodiscovery id> (for
QA) or [Link]-8573-<autodiscovery id> (for STG) where
<autodiscoveryId> is replaced with the value for your API in that environment. Set the logging
level to DEBUG.
4. Click apply and call your api, you will now see the message before and after the api call in the
runtime logs:
This will only work in non-prod environments by design. Any data logged should not contain PII or
sensitive information. Once you no longer need to have the message logged, set the logging level to
INFO or ERROR rather than DEBUG.
5/24/2022 8
API Implementation and Management Guidelines
Connector logging
In addition to the loggers added to flows, connector logging can also be enabled when more visibility is
needed into a connector’s internal process. This can be done locally by adding the connector’s package
to an async logger in the [Link] file. In cloudhub, add the connector’s package in the runtime
manager settings under logging.
Exception Handling
All errors and responses must match the published api specification.
Global Error Handler
The global error handler is defined in the global configuration and provides the expected handling for
the common errors defined for the apps. In many cases, this is sufficient for the error handling of the
API and should be used if flow error handlers are not doing anything more than the global error handler
already provides (set the response code and build an error response).
Flow Error Handling
Flow error handling can be implemented when the global error handling is not sufficient. Some cases
for this include retries, transaction management, notifications, and alternative processes.
Any time flow error handling is uses on a flow, an error handler without an error type should be
included.
Use the correct exception strategy for the specific use case: error continue vs propagate scopes vs try
scope vs until successful scope.
Use error mappings when calling third party APIs and define exception handlers to handle these error
codes. Proper response codes should be returned from our APIs regardless of underlying third-party
implementations.
Transaction Management
Transaction management can be used to define the steps in a flow that must succeed or fail as a single
unit.
When running in a transactional scope, the entire process must run in the same thread. This can change
the way a scope executes, or how a transaction is handled. How Transactions Affect Scopes and Routers
Until Successful / Retries
Retries should only be done for safe/idempotent processes and not for processes where duplicate
processing could occur.
The total time for all configured retries in a flow must be within the client-configured timeout. If the
client is only going to wait for a response at most 60 seconds, then a response taking longer than that
after multiple retries does not make sense even if it ultimately was successful.
HTTP response code and error responses
Proper http response codes and error messages should always be returned to client apps based on the
published raml specification.
5/24/2022 9
API Implementation and Management Guidelines
Error notifications
Error notifications can be sent when requested. This is particularly useful for async and scheduled
processes where stakeholders wish to be notified of any errors.
Notifications should include the environment and application in which the error occurred.
Large numbers of notifications should be avoided, so throttling of notifications should be included for
potentially high-volume processes.
Notifications must be sent to people that can address the issue and may optionally also include other
interested stakeholders.
Notifications should be set up to send to distribution lists rather than individuals. Business users should
only receive notifications from production except when specifically testing in QA.
DataWeave
DataWeave is the programming language designed by MuleSoft for data transformation. It is also the
expression language Mule runtime engine uses to configure components and connectors.
DataWeave enables you to build a simple solution for a common use case for integration developers:
read and parse data from one format, transform the data, and write it out as a different format. 2
DataWeave Quickstart
DataWeave Language Guide
DataWeave Functions Reference
DataWeave Cookbook
DataWeave Playground
Object Store
Object Store can be used for simple key-value storage. It is used to support Mule Runtime components
including the Cache and Oauth modules and can also be used for storing watermarks and access tokens.
Object Store V2 is enabled by default, however it should not be used for multi worker and high
concurrency processes. It can be disabled by overriding the Boolean flag from the parent pom in the
project pom.
Different Types of Object Stores Explained
2
[Link]
5/24/2022 10
API Implementation and Management Guidelines
Test
Unit Tests
MUnit tests should fully cover all implementation flows. Insufficient coverage or test failures will break
the build in the Github pipeline which deploys changes into the STG environment.
MUnit tests for APIs should assert that the response matches what was defined in the RAML
specification. The raml datatype to json schema utility can convert a raml datatype into a JSON schema
which can then be used to assert a json response is valid.
A Good MUnit Test Has These 5 Parts
Helpful Tools for Writing MUnit Tests
Integration Tests
Unlike unit tests, integration tests should make connections from the application and verify all the
pieces are working together as expected.
Integration testing must be done in the QA environment for final verification. You may also do some
developer integration testing in STG.
End-to-end testing of APIs and integrations with source systems and target systems should also be done
in the QA environment.
Postman Shared Workspace
A shared workspace for Mulesoft is used to collaborate test API calls amongst the development team
and QA testers. You can request access to Postman via Okta if you do not see it on your dashboard.
The collections in the workspace are set up to easily change between test environments and use the
MuleSoft development team credentials to call the APIs.
Production calls should never be set up or made from the shared workspace. Only production apps
should call production APIs. All testing should be done in the non-prod environments.
Newman
Newman can be used to automate running your postman tests from the CICD pipeline.
Performance Tuning and Load Testing
The purpose of load testing and performance tuning applications is to ensure they are sized and tuned
properly prior to going to production. This testing should be done in the QA environment and the
number of requests, worker count, and worker size should be as close to prod as possible. Anypoint
monitoring provides built-in dashboards that can be monitored to identify any issues.
Management
Security
Transport Security
All Incoming traffic to Mule APIs will be going through Cloudhub Dedicated Load Balancers. Traffic from
client applications to the dedicated load balancer is secured with one way TLS and SSL offloading
5/24/2022 11
API Implementation and Management Guidelines
happens at the dedicated load balancer. The dedicated load balancers establish new SSL connections
with the cloudhub workers running the apps.
Traffic from the cloudhub VPC to Werner’s internal network is secured over a VPN. Outbound traffic
from cloudhub should use secure transport.
API Security
All APIs deployed to mule runtimes have to be linked to API manager using the API id and auto discovery
configuration. This associates the API deployed to the runtime with the policies, contracts, and SLAs
defined and managed in the control plane (API Manager).
Currently, all APIs include client id enforcement using client ids and secrets generated in Anypoint
platform. Recommendation is that all external facing APIs be secured with either JWT validation policy
or Open ID Connect OAuth 2.0 token enforcement policy.
Data Security
Mulesoft runtimes are stateless and data is secure as it runs through the system.
Developers should avoid logging payload data as that breaks the stateless philosophy and care should be
taken before logging and data that is sensitive or may contain PII data.
API Policies
Table 1- Overview of security policies recommended to support the security requirements
Level Good for API security policies
Tier1 Experience APIs, Client ID enforcement
Process APIs and
System APIs
Tier2 System APIs, Client ID enforcement
Process APIs IP Whitelist
JWT validation (optional) or
OpenID Connect OAuth 2.0 Token Enforcement
Policy (optional)
Tier3 Experience APIs CORS Policy
Client ID enforcement
JWT validation policy or
OpenID Connect OAuth 2.0 Token Enforcement
Policy
There are wide range of other policies that can be applied on APIs. For details on more out of the box
policies refer to the Included Policies Directory.
Client Id Enforcement Policy
Required for all APIs. Requires the Anypoint Platform supplied client id and password for the client
application be passed in the header.
5/24/2022 12
API Implementation and Management Guidelines
In some cases, external partners have only been able to support basic authentication. If Basic Auth is
used in place of client id enforcement, the RAML should reflect this and the username and password for
basic auth should be set to the client id and secret from Anypoint platform.
IP Whitelist Policy
Can be used to restrict callers to a subset of IP addresses. Should be applied to system and process APIs
to ensure only experience APIs within the VPC can call them.
Anypoint Audit Logs
All changes made by users and connected apps within the Anypoint org are logged and available via
audit logs.
Anypoint Access Management
Access to the Anypoint platform for internal users is managed using Okta and SSO. All IT associates and
contractors should see the MuleSoft tile on their SSO dashboard.
Expanded access is provided to developers via an Okta group upon completion of fundamentals training
and certification.
Exchange assets that need to be shared with external partners can be shared via the public portal or
external users can be invited into the Werner Anypoint org. Unlike internal users, external users will not
see all the available API specifications in Exchange. Each asset needs to be shared to these users
specifically in Exchange. Reach out to an org admin for assistance to set up a external users.
MFA is required for all users not coming through SSO.
API Access Management
Access to APIs is managed with contracts to client applications. A client application is this context simply
means an app that wants to call an API. This may be an external (to MuleSoft) application or another
API or integration app.
Existing contracts for an API can be viewed in API manager.
Access to an API is requested in Exchange.
If the client application already exists and you are not an owner, an org admin can add you as an owner
so it appears in the drop down of applications when requesting access and under My Applications in
Exchange.
If the application does not already exist, you can create a new application when requesting access. Non-
prod and prod should always have separate client applications and the name of the app should indicate
whether it is for prod or non-prod.
If you are unsure if a client application already exists, you can use the anypoint-sapi to retrieve a list of
all existing client apps or reach out to an org admin.
Requests for non-prod are usually granted the access automatically.
5/24/2022 13
API Implementation and Management Guidelines
If an SLA tier is applied on an API and approval type was configured as Manual, an email will be sent to
the organization administrator to approve the request. Manual approval is set for all APIs in PROD
environments.
Deployment
CICD
The following are the sequence of the events needed to move code through environments and deploy
into production:
1. Developers clone the github repo locally. Once the code is updated and tested locally, the
changes are pushed into the remote github develop branch.
2. Github STG pipeline monitors the code repository for commits to the develop branch. Maven
builds and runs the munit tests for the project for any committed change.
3. If builds and tests succeed and munit coverage sufficient, the code is deployed into STG
environment.
4. Once developer testing is complete in STG and changes are ready to move into QA for
integration and acceptance testing, a pull request is initiated in github from develop branch into
main.
5. The pull request needs to be peer reviewed and approved by two other developers. The
implementation peer review checklist should be used as a guide for the developer and peer
reviewers.
6. Once approved, the code can be merged into the main branch which will trigger the release
pipeline action automatically.
7. The release action builds a release candidate using the maven plugin. If the build is successful,
the code repo gets tagged and the artifact is stored in Github. The release candidate is also then
automatically deployed into QA.
8. Automated integration tests may be executed following a successful deployment into QA.
9. Integration, functional, regression, acceptance, load testing, etc should now be completed in the
QA environment. The prod deploy step is left pending and awaiting final manager approval
following testing and verification.
10. Once the changes are deemed ready to be deployed into production, a CAB is required for all
production deployments. Once CAB is approved, a manager from the release managers team in
github can approve the prod deploy step in github. This will deploy the release candidate into
production and create a release tag in github.
5/24/2022 14
API Implementation and Management Guidelines
Figure 3 - MuleSoft CICD Pipeline
API Manager – Asset Version
Following deployment of the jar into an environment, an automated process will update the asset
version in API manager to match the version from the deployed jar. This will ensure that Exchange
always shows the API version that is running in each environment. It is important that the artifact
version on APIs properly reflect the raml version it is implementing.
5/24/2022 15
API Implementation and Management Guidelines
Anypoint Monitoring and Alerting
Built-in Dashboards
API Analytics Dashboard provides a view of the requests to an API, response status and times, failures,
performance, and client applications.
Built-in application dashboards provide information on the deployed application including jvm metrics,
inbound and outbound request info, connector performance, and more.
Custom Dashboards
Custom dashboards can be used to provide a view across multiple APIs or a specific process.
The MuleSoft Dashboard is a custom dashboard for a view across all deployed APIs and apps at Werner.
Log Search
Anypoint monitoring includes a Log Search feature where logs are aggregated across all apps and
environments. Predefined filters are provided such as application, environment, log level, worker and
more.
When viewing logs for a deployed cloudhub application, this is a preferred approach over viewing them
in runtime manager. These logs are viewable and sharable to all anypoint users in the org (which
includes all of IT), unlike runtime manager logs which are limited to the mulesoft development team.
Runtime manager logs are also limited in size so will not be as complete as the logs found in log search.
API Manager Alerts
In API Manager, alerts for request count, response codes, policy violations, and response times can be
set up for an API. Response code 503 alerts and client id enforcement policy violation alerts are set up
on most of the production APIs.
Runtime Manager Alerts
Operational alerts can be set up in runtime manager for monitoring the health of the servers.
Alerts currently set up for all cloudhub apps include: CPU and memory usage, deployment status, and
workers not responding.
Monitoring Alerts
Basic Alerts trigger email notifications when a measured metric passes a specific threshold.
Advanced Alerts are added and configured in the Alert Tab of any graph within Custom dashboards,
letting you build and visualize an alert using existing queries.
5/24/2022 16
API Implementation and Management Guidelines
Figure 4 - An advanced alert set up to trigger when the number of failures is greater than 100 in a 5 minute period. When
triggered, an email is sent to the app team letting them know.
Reports
Reports give a view of overall performance of applications running in an environment. Available reports
include Requests, Performance, Failures, CPU Utilization, and Memory Utilization.
Figure 5 - Sample Reqeusts Report
Troubleshooting
The MuleSoft Knowledge Base in confluence includes troubleshooting articles.
5/24/2022 17