MCOP Micro services Proposal
Overview
MCOP is an Ordering platform aimed at enabling use cases where multiple business entities are involved
in order fulfilment. MCOP makes the global selling, global listing and NAFN use cases possible as it allows
Amazon entities to share responsibilities when fulfilling customer orders. MCOP currently handles
integration with COW, CROW, PCE and RAF for order creation and fulfillment process.
This document details out concerns with monolithic nature of current MCOP system, various problems
and limitations due to it and proposes solutions to resolve the same.
Current Architecture
As of today, MCOP mainly comes into existence once a customer order is created. Its current
responsibility includes creating the secondary order/s, syncing order updates, chaining fulfillment
events, posting accounting entries and handling reversals.
MCOP application is currently hosted among 3 services: MultiCompanyOrderingService, MCFNSilService
and OrderUpdateNotificationService. Among them, OUNS hosts 1 APIs which is CASE plugin to capture
Order Update Events. MCFN consists of 7 APIs which support partial handling of shipment chaining.
MCOS has most monolithic nature with 40+ APIs across functional areas.
PCE, COW, CROW and RAF process the customer orders(reversals) by invoking MCOP APIs at various
hook points. MCOP, in turn, handles different businesses by onboarding corresponding plugins to its
system.
1
[Link] confidential
Limitations
Following are major limitations of current monolithic architecture:
- Delayed business launches: Current architecture of MCOP results in changes across multiple
modules to be deployed simultaneously. This increases the probability of changes in one MCOP
module impacting another. In last 6 months, we have seen 5 instances among 20 deployments
where multiple business launches got delayed because of the bugs introduced by changes in
unrelated modules. Entire deployment needed to be rolled back and caused delays in launch. On an
average with decentralized design, we could have saved 20 days of delays in business launches.
Deployment Issue Effected Delayed Modules Delay
Module
09/13 Deployment Bug in changes to Accounting Reversal 1 week
COGS ARH Chaining,
Platform
refactoring for
metrics
03/08 Deployment Refactoring NAFN NAFN Purchase 1.5 weeks
logic migration to Creation,
NafnOrderService Reversal Chaining
05/07 Deployment Bug in CROW ARH Reversal Shipment 1.5 weeks
while migrating Chaining
mapping logic changes, Prime
2
[Link] confidential
day JMX activity
04/05 Deployment GetMappings API Tooling Purchase 1 week
changes to exception Creation
handling effecting 1 changes,
external client. Shipment
Chaining Changes
- Increased development burden: MCOP code base has currently 1000+ files and over 68000 LoC. The
package takes significant time to load on IDE and we recently had issues while running the test cases
as the class path is too large to load. Due to bloated dependencies, the package is encountering Fat
Jar issues. Complete build takes around 9 min on developer desktop. By splitting and refactoring
appropriately, we can bring down individual build times to around 4 min which results in saving 5
Developer Days per year. (5 min saving * 3 builds/day * 6 developers)
Generally, feature requests correspond to different logical areas and code base is quite orthogonal
inside the monolithic package. There are some common utilities class which serve across board but
they are pretty much stable now. This is a good indication to support splitting.
- Separation of concerns: Although 3 distinct services exist, the responsibility structure is not well
defined. There is no clear segregation of APIs. MultiCompanyOrderingService has grown extensively,
compassing all major logic and catering over 40 APIs. OrderUpdateNotificationService &
MCFNSilService consist few APIs without well-defined responsibility. Changes related to Shipment
chaining often needs coordinated deployment between MCOS and MCFN since the functionality is
spread among them. These services incur a minimum hosting IMR without much benefit e.g. OUNS
has an IMR cost of 1.6K per month (minimum fleet size with redundancy) with 1 API and CPU
utilization of 3-5%
- Scaling Concerns: Currently all APIs serving MCOP use cases are hosted on single service. MCOP APIs
have different resource utilization requirements & traffic surge predictions. Current model requires
MCOP service to scale up assuming the worst utilization impact which is inefficient. For
consideration, CreateShadowOrder API takes 2700ms, GetMCOPOrderInfo takes 800ms,
SendChainingStatus takes 450ms, CheckLastTouchForShipment takes 8ms. These API operate in
distinct functional areas but peak scaling will be performed based on worst utilization impact.
Efficient monitoring for monolithic application like MCOS is quite difficult due to varying range of
metrics and thresholds. Currently, the Log Scan monitor is in alarm almost 90% of the time and does
not serve as efficient indicator. These pose a challenge to Continuous deployment.
Proposal
It has been observed that most of business launches and platform enhancements requires changes in
partial set of business functions/capabilities. Deployment will usually consist of multiple such changes in
3
[Link] confidential
orthogonal areas. The below table consolidates this idea by mentioning the recent business features
launched in respective areas.
Deployment Feature Module
Checksum Verifier Purchase Creation & Updates
06/13 Deployment
Promise Sync Changes Purchase Creation & Updates
PFW Handling for 18 Purchase Creation & Updates
06/14 Deployment
BO Charge Override Changes Shipment Chaining
Checksum Verifier Enforcement Purchase Creation & Updates
06/21 Deployment New Mapping for ShipGen Shipment Chaining
Filter orders for AGLSwap step Shipment Chaining
Order Cancellation Purchase Creation & Updates
07/25 Deployment Fix EntityReference Copier Shipment Chaining
Client call for Shipment Notification Shipment Chaining
PFW Chaining for 2 Purchase Creation & Updates
Fix Tooling
GetMCOPOrderInfoAndMappingsForOrde
08/14 Deployment r
bug
Order Cancellation for Sign Failures Purchase Creation & Updates
Initiate new mapping in reversal Reversal Chaining
New Mapping for ShipComplete Shipment Chaining
OWEN propagation fix Purchase Creation & Updates
08/29 Deployment
Handling ShipEditability Restrictions Purchase Creation & Updates
IBA changes for FRW ARH Accounting
COGS Generation for IBA Shipment Chaining
09/12 Deployment
New Item Mapping in Reversal Reversal Chaining
Optimisation in reversal mapping Reversal Chaining
09/25 Deployment
Remove fallback in COGS Generation Shipment Chaining
Proposal is to segregate MCOP in to multiple individual areas so that each of them can move at their
own pace in terms of business launches, innovation and building operational stability. Division of MCOP
into multiple Microservices is based upon the data presented in table above with core idea being
separation of functional areas.
MCOP Purchase Creation & Updates
Scope: This module would cover all responsibilities related to Purchase Creation & Order Update
propagation. It would be responsible for keeping the customer order and secondary orders in sync,
which involves creation and propagation of changes. It would host PCE plugins to intercept sign events,
expose APIs to retry Purchase Creation from Herd, perform all necessary data translation from source to
destination marketplace & invoked PCE on the destination stack. It would also expose APIs necessary to
execute PFW & NFS workflows to send the order to fulfillment stage.
4
[Link] confidential
Dependencies: OMA/PCE (write APIs), PFW workflow
Proposed artifact nature: Service
MCOP Fulfillment Chaining
Scope: This module would be concerned with the fulfillment aspects of order execution and provide
virtual order fulfillment via Multi Company Fulfillment Network (MCFN). This corresponds to handling
the chaining of the shipment workflows namely Shipment generation, Shipment Complete, Shipment
Partial & Shipment Cancel. It would intercept the COW workflows, perform data translations, notify and
initiate corresponding workflows on destination stack(s) and chain them to completion.
Dependencies: COW Shipment workflows, OFS, OMA/PCE
Proposed artifact nature: Service
MCOP Reversal Chaining
Scope: This module would be responsible for interactions with CROW to create and handle reversals.
Reversal can be created on either side and this would provide bidirectional chaining. This module would
expose various APIs called by CROW graphs to understand and chain MCOP reversals. Currently it would
also intercept Refund Accounting workflows but once it is migrated to RAF, this responsibility can be
handed off. The traffic to this module would be much lesser and peak events have an off-set from
normal peak events.
Dependencies: CROW
Proposed artifact nature: Service
MCOP Accounting
What would this module handle? This module would be responsible for posting the accounting entries
with RAF (Retail Accounting Framework). In current architecture, different businesses handle various
accounting requirements via plugin model in RAF which is extended to MCOP. This module would be
responsible for handling data translations & data sharing needs between the orders for accounting
purposes
Dependencies: RAF
Proposed artifact nature: Service
MCOP Analytics/Tooling*
What would this module handle? This layer would take care of MCOP tooling for operational support
and building analytics reports that can aid in business expansions for clients. It would have debugging
5
[Link] confidential
capabilities for oncall support and auditing purposes. As of now, analytics systems do not exist but
MCOP hosts a set of debug APIs exposing mappings which are used by clients. Mappings include the
order/purchase id mappings, shipment id mappings, reversal id mappings for which MCOP is the
authoritative source. It also involves MCOP integrations with Grass and Hank which aid in debugging. In
long term, some mappings can be ported over to order document (maybe reversal document) & existing
ordering storage systems can be reused. This layer can assist in providing abstraction for that initiative.
Proposed artifact nature: Service
MCOP CrossMarketplaceProxy
What would this module handle? Data sharing layer that can aid in replicating data between source and
destination marketplaces. It will be used across the platform for data replication requirements. Further
it is being developed as independent offering for other clients with similar requirements. This will be
single point for cross marketplace calls (both synchronous and synchronous). All other modules have
requirement for cross region invocation and data transfer and they can re-use this module.
Proposed artifact nature: Service
MCOP Rule Engine
What would this module handle? This layer would involve business identification rules, plugin
invocation rules & self-service portal. MCOP behavior can be altered at multiple joints based on client.
This module would define such behavior. This include behavior specifications such as Order Types,
Chaining behavior, Parent-Child relationships, OPE definitions. Separating it out now would help in long
term when MCOP would be integrated with Ordering Central where client can configure accordingly.
Proposed artifact nature: Package
IMR Calculations
Below table shows the IMR costing for 3 existing service with corresponding usage data. These include
only Production hosts. Due to lack of distinct responsibility & unclear division, fleet are under-utilized
even though they are running at bare minimum capacity.
Current IMR Cost
Service CPU Hosts TPS Monthly
Peak AVG IMR
OUNS 3-5% 30 153 87 1.6K
MCFNSilService 3-5% 25 5 2 1.1K
MCOS 10-15% 80 1 0.5 4.6K
7.3K
6
[Link] confidential
Above proposal gives us 2 additional services to maintain. Assuming no optimization or change in API
structure, the total TPS would remain constant. By properly splitting the requests in appropriate
modules, we can improve the overall capacity utilization
Proposed IMR Cost
Service Estimated Hosts Monthly
IMR
Purchase Creation 25 2.2K
Shipment Chaining 25 2.2K
Reversal Chaining 15 1.3K
Accounting 20 1.5K
Analytics/Tooling 15 1.3K
8.5K
Appendix
1. Database Access Pattern
In current architecture, MCOP owns multiple DynamoDB tables. Following table shows the various
tables, their usages and how Micro Services will impact them. Most of the tables serve a purpose in
particular service and are not accessed beyond that. They should be accessed independently and
treated as a unique resource.
Table Name Description Usage
MCOSFieldMappings Contains aggregated view of all All modules
mappings.
OrderIdMapping Temporary mapping used during Only Purchase Creation
PCE Sign Purchase.
ShadowOrderCreationStatus Versioned status describing Order Only Purchase Creation
Creation Status progress.
PrefulfillmentStatusMapping Sync order condition. Only Purchase Creation
ShipmentIdMapping Mapping between Shipment. No Only Fulfillment chaining
context of Order here.
ShipmentStatus Sync the shipment status. Only Fulfillment chaining
ShipmentIdOrderIdMap Mapping of B2B order created Only Purchase Creation
against shipment.
CrowReversalMapping Reversal mapping. Only Reversal module
7
[Link] confidential
PluginRegistration Plugin endpoints. All modules
2. Impact on Roadmap Initiatives
2.1 Impact on Standardization
MCOP and business teams have integrated with many systems apart from Ordering to completely
support their usecase. MCOP-Standardization is an initiative to make those systems under MCOP orders
and behave accordingly. Having Micro Services will not impact this directly.
2.2 Impact on Externalization
Traditionally, MCOP had support to create orders for Amazon entity based on order for another entity.
MCOP Externalization is a project to reuse this functionality to entities external to Amazon. In current
architecture, MCOPExternalizationService has been created which acts as an integration layer. Splitting
monolithic system into Micro Services will help this initiative. Now the Externalization service clearly
knows exact hook points for various events during an order’s lifecycle. For example, for order creation it
will be invoking the Purchase Creation Service, for reversal creation it will invoke Reversal Micro Service.
3. Execution Strategy
This section presents couple of options to initiate the project in phases instead of a big bang launch. First
phase would be to split the code into simple, independent packages. In next phase, two of the modules
will be extracted out and functionality will be added to existing services.
3.1 Component Split
This proposal is a watered-down version of Micro Services. The idea is to split the existing monolithic
architecture into smaller independent components which will be co-deployed. Anyways this will be the
first phase of Micro Services proposal as well.
Division
Proposal is to break down monolithic package MultiCompanyOrderingService into smaller logically-
independent packages. (Same logical areas as mentioned earlier). Each module will exist as a separated
package. To promote code re-use, there would be set of helper libraries which would be invoked across
modules. Samples include
- Replication Library - Mapping Library
- Messaging Library - Document Handling Library
- Persistence Library - Rule Engine Library
Hosting
Under this proposal, there is no change to existing hosting infrastructure. They will continue as it is but
the packages deployed will change.
Limitations addressed
- Many developer pain points would be addressed by this. Merge Conflicts would be reduced.
Changes limited to one area could be implemented & tested independently.
8
[Link] confidential
- Further these can act as building blocks aiding in development of Micro Services.
Pending Limitations
- Time to market problem would still exist since all components would be deployed together. They
will run in same runtime environment and rollback would affect each component.
- This doesn’t handle the scaling concerns also.
3.2 Extending Current Services
Here the idea is to maintain the 3 existing services but adding functionality to them as per the earlier
proposal. It can be treated as phase 1 for Micro Service proposal.
MCFNSilService
Currently this service only hosts couple of APIs to translate the shipment data and open COW
workflows. All the fulfillment chaining logic from MultiCompanyOrderingService will be ported here;
COW shipment workflows will interact with only this service.
OrderUpdateNotificationService
This service currently hosts CASE plugin for order update events. Entire Purchase Creation flow and
Order Update flow will be ported here. It will host the CASE plugins & V2 plugin architecture. However,
the service name would be a misnomer.
MultiCompanyOrderingService
The existing service will continue hosting the remaining APIs. Accounting, reversals and all other existing
modules will remain as such.
Limitations addressed
- Two of the significant modules i.e. Purchase Creation & Fulfillment Chaining will be separated out
and can be independently developed & deployed.
- Cost associated with spinning up new services can be avoided.
Pending Limitations
- Some modules would still be hosted together. With respect to those, the problems mentioned
would not be solved.
- IMR impact due to this needs to be investigated. Check if we can reduce costs.
9
[Link] confidential