Concurrent Manager

 Concurrent Manager
 Output-post-processor
 XML Post Processor
 XML-publisher
 Concurrent Manager Queries

Concurrent Manager

Concurrent Mangers log files are located in the $APPLCSF/$APPLLOG location.

Where is Concurrent Manager log file location.

By default standard location is $APPLCSF/$APPLLOG , in some cases it can go to $FND_TOP/log as

well

cd $APPLCSF/$APPLLOG

For ICM Log                                       –> ls -lrt *$TWO_TASK*

For Standard manager Log                  –> ls -lrt w*.mgr

For Conflict Resolution manager Log –> ls -lrt c*.mgr

Question 1: What are different types of concurrent manager?

Answer There are several types of concurrent manager. Important ones:

Internal Manager

Standard Manager.

Conflict Resolution Manager

Internal monitors

Service Manager

Transaction Manager

Custom Manager Defined

Read more on below link

Oracle Concurrent Manager

Question 2: What is Internal Concurrent Manager?

Answer : It is responsible for controlling all other concurrent managers. Its main task is to make
sure that all other concurrent managers are up and running. It controls other manager through the
requests made to the service manager. It also starts, stops and restarts the service manager for all
nodes.

Question 3: What is Conflict Resolution Manager (CRM)?

Answer: It takes care of resolving the program incompatibilities and checks if a request in queue
can be run in parallel with the running request. If a program is identified as run alone, then it
prevents concurrent managers from starting other programs in the same conflict domain.

Question 4: What is Standard Manager?

Answer : Standard Manager is the master concurrent manager. It is always running and can take
care of processing any concurrent request. If at all, no other manager is assigned to a program,
that program will be picked by standard manager.

Question 5:

What happened behind the scene when a concurrent request is submitted?

Answer

1) Once a concurrent request is submitted by the user, the table FND_CONCURRENT_REQUESTS is

automatically updated with the details of the request. The table is also updated with the
information about the schedule of the concurrent request whether it’s immediately scheduled or
scheduled at a fixed time.

2)a) If the request is incompatible/constraints defined ,Once the request time to run is arrived,it
status is set to pending/[Link] the conflict resolution manager takes care of the request and
finds out what are the incompatibilities and set the status pending normal when the
incompatibilities are cleared.
b) If there are no incompatibilities then Once the request time to run is arrived,it status is set to
pending/Normal

c) ALL the standard concurrent Manager and special manager continuously poll the
FND_CONCURRENT_REQUESTS table .The job of a concurrent manager is to execute concurrent
requests that are in Pending / Normal phase / status and that it is qualified to run according to its
specialization rules.

Concurrent Manager Processes

– Act independently

– Select only requests that: (a) match the manager specialization rules, (b) are Pending/Normal, (c)
have a requested start time <= sysdate

d) Once the request is processed, the FND_CONCURRENT_REQUESTS table is updated with the
status.

Question 6: Business user create the task that Concurrent request is taking a lot of time to
complete. What will be your approach for debugging it?

Answer

1) First find the status of the concurrent request. It may be scheduled later on or it could be
pending/standby mode or all the concurrent manager are occupied running other request. If it is
pending/standy, we need to find the incompatible program running and inform [Link] times the
users schedule the request to run at a later time.

2) Find out the database sid of the concurrent request and check it is waiting on any locks. We will
kill the blocking session to order to complete the job

3) We can run a trace on the request id to find the sql running and then generate the explain plan
for it. You can see if the sid is stuck on particular sql . If it is particular sql, then it is good to check
the statistics of the table involved . We can look for tuning option for that query

4) We can check the parameters with which the request is run. (For example, once a user came
saying the request is not printing the output. On Checking the possible things, it was realized that
he scheduled the request with print copies = 0.)
Question 7: What happens when the internal concurrent manager dies abruptly? Are all the
managers also killed immediately after it?

Answer

No ALL the standard manager keep running and executing the request. if the internal manager dies,
below queue control request are not performed

a) Starts all other processes.

b) Executes “control requests” submitted by the administrator.

c) Activate/Deactivate/Abort Concurrent Manager

d) Terminate Concurrent Request

e) Monitors processes, restarting any that failed.

f) Sets the target number of processes for each service based on the current work shift.

Question 8: Does the internal manager run or schedule any request for itself?

Answer

No, the internal manager does not run or schedule any requests. It has nothing to do with
scheduling requests, or deciding which manager will run a particular request. Its function is only to
run ‘queue control’ requests

a) Starts all other processes.

b) Executes “control requests” submitted by the administrator.

c) Activate/Deactivate/Abort Concurrent Manager

d) Terminate Concurrent Request

e) Monitors processes, restarting any that failed.

f) Sets the target number of processes for each service based on the current work shift.

Question 9: How do I process more concurrent requests in parallel?

Answer

We can increase the target processes of the concurrent manager in order to increase the
parallelism. This can be done using define concurrent manager form or through direct update from
sqlplus

Question 10 : If the internal manager goes down, do I need to kill all the managers before restarting
the internal manager?

Answer

No, if the internal manager goes down you need not kill all the managers. You can simply start the
internal manager using startmgr.

Concurrent Manager troubleshooting

Question 11 : What are the problems u have faced while shutting down applications?

Answer

While shutting down application generally concurrent manager won’t go down because some or the
other request may be running. We will see what are the concurrent requests running by querying
fnd_concurrent_requests, fnd_concurrent_program_vl, v$session, v$process and v$sqltext.

If that request is only doing some select statement then we will kill those requests, otherwise we
will check what time it will take to complete by querying the previous runs of that request and
then we will decide what to do.

Oracle Concurrent Manager Secrets

Question 12: What are Internal Monitors ?

Answer: Internal Monitors are used specifically in PCP to allow for ICM failover to other available
middle tier nodes.

a) Place an Internal Monitor on any node where the ICM can start in case of a failure.

b) Internal Monitors are seeded on every registered node by default.

c) If the ICM goes down, the Internal Monitor will attempt to start a new ICM on the local node.

d) If multiple ICMs are started, only the first will stay active. The others will gracefully exit.

Question 13: Can I delete concurrent manager?

Answer:

Yes, you can delete any concurrent manager. For deleting, query for the manager in the defined
concurrent manager form and then delete the row.

Deleting the predefined concurrent managers is not recommended and it should never be done.
Deletion may cause instability in the system.

Question 14: How can you know which trace file is created for the particular request?

Answer

You can find out the same using the script given below. The trace will be located in the udump
location of the database server.

prompt

accept request prompt ‘Please enter the concurrent request id for the appropriate concurrent
program:’

prompt

column traceid format a8

column tracename format a80

column user_concurrent_program_name format a40

column execname format a15

column enable_trace format a12

set lines 80

set pages 22

set head off

SELECT ‘Request id: ‘||request_id, ‘Trace id: ‘||oracle_Process_id, ‘Trace flag: ‘||
req.enable_trace, ‘Trace Name: ‘||[Link]||’ ‘||lower([Link])||’ora’||
oracle_process_id||’.trc’, ‘Prog. Name: ‘||prog.user_concurrent_program_name, ‘File name: ‘||
execname.execution_file_name||execname.subroutine_name , ‘Status :’||decode(phase_code,
‘R’, ‘Running’)||’ ‘||’-‘||decode(status_code, ‘R’, ‘Normal’), “SID Serial: “||[Link]||” , “||
[Link]#, “Module : “||[Link]

from fnd_concurrent_requests req,

v$session ses, v$process proc,

v$parameter dest, v$parameter dbnm,

fnd_concurrent_programs_v1 prog,

fnd_executables execname

where req.request_id = &request

and req.oracle_process_id=[Link](+)

and [Link] = [Link](+)

and [Link]=’user_dump_dest’

and [Link]=’db_name’

and req.concurrent_program_id =

prog.concurrent_program_id

and req.program_application_id =

prog.application_id

and prog.application_id =

execname.application_id

and

prog.executable_id=execname.executable_id;

Top 30 Most Useful Concurrent Manager Queries

Question 15: Explain how parallel concurrent processing(PCP) works?

Answer

In case of parallel concurrent processing, all the managers are assigned a primary and a secondary
node. The managers are started in their primary node by default. In case of node failure or Oracle
instance failure, all the concurrent managers on that node are switched to their secondary nodes.
Once the primary node is available again the concurrent managers on the secondary nodes are
migrated back to the primary node. During the migration process, a manager may be spread across
both primary and secondary nodes.
In case of parallel concurrent processing, it may happen that in a node where parallel concurrent
processing is configured, the Oracle instance may or may not be running. The node which is not
running Oracle, the concurrent managers connects via Net8 to a node which is running Oracle.

The internal concurrent manager can run on any node, and can activate and deactivate concurrent
managers on all nodes. Since the internal concurrent manager must be active at all times, it needs
high fault tolerance. To provide this fault tolerance, parallel concurrent processing uses internal
monitor processes. The job of the internal monitor process is to constantly monitor the internal
manager and start it when it fails. Only one internal monitor process can be active on a single
node. You decide which nodes have an internal monitor process when you configure your system.
You can also assign each internal monitor process a primary and a secondary node to ensure fail
over protection. Internal monitor processes, like concurrent managers, can be assigned work shifts,
and are activated and deactivated by the internal concurrent manager.

Parallel Concurrent Processing

Question 16: What are the circumstances in which you need to bounce the concurrent manager?

Answer: There can be many situation where you need to bounce the concurrent manager

a) When you modify the definition of the printers

b) When you modify the environment variables. Suppose you have changed the APPLTMP and
APPLPTMP variable.

c) When all the requests are pending and hanging and no processing happening

d) patch application requires the bounce of the CM

e) We have many global hangs in the system due to locks by several concurrent manager and other
processes

Question 17: What are the reasons a concurrent manager hangs?

Answer:

The concurrent manager hangs due to many reasons. A few of them are:

– Long running jobs

– The internal manager was activated by someone other then owner of the application system

– The operating system files system is full

– It’s not able to create the log file

– You’ve shut down the internal manager, but actual has a number in it
– The database is hanging may be because the archive log files have filled

– Pending/standby requests are too many

Question 18: How can we enable/disable Conflict Resolution Manager?

Answer: It can be done using profile options “Concurrent: Use ICM”. Set it to “Y” to enable Conflict
Resolution manager. To disable it, set the profile option to “N”.

Question 19: What are Transaction Managers?

Answer : Transaction managers provide synchronous job processing by continually monitoring a

DBMS pipe for requests to come through from a client-side application. The job of a transaction
manager is to process this job immediately and send information back to the client using the pipe.

a) Transaction Managers Provide Synchronous Job Processing

b) A client makes a request for a specific transaction manager to run a program, and waits for the
results of that program

c) Product teams’ programs are linked directly into the transaction manager executables

d) PO, CRP, INV, AR, and OE all ship transaction managers

Question 20: How the mechanism of viewing log and output files works from browser?

Answer

The sequence of events is as follows:

1. A user within an Applications session requests to view a log or out file.

2. The browser receives the request and spawns the cgi program [Link]

3. [Link] logs into the database and queries FND_CONCURRENT_REQUESTS to discover on

which node the files for this request are stored.

4. [Link] constructs the service name for the file server on that node. And makes the tns call
to contact the listener for this service name.

5. The listener responds by spawning the local FNDFS executable, as defined in it’s [Link] file.
Now, FNDFS and [Link] are able to communicate directly now, using RPC calls.

6. [Link] requests FNDFS to transfer the file that was selected by the user.

7. FNDFS transfers the file contents to a temporary file directory on the web server’s node.
8. The web server displays the file contents to the user.

Question 21: Why does the Concurrent manager put a concurrent program into a queue? Why
doesn’t the manager simply let the program run?

Answer: Because at any given point in time a concurrent manager can run no more than say 10
programs concurrently. This figure of 10 is configurable of course. First the manager puts a
submitted program into a queue, next the manager checks if there is a slot available (i.e. Less than
10 programs are currently running). If a slot is found available, the concurrent manager then runs
the program, or else it keeps the concurrent program in a queue with status Pending.

Question 22: What to do if a request is Inactive/No Manager

Answer: This is quite tricky questions

a) Make certain that there is at least one active manager with specialization rules that allow the
program.

b) If you have confirmed the previous point, then the problem may be a stale Worker Request View

– The view is used internally to map requests to managers

– The view is regenerated when managers are created, or specialization rules are altered

c) You can manually regenerate the view

FNDLIBR FND FNDCPBWV apps/apps SYSADMIN ‘System Administrator’ SYSADMIN

Question 23: What are Service Managers

Answer: Service Managers are spawned on the middle-tier nodes of a GSM enabled system in order
to act as an agent of the ICM. When the ICM sees that it needs An Service Manager to perform some
function, such as start a concurrent manager process, on a middle-tier node, it will make remote
procedure control (RPC) calls to the Apps listener on that node to start the Service manager. Once
the Service manager has been started and initialized, the ICM communicates directly to the SM
through RPC, giving it information to manage the services on that node. The SM is spawned from
the APPS TNS [Link] APPS TNS Listener must be started on every middle-tier node in the
system, and started by the user that starts ICM (e.g. applmgr) .TNS Listener spawns Service
Manager to run as agent of ICM for the local node
The Service Manager is started by ICM on demand when needed. If no management actions are
needed on a node a Service Manager will not be started by ICM until necessary. When ICM exits its
Service Managers exit as well.

Question 24: What all is performed by ICM Process Monitor?

Answer: The ICM itself and each process spawned by the ICM has an entry in
FND_CONCURRENT_PROCESSES and holds a uniquely named DBMS lock.

The unique lock of the ICM has a format of FNDCPLK_ICM. This database session lock is the method
that the ICM ensures each PMON cycle that manager and service processes are still alive. If the ICM
can get the DBMS session lock of a process, the ICM will start a new process for that manager or
service.

Output Post Processor

 What is Output Post Processor?

Concurrent Processing now uses the Output Post Processor (OPP) to enforce post-processing
actions for concurrent [Link]-processing actions are actions taken on

concurrent request output. An example of a post-processing action is that used in Concurrent

Processing support of XML Publisher.

If a request is submitted with an XML Publisher template specified as a layout for the concurrent
request output, then after the concurrent manager finishes running

the concurrent program, it will contact the OPP to apply the XML Publisher template and create
the final output.

OPP runs as a service that can be managed through Oracle Applications Manager (OAM) from
the System Activity page (Navigation: Applications Dashboard > Applications

Service (from the dropdown list) > Go).

How to Increase the number of Output Post Processors?

[Link] on to Applications with “System Administrator” responsibility.
[Link] to Concurrent -> Manager -> Define.
[Link] for the “Output Post Processor” service.
[Link] on “Work Shifts” and increase the number of processes

How to get OPP manager log file location?

$APPLCSF/log/<SID>/FNDOPP####.txt       OR
1,System Administrator > Concurrent > Manager > Administer
2,Search for ‘Output Post Processor’
3,Click the ‘Processes’ button .
4,Click the Manager Log button. This will open the ‘OPP’
Upload the OPP log file.

cd $APPLCSF/$APPLLOG

 ls -ltrh FNDOPP*

In some cases, Output Post Processor is not start up and it shows Actual and Target are
showing different values when we query for Output Post Processor.
The log files shows that no error [Link] this case apply the following possible solution for
starting the OPP.

1. Shutdown the internal manager by using [Link] stop apps/apps

2. Make sure there is no FNDLIBR processe running:
$ ps -ef| grep FNDLIBR OR ps -ef|grep applprod|grep FNDLIBR
3. If there is any FNDLIBR processe please kill it $ kill -9 pid
4. Run [Link] script as document from Note 134007.1
5. Restart the internal manager by using [Link] start apps/apps

or How to kill and start Output Post Processor (OPP) Background:

This article explains how to kill OPP and restart the same

Solution:
1,System Administator > Concurrent > Manager > Administer
2,Query "Output Post Processor" -> Processes button
Get the sytem id of "Output Post Processor"
3,ps -ef|grep [system id]
4,kill -9 [system id] in Unix
5,System Administator > Concurrent > Manager > Administer
6,Query "Output Post Processor" -> Restart button

Fixing Output Post-processor actions failed issues in EBS R12

Environment: Oracle EBS 12.1.3, Oracle Database 11gR2, RedHat Linux5

Symptoms:

Users unable to open the out files.

Concurrent requests failed with “Post-processing of request failed error message”

One or more post-processing actions failed. Consult the OPP service log for details.

No further attempts will be made to post-process this request.

Cause:
The concurrent manager process was able to successfully invoke the Output Post-Processor
(OPP) but encountered a timeout as the OPP takes longer than the value
assigned to complete the job.

 Solution:

1,Increase the value of profile Concurrent: OPP Response Timeout . Bounce Apache and retest.
2,If the issue still exists, perform the following steps.
3,Increase the number of Output Post Processors as follows:
4,Increase the number of processes for Output Post Processor.
5,Additionally, ensure there is a setting of
[Link]:0:max_threads=5 under Parameters.

****

We had an incident for XML reports not being Post Processed.  I checked the logs for the reported
request id and one of the processes on amxp05cm01b was giving OutOfMemoryError errors.  So I
have restarted OPP on that node.  Can you please monitor the logs to check that the issue is not on-
going?  We already have the max memory 2GB per process allocated, so if it does re-occur we need
to raise an SR with Oracle.
grep 93399877 *OPP*

[Link]:[10/6/17 11:52:39 AM] [OPPServiceThread0] Post-processing request 93399877.

[Link]:[10/6/17 11:52:39 AM] [334418:RT93399877] Executing post-processing actions

for req                                      uest 93399877.

[Link]:[10/6/17 11:52:39 AM] [334418:RT93399877] Starting XML Publisher post-

processing ac                                      tion.

[Link]:[10/6/17 11:52:39 AM] [334418:RT93399877]

[Link]:[10/6/17 11:52:39 AM] [334418:RT93399877] Output file was found but is zero
sized -                                       Deleted

[Link]:[10/6/17 11:52:39 AM] [UNEXPECTED] [334418:RT93399877]

[Link]:                                       Java heap space

[Link]:[10/6/17 11:52:39 AM] [334418:RT93399877] Completed post-processing actions

for req                                      uest 93399877

XML Post Processor

XML Post Processor output was not comingin PDF format instead of it comioig ni XML format

--- Error messges shows the problem with OPP.

Troubleshooting:

  don’t this this program (Statement Generation Program – Request id’s 88960874 and 88960963 ) is
running through OPP as it was running from Standard Manager2.

 I checked ARXSGPO template and looks like default output type defined as “PDF”.

I did see this program was submitted today since a week, as this was executed at first on Thursday
06/01/2017 @ 13:58. There was another concurrent request “Statement Generation Program (XML
Report Publisher)” was executed on May 26th.
Even though the Concurrent Program says XML the Statement Print should come out in PDF based on
below set up:

We have the soft link missing for [Link] and this rdf was updated on May 28 th with CR #
42406 applied at May 28, 2017 10:03:04 AM CDT on OPRD(adpatch) (application: Oracle
Accounts Receivables). Just recreated the link to point the standard program to custom program. I
had taken the backup of the standard report before creating the soft link. I have submitted the
“Statement Generated Program” and its completed successfully and also spawns the child program
as well. I can see the output in PDF format. Please verify the same and let us know further.

XML Publisher

XML Publisher is a simple engine. It needs only 3 input values to generate report.
[Link] to display on report(in XML format)
[Link] model/presentation(in XSL-FO format or RTF)
[Link] output type (PDF, Doc, HTML, etc)
Points to Note:
o XML publisher doesn’t care about how you are generating xml data. Only it needs is, your actual
data(text file, excel file, data in DB) must be converted in xml format(calling XML Data) as input to
the engine. To get this, you can write your own xml data or use Oracle Reports to generate xml data or
use any other third party tools to generate xml data from actual data.
o XSL-FO stands for Extensible Stylesheet Language Formatting Objects. Used to present XML Data
in readable and user friendly format. But normal user understands very little in this, so there is an
alternative i.e. user friendly RTF. You do presentation work on RTF file and give this as input to the
XML Publisher engine. XML Publisher converts RTF into XSL-FO.

What is new here?

We are using Oracle Reports(rdf file) to generate XML data.

Run CP
|
———–Oracle Reports(RDF)—–>Generates XML Data–>
| |—>XML Pub Engine–>CP Output(in PDF or etc)
–Finds template from XML Publihser Admin resp–>RTF file–>

Summary Steps:
1. Create Oracle Report(XXXX_TEST.rdf)
2. Define/Register Concurrent Program(XXXX_TEST)
3. Create Template Definition and Template(XXXX_TEST.rtf)
4. Run CP and Show XML Publisher report in PDF format(XXXX_TEST_OUTPUT.pdf)

Concurrent Manager 

Manager status 

select decode(CONCURRENT_QUEUE_NAME,'FNDICM','Internal Manager','FNDCRM','Conflict

Resolution Manager','AMSDMIN','Marketing Data Mining Manager','C_AQCT_SVC','C AQCART
Service','FFTM','FastFormula Transaction Manager','FNDCPOPP','Output Post
Processor','FNDSCH','Scheduler/Prereleaser Manager','FNDSM_AQHERP','Service Manager:
AQHERP','FTE_TXN_MANAGER','Transportation Manager','IEU_SH_CS','Session History
Cleanup','IEU_WL_CS','UWQ Worklist Items Release for Crashed session','INVMGR','Inventory
Manager','INVTMRPM','INV Remote Procedure Manager','OAMCOLMGR','OAM Metrics
Collection Manager','PASMGR','PA Streamline Manager','PODAMGR','PO Document Approval
Manager','RCVOLTM','Receiving Transaction Manager','STANDARD','Standard
Manager','WFALSNRSVC','Workflow Agent Listener Service','WFMLRSVC','Workflow Mailer
Service','WFWSSVC','Workflow Document Web Services Service','WMSTAMGR','WMS Task
Archiving Manager','XDP_APPL_SVC','SFM Application Monitoring
Service','XDP_CTRL_SVC','SFM Controller Service','XDP_Q_EVENT_SVC','SFM Event Manager
Queue Service','XDP_Q_FA_SVC','SFM Fulfillment Actions Queue
Service','XDP_Q_FE_READY_SVC','SFM Fulfillment Element Ready Queue
Service','XDP_Q_IN_MSG_SVC','SFM Inbound Messages Queue
Service','XDP_Q_ORDER_SVC','SFM Order Queue Service','XDP_Q_TIMER_SVC','SFM Timer
Queue Service','XDP_Q_WI_SVC','SFM Work Item Queue Service','XDP_SMIT_SVC','SFM SM
Interface Test Service') as "Concurrent Manager's Name", max_processes as "TARGET
Processes", running_processes as "ACTUAL Processes" from apps.fnd_concurrent_queues
where CONCURRENT_QUEUE_NAME in
('FNDICM','FNDCRM','AMSDMIN','C_AQCT_SVC','FFTM','FNDCPOPP','FNDSCH','FNDSM_AQHER
P','FTE_TXN_MANAGER','IEU_SH_CS','IEU_WL_CS','INVMGR','INVTMRPM','OAMCOLMGR','PAS
MGR','PODAMGR','RCVOLTM','STANDARD','WFALSNRSVC','WFMLRSVC','WFWSSVC','WMSTAM
GR','XDP_APPL_SVC','XDP_CTRL_SVC','XDP_Q_EVENT_SVC','XDP_Q_FA_SVC','XDP_Q_FE_READ
Y_SVC','XDP_Q_IN_MSG_SVC','XDP_Q_ORDER_SVC','XDP_Q_TIMER_SVC','XDP_Q_WI_SVC','XDP
_SMIT_SVC');

Concurrent Queue 

SELECT a.concurrent_queue_name, SUBSTR(b.os_process_id,0,10) "OS Proc",

b.oracle_process_id "Oracle ID",  b.process_status_code

FROM fnd_concurrent_queues a, fnd_concurrent_processes b

WHERE a.concurrent_queue_id=b.concurrent_queue_id

AND b.process_status_code='A'

ORDER BY b.process_status_code;

Concurrent Manager log location 

select fcp.node_name, 

TO_CHAR(fcp.last_update_date, 'DD-MON-RR HH24:MI:SS'), 

fcp.logfile_name 

FROM fnd_concurrent_processes fcp, fnd_concurrent_queues fcq 

WHERE fcp.concurrent_queue_id = fcq.concurrent_queue_id 

AND fcp.queue_application_id = fcq.application_id 

AND fcq.concurrent_queue_name = 'FNDCPOPP' 
order by 2 desc;