Maintenance Guidelines
This document provides a set of mandatory guidelines that need to be implemented in your Calypso Installation in order to qualify for Product Support.
1. GG - General Guidelines
GG.1 - Ensure that Server and Client Request logs are enabled for all processes.
For this, you need to set:
LOG_CATEGORY=Monitoring.ClientRequest,Monitoring.ClientRequest.Stacktrace,Monitoring.ServerRequest and LOG_LEVEL=INFO.
GG.2 - Ensure that GC logs are enabled for all processes:
Xloggc:<gclog.log> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
GG.3 - Monitor Alerts generated in the system.
2. DB - Database
Calypso recommendations for database optimization must be followed. These are also part of the Database Administration Guide.
DB.1 - Set GLOBAL_TEMP_TABLE_STATS=SHARED.
DB.2 - Do not use Oracle Auto Stats Gather feature, please turn off this feature as it doesn’t work with Calypso.
DB.3 - Daily gather schema stats for CALYPSO, SYS and SYSTEM schemas using dbms_stats.gather_schema_stats with sample 20% (if db size is small use 100%).
DB.4 - Weekly gather schema stats using dbms_stats.gather_schema_stats with sample 100%.
DB.5 - Please lock the statistics for the following tables while system is running, make sure you have enough records before you gather stats and lock it:
| • | PS_EVENT |
| • | PS_EVENT_INST |
| • | TOKENSTORE |
do a count() on the ps_event,
make sure you have count() > 10K:
3. HA - High Availability
HA.1 - Ensure there are odd number of nodes. There is no Support for even nodes.
4. ES/DS - Event Server/ Data Server
The Event Server properties (<CALYPSO_HOME>/client/resources/appConfig) changed for below must also be reflected in Event Server and Data Server properties files.
Event Server must run in a separate JVM.
|
Component |
Description |
Automation / Alert |
|||
|---|---|---|---|---|---|
|
ES.1 - Event Server - Memory sizing |
Summary For all Calypso processes ensure -Xms is same as -Xmx. Ensure that each process is sized properly by evaluating the GC logs and ensuring that process is not under stress - GC activity, memory thrashing, Peak memory usage > 90%. Details GC logs should determine the Data Server failover parameters. |
Alert if Failover settings are not inline with our recommendations |
|||
|
ES.2 - Event Server |
Summary Event Server must run as a separate component. Running it within the Data Server is not supported in production. Event Server within Data Server should be limited to non-production environments only. |
Alert: Event Server should run separately from the Data Server | |||
|
ES.3 - Event Server - Calypso topic memory |
Summary Memory allocated for "calypso" topic should be 50% - 55% of total memory allocated to Event Server. Older versions of Calypso may default to a value as high as 75%. This is automatically done when Calypso deployLocal is executed. However, clients may tweak this to adjust. Any changes to Event Server topic memory outside of these limits is not recommended. Do not use Event server for any other custom queues or topics. Details The value - <max-size-bytes> should be used to set the topic value. <address-setting match="jms.topic.calypso"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <max-size-bytes>12025908428</max-size-bytes> -- Ensure this is 50 - 55% of your DS Memory or ES memory if this container is separate. <page-size-bytes>25085760</page-size-bytes> <address-full-policy>PAGE</address-full-policy> <message-counter-history-day-limit>10</message-counter-history-day-limit> </address-setting> |
Alert: Calypso topic memory should be within approved limits Look to stop ES from starting up if configuration is outside of recommended tolerances |
|||
|
ES.4 - Event Server - EVENT_LIMIT_SIZE |
Summary This sets the max size of a Calypso PS Event that be published within the system. It defaults to 1 MB. If clients change this limit due to large events published in the system, corresponding changes must be done to the Large message configuration. Details Event Server Large Message Configuration Monitoring applications logs for events that cannot be published. If these messages are too frequent and impact business, you may need to change the EVENT_LIMIT_SIZE. |
||||
|
ES.5 - Event Server - Large Message Configuration |
Summary Large messages are defined as those that exceed the threshold set in Event Server. For optimal performance, set the threshold to the same value as EVENT_LIMIT_SIZE. This ensures that all messages are held in memory and not paged to disk. However, If EVENT_LIMIT_SIZE is set to a higher number ( 2-4 MB), only set the Large Message threshold to a higher number if there is sufficient memory allocated to the event server. Enable compression of large messages, especially when there are a large number of clients or network has a higher latency. Details Ensure ES is sized appropriately to allow for all events to be held in memory. Monitor Event server storage folder for large message paging. If too many files are being created and not consumed immediately, please adjust the threshold to keep the messages in memory. <min-large-message-size>204800</min-large-message-size> in connection factories for hornetq-ra, RemoteConnectionFactory, mdb-hornetq-ra. These are usually added for Netty connection factories. But, can be added to in-vm too. |
||||
|
DS.1 - Data Server - GC Settings |
Summary Limit GC time to less than failover trigger time. Details DS failover trigger If using G1 GC then you can set
Sets a target value for desired maximum pause time. The default value is 200 milliseconds. Set this value such that it GC pause time is reduced to avoid failover due to long pause. This argument will act as a soft target for the max GC pause.
Set -Xms ** equal to about ~30% of -Xmx. Setting additional GC options for better memory management. Added to both G. |
||||
|
DS.2 - All Calypso Servers - JVM Setting |
Summary Set MaxMetaSpace. Details Sets the maximum amount of native memory that can be allocated for class metadata. By default, the size is not limited. Out of box we have set this value to 1 GB for calypso servers you can increase as per your requirement but it is not recommended to keep it unbounded.
|
3 | |||
|
DS.3 - Data Server - Minimum Connection Pool Size |
Summary Minimum connection pool should be atleast 25. Details <min-pool-size>5</min-pool-size> <max-pool-size>300</max-pool-size> |
Enforce via automation to prevent a setting < 25 (or 20) |
5. RM - Recommended Routine Maintenance
To ensure the optimal operation of your system, you should perform routine maintenance. Below is a set of processes and scheduled tasks that should be run on a regular basis. You may tailor them to your environment by changing the execution frequency.
Scheduled tasks are configured using Configuration > Scheduled Tasks from the Calypso Navigator. The Calypso Scheduler must be running for Scheduled Tasks to function.
RM.1 - Login Attempts
It is recommended to purge the login attempts on a regular basis using Web Admin > Data Server > Server > Login Attempts.
You can also use the PURGE scheduled task to purge login attempts, see below.
Purge the consumed events on a regular basis using Utilities > Maintenance > Monitoring > Clean-up > Purge Consumed Events from the Calypso Navigator.
Refer to Calypso Utilities Menu documentation for details.
You can also use the PURGE scheduled task to purge consumed events, see below.
Provided there are no pending events in Web Admin, you can reset the event seed.
If events are pending, wait until the engines are finished processing, if they have not finished.
If the events are still pending after a few minutes, investigate why the events cannot be processed (this can happen because of faulty workflow setup or engine exceptions).
Purge unconsumed events only as a last resort after determining how to reprocess the Trades, Transfers, and Messages that created the problem.
You can purge unconsumed events using Utilities > Maintenance > Monitoring > Clean-up > Purge Unconsumed Events from the Calypso Navigator.
The PURGE scheduled task allows deleting a number of unused objects. The recommended execution frequency is specified below for each object.
The configuration window appears as shown below.
Specify the desired attributes and save the configuration.
You can configure multiple PURGE scheduled tasks to purge different objects, and execute with different frequencies.
| • | PURGE EVENTS - Select true to purge all consumed events, including guarantee-delivery events. (Run on a daily basis.) |
| • | PURGE CLIENTS CACHE - Select true to purge all client caches. (Run periodically.) |
| • | Restart Engines - Select an engine to be restarted. |
| • | PURGE_COMPLETED_TASKS - Select true to purge all completed tasks. Run periodically. |
Ⓘ [NOTE: The scheduled task allows deleting completed tasks for relatively small volumes - If the table bo_task has a large number of records, it is recommended to use the Archive scheduled task instead with attribute "Archive completed Tasks of type" and a date range]
| • | PURGE_LOG_FILES - Set to true to purge all log files, in particular log files from a previous version of the system. (Run periodically.) |
| • | PURGE_MARGINCALL_POSITIONS - Select ALL, SECURITIES, or CASH to purge the corresponding margin calls on or before a given date. Enter the date in the MARGINCALL_DATE attribute. (Run periodically.) |
| • | PURGE LOGIN ATTEMPTS - Select true to purge login attempts that have been in the system for a certain period. Select the period from the LOGIN ATTEMPTS OLDER THAN attribute. (Run periodically.) |
| • | PURGE PAGERS - Set to true to purge Pagers that have been in the system for a certain period. Select the period from the PAGERS OLDER THAN attribute. |
RM.4 - Scheduled Task COMPLETE_TASKS
This scheduled task allows completing tasks in bulk.
| • | Days in past (mandatory) - Number of days in the past to process tasks with respect to the valuation date. |
| • | Tasks to Complete (mandatory) - Enter a comma-separated list of task event types to be processed. |
| • | Tasks to Exclude - Enter a comma-separated list of task event types to be excluded as needed. |
| • | Archive/Delete/Complete (mandatory) - Select the process to be applied: – Complete - The tasks are moved to COMPLETE status. – Complete & Delete - The tasks are moved to COMPLETE status and are deleted. – Complete & Archive - The tasks are moved to COMPLETE status and are archived. – Only delete already completed tasks - Tasks already in COMPLETE status are deleted. – Only archive already completed tasks - Tasks already in COMPLETE status are archived. |
| • | Max batch size (mandatory) - You can specify the number of tasks to be processed at a time. Once a batch is completed, the next batch is started. |
| • | Max tasks to complete (mandatory) - Maximum number of tasks to be processed - The scheduled task will stop after the maximum number of tasks is reached. |
| • | Output file - Enter the full path to a file name to store processing information as needed. |
| • | Log detailed processing - True to display |
6. Arch - Archiving
The archiving capability allows administrators to move data from the current tables to history tables. This reduces the size of the current tables. It also makes it easier to use database utilities (such as Oracle's DataPump utility) for exporting archived data to an external storage media such as tapes, CDROMs, or data warehouses.
Access to archived objects is limited. An administrator can restore them (move them from the history tables back to the current tables). A few reports can include archived objects as well as current objects. But most Calypso applications will not be able to access archived objects.
Archiving is more then just activating a set of scheduled tasks. It needs business review and sign off to be implemented. Please reach out to your Customer Success Manager for further details on Calypso’s Health Check offering.
Archiving should be run using the following procedure.
First and foremost, a trade should never be archived if all the related objects have not been previously archived. Therefore, the archiving should be done in the following order.
Arch.1 - Scheduled task ARCHIVE_TASK - To archive completed exception tasks of type EX_GATEWAY from the current database tables to separate history tables in the database, or simply remove/delete completed exception tasks of type EX_GATEWAY
Arch.2 - Clean-up Database window - To archive Market Data and Analysis Outputs.
Arch.3 - Scheduled task Archive - To archive Postings and CREs.
Arch.4 - Scheduled task ARCHIVE_ADVICE_DOCUMENT - To archive saved documents.
Arch.5 - Scheduled task ARCHIVE_MESSAGE - To archive messages.
Arch.6 - Scheduled task Archive - To archive Transfers.
Arch.7 - Scheduled task ARCHIVE_POSITION - To archive positions computed by the Liquidation engine.
Arch.8 - Scheduled task Archive - To archive Audit Data.
Arch.9 - Scheduled task ARCHIVE_TRADE_GRAPH - To archive trades once the positions have been archived.
Arch.10 - Scheduled task MATURE_TRADE
This scheduled task applies an action to OTC mature trades. It should be run on a daily basis.
The configuration window appears as shown below.
Specify the attributes as applicable, and save the configuration.
| • | APPLY_ACTION - Select the action to apply. The action needs to be available within the trade workflow. For example, if you select the action MATURE, the trade workflow must have a transition VERIFIED/MATURE/MATURED. |
| • | FILTER_TRADE_STATUS - Select the trade status to which you want to apply the MATURE action. If not specified, only VERIFIED trades are selected. |
Arch.11 - Scheduled task MATURE_POSITIONTRADE
This scheduled task applies an action (previously defined in the workflow) to position-based trades which have a Value Date < Scheduled Task Process Date. This should be run on a daily basis.
The configuration window appears as shown below.
Specify the attributes as applicable, and save the configuration.
Arch.12 - Scheduled task ERS_HOUSEKEEPING.
ou can use the scheduled task ERS_HOUSEKEEPING to delete ERS Risk Adhoc and Official results, and trade level historical results. Caution should be taken when using this scheduled task as IT WILL PERMANENTLY REMOVE THE ROWS FROM THE DATABASE.
| • | Adhoc Days – Enter the number of calendar days, relative to the scheduled task val date, for which TO KEEP adhoc results in your database. Data older than this value will be deleted from the following tables: |
| – | ERS_LOG |
| – | ERS_RESULT |
| – | ERS_RUN_PARAM |
| – | ERS_RUN |
| – | ERS_JOB_EXEC |
For example, set to 365 to remove results older than 1 year. If not set, nothing is removed from the database.
| • | Trade Result Days – Enter the number of calendar days, relative to the scheduled task val date, for which TO KEEP trade level official historical results in your database. Data older than this value will be deleted from the following table: ERS_RESULT_HISTORY. |
Depending on your trade volumes and policies, a value between 7 and 30 is recommended. If not set, nothing is removed from the database.
'Trade Result Days' only cleans trade level official historical results (i.e. aggregation not used - GROUP_LEVEL=0 in the database), whereas 'All Results Days' cleans all official historical results (i.e. aggregated or not - GROUP_LEVEL not considered).
See "Results Date for Comparison" below for additional details.
| • | All Results Days – Enter the number of calendar days, relative to the scheduled task val date, for which TO KEEP official historical results in your database. Data older than this value will be deleted from the following tables: |
| – | ERS_LOG |
| – | ERS_SCENARIO_RULE_AUDIT |
| – | ERS_RESULT_HISTORY |
| – | ERS_RUN_PARAM |
| – | ERS_RUN_HISTORY |
| – | ERS_RESULT_DRILLDOWN |
| – | ERS_JOB_EXEC |
| – | ERS_MKTDATA_QUOTE_VALUE |
| – | ERS_MKTDATA_PORTFOLIO |
For example, set to 365 to remove results older than 1 year. If not set, nothing is removed from the database.
| • | DateRule – You can select a date rule to define dates that will not be removed from the database. For example, if you set Adhoc Days to 90, and DateRule to “End Month”, then any End of Month results will not be removed from the database over the past 90 days. |
| • | DateRule Start Year – Enter the start year of the date rule, if any. |
| • | Wait – Whether the scheduled task should monitor the completion of the housekeeping process or not. |
| – | If false, the scheduled task will complete immediately, whether the housekeeping has completed or not. |
| – | If true, the scheduled task will not complete until the housekeeping process has completed. The true setting is required if you wish to chain scheduled tasks. |
Results Date for Comparison
The date that is evaluated for housekeeping varies based on the results type.
| • | Official results generated by ERS_ANALYSIS scheduled task: |
| – | ERS_ANALYSIS val date < (ERS_HOUSEKEEPING val date - 'All Results Days' parameter) |
| – | ERS_ANALYSIS val date < (ERS_HOUSEKEEPING val date - 'Trade Result Days' parameter) |
| • | Adhoc results made official: |
| – | Official results val date < (ERS_HOUSEKEEPING val date - 'All Results Days' parameter) |
| – | Official results val date < (ERS_HOUSEKEEPING val date - 'Trade Result Days' parameter) |
| • | Adhoc results: Adhoc physical run date < (ERS_HOUSEKEEPING val date - 'Adhoc Days' parameter) |
How are we doing?