FAQ

Changing DB connection information

How to manually change DB connection information.

• Change DB Data Source information (execution of value update query set as DB data)

• Note: The query statement is reorganized with the changed DB_IP and DB_PORT values ​​to reflect the update.

su ipadb
cd /rpa/pkgs/mariadb/bin
./mysql -u root -p
MariaDB [(none)]> SELECT * FROM catalog.tb_data_source;
MariaDB [(none)]> UPDATE catalog.tb_data_source_info SET CONNECT_URL='jdbc:mysql://DB_IP:DB_PORT' WHERE DATA_SOURCE_ID='DS_2bc4eb2ee9b74807941fce8fac7bde87';
엔터
Ctrl-C

• Changed Tomcat comm.Properties (Web application related, 3 places in total – tenant/admin/user portal)

su ipaadm
cd /rpa/apps/admin/admin/WEB-INF/classes/properties
vi comm.properties

• Change the jdbcUrl property below.

jdbcUrl=jdbc:mysql:// DB_IP: DB_PORT/rpa?useLegacyDatet~~~생략~~~~
wq!
cd /rpa/apps/admin/user/WEB-INF/classes/properties
vi comm.properties

• Change the jdbcUrl property below.

jdbcUrl=jdbc:mysql:// DB_IP: DB_PORT/rpa?useLegacyDatet~~~생략~~~~
wq!
cd /rpa/apps/admin/tenant/WEB-INF/classes/properties
vi comm.properties

• Change the jdbcUrl property below.

jdbcUrl=jdbc:mysql:// DB_IP: DB_PORT/rpa?useLegacyDatet~~~생략~~~~
wq!

• Change RPA application.properties (single file)

cd /rpa/properties
vi application.properties

• Change the properties below.

ipa.db.server.url= DB_IP
ipa.db.server.port=DB_PORT
wq!

• Change ActiveMQ settings

cd /rpa/pkgs/apache-activemq-5.15.9/conf
vi activemq.xml

• change line 39

<bean id="mysql-ds" class="org.apache.commons.dbcp2.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="org.mariadb.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://DB_IP:DB_PORT/activemq?relaxAutoCommit=true&amp;serverTimezone=GMT&amp;"/>
wq!

• Restart Tomcat, RPA Application, and Active MQ.

Server Restart Procedure

RPA Service and Tomcat are installed as ipaadm, MariaDB is installed as ipadb
It is installed and the installation folder is /rpa as an example.

Server startup should never be done with root authority, and the startup sequence is as follows.

Activation

• MariaDB (run as ipadb account)

su ipadb
cd /rpa/bin
./mysql-start.sh

or,

cd /rpa/pkgs/mariadb/bin
./mysqld_safe --defaults-file=/rpa/pkgs/mariadb/conf/mysqld.conf &

ps -ef|grep mysql

• ActiveMQ (run as ipaadm account)

su ipaadm
cd /rpa/bin
./activemq-run.sh

Or,
cd /rpa/pkgs/apache-activemq-5.15.14/bin
./activemq start

ps -ef|grep activemq

• RPA Service (run as ipaadm account)

- run.sh : There is a procedure to input once again to the y/n confirmation command of “Do you really want to start?”, and if you want to start only a specific service module you want in the RPA service, you can specify it as an argument and start it single.

(example : ./run.sh comm)

. run-y.sh : Immediately start all RPA service modules sequentially.

. ps-rpa.sh : Displays the overall process status whether RPA services are up normally.

cd /rpa/bin
./run.sh
ps -ef|grep rpa

• Tomcat (run as ipaadm account)

cd /rpa/bin
./tomcat-run.sh

Or,

cd /rpa/pkgs/tomcat/bin
./startup.sh

ps -ef|grep tomcat

End

• Tomcat (run as ipaadm account)

cd /rpa/bin
./tomcat-stop.sh

 or
 cd /rpa/pkgs/tomcat/bin
./shutdown.sh

• Check process termination with ps -ef|grep tomcat command

• RPA Service (run as ipaadm account)

cd /rpa/bin
./stop.sh

• Check process termination with ps-rpa.sh script

• ActiveMQ (run as ipaadm account)

cd /rpa/bin
./ activemq-stop.sh

Or,
cd /rpa/pkgs/apache-activemq-5.15.14/bin
./activemq stop

Check process termination with ps -ef|grep activemq command

• MariaDB (run as ipadb account)

are ipadb
cd /rpa/bin
./mysql-stop.sh

or
cd /rpa/pkgs/mariadb/bin
./mysqladmin
-u root -p shutdown

• Enter DB ROOT password: rpago!23

• Check process termination with ps -ef|grep mariadb command

Deleting the RPA solution

Stop the service by following the server restart procedure.

Check process termination with ps -ef|grep tomcat command
Check process termination with ps -ef|grep rpa command
Check process termination with ps -ef|grep activemq command
Check process termination with ps -ef|grep mariadb command

Run as root account in case of permission error

cd /rpa
rm -rf apps
rm -rf bin
rm -rf certificate
rm -rf logs
rm -rf pkgs
rm -rf properties

How to set up my.cnf file when installing DBMS individually

If you installed MariaDB or MySQL separately, you need to change the my.cnf file as follows. File changes must be made prior to installing the RPA solution.

After shutting down the DBMS, refer to the following to change my.cnf. After that, start the DBMS.
It is located in the /etc/mysql/my.cnf path during default installation, and root privileges are required to modify it.

[mysqld]

default_authentication_plugin=mysql_native_password
explicit_defaults_for_timestamp = 1
log_bin_trust_function_creators=1
open_files_limit = 20480
max_connections = 5000
max_allowed_packet = 64M
default-time-zone='+0:00'
innodb-strict-mode=0
collation-server=utf8_general_ci
character-set-server=utf8
sql_mode="STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"

[client]

default-character-set=utf8

Check DBMS character-set

The Brity RPA solution uses utf8 character-set and utf-general-ci COLLATION when using MySQL or MariaDB.

If you installed MariaDB through the installation package, you do not need to check the following.

cd /usr/bin
./mysql -u root -p or ./mysql -port 4406 -u root -p enter
Enter password

mysql> show variables like 'char%';
Confirm that charset is utf8 by executing the command.

mysql> show variables like 'coll%';
Confirm that collation is utf8_general_ci by executing the command.

When DBMS is set to utf8 and utf-general-ci, RPA solution must be installed to avoid COLLATION conflict.
When COLLATION crashes, you need to change the DBMS settings and reinstall the RPA solution.

DB Redundancy Configuration Architecture (Reference)

DB redundancy requires a separate configuration in advance if necessary.

* DB duplication configuration and technical support are not provided when installing Brity RPA Server.


When configuring DB redundancy, please refer to the architecture diagram and link below.

[Note] Red Hat Pacemaker https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_administration/ch-startup-haaa

application/comm. properties Description of each property value

application.properties property description

• Default installation path: /rpa/properties/application.properties

• SERVER BASIC SETTINGS

The RPA server IP/PORT settings currently installed on the server.

• SERVER DETAIL SETTINGS

You can set the OCR server and server token timeout, etc.

• DATABASE SETTINGS

You can configure DB settings for each RPA server.

• KNOX API SETTINGS

Knox api linkage (mail and messenger) related information settings. This property is exclusive to Samsung affiliates. Set the token/account information issued after applying for Knox linkage use.

• LOG LEVEL SETTINGS

Log level settings stored in /rpa/logs.

• SMTP API SETTINGS

This is the setting for SMTP mail linkage.
You can set it in Tenant Portal > Manage Settings (see SMTP Settings).

• CERTIFICATION SETTINGS

Certificate settings for the RPA server. Each server is set with a private certificate provided during installation.
When configuring redundancy, the AP2 server must have a different _02 certificate setting than that of the AP1 server.

• SAAS SERVICE SETTINGS

This is the setting for the Brity RPA SaaS. The customer does not need to modify it separately.

• CUSTOMIZABLE SETTINGS

A property value that can be changed.

comm.properties property description

• Default installation path: /rpa/apps/admin/{admin/tenant/user}/WEB-INF/classes/properties/comm.properties

• SERVER BASIC SETTINGS

Default settings such as language/timezone of the installed RPA portal.

• SERVER DETAIL SETTINGS

You can configure the certificate and queue of the RPA portal, and the RPA server associated with the portal.

• DATABASE SETTINGS

You can configure DB connection settings for RPA portal.

• SAAS SERVICE SETTINGS

This is the setting for the Brity RPA SaaS. The customer does not need to make any separate modifications.
rpa.service.saas should always remain false.

• CUSTOMIZABLE SETTINGS

Portal attribute values ​​that change frequently.

OCR Server Configuration

How to configure the OCR server

• License activation after installing ABBYY SDK

• Add or modify the following property in /rpa/apps/textrecognitionServer/application.properties

abbyy.classification=true //add property
abbyy.enginePath=/opt/ABBYY/FREngine12/Bin //Modify to fit ABBYY SDK path
abbyy.serialNumber=jYRy6SdjWv4RE8pCZTNN //Project ID issued

• Restart

/rpa/apps/textrecognitionServer/run.sh

Property Changes for Portal Linkage

• Add useOcr=Y property to the following files

• N/A

  • /rpa/apps/gateway/application.properties

  • /rpa/apps/admin/admin/WEB-INF/classes/properties/comm.properties

  • /rpa/apps/admin/user/WEB-INF/classes/properties/comm.properties

  • /rpa/apps/admin/tenant/WEB-INF/classes/properties/comm.properties

• Add the following properties to /rpa/properties/application.properties

useOcr=Y
ipa.server.ip.ocr={Installed server IP}
ipa.server.port.ocr={Server port to be installed}

• Restart the gateway server

/rpa/apps/gateway/stop.sh
/rpa/apps/gateway/run.sh

• Restart the portal

/rpa/bin/tomcat-run.sh
/rpa/bin/tomcat-stop.sh

• Enable OCR function by tenant

• N/A

  • After accessing the /tenant portal, manage settings → useOcr Check whether to use by tenant

Server Data Aging Settings

Working with Server Data Aging

• The RPA server can set the data retention period as shown below, and for this purpose, a scheduled data aging task is performed inside the server every day.

• It is also possible to explicitly and manually delete old data using the portal's functionality.

These operations delete unnecessary data in the DB whose retention period has passed. Still, even after the deletion, the physical file's capacity in the DB's corresponding tablespace does not significantly decrease.

For the gradual increase in storage space due to high usage, consider having the server administrator increase the storage accordingly.

To secure additional storage space for data deleted through aging, you can secure more storage space by periodically performing the following MariaDB efficiency operation in a situation with sufficient DB storage space. (recommendation for regular maintenance work, etc.)

Officially, MariaDB provides the OPTIMIZE TABLE function as follows,

https://mariadb.com/kb/en/optimize-table/

As for MariaDB work, each customer has different versions, operators, configuration architectures, etc., and it does not fall within the scope of RPA technical support. Therefore, backup/execution/procedures for optimization work are reviewed by DBA or contracted maintenance company, etc. I hope you do.

• In summary, the ways that RPA server administrators/operators can reduce DB storage can be reviewed as follows.

1) If you do not need to retain up to 30 days of data provided by default, try reducing the retention value. For example, add "scheduler.log.keepdays=30" to the server's application.properties file. After setting the desired retention period number (days), restart the scheduler service.

2) Admins of each tenant check and delete versions that are no longer in use among the old versions of each project accumulated.

3) Perform DB table optimization to secure physical DB capacity by using the time available for PM or service off.

• If online is performed during the service, problems may occur due to the impact of the transaction.

• Locking the table, copying the contents to a new table, deleting the old table, and changing the new table name to the original table. Space must be reserved.

• If a development/stage environment can be tested in advance, run it first and check the required time or impact.

• It is recommended to start with a table with small data size.

• As for execution time, in my previous experience, when I optimized for a table of about 200GB, there was a report that took about 3 hours.

Working with OPTIMIZE TABLE in MariaDB

MariaDB's OPTIMIZE TABLE operation can be reviewed in the following scenario.

The scenarios below do not include data backup/recovery. Please refer to the example scenarios and proceed with the procedure and action plan tailored to the customer company with the DBA. (In RPA technical support, MariaDB maintenance is not supported)

• EXAMPLE OPTIMIZE TABLE WORK SCENARIOS

1) Check DB server disk capacity before operation (by schema): Record it for comparison with After.

cd /data/mariadb/data
du -h --max-depth=1

2) Connect to MariaDB.

cd /rpa/pkgs/mariadb/bin
./mysql -u root -p
Enter password

3) Check the total DB capacity before operation. (perform a query)

SELECT SUM(data_length+index_length)/1024/1024 used_MB, SUM(data_free)/1024/1024 free_MB FROM information_schema.tables;

You can easily check the status of how much you are using and how much DATA_FREE is.
(Record data before/after work to check Before & After)
☞ DATA_FREE: bytes allocated but not used
    The data_free column shows the amount of free space allocated to the table that is not in use.
    OPTIMIZE TABLE might help you free up this space.
    (This will reduce the space occupied by resolving table fragmentation)

4. Select target tables to reduce with OPTIMIZE operation (confirm with a query)

-- Look up the 20 tables that use the most data
select concat(round(data_length/(1024*1024),2),'M') data, t.* from information_schema.TABLES t order by DATA_LENGTH desc limit 20;
 
-- Look up the top 20 tables with the highest amount of DATA_FREE
select concat(round(DATA_FREE/(1024*1024),2),'M') data, t.* from information_schema.TABLES t order by DATA_FREE desc limit 20;

Among the tables with high DATA_FREE, especially tb_asset_file or tb_job_related_file are checked and listed.

5) Perform OPTIMIZE

With enough maintenance time to put down the RPA service, each command is executed sequentially for each selected table.

-- command
OPTIMIZE TABLE tablename

Example) When optimizing the rpa.tb_job_related_file table

MariaDB [(none)]> optimize table rpa.tb_job_related_file;
+-------------------------+----------+----------+-------------------------------------------------------------------+
| Table                   | Op       | Msg_type | Msg_text                                                          |
+-------------------------+----------+----------+-------------------------------------------------------------------+
| rpa.tb_job_related_file | optimize | note     | Table does not support optimize, doing recreate + analyze instead |
| rpa.tb_job_related_file | optimize | status   | OK                                                                |
+-------------------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (0.074 sec)
As a result, the message "Table does not support optimize, doing recreate + analyze instead" is displayed, but optimize is actually applied.

6) When the job is complete, rerun the capacity check query to see how much data has been reduced.

7) Organize the work time and effect, reflect it into the policy of how often the efficiency work will be carried out, and include it in the operation work.

Redundancy - Session timeout issue "Connection Reset by Peer"

• When redundancy is configured, a communication error occurs due to session timeout. In this case, a functional error occurs, such as a mismatch of state information between the server and the bot.

• I get a "Connection Reset by Peer" error in the server log.

• When configuring redundancy, this phenomenon may occur depending on the value set on the equipment, such as L4.idle Timeout.

Solution

1) Check whether the TCP 'RST' flag transmission option can be turned on after L4 session timeout (request confirmation from the person in charge of infrastructure)

2) Added a reconnection attempt option in the server property, but check the session timeout value of L4 and set it to be smaller.

• Properties added to common application.properties (applies to hotfix 3.1.0.0908 or later)

• rpa.httpclient.maxIdleTime=300s

• rpa.httpclient.maxLifeTime=300s

Setting up Job Triggers and Processflow Waitmail Polling Cycles

Job Triggers (Event Server) Setting the waitmail polling cycle

1) Add the following content to the /apps/core/run.sh file. (in ms)
- added "-Devent.polling.queue.interval=600000" (ms 600sec=10 minutes)
- If the value is not set, it will be operated as the default value (every 10 seconds)

2) Restart core for the settings to take effect.
    cd /rpa/apps/core
    ./stop.sh
    ./run.sh

Processflow waitmail PollingSet the Cycle

1) Add the following content to the /apps/workflow/run.sh file. (in seconds)
- Added "-Dworkflow.event.mail.interval=600" (sec 600sec=10 minutes)
- If the value is not set, it will be operated as the default value (1 minute cycle)

2) Restart workflow to apply the settings
    cd /rpa/apps/workflow
    ./stop.sh
    ./run.sh

Summary of questions related to the waitmail polling cycle

1) What is the JobTrigger & Process Flow Server Default?
- Job triggers related to the server default value are 10-second cycles and process flows are 1-minute cycles.


2) Are email job triggers repeated every 10 seconds, and are they executed by the number of job triggers (10)?
- Access the mail server as many registered email jobs per cycle.


3) How often does the JobTrigger access to the Nox API occur internally?
- To be precise, it connects via pop3, not API. One access per account per cycle.
Example) If the cycle is 10 seconds and five email job triggers are registered, the number of connections per minute is [1 * 6 = 5].
Since the cycle is 10 seconds, there is one repetition per minute, and there are six email job triggers, so five connections per cycle ==> 5 * 6 = 5


4) How often does the waitmail card go through the Knox API, including the KNOX login?
- ProcessFlow also attempts to access POP3 to the mail server as many waitmail cards per cycle.

Troubleshooting

Install Runtime Library additionally (libtinfo.so.5)

After initial installation, in some cases, depending on the Linux OS, an additional runtime library called libtinfo.so.5 may be required in relation to the case that MariaDB does not start normally or there is an error in connection. In this case, you need to check the error message and install the required library yourself. If the following error occurs on CentOS or RHEL, you need to install the library below additionally.

libtinfo.so.5: cannot open shared object file: No such file or directory

If you cannot connect to the Internet, install it yourself using the bundled package as follows. (CentOS)

cd /rpa/install/lib/ncurses
sudo rpm -i --nosignature ncurses-compat-libs-6.1-7.20180224.el8.x86_64.rpm

In an environment where an Internet connection is available, installation is possible as follows.

sudo yum install libncurses*

# For Ubuntu, use the following command instead
sudo apt install libncurses5

In some cases, the following error message may occur during the process of creating tables when installing MariaDB in Ubuntu environment.

/rpa/pkgs/mariadb/bin/mysqld: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory

In an environment where internet connection is available, you can install related libraries with the following command.

sudo apt-get install libaio1 libaio-dev

Or,
sudo yum install libaio

View IPv6 is enabled (disable IPv6)

If you use IPv6 and IPv4 at the same time, it may not work properly, so you may need to change it to not use IPv6.

You can check whether IPv6 is enabled on the server using CentOS 8 with the command below.

ip a | grep inet6 (disable IPv6)

If IPv6 setting is enabled, you can check the inet6 keyword as above, and if it is not set, nothing is output.

You can use the sysctl command as a way to disable IPv6 addresses.

To disable IPv6 addresses using the sysctl command, follow the steps below.

1. Use the command below to create a new sysctl configuration file /etc/sysctl.d/70-ipv6.conf.

# sudo vi /etc/sysctl.d/70-ipv6.conf

2. When the file creation is complete, enter the following command and save it.

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

3. Enter the following command to disable IPv6 enabled.

# sudo sysctl --load /etc/sysctl.d/70-ipv6.conf

4. Enter the following command to check whether IPv6 is normally disabled.

# ip a | grep inet6

If there is no result message after entering the above command, you can check that the IPv6 settings are disabled for all network cards. (This setting is maintained even after rebooting)

If the IPv6 keyword is still found after disabling IPv6 using the above sysctl command in CentOS 8, this is because Network Manager is used by default in CentOS 8, so use the nmcli command to process it as follows. You need to reboot after that.

Enter the following command to disable IPv6 enabled state using nmcli command.

# sudo nmcli connection modify interface ipv6.method ignore

Finally, reboot CentOS 8 by entering the command below.

# sudo reboot

You can check whether IPv6 is enabled on the server using Ubuntu using the command below.

# ip a | grep inet6

If IPv6 setting is enabled, you can check the inet6 keyword as above, and if it is not set, nothing is output.

Ubuntu can also use the sysctl command as a way to disable IPv6 addresses.

To disable IPv6 addresses permanently on Ubuntu using the sysctl command, follow the steps below.

(Compared with CentOS 8, the files and contents to be modified are slightly different)

1. Using the command below, the sysctl configuration file, /etc/sysctl. Open conf with vi editor.

# sudo vi /etc/sysctl. conf

2. Append the following commands to the end of the file and save it.

net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1

3. Enter the following command to apply the settings.

# sudo sysctl -p

4. Enter the following command to check whether IPv6 is normally disabled.

# ip a | grep inet6

If there is no result message after entering the above command, you can check that the IPv6 settings are disabled for all network cards. (This setting is maintained even after rebooting)

If IPv6 keyword is still found after disabling IPv6 using the above sysctl command in Ubuntu, the following processing is required to force the OS to read the sysctl configuration file as a kernel parameter.

/etc/rc.local file is created.

# sudo vi /etc/rc.local

Fill in the contents of the /etc/rc.local file as follows and save it.

#!/bin/bash
# /etc/rc.local

/etc/sysctl.d
/etc/init.d/procps restart

exit 0

Reset the execute permission on this file as follows.

# sudo chmod 755 /etc/rc.local

In addition to the above methods, there is a way to disable IPv6 by setting GRUB, but this document does not cover it.

Viewing error log

If you installed the solution under /rpa, you can check the log of each service as shown below.

• Check MariaDB log

cd /rpa/logs/mariadb
tail -f mariadb.err

• Check Tomcat log

cd /rpa/logs/admin
tail -f catalina.out

• Check RPA AP log

cd /rpa/logs

Check the error log under the asset, auth, comm, event, gateway, interface, scheduler, tenant, extrecognitionServer, workflow folder.

Viewing RPA service access

You can check if there is a problem with Linux internal/external connection with the command.

For example, if gateway communication succeeded inside Linux but failed on the bot PC, it can be expected that the firewall of the corresponding port is not open. (You can also check through the web browser, not the Curl command.)

If the connection is normal, the version of the installed service is displayed.

(Example)

• Check gateway communication status (note that it is https)

curl https://182.193.17.236:8777/version --insecure

• Check auth communication status (note that it is http)

curl http://182.193.17.236:9091/auth/version

• check communication status

curl https://182.193.17.236:9001/communication/version -- insecure

See command for confirmation

You can use the commands below to check the for abnormal status.

When mysql command doesn't work

alias mysql=/rpa/pkgs/mariadb/bin/mysql
export PATH=$PATH:/rpa/pkgs/mariadb/bin/

Check the Date

date

SERVICE PORT LISTEN STATUS VIEW

While logged into the ipaadm account

lsof -i -nP | grep LISTEN

Check the operating system bits

getconf LONG_BIT

Check if port is open without telnet

Among the parameters of curl, the telnet scheme is also allowed in the url.

curl -v telnet://ip:port

Example) curl -v telnet://70.70.189.84:22

If it is open, it will appear as Connected to in the result.

Adjusting the size of heap memory for java execution of RPA service

If an error occurs while an OOM (Out Of Memory) error message of "java.lang.OutOfMemoryError: Java heap space" is recorded in the log of a specific RPA service module, increase the heap memory setting related to java execution of the RPA service module.

For example, if an asset-related problem fails to import common resources. In that case, the corresponding service module is core (from version 2.0, three services, including interface, asset, and event, are integrated into core). The execution script to be modified (run .sh) is located in the path below. Open run.sh with vi editors. 

cd /rpa/apps/core
vi run.sh

Find the "-Xms" and "Xmx" items among Java's execution options and modify the memory size higher than the currently set value. For reference, the above options are defined as follows.

-Xms : Initial heap size setting 
-Xmx : Set the maximum heap size

If you give an example of extending what is initially set to 1G to 2G.

Before change: -Xms1G -Xmx1G
After change: -Xms2G -Xmx2G

Save the modifications and restart the service module to take effect immediately.

As the memory usage rate of the service module will increase as the memory option value increases, check the memory usage by using the top or free command below on the server in advance, and proceed when there is enough physical memory. If there is insufficient memory, applying it after adding physical memory first is recommended.

Checking memory status with top command (Linux server)

It shows various usage status of Linux server. (When running without options, the screen refreshes every 3 seconds)

The screen is updated every second using the delay option below, and if you press Shift + m together on the screen, the processes with the highest memory usage are sorted and displayed first.

top -d 1

Execute the /rpa/bin/ps-rpa.sh command to check the starting (-Xms) maximum (-Xmx) memory setting.

Compare the PID of the top command to check that the physical memory (RES) value does not exceed the heap memory. If the heap memory is insufficient, increase the maximum memory Xmx value.

ipaadm 338 1 1 12:34 tty1 00:01:01 java -jar -DDEV_HOME=/rpa/logs -Dspring.config.location=/rpa/properties/application.properties,classpath:/application.properties -Xms2G -Xmx2G rpa_auth.jar
ipaadm 379 1 1 12:34 tty1 00:00:52 java -jar -DDEV_HOME=/rpa/logs -Xms1G -Xmx1G -Dspring.config.location=/rpa/properties/application.properties,classpath:/application.properties rpago_api_gateway.jar

To configure Java heap memory, modify run.sh of each service and restart it.

vi /rpa/apps/auth/run.sh
vi /rpa/apps/gateway/run.sh
vi /rpa/apps/tenant/run.sh
vi /rpa/apps/core/run.sh
vi /rpa/apps/comm/run.sh
vi /rpa/apps/scheduler/run.sh
vi /rpa/apps/workflow/run.sh
vi /rpa/apps/textrecognitionServer/run.sh

If there is available memory, Linux uses it for cache and other purposes to make the most of it and returns it when there is a memory request.

The actual memory used is (used - buff/cache), and the avail Mem value is the available memory.

It is not a situation where memory is insufficient simply because there are few free Mem, and if there are a lot of swaps used, performance degradation occurs.

To be considered a normal situation, the amount of (free + buff/cache) must be at least 20% of the total memory (Mem total). If this value does not exceed 20% and swap usage occurs frequently, physical memory expansion is considered necessary.

Using OutOfMemory check script in the error log

The method to check whether OutOfMemory has occurred in the server log file is as follows.

Extract and check the OutOfMemory log for the entire error log

1) Prepare the script file
Place the oom-check.sh file in /rpa/bin/oom-check.sh .

2) Run the script
cd /rpa/bin
./oom-check.sh

3) Run the script repeatedly
./oom-check.sh -t

Check memory status and take action

1) Check the result
 If OutOfMemory is found in the /rpa/logs/*/error.log file, the contents of the file are displayed on the screen. 
If there is no result, nothing is displayed.

Displays the Heap Memory status of RPA services.
You can check the number and duration of Minor GC and Full GC executions.

2) RPA Service Memory Check option
ETIME : Process Elapsed Time
YOUNG : Young Area Heap Size (GB)
YGC : Minor GC Count
YGCAT : Minor GC Average Time (sec)
OLD   : Old Area Heap Size (GB)
FGC   : Full GC Count
FGCAT : Full GC Average Time(sec)

3) Memory stabilization conditions
YGCAT : Around 0.05 sec
FGCAT: Around 1 sec

• Download oom-check.sh

oom-check.sh

• oom-check.sh source content

#!/bin/bash
grep -r 'OutOfMemory' ../logs/*/error.log

#Omitted below

Tomcat Log Time KST

Tomcat's logs are logged using a mixture of KST and UTC, as shown below. (Basic installation status)
Files to be changed
3 logback .xml configuration files in use by each web application
☞ location
/rpa/apps/admin/admin/WEB-INF/classes/logback.xml
/rpa/apps/admin/user/WEB-INF/classes/logback.xml
/rpa/apps/admin/tenant/WEB-INF/classes/logback.xml

<configuration>
    <!-- Console Log -->
    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>
                [%-5p] [Thread Id=%t] [%date{"yyyy-MM-dd HH:mm:ss.SSSZ"}{KST}] %13F:%04L %m%n
            </Pattern>
        </layout>
    </appender>

Server Operation Guide

OS Parameter Application Guide

This section shows recommended values for OS kernel parameters that manage and control Linux systems. 
There are differences in setting methods depending on the OS, which is not directly controlled by the solution. 
Therefore, server operators are advised to use this guide to apply recommended values for kernel parameters.

Factors affecting the solution include:

NetWork Performance Tuning
File System Performance Tuning

Setting

OS kernel parameters are values that manage and control the Linux system, and you can optimize the system by setting kernel parameter values.

The primary setup method is as follows. 
- Just set the value in the file using the vi /etc/systl.conf command. (Applicable permanently) 
- You can also control the values of kernel variables with the sysctl command.

Detailed setup method

The detailed setting method is as follows.

1) How to query setting values
  - sysctl [kernel parameter]
  ex) # sysctl net.ipv4.tcp_max_syn_backlog

2) How to apply immediately
   - sysctl -w [kernel parameter]=[value to set]
    ex) # sysctl -w net.ipv4.tcp_max_syn_backlog=1024

3) On permanent setting
    # vi /etc/sysctl.conf
    net.ipv4.tcp_max_syn_backlog=1024 
    net.core.somaxconn=1024 
    net.ipv4.tcp_fin_timeout=60      
    net.ipv4.tcp_keepalive_intvl=75    
    net.ipv4.tcp_keepalive_probes=9    
    net.ipv4.tcp_keepalive_time=7200    
    net.ipv4.tcp_syn_retries=6
    net.ipv4.tcp_retries2=15

4)  How to apply after setting
    # Apply with the /sbin/sysctl -p command.

Recommended settings for each OS parameter item

Resource parameter recommended settings

WAS Recommended Setpoint Application Guide

This section provides recommended guides for Tomcat and Spring Boot settings required to run the solution. Most recommended values are applied during installation. Operators must be aware of the settings and be able to adjust and manage them when necessary. Additionally, you must be able to check essential logs and setup methods for analysis when an operational issue occurs.

Common Application Guide

Tomcat Application Guide

Spring Boot Application Guide

Accesslog Settings Activation Guide

This section provides a guide to activating Accesslog settings. Accesslog is essential for recording client requests, diagnosing problems, monitoring, and performance tuning, so it is recommended to set it up as it can be used to identify the cause when a failure occurs.

Adding Access Log to a gateway service

1) Add parameters to run.sh in the gateway service folder (e.g. /rpa/apps/gateway)
Add -Dlogging.config=./logback.xml -Dreactor.netty.http.server.accessLogEnabled=true

2) Upload the attached logback.xml file to the gateway service folder (e.g. /rpa/apps/gateway)
attachment: logback.xml

3) Restart the gateway service and check the /rpa/logs/gateway/access log.log file.

Adding Access Log to auth, batch, comm, core, scheduler, tenant, textrecognitionServer, workflow service

1) Access Log Create application.properties in the service folder you want to add (e.g. /rpa/apps/auth), and add the contents if it already exists.
Change the directory (server.tomcat.accesslog.directory) to suit each service.

2) Add apllication.properties to run.sh execution shell

3) Restart the service and check if /rpa/logs/service name/access-log is created.

Guide to periodic deletion of log files

This section provides a guide to deleting logs that accumulate periodically. Since log files that are continuously accumulating may cause a shortage of disk space, you must check the log directory and apply a deletion method. It is recommended that detailed operation plans, such as deletion cycle and backup settings, be applied according to each company's policy and environment.

Create a shell script to automatically delete logs

Example

1) Files modified for more than 11 days will be deleted. Open the file and write as below.
> cd /rpa/tools
> vi log_del.sh

#!/bin/sh
find /rpa/logs/*/* -mtime +10 -delete
find /rpa/logs/*/archived/*.* -mtime +10 -delete
find /rpa/logs/admin/*.* -mtime +10 -delete
cp /rpa/logs/admin/catalina.out /rpa/logs/admin/catalina_$(date +%F_%H-%M-%S).log
cat /dev/null > /rpa/logs/admin/catalina.out

2) Save and exit (:wq)

3) Give execution permission
> chmod 755 log_del.sh

Register schedule (crontab), root privileges

Example

1) Registered in crontab, executes delete shell every day at 00:30.
>crontab -e
30 00 * * * /home/rpco/tools/log_del.sh

2) Save and exit (:wq)

Portal Instance Separation Guide (User, Admin, Tenant Separation)

This section explains how to detach a Brity RPA Orchestrator Portal instance.
Brity RPA has user, admin, and tenant portals for each user's permission.
The three portals are installed to run as one service.
This section will explain how to separate the three portals and the pros and cons.

Advantages and Disadvantages

Operating the three portals separately in each instance has advantages and disadvantages and can be set based on the customer's policy.

Advantage: Minimizes impact in case of failure through separation of portal instances
Disadvantage: Increase in server resource usage such as CPU/Memory and management targets due to starting instances individually

Command Execution Guide

It is not provided separately in the basic installation package and can be manually set separately as follows:

Portal Instance Detach Command

The instance detachment is performed through the command below.

1) Existing folders will be backed up.
tomcat folderbackup /home/rpa/pkgs/tomcat_bak
admin folderbackup /home/rpa/apps/admin_bak
Tomcat Runshell /home/rpa/bin/tomcat-run.sh.bak
Tomcat shutdownshell /home/rpa/bin/tomcat-stop.sh.bak

2) Perform commands
The commands below are based on the RPA path /home/rpa; please change your installation path.

cd /home/rpa/apps
cp -R admin admin_bak
cd /home/rpa/pkgs
cp -R tomcat tomcat_bak
mkdir tomcat/tomcat_admin
mkdir tomcat/tomcat_user
mkdir tomcat/tomcat_tenant
cp -R tomcat_bak/* tomcat/tomcat_admin
cp -R tomcat_bak/* tomcat/tomcat_user
cp -R tomcat_bak/* tomcat/tomcat_tenant
cd /home/rpa/apps
mkdir portal_admin
mkdir portal_user
mkdir portal_tenant
ln -s /home/rpa/apps/admin/user /home/rpa/apps/portal_user/user
ln -s /home/rpa/apps/admin/admin /home/rpa/apps/portal_admin/admin
ln -s /home/rpa/apps/admin/tenant /home/rpa/apps/portal_tenant/tenant
f=server.xml
cd /home/rpa/pkgs/tomcat/tomcat_user/conf
sed -i 's/8080/8080/g' $f
sed -i 's/8009/8009/g' $f
sed -i 's/8443/8443/g' $f
sed -i 's/8005/8005/g' $f
sed -i 's/\/apps\/admin/\/apps\/portal_user/g' $f
sed -i 's/\/logs\/admin/\/logs\/admin\/user/g' $f
sed -i 's/\/logs\/admin/\/logs\/admin\/user/g' logging.properties
cd /home/rpa/pkgs/tomcat/tomcat_admin/conf
sed -i 's/8080/8090/g' $f
sed -i 's/8009/8010/g' $f
sed -i 's/8443/8444/g' $f
sed -i 's/8005/8006/g' $f
sed -i 's/\/apps\/admin/\/apps\/portal_admin/g' $f
sed -i 's/\/logs\/admin/\/logs\/admin\/admin/g' $f
sed -i 's/\/logs\/admin/\/logs\/admin\/admin/g' logging.properties
cd /home/rpa/pkgs/tomcat/tomcat_tenant/conf
sed -i 's/8080/8070/g' $f
sed -i 's/8009/8011/g' $f
sed -i 's/8443/8445/g' $f
sed -i 's/8005/8007/g' $f
sed -i 's/\/apps\/admin/\/apps\/portal_tenant/g' $f
sed -i 's/\/logs\/admin/\/logs\/admin\/tenant/g' $f
sed -i 's/\/logs\/admin/\/logs\/admin\/tenant/g' logging.properties
f=comm.properties
cd /home/rpa/apps/admin/user/WEB-INF/classes/properties
sed -i 's/\/user\/auth/\/tenant\/auth/g' $f
sed -i 's/\/admin\/admin/\/portal_user\/user/g' $f
sed -i 's/8080\/user/8080\/user/g' $f
sed -i 's/8080\/admin/8090\/admin/g' $f
sed -i 's/8080\/tenant/8070\/tenant/g' $f
sed -i 's/portalSSO=true/portalSSO=false/g' $f
cd /home/rpa/apps/admin/admin/WEB-INF/classes/properties
sed -i 's/\/user\/auth/\/admin\/auth/g' $f
sed -i 's/\/admin\/admin/\/portal_admin\/admin/g' $f
sed -i 's/8080\/user/8080\/user/g' $f
sed -i 's/8080\/admin/8090\/admin/g' $f
sed -i 's/8080\/tenant/8070\/tenant/g' $f
sed -i 's/portalSSO=true/portalSSO=false/g' $f
cd /home/rpa/apps/admin/tenant/WEB-INF/classes/properties
sed -i 's/\/user\/auth/\/tenant\/auth/g' $f
sed -i 's/\/admin\/admin/\/portal_tenant\/tenant/g' $f
sed -i 's/8080\/user/8080\/user/g' $f
sed -i 's/8080\/admin/8090\/admin/g' $f
sed -i 's/8080\/tenant/8070\/tenant/g' $f
sed -i 's/portalSSO=true/portalSSO=false/g' $f
cd /home/rpa/bin
cp tomcat-run.sh tomcat-run.sh.bak
sed -i 's/.\/startup.sh/#.\/startup.sh/g' tomcat-run.sh
sed -i 's/cd /#cd /g' tomcat-run.sh
sed -i 's/sudo -u/sudo /g' tomcat-run.sh
f=tomcat-run.sh
sed -i '19 i cd /home/rpa/pkgs/tomcat/tomcat_user/bin' $f
sed -i '20 i ./startup.sh' $f
sed -i '21 i cd /home/rpa/pkgs/tomcat/tomcat_admin/bin' $f
sed -i '22 i ./startup.sh' $f
sed -i '23 i cd /home/rpa/pkgs/tomcat/tomcat_tenant/bin' $f
sed -i '24 i ./startup.sh' $f
sed -i '12 i cd /home/rpa/pkgs/tomcat/tomcat_user/bin' $f
sed -i '13 i ./startup.sh' $f
sed -i '14 i cd /home/rpa/pkgs/tomcat/tomcat_admin/bin' $f
sed -i '15 i ./startup.sh' $f
sed -i '16 i cd /home/rpa/pkgs/tomcat/tomcat_tenant/bin' $f
sed -i '17 i ./startup.sh' $f
cp tomcat-stop.sh tomcat-stop.sh.bak
sed -i 's/.\/shutdown.sh/#.\/shutdown.sh/g' tomcat-stop.sh
sed -i 's/cd /#cd /g' tomcat-stop.sh
f=tomcat-stop.sh
sed -i '20 i cd /home/rpa/pkgs/tomcat/tomcat_user/bin' $f
sed -i '21 i ./shutdown.sh' $f
sed -i '22 i cd /home/rpa/pkgs/tomcat/tomcat_admin/bin' $f
sed -i '23 i ./shutdown.sh' $f
sed -i '24 i cd /home/rpa/pkgs/tomcat/tomcat_tenant/bin' $f
sed -i '25 i ./shutdown.sh' $f

Changes and start-up procedures

1. Change the port according to the separation of instances
- user portal https://ip:8080/user => https://ip:8080/user
- admin portal https://ip:8080/admin => https://ip:8090/admin
- tenant portal https://ip:8080/tenant => https://ip:8070/tenant
* An additional firewall needs to be opened for added webports 8090 and 8070

2. Web application source folder symbolic link
- user portal /home/rpa/apps/admin/user => /home/rpa/apps/portal_user/user
- admin portal /home/rpa/apps/admin/admin => /home/rpa/apps/portal_admin/admin
- tenant portal /home/rpa/apps/admin/tenant => /home/rpa/apps/portal_tenant/tenant

3. Detach the Tomcat instance
- Run with one existing instance /home/rpa/pkgs/tomcat/bin/startup.sh
- Can be run individually with three instances of change
=> user portal operation /home/rpa/pkgs/tomcat/user/bin/startup.sh
=> user portal exit /home/rpa/pkgs/tomcat/user/bin/shutdown.sh
=> admin portal operation /home/rpa/pkgs/tomcat/admin/bin/startup.sh
=> admin portal exit /home/rpa/pkgs/tomcat/admin/bin/shutdown.sh
=> tenant portal operation /home/rpa/pkgs/tomcat/tenant/bin/startup.sh
=> tenant portal exit /home/rpa/pkgs/tomcat/tenant/bin/shutdown.sh

4. Starting Tomcat
- Tomcat starts up to three instantaneous, simultaneous starts via /home/rpa/bin/tomcat-run.sh as before.

5. Shutting down Tomcat
- Tomcat shutdown is configured to terminate three instantaneously via /home/rpa/bin/tomcat-stop.sh as before.

Setting up a Suit

After setting, the original method is as follows. Please be careful when deleting.

cd /home/rpa/apps
rm -rf portal_admin
rm -rf portal_tenant
rm -rf portal_user
cp /home/rpa/apps/admin_bak/user/WEB-INF/classes/properties/comm.properties /home/rpa/apps/admin/user/WEB-INF/classes/properties/comm.properties
cp /home/rpa/apps/admin_bak/user/WEB-INF/classes/properties/comm.properties /home/rpa/apps/admin/user/WEB-INF/classes/properties/comm.properties
cp /home/rpa/apps/admin_bak/user/WEB-INF/classes/properties/comm.properties /home/rpa/apps/admin/user/WEB-INF/classes/properties/comm.properties
cd /home/rpa/pkgs/tomcat
rm -rf tomcat_admin
rm -rf tomcat_user
rm -rf tomcat_tenant
cd /home/rpa/bin
rm tomcat-run.sh tomcat-stop.sh
mv tomcat-run.sh.bak tomcat-run.sh
mv tomcat-stop.sh.bak tomcat-stop.sh

Server Resource Expansion Guide

When installing an orchestrator, the solution will give you basic recommended specifications. However, as the period of use increases, the total usage increases, such as the number of server users, bots, jobs performed, and scheduled tasks, creating a limit that can be used with the initial server resources. In this case, the operator should consider increasing server resources. Operators should consider the infrastructure configuration from the initial installation to consider the increase in users and bots.
This section provides a baseline guide for Brity RPA orchestrators to increase server resources.

Comparison of resource expansion plans

• Scale-up

1) Advantages
Capacity can be increased without additional network connectivity.
It has fewer management costs and operational issues than the scale-out method, and it can be applied relatively easily by raising the specifications of server resources.

2) Disadvantages
Performance improvements and load balancing are limited.
Due to the significant burden on one server, there is no backup plan in the event of a failure.
If you replace an existing server to improve performance, you will experience service downtime.

• Scale-out

1) Advantages
Multiple nodes can distribute the load to provide smooth service.
A backup plan can be established in the event of a failure, and flexible resource operation is possible.


2) Disadvantages
As the number of servers increases, server management becomes more complex, and initial costs, such as securing infrastructure such as network equipment, are involved when introducing the initial architecture.
As the node expands, it may be challenging to identify the cause of various failures, such as L4, which occur when a problem occurs.
If there are fewer users or the need for expansion is eliminated, the use of the resources that have been secured may be compromised.

• Comparing the advantages and disadvantages of scale-up and scale-out

Scale-up expansion criteria

1) The CPU usage of the server may vary depending on the throughput or workload, and in general, if the CPU usage is more than 70% on average, consider whether to increase it.

2) Memory usage depends on the amount of concurrent execution of the project, but if the memory usage is more than 80% on average, you should consider adding it. Detailed memory usage is Memory calculation criteria. Note:

3) The exact criteria may vary depending on the purpose of the server, the nature of the work, etc. Therefore, it should be judged according to the customer's agreed procedures and criteria for expanding the server.

4) Other disk expansion standards are found in the following installation manual. Disk capacity calculation criteria Note:

Scale-out expansion criteria

1) The redundancy configuration must be applied by default to consider scaling out.

2) One node considers expansion based on the server resources of the recommended installation specifications.
Detailed recommended specifications are available in the installation manual. Note:

3) Generally, when the number of bots operating at 80% or more increases to 100, it is presented as a standard for adding nodes for load balancing.

4) Threshold settings for each node can be set in the resource monitoring function as follows.

Server Configurations

Server Diagram

Default installation path: /RPA

Redundant server diagram

RPA servers can configure duplication depending on the switch configuration.
※ Separately configure hardware (L4/L7) or software reverse proxy (nginx) switch. We have not separately explained how to configure RDBMS for redundancy regardless of the application.

Service Health Check

1. Call the Health Check API

Use the API call format and module name below to call the API locally. (Method: GET) 

For example, http://127.0.0.1:9091/auth/version

Exceptions)
 •Gateway, Comm services must use HTTPS
 •gateway is used without module name (https:// server address: 8777/version)

2. Call Result

When the service functions normally, calling the API will display version information.
RPA Portal displays the HTML of the login page.

Log location and file management

1. Module-specific log location
/{Installation Location}/logs/{Module Name}/server.log #Full log
/{Installation Location}/logs/{Module Name}/error.log #Error log
   Ex) /rpa/logs/auth/serverlog

2. Keep track of errors.
Refer to the module relationship diagram between functions to check the relevant logs first.

3. You should manage your error files. The default log storage is set to 1 day, and the maximum file size is 10 MB. A new log file is automatically created when the maximum size is exceeded. Periodically, you should delete log files you don't need to free up server space.