Skip to content

(SUP-3769) Update Readme for new release #113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 7, 2022
Merged

(SUP-3769) Update Readme for new release #113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8ef76de
(SUP-3769) Update Readme for new release
MartyEwings Nov 7, 2022
0b597b2
(SUP-3769)Update Metadata
MartyEwings Nov 7, 2022
61df45e
(SUP-3769) Update Readme for new release
MartyEwings Nov 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 36 additions & 89 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,78 +6,50 @@ Table of Contents
- [What does this module provide?](#what-does-this-module-provide)
- [Usage](#usage)
- [Items you may want to configure](#items-you-may-want-to-configure)
- [Disable Maintenance](#disable-maintenance)
- [General PostgreSQL Recommendations](#general-postgresql-recommendations)
- [Tuning](#tuning)
- [Maintenance](#maintenance)
- [PostgreSQL Settings](#postgresql-settings)
- [maintenance_work_mem](#maintenance_work_mem)
- [work_mem](#work_mem)
- [autovacuum_work_mem](#autovacuum_work_mem)
- [autovacuum_max_workers](#autovacuum_max_workers)
- [checkpoint_segments and checkpoint_completion_target](#checkpoint_segments-and-checkpoint_completion_target)
- [How to Report an issue or contribute to the module](#how-to-report-an-issue-or-contribute-to-the-module)
- [Maintenance](#maintenance)
- [Disable Maintenance](#disable-maintenance)
- [Deprecated functionality](#deprecated-functionality)
- [Backups](#backups)
- [PE PostgreSQL Tuning](#pe-postgresql-tuning)
- [Supporting Content](#supporting-content)
- [Articles](#articles)
- [Videos](#videos)
- [How to Report an issue or contribute to the module](#how-to-report-an-issue-or-contribute-to-the-module)

# Overview

This module provides tuning, and maintenance for PE PostgreSQL.

## What does this module provide?

This module provides the following functionaility

1. Customized settings for PE PostgreSQL
1. Maintenance to keep the `pe-puppetdb` database lean and fast
This module provides maintenance tasks to keep the `pe-puppetdb` database lean and fast

## Usage

In order to use this module, classify the node running PE PostgreSQL with the `pe_databases` class.
The Primary Server and Replica run PE PostgreSQL in most instances, but there may be one or more servers with this role in an XL deployment

To classify via the PE Console, create a new node group called "PE Database Maintenance".
Then pin the node(s) running pe-postgresql to that node group.
It is not recommended to classify using a pre-existing node group in the PE Console.

## Items you may want to configure


> WARNING: The backup functionality in this module has been removed.
Please refer to the [PE Backup and Restore documentation](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html) for details on how to backup.
You should ensure the parameter `pe_databases::manage_database_backups` and any parameters from the `pe_databases::backup` class are removed from classification or hiera.
You should also clean up associated crontab entries.

### Disable Maintenance
This module is bundled with all currently support versions of Puppet Enterprise, and enabled by default. When with Puppet Enterprise, see the following documentation for instructions on how to enable or disable.

The maintenance systemd timers will perform a `pg_repack` on various `pe-puppetdb` tables to keep them lean and fast.
pg_repack is a non blocking operation and should have no impact on the operations of Puppet Enterprise, however, if for some reason you experience issues you can disable the maintenance systemd timers.
You can do so by setting `pe_databases::disable_maintenance: true` in your hieradata.
https://puppet.com/docs/pe/latest/pe_database_maintenance.html#enable_pe_database_module


# General PostgreSQL Recommendations

## Tuning

Under normal operating conditions, there is very little tuning needed for PE PostgreSQL.
## Items you may want to configure

If you are using a Monolithic installation of Puppet Enterprise then the default settings are well-tuned.
If you are using a dedicated node for PE PostgreSQL, then some of the settings can be tuned upwards, but likely with little difference in performance.

This module attempts to provide default settings that accommodate both a Monolithic install and a dedicated PE PostgreSQL instance.
Those defaults change based on the `$all_in_one_pe` parameter.
### Maintenance

## Maintenance
This module provides systemd timers to pg_repack tables in the `pe-puppetdb` database. These times are configurable using the corresponding parameters, but default to:

This module provides systemd timers to pg_repack tables in the `pe-puppetdb` database:
- facts tables are pg_repack'd Tuesdays and Saturdays at 4:30AM
- catalogs tables are pg_repack'd Sundays and Thursdays at 4:30AM
- reports table is pg_repack'd on the 10th of the month at 05:30AM on systems with PE 2019.7.0 or less
- resource_events table is pg_repack'd on the 15th of the month at 05:30AM on systems with PE 2019.3.0 or less
- other tables are pg_repack'd on the 20th of the month at 5:30AM

> Note: You may be able to improve the performance (reduce time to completion) of maintenance tasks by increasing the [maintenance_work_mem](#maintenance_work_mem) setting.
Classify the following parameters to change the values:
```
$facts_tables_repack_timer = 'Tue,Sat *-*-* 04:30:00',
$catalogs_tables_repack_timer = 'Sun,Thu *-*-* 04:30:00',
$other_tables_repack_timer = '*-*-20 05:30:00',
$reports_tables_repack_timer = '*-*-10 05:30:00',
$resource_events_tables_repack_timer = '*-*-15 05:30:00',
```


Please note that when using `pg_repack` as part of the pe_databases module, unclean exits can leave behind the schema when otherwise it should have been cleaned up. This can result in the messages similar to the following:
Expand All @@ -91,56 +63,28 @@ DETAIL: The trigger was probably installed during a previous attempt to run pg_r
The module now contains a task `reset_pgrepack_schema` to mitigate this issue. This needs to be run against your Primary or PE-postgreSQL server to resolve this and it will drop and recreate the extension, removing the temporary objects.


## PostgreSQL Settings
#### Disable Maintenance

### [maintenance_work_mem](https://www.postgresql.org/docs/11/runtime-config-resource.html)

You can improve the speed of vacuuming, reindexing, and backups by increasing this setting.
Consider a table that is 1GB.
If `maintenance_work_mem` is `256MB`, then to perform operations on the table a quarter of it will be read into memory, operated on, then written out to disk, and then that will repeat three more times.
If `maintenance_work_mem` is `1GB` then the entire table can be read into memory, operated on, then written out to disk.

Note: that each autovacuum worker can use up to this much memory, if you do not set [autovacuum_work_mem](https://www.postgresql.org/docs/11/runtime-config-resource.html) as well.

### [work_mem](https://www.postgresql.org/docs/11/runtime-config-resource.html)

Puppet Enterprise ships with a default `work_mem` of `4MB`.
For most installations, this is all that is needed, however, at a larger scale you may need to increase to `8MB` or `16MB`.
One way to know if you need more `work_mem` is to change the `log_temp_files` setting to 0 which will log all temporary files created by PostgreSQL.
When you see a lot of temporary files being logged over the `work_mem` threshold then it’s time to consider increasing `work_mem`, however, you should first run a `REINDEX` and `VACUUM ANALYZE` to see if that reduces the number of temporary files being logged.

Another way to determine the need for an increase in `work_mem` is to get the query plan from a slow running query (accomplished by adding `EXPLAIN ANALYZE` to the beginning of the query).
Query plans that have something like `Sort Method: external merge Disk` in them indicate a possible need for for more `work_mem`.

This is discussed on the [Tuning Your PostgreSQL Server Wiki](https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server#work_mem)

### [autovacuum_work_mem](https://www.postgresql.org/docs/11/runtime-config-resource.html)

This setting is essentially `maintenance_work_mem` but for autovacuum processes only.
Usually you will set `maintenance_work_mem` higher and this lower, since `autovacuum_work_mem` is used by `autovacuum_max_workers` number of autovacuum processes.

### [autovacuum_max_workers](https://www.postgresql.org/docs/11/runtime-config-autovacuum.html)
The maintenance systemd timers will perform a `pg_repack` on various `pe-puppetdb` tables to keep them lean and fast.
pg_repack is a non blocking operation and should have no impact on the operations of Puppet Enterprise, however, if for some reason you experience issues you can disable the maintenance systemd timers.
You can do so by setting `pe_databases::disable_maintenance: true` in your hieradata.

The larger your database the more autovacuum workers you may need.
The default of `3` is reasonable for most small or medium installations of Puppet Enterprise.
When you’re tracking the size of your database tables and indexes and you notice some of them seem to keep getting larger then you might need more autovacuum workers.

If you’ve installed PE PostgreSQL on its own node, then we recommend `CPU / 2` as a default for this setting (with a maximum of 8).
For a Monolithic installation, increasing this setting means you likely need to compensate by reducing other settings that may cause your CPU to be over-subscribed during a peak.
Those settings would be PuppetDB Command Processing Threads and Puppet Server JRuby Instances.

### [checkpoint_segments and checkpoint_completion_target](https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server#checkpoint_segments_checkpoint_completion_target)

We suggest a middle ground of `128` for `checkpoint_segments` and `.9` for `checkpoint_completion_target`.
As mentioned in the PostgreSQL Wiki, the larger value you use for `checkpoint_segments` affords you better performance but you sacrifice in potential recovery time.
## Deprecated functionality

If you see messages like “LOG: checkpoints are occurring too frequently (xx seconds apart)” then you definitely want to increase your `checkpoint_segments`.

## How to Report an issue or contribute to the module
### Backups
> WARNING: The backup functionality in this module has been removed.
Please refer to the [PE Backup and Restore documentation](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html) for details on how to backup.
You should ensure the parameter `pe_databases::manage_database_backups` and any parameters from the `pe_databases::backup` class are removed from classification or hiera.
You should also clean up associated crontab entries.

If you are a PE user and need support using this module or are encountering issues, our Support team would be happy to help you resolve your issue and help reproduce any bugs. Just raise a ticket on the [support portal](https://support.puppet.com/hc/en-us/requests/new).
If you have a reproducible bug or are a community user you can raise it directly on the Github issues page of the module [puppetlabs/puppetlabs-pe_databases](https://github.com/puppetlabs/puppetlabs-pe_databases/issues). We also welcome PR contributions to improve the module. Please see further details about contributing [here](https://puppet.com/docs/puppet/7.5/contributing.html#contributing_changes_to_module_repositories)
### PE PostgreSQL Tuning

Recent versions of PE PostgreSQL included in all supported versions of Puppet Enterprise have superseded or improved upon the parameters previously tuned by this module, as such they are no longer required.
Any classifications of the parameters relating to this functionality should be removed, however if set will harmlessly omit a warning of this deprecation.

---

Expand All @@ -160,7 +104,10 @@ This Module has the following specific Article(s) available:

The [Support Video Playlist](https://youtube.com/playlist?list=PLV86BgbREluWKzzvVulR74HZzMl6SCh3S) is a resource of content generated by the support team

## How to Report an issue or contribute to the module

If you are a PE user and need support using this module or are encountering issues, our Support team would be happy to help you resolve your issue and help reproduce any bugs. Just raise a ticket on the [support portal](https://support.puppet.com/hc/en-us/requests/new).
If you have a reproducible bug or are a community user you can raise it directly on the Github issues page of the module [puppetlabs/puppetlabs-pe_databases](https://github.com/puppetlabs/puppetlabs-pe_databases/issues). We also welcome PR contributions to improve the module. Please see further details about contributing [here](https://puppet.com/docs/puppet/7.5/contributing.html#contributing_changes_to_module_repositories)


---
Expand Down
2 changes: 1 addition & 1 deletion metadata.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"name": "puppetlabs-pe_databases",
"version": "2.3.0",
"author": "puppetlabs",
"summary": "A Puppet Module for Backing Up / Maintaining / Tuning Your Puppet Enterprise Databases",
"summary": "A Puppet Module for Maintaining Your Puppet Enterprise Databases",
"license": "Apache-2.0",
"source": "https://github.com/puppetlabs/puppetlabs-pe_databases",
"project_page": "https://github.com/puppetlabs/puppetlabs-pe_databases",
Expand Down