Scaling Postgres

In this episode of Scaling Postgres, we discuss automatic indexing, function pipelines, with hold cursors and the different query scans. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.depesz.com/2021/10/22/why-is-it-hard-to-automatically-suggest-what-index-to-create/ https://blog.timescale.com/blog/function-pipelines-building-functional-programming-into-postgresql-using-custom-operators/ https://www.cybertec-postgresql.com/en/with-hold-cursors-and-transactions-in-postgresql/ https://arctype.com/blog/postgres-explain-analyze-scan/ https://postgres.ai/blog/20211018-postgresql-lock-trees https://blog.crunchydata.com/blog/postgres-queries-for-projecting-monthly-revenue-run-rate https://about.sourcegraph.com/blog/postgres-text-search-balancing-query-time-and-relevancy/ https://elephanttamer.net/?p=68 https://aws.amazon.com/blogs/database/upgrade-amazon-rds-and-amazon-aurora-postgresql-version-9-6/ https://postgresql.life/pos

In this episode of Scaling Postgres, we discuss one method to shard a database, federating your database, the benefits of configuring linux HugePages and performance tuning. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.notion.so/blog/sharding-postgres-at-notion https://blog.crunchydata.com/blog/multi-cluster-postgres-federation-on-kubernetes-with-postgres-14-using-gitops https://www.percona.com/blog/why-linux-hugepages-are-super-important-for-database-servers-a-case-with-postgresql/ https://www.enterprisedb.com/blog/webinar-recap-tuning-tips-maximize-postgres-performance https://aws.amazon.com/blogs/database/tune-sorting-operations-in-postgresql-with-work_mem/ https://www.cybertec-postgresql.com/en/show-tables-in-postgresql-whats-wrong-with-it/ https://blog.rustprooflabs.com/2021/10/postgres-14-query-id https://www.highgo.ca/2021/10/15/backup-label-in-postgresql/ https://www.postgresonline.com/journal/index.php?/archives/401-Multirange

In this episode of Scaling Postgres, we discuss select for update, the pgx framework to generate extensions, cool additions to Postgres 14 and full-text search. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.migops.com/blog/2021/10/05/select-for-update-and-its-behavior-with-foreign-keys-in-postgresql/ https://blog.rustprooflabs.com/2021/10/pgdd-extension-using-pgx-rust https://www.enterprisedb.com/blog/cool-new-contributions-postgresql-14 https://arctype.com/blog/postgres-full-text-search/ https://www.enterprisedb.com/postgresql-database-backup-recovery-what-works-wal-pitr https://yugonagata-pgsql.blogspot.com/2021/10/transition-tables-in-imcremental-view.html https://www.enterprisedb.com/blog/how-set-kerberos-authentication-using-active-directory-postgresql-database https://www.cybertec-postgresql.com/en/openstreetmap-service-by-cybertec/ https://blog.crunchydata.com/blog/postgresql-14-on-kubernetes https://postgresql

In this episode of Scaling Postgres, we discuss the release of Postgres 14, how best to use JSON, why are your indexes not being used and sequence gaps. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-14-released-2318/ https://www.citusdata.com/blog/2021/10/01/how-we-shipped-postgres-14-on-azure-within-a-day/ https://www.instaclustr.com/postgresql-14/ https://www.migops.com/blog/2021/09/29/upgrading-postgresql-9-6-to-postgresql-13/ https://www.cybertec-postgresql.com/en/json-postgresql-how-to-use-it-right/ https://www.depesz.com/2021/09/27/using-recursive-queries-to-get-distinct-elements-from-table/ https://www.pgmustard.com/blog/why-isnt-postgres-using-my-index https://postgres.ai/blog/20210923-zero-downtime-postgres-schema-migrations-lock-timeout-and-retries https://www.cybertec-postgresql.com/en/gaps-in-sequences-postgresql/ https://mydbanotebook.org/post/auto-increment/ https:/

In this episode of Scaling Postgres, we discuss Postgres 14 RC1, little improvements in PG14, an index advisor and how to handle 1 million insert statements. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-14-rc-1-released-2309/ https://blog.crunchydata.com/blog/postgres-14-its-the-little-things https://pganalyze.com/blog/introducing-pganalyze-index-advisor https://dev.to/yugabyte/when-you-have-millions-of-insert-statements-in-a-file-3nck https://fluca1978.github.io/2021/09/23/OracleAlterSequence.html https://www.cybertec-postgresql.com/en/building-an-oracle-to-postgresql-migrator-lessons-learned/ https://b-peng.blogspot.com/2021/09/reserved-connections.html https://b-peng.blogspot.com/2021/09/enable-shared-relcache.html https://postgresql.life/post/michael_goldberg/ https://www.rubberduckdevshow.com/episodes/14-background-job-processing/

In this episode of Scaling Postgres, we discuss one cause of broken indexes, Postgres trademark issues, percentiles vs. averages and logical replication improvements. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://elephanttamer.net/?p=61 https://www.postgresql.org/about/news/trademark-actions-against-the-postgresql-community-2302/ https://postgresql.fund/blog/postgres-core-team-attacks-postgres-community/ https://postgresql.fund/blog/respecting-majority-questioning-status-quo-as-a-minority/ https://blog.timescale.com/blog/how-percentile-approximation-works-and-why-its-more-useful-than-averages/ http://amitkapila16.blogspot.com/2021/09/logical-replication-improvements-in.html https://www.cybertec-postgresql.com/en/postgresql-create-indexes-after-bulk-loading/ https://www.highgo.ca/2021/09/17/a-quick-test-for-postgres_fdw-batch-insertion/ https://www.cybertec-postgresql.com/en/postgresql-schedulers-comparison-table/ https:/

In this episode of Scaling Postgres, we discuss having boundless text fields, revoking public schema permissions, less index bloat in PG14 and comparing hardware performance. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://brandur.org/text https://www.depesz.com/2021/09/10/waiting-for-postgresql-15-revoke-public-create-from-public-schema-now-owned-by-pg_database_owner/ https://www.cybertec-postgresql.com/en/index-bloat-reduced-in-postgresql-v14/ https://dev.to/aws-heroes/aurora-postgresql-db-r6g-compared-to-db-r5-with-ybio-1bnd https://brandur.org/sqlc https://blog.rustprooflabs.com/2021/09/psycopg3-initial-review https://www.varrazzo.com/blog/2020/03/06/thinking-psycopg3/ https://fluca1978.github.io/2021/09/10/pgbackrest-jq.html https://postgresql.life/post/akshay_joshi/ https://www.rubberduckdevshow.com/episodes/12-programming-with-dates-times-zones

In this episode of Scaling Postgres, we discuss connection scaling, how to cache your sequences, a message queuing implementation and the harms of subtransactions. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://richyen.com/postgres/2021/09/03/less-is-more-max-connections.html https://dev.to/yugabyte/uuid-or-cached-sequences-42fi https://blog.crunchydata.com/blog/message-queuing-using-native-postgresql https://postgres.ai/blog/20210831-postgresql-subtransactions-considered-harmful https://blog.timescale.com/blog/how-to-create-lots-of-sample-time-series-data-with-postgresql-generate_series/ https://blog.anayrat.info/en/2021/09/01/partitioning-use-cases-with-postgresql/ https://www.postgresql.fastware.com/blog/logical-replication-tablesync-workers https://dev.to/ftisiot/solving-the-knapsack-problem-in-postgresql-40cc https://www.cybertec-postgresql.com/en/postgresql-the-power-of-a-single-missing-index/ http://postgis.net/202

In this episode of Scaling Postgres, we discuss how to use aggregate filters, the importance of vacuum with insert-only tables, how to fix out-of-sync sequences and faster functional index options. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://blog.crunchydata.com/blog/fast-flexible-summaries-with-aggregate-filters-and-windows https://blog.crunchydata.com/blog/insert-only-tables-and-autovacuum-issues-prior-to-postgresql-13 https://www.cybertec-postgresql.com/en/fixing-out-of-sync-sequences-in-postgresql/ https://dev.to/aws-heroes/index-only-scan-on-functional-indexes-9o2 https://www.percona.com/blog/postgresql-database-security-os-authentication/ https://www.depesz.com/2021/08/26/how-to-get-advisory-lock-in-shell/ https://www.migops.com/blog/2021/08/27/announcing-pg_dbms_job-in-postgresql-for-oracle-dbms_job-compatibility/ https://www.highgo.ca/2021/08/23/tuning-the-pgpool-ii-load-balancer/ https://arctype.com/blog/postgre

In this episode of Scaling Postgres, we discuss how to use foreign data wrappers, compression available in timescaledb, working with fuzzy search and describing Postgres query execution. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://blog.crunchydata.com/blog/understanding-postgres_fdw https://blog.rustprooflabs.com/2021/08/timescale-compression-openstreetmap-tags https://arctype.com/blog/postgres-fuzzy-search/ https://dev.to/yugabyte/postgres-query-execution-simple-query-2i71 https://www.highgo.ca/2021/08/20/how-batch-insertion-was-done-in-postgres_fdw/ https://dev.to/yugabyte/pgbench-client-jobs-68g https://blog.crunchydata.com/blog/waiting-for-postgis-3.2-secure-cloud-raster-access https://www.migops.com/blog/2021/08/20/validate-postgresql-extension-upgrade-scripts-using-pg_validate_extupgrade/ https://www.cybertec-postgresql.com/en/pg_timetable_v4-is-out/ https://postgresql.life/post/andrey_borodin/ https://www.rubber

In this episode of Scaling Postgres, we discuss new releases for Postgres & pgbouncer, have a performance discussion, describe the different index types and how to use them. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-134-128-1113-1018-9623-and-14-beta-3-released-2277/ https://www.pgbouncer.org/2021/08/pgbouncer-1-16-0 https://www.rubberduckdevshow.com/episodes/8-database-performance/ https://pganalyze.com/blog/postgres-create-index https://www.youtube.com/watch?v=gBGUuMrO6MI https://www.cybertec-postgresql.com/en/postgresql-vs-redis-vs-memcached-performance/ https://www.percona.com/blog/impact-of-network-and-cursor-on-query-performance-of-postgresql/ https://www.ongres.com/blog/noset-postgresql-extension-for-blocking-set-reset-execution/ https://www.percona.com/blog/migrating-postgresql-to-kubernetes https://www.highgo.ca/2021/08/09/horizontal-scalability-options-in-postgresq

In this episode of Scaling Postgres, we discuss ways to work around the planner, working with grouping sets, using a lateral join and not-so-easy Postgres issues. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://heap.io/blog/when-the-postgres-planner-is-not-very-smart https://www.cybertec-postgresql.com/en/postgresql-grouping-sets-rollup-cube/ https://fluca1978.github.io/2021/08/07/PostgreSQLLateralJoin.html https://pgdash.io/blog/postgresql-six-not-so-easy-pieces.html https://www.postgresql.fastware.com/blog/logical-decoding-of-two-phase-commits https://blog.crunchydata.com/blog/using-cert-manager-to-deploy-tls-for-postgres-on-kubernetes https://postgresql.life/post/bryn_llewellyn/ https://www.rubberduckdevshow.com/episodes/7-writing-secure_code/

In this episode of Scaling Postgres, we discuss JSONB vs. window functions, a Postgres security checklist, implementing full-text search and using NFL timeseries data. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://swarm64.com/post/postgresql-jsonb-vs-window-functions/ https://www.cybertec-postgresql.com/en/postgresql-security-things-to-avoid-in-real-life/ https://blog.crunchydata.com/blog/postgres-full-text-search-a-search-engine-in-a-database https://blog.timescale.com/blog/hacking-nfl-data-with-postgresql-timescaledb-and-sql/ https://ardentperf.com/2021/07/26/postgresql-logical-replicas-and-snapshots-proceed-carefully/ https://www.pgmustard.com/blog/max-parallel-workers-per-gather https://fluca1978.github.io/2021/07/27/pgbackrestAsync.html https://b-peng.blogspot.com/2021/07/deploying-pgpool2-exporter-with-docker.html https://www.percona.com/blog/improve-postgresql-query-performance-insights-with-pg_stat_monitor/ https:/

In this episode of Scaling Postgres, we discuss the current status of zheap, the gaps in sequences, how to efficiently paginate and how to set up patroni with etcd and haproxy. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.cybertec-postgresql.com/en/postgresql-zheap-current-status/ https://incident.io/blog/one-two-skip-a-few https://pganalyze.com/blog/pagination-django-postgres https://arctype.com/blog/postgres-patroni/ https://fluca1978.github.io/2021/07/20/PostgreSQLExtensions.html https://www.highgo.ca/2021/07/23/the-amazing-buffer-tag-in-postgresql/ https://www.paulox.net/2021/07/19/maps-with-django-part-2-geodjango-postgis-and-leaflet/ https://blog.crunchydata.com/blog/using-postgis-and-pg_featureserv-with-qgis https://blog.crunchydata.com/blog/waiting-for-postgis-3.2-st_makevalid https://blog.crunchydata.com/blog/waiting-for-postgis-3.2-st_contour-and-st_setz https://postgresql.life/post/rafia_sabih/ https:/

In this episode of Scaling Postgres, we discuss understanding lateral joins, generating JSON with Postgres, the dbcritic utility, and fetch first rows vs. limit. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.cybertec-postgresql.com/en/understanding-lateral-joins-in-postgresql/ https://blog.crunchydata.com/blog/generating-json-directly-from-postgres https://www.channable.com/tech/dbcritic-constructively-criticizing-your-postgres-schema https://www.cybertec-postgresql.com/en/postgresql-limit-vs-fetch-first-rows-with-ties/ https://elephanttamer.net/?p=58 https://www.youtube.com/watch?v=znPL6iDDou8 https://fluca1978.github.io/2021/07/13/PostgreSQLWalTraffic.html https://www.postgresql.fastware.com/blog/improved-logging-by-libpq-in-postgresql-14 https://www.highgo.ca/2021/07/14/first-contact-with-the-pg_filedump/ http://amitkapila16.blogspot.com/2021/07/logical-replication-of-in-progress.html https://postgresql.life/post

In this episode of Scaling Postgres, we discuss how to delay replication, working with materialized view permissions, paranoid configuration options and addressing memory overcommit. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.cybertec-postgresql.com/en/simulating-temporal-tables-with-dblink-and-replication-delay/ https://blog.rustprooflabs.com/2021/07/postgres-permission-mat-view https://ardentperf.com/2021/07/06/paranoid-sql-execution-on-postgresql/ https://sosna.de/posts/pgaas-memory-overcommit/ https://pganalyze.com/blog/postgresql-partitioning-django https://www.migops.com/blog/2021/07/02/credcheck-extension-to-enforce-username-and-password-checks-in-postgresql/ https://blog.crunchydata.com/blog/the-next-generation-of-kubernetes-native-postgres https://blog.crunchydata.com/blog/getting-started-with-pgo-postgres-operator-5.0 https://b-peng.blogspot.com/2021/07/postgres-disaster-recovery-on-k8s-zalando.html http

In this episode of Scaling Postgres, we discuss tracking per-operation explain analyze, Postgres 14 contribution stats, parallel execution on foreign servers and incremental view maintenance. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.pgmustard.com/blog/calculating-per-operation-times-in-postgres-explain-analyze https://www.highgo.ca/2021/06/28/some-interesting-statistics-about-pg-14-contributions/ https://www.highgo.ca/2021/06/28/parallel-execution-of-postgres_fdw-scans-in-pg-14-important-step-forward-for-horizontal-scaling/ https://yugonagata-pgsql.blogspot.com/2021/06/implementing-incremental-view.html https://lesovsky.medium.com/updates-of-postgresql-observability-diagram-66f0bbbfe021 https://www.migops.com/blog/2021/07/01/ora2pg-now-supports-oracle_fdw-to-increase-the-data-migration-speed/ http://postgis.net/2021/07/02/postgis-3.1.3/ https://www.cybertec-postgresql.com/en/postgis-setup-with-ubuntu-20-04-2/ https

In this episode of Scaling Postgres, we discuss practical partitioning, understanding pg_repack, explaining buffers and enforcing multi-row constraints. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-14-beta-2-released-2249/ https://www.postgresvision.com/ https://vyruss.org/blog/practical-partitioning-in-production-with-postgres-postgres-vision-2021.html https://www.percona.com/blog/2021/06/24/understanding-pg_repack-what-can-go-wrong-and-how-to-avoid-it/ https://www.depesz.com/2021/06/20/explaining-the-unexplainable-part-6-buffers/ https://www.cybertec-postgresql.com/en/postgresql-constraints-over-multiple-rows/ https://www.cybertec-postgresql.com/en/from-md5-to-scram-sha-256-in-postgresql/ https://www.depesz.com/2021/06/23/a-tale-of-making-company-wide-standard-psqlrc/ https://fabianlindfors.se/blog/schema-migrations-in-postgres/ https://pganalyze.com/blog/geodjango-postgis htt

In this episode of Scaling Postgres, we discuss Redis vs. Postgres, working in the hard quadrant, how Postgres optimizes joins and working with materialized views. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://spin.atomicobject.com/2021/02/04/redis-postgresql/ https://momjian.us/main/blogs/pgblog/2021.html#June_16_2021 https://www.youtube.com/watch?v=XA3SBgcZwtE https://www.cybertec-postgresql.com/en/creating-and-refreshing-materialized-views-in-postgresql/ https://blog.crunchydata.com/blog/fun-with-pg_checksums https://www.cybertec-postgresql.com/en/cross-join-in-postgresql/ https://paquier.xyz/postgresql-2/postgres-14-monitoring-copy/ https://www.youtube.com/c/percona/videos https://pgsqlpgpool.blogspot.com/2021/06/promoting-specied-node-in-pgpool-ii.html https://blog.timescale.com/blog/how-and-why-to-become-a-postgresql-contributor/ https://postgresql.life/post/josh_berkus/ https://www.rubberduckdevshow.com/

In this episode of Scaling Postgres, we discuss only using bigint, high availability, Postgres autocommit and synchronized sequential scans. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://blog.rustprooflabs.com/2021/06/postgres-bigint-by-default https://www.percona.com/blog/2021/06/11/postgresql-ha-with-patroni-your-turn-to-test-failure-scenarios/ https://www.cybertec-postgresql.com/en/disabling-autocommit-in-postgresql-can-damage-your-health/ https://www.cybertec-postgresql.com/en/data-warehousing-making-use-of-synchronized-seq-scans/ https://blog.crunchydata.com/blog/better-range-types-in-postgres-14-turning-100-lines-of-sql-into-3 https://www.youtube.com/watch?v=gDFIc2uaMc8 https://tapoueh.org/blog/2021/06/postgresql-as-a-microservice/ https://patternmatchers.wordpress.com/2021/06/11/ignore-nulls-in-postgres/ https://paquier.xyz/postgresql-2/postgres-14-memory-dumps/ https://blog.crunchydata.com/blog/postgresql-on-linux