More resources

Filter by category

View all

Deep Dive

Best Practices

Tutorials

Deep Dive

Zoom Developer Forum

.avif) Google Meet Developer Forum

Microsoft Teams Developer Forum

Announcements

Thank you! Your submission has been received!

Oops! Something went wrong while submitting the form.

\ A guide to programmatic screen capture on macOS\ \ Deep Dive](/content/blog/macos-screencapture-api/index.html)

\ Joining multiple simultaneous meetings through the Zoom SDK\ \ Zoom Developer Forum](/content/blog/zoom-sdk-simultaneous-meeting-limitations/index.html)

\ How to get transcripts from Microsoft Teams\ \ Tutorials](/content/blog/how-to-get-transcripts-from-microsoft-teams/index.html)

\ How to Get Recordings and Transcripts from Slack Huddles\ \ Tutorials](/content/blog/get-recordings-and-transcripts-from-slack-huddles-api/index.html)

\ Understanding the Zoom On Behalf Of (OBF) Token\ \ Deep Dive](/content/blog/zoom-obf/index.html)

\ What is a meeting bot?\ \ Deep Dive](/content/blog/what-is-a-meeting-bot/index.html)

\ Zoom SDK Credentials\ \ Deep Dive](/content/blog/zoom-sdk-credentials/index.html)

\ What is a Zoom Access Key (ZAK) token?\ \ Deep Dive](/content/blog/what-is-a-zoom-access-key-zak-token/index.html)

\ How WebSockets cost us $1M on our AWS bill\ \ Deep Dive](/content/blog/how-websockets-cost-us-1m-on-our-aws-bill/index.html)

\ The fix for a segfault that never shipped\ \ Deep Dive](/content/blog/the-fix-for-a-segfault-that-never-shipped/index.html)

\ Postgres Postmaster does not scale\ \ Deep Dive](/content/blog/postgres-postmaster-does-not-scale/index.html)

\ How to access audio streams in the Zoom Web SDK?\ \ Zoom Developer Forum](/content/blog/how-to-access-audio-streams-in-the-zoom-web-sdk/index.html)

\ How to determine if audio is connected on the Zoom Web SDK?\ \ Zoom Developer Forum](/content/blog/how-to-determine-if-audio-is-connected-on-the-zoom-web-sdk/index.html)

\ Recall.ai Series B: The API for Meeting Recording\ \ Announcements](/content/blog/recall-ai-series-b-the-api-for-meeting-recording/index.html)

\ How to build a Microsoft Teams Bot\ \ Tutorials](/content/blog/how-to-build-a-microsoft-teams-bot/index.html)

\ What is the Google Meet Media API?\ \ Best Practices](/content/blog/what-is-the-google-meet-media-api/index.html)

\ Postgres LISTEN/NOTIFY does not scale\ \ Deep Dive](/content/blog/postgres-listen-notify-does-not-scale/index.html)

\ Code Walkthrough: Generating Meeting Action Items With Recall.ai\ \ Tutorials](/content/blog/generating-meeting-action-items-with-recall-ai/index.html)

\ What are speaker labels and how to obtain them\ \ Tutorials](/content/blog/speaker-labels-and-names-explained/index.html)

\ 7 APIs to get Zoom transcripts: A comprehensive guide\ \ Best Practices](/content/blog/7-apis-to-get-zoom-transcripts-a-comprehensive-guide/index.html)

\ How to get transcripts from Zoom\ \ Tutorials](/content/blog/how-to-get-transcripts-from-zoom/index.html)

\ How to record Microsoft Teams\ \ Tutorials](/content/blog/how-to-get-recordings-from-microsoft-teams/index.html)

\ New Recall.ai Pricing for 2026: $0.50 per Hour of Meeting Recording\ \ Announcements](/content/blog/new-recall-ai-pricing-for-2026/index.html)

\ Open source Microsoft Teams recording software\ \ Microsoft Teams Developer Forum](/content/blog/microsoft-teams-recording-software-open-source/index.html)

\ How to pick a meeting note taker\ \ Best Practices](/content/blog/how-to-pick-a-meeting-note-taker/index.html)

\ How to Get a Recording from the Google Meet API\ \ Best Practices](/content/blog/how-to-get-a-recording-from-the-google-meet-api/index.html)

\ How to get Google Meet transcripts programmatically\ \ Best Practices](/content/blog/how-to-get-transcripts-from-google-meet-developer-edition/index.html)

\ What is Zoom RTMS?\ \ Best Practices](/content/blog/what-is-zoom-rtms/index.html)

\ How to build a desktop recording app (Like Granola)\ \ Tutorials](/content/blog/how-to-build-a-desktop-recording-app/index.html)

\ Speaker Diarization: What it is and how to get diarized transcripts\ \ Best Practices](/content/blog/speaker-diarization/index.html)

\ Understanding Zoom RTMS Behavior and Its Constraints\ \ Best Practices](/content/blog/understanding-zoom-rtms-behavior-and-its-constraints/index.html)

\ How to build a meeting notetaker\ \ Tutorials](/content/blog/how-to-build-a-meeting-notetaker/index.html)

\ How to build a meeting bot \ \ Tutorials](/content/blog/how-to-build-a-meeting-bot/index.html)

\ When to build a Microsoft Teams bot with Puppeteer\ \ Microsoft Teams Developer Forum](/content/blog/puppeteer-microsoft-teams-bot/index.html)

\ How to join a Zoom meeting using the Zoom Meeting SDK\ \ Zoom Developer Forum](/content/blog/how-to-join-a-zoom-meeting-using-the-zoom-meeting-sdk/index.html)

\ Microsoft Teams open source bot\ \ Microsoft Teams Developer Forum](/content/blog/microsoft-teams-open-source-bot/index.html)

\ How to Build a Google Meet Bot from Scratch\ \ Tutorials](/content/blog/how-i-built-an-in-house-google-meet-bot/index.html)

.avif)\ Is there open source Google Meet recording software?\ \ Google Meet Developer Forum](/content/blog/is-there-open-source-google-meet-recording-software/index.html)

.avif)\ Puppeteer Google Meet Bot\ \ Google Meet Developer Forum](/content/blog/puppeteer-google-meet-bot/index.html)

\ Zoom recording software open source\ \ Zoom Developer Forum](/content/blog/zoom-recording-software-open-source/index.html)

\ How to build a Chrome recording extension using tabCapture API\ \ Tutorials](/content/blog/how-to-build-a-chrome-recording-extension/index.html)

\ Zoom Signature Invalid\ \ Zoom Developer Forum](/content/blog/zoom-signature-invalid/index.html)

\ How to get automatic recordings from a Zoom bot\ \ Zoom Developer Forum](/content/blog/zoom-bot-automatic-recording/index.html)

\ Retrieving participant names through the Zoom SDK\ \ Zoom Developer Forum](/content/blog/zoom-sdk-retrieving-participant-names/index.html)

\ How to Access Live Transcription Data from Zoom Web SDK (Component View)\ \ Zoom Developer Forum](/content/blog/how-to-access-live-transcription-data-from-zoom-web-sdk-component-view/index.html)

\ How to build a Zoom bot from scratch\ \ Tutorials](/content/blog/how-to-build-a-zoom-bot/index.html)

\ Create a Google Meet Programmatically\ \ Tutorials](/content/blog/create-a-google-meet-programmatically/index.html)

\ How to join Zoom using Puppeteer\ \ Zoom Developer Forum](/content/blog/how-to-join-zoom-using-puppeteer/index.html)

\ Handling Zoom SDK recording permissions\ \ Zoom Developer Forum](/content/blog/zoom-sdk-recording-permissions/index.html)

\ How to Enable a Zoom Bot to Skip the Waiting Room\ \ Zoom Developer Forum](/content/blog/zoom-bot-skip-waiting-room/index.html)

\ How do I sync audio and video when processing raw audio/video from the Zoom SDK?\ \ Zoom Developer Forum](/content/blog/how-do-i-sync-audio-and-video-when-processing-raw-audio-video-from-the-zoom-sdk/index.html)

\ Fixing DLL warnings in the Zoom Windows Meeting SDK\ \ Zoom Developer Forum](/content/blog/fixing-dll-warnings-in-the-zoom-windows-meeting-sdk/index.html)

\ Zoom join tokens for local recording\ \ Best Practices](/content/blog/zoom-join-tokens-for-local-recording/index.html)

\ How to get recordings from Zoom using a Zoom recording bot API\ \ Tutorials](/content/blog/how-to-get-recordings-from-zoom/index.html)

\ How to manage ZAK Tokens for Zoom bots\ \ Zoom Developer Forum](/content/blog/how-to-manage-zak-tokens-for-zoom-bots/index.html)

\ How to build a Zoom notetaker\ \ Tutorials](/content/blog/how-to-build-a-zoom-notetaker/index.html)

\ Recall.ai Announces Participant Emails\ \ Announcements](/content/blog/recall-ai-announces-participant-emails/index.html)

\ How often should I expect to receive the Zoom recording completed webhook for a single meeting?\ \ Zoom Developer Forum](/content/blog/how-often-should-i-expect-to-receive-the-zoom-recording-completed-webhook-for-a-single-meeting/index.html)

\ Why are waiting rooms causing issues with the Zoom Web Meeting SDK?\ \ Zoom Developer Forum](/content/blog/why-are-waiting-rooms-causing-issues-with-the-zoom-web-meeting-sdk/index.html)

\ Why is my Zoom recording completed webhook so delayed?\ \ Zoom Developer Forum](/content/blog/why-is-my-zoom-recording-completed-webhook-so-delayed/index.html)

\ Why aren't transcription webhooks firing for my Zoom meetings?\ \ Zoom Developer Forum](/content/blog/why-arent-transcription-webhooks-firing-for-my-zoom-meetings/index.html)

\ How do I track participants across Zoom breakout rooms?\ \ Zoom Developer Forum](/content/blog/how-do-i-track-participants-across-zoom-breakout-rooms/index.html)

\ Can I identify meeting participants by email on Zoom?\ \ Zoom Developer Forum](/content/blog/can-i-identify-meeting-participants-by-email-on-zoom/index.html)

\ How do I capture participant data in real time on Zoom?\ \ Zoom Developer Forum](/content/blog/how-do-i-capture-participant-data-in-real-time-on-zoom/index.html)

\ Tracking persistent Zoom meeting ID's for participants who leave and rejoin\ \ Zoom Developer Forum](/content/blog/tracking-persistent-zoom-meeting-ids-for-participants-who-leave-and-rejoin/index.html)

\ Debugging issues with automatic cloud recording\ \ Zoom Developer Forum](/content/blog/debugging-issues-with-automatic-cloud-recording/index.html)

\ Tracking Zoom meetings when using Zoom PMI\ \ Zoom Developer Forum](/content/blog/tracking-zoom-meetings-when-using-zoom-pmi/index.html)

\ Encoding raw video from the Zoom SDK to MP4\ \ Zoom Developer Forum](/content/blog/encoding-raw-video-from-the-zoom-sdk/index.html)

\ Handling crashes in the Zoom Windows Meeting SDK\ \ Zoom Developer Forum](/content/blog/handling-crashes-in-the-zoom-windows-meeting-sdk/index.html)

\ Encoding raw video frames from the Zoom SDK to JPEG\ \ Zoom Developer Forum](/content/blog/decoding-raw-video-frames-from-the-zoom-sdk/index.html)

\ Recording meetings without a local recording token\ \ Zoom Developer Forum](/content/blog/recording-meetings-without-a-local-recording-token/index.html)

\ Resolving intermittent errors while downloading Zoom transcripts\ \ Zoom Developer Forum](/content/blog/resolving-intermittent-errors-while-downloading-zoom-transcripts/index.html)

\ Applying account-level settings to meetings created via the Zoom API\ \ Zoom Developer Forum](/content/blog/applying-account-level-settings-to-meetings-created-via-the-zoom-api/index.html)

\ Why can't my bot join meetings hosted by people outside of my Zoom organization?\ \ Zoom Developer Forum](/content/blog/why-cant-my-bot-join-meetings-hosted-by-people-outside-of-my-zoom-organization/index.html)

\ Zoom Cloud Recording API: Transcript Tutorial, Common Pitfalls, and Workarounds\ \ Tutorials](/content/blog/zoom-transcript-api/index.html)

\ Zoom Meeting Bots: What They Are and Solutions\ \ Deep Dive](/content/blog/zoom-bots/index.html)

\ Can I access gallery view video on the Linux SDK?\ \ Zoom Developer Forum](/content/blog/can-i-access-gallery-view-video-on-the-linux-sdk/index.html)

\ How to access the Zoom Archive Files API\ \ Zoom Developer Forum](/content/blog/how-to-access-the-zoom-archive-files-api/index.html)

\ Resolving Web SDK local recording errors\ \ Zoom Developer Forum](/content/blog/resolving-web-sdk-local-recording-errors/index.html)

\ How to resolve permissions errors with the Zoom Users API\ \ Zoom Developer Forum](/content/blog/how-to-resolve-permissions-errors-with-the-zoom-users-api/index.html)

\ How to track participant name changes in breakout rooms\ \ Zoom Developer Forum](/content/blog/how-to-track-participant-name-changes-in-breakout-rooms/index.html)

\ Why is the transcript.completed webhook not firing?\ \ Zoom Developer Forum](/content/blog/zoom-sdk-webhook-not-firing/index.html)

\ Debugging Zoom SDK failed video capture\ \ Zoom Developer Forum](/content/blog/zoom-sdk-debugging-video-capture/index.html)

\ Capturing video and audio streams through the Zoom SDK\ \ Zoom Developer Forum](/content/blog/zoom-sdk-receiving-video-streams/index.html)

\ Can you suppress the Zoom recording consent prompt through the SDK?\ \ Zoom Developer Forum](/content/blog/zoom-sdk-suppressing-consent-prompt/index.html)

\ How to stream video to a meeting through the Zoom SDK\ \ Zoom Developer Forum](/content/blog/zoom-sdk-streaming-video-to-meeting/index.html)

\ Sending chat messages through the Zoom SDK\ \ Zoom Developer Forum](/content/blog/zoom-sdk-sending-chat-messages/index.html)

\ Handling Zoom app OAuth redirect URL requirements\ \ Zoom Developer Forum](/content/blog/zoom-app-oauth-redirect-url-requirement/index.html)

\ Debugging missing Zoom transcription webhooks\ \ Zoom Developer Forum](/content/blog/zoom-sdk-missing-transcription-webhooks/index.html)

\ Handling Zoom SDK recording compliance requirements\ \ Zoom Developer Forum](/content/blog/zoom-sdk-compliance-requirements/index.html)

\ Handling Zoom app verification\ \ Zoom Developer Forum](/content/blog/zoom-app-domain-verification/index.html)

\ How to join registration-required meetings with a Zoom bot\ \ Zoom Developer Forum](/content/blog/zoom-sdk-joining-registration-required-meetings/index.html)

\ How to use a Zoom bot to join meetings without passcodes\ \ Zoom Developer Forum](/content/blog/zoom-sdk-joining-meetings-without-passcodes/index.html)

\ Debugging stuck Zoom SDK bots\ \ Zoom Developer Forum](/content/blog/zoom-sdk-docker-issues/index.html)

\ How to create a voice agent to send to Zoom meetings\ \ Zoom Developer Forum](/content/blog/zoom-sdk-ai-voice-agent/index.html)

\ How to capture Zoom audio and video with a meeting bot\ \ Zoom Developer Forum](/content/blog/create-audio-video-capture-bots/index.html)

\ How to transcribe a Zoom meeting in real time\ \ Zoom Developer Forum](/content/blog/accessing-real-time-transcription-api/index.html)

\ How to receive raw video from the Zoom SDK\ \ Zoom Developer Forum](/content/blog/accessing-host-video-stream/index.html)

\ Announcing the Recall.ai and Speechmatics Integration\ \ Announcements](/content/blog/announcing-the-recall-ai-and-speechmatics-integration/index.html)

\ Can I access raw audio data using the Zoom Web SDK?\ \ Zoom Developer Forum](/content/blog/can-i-access-raw-audio-data-using-the-zoom-web-sdk/index.html)

Go To Blog

Deep Dive

Postgres LISTEN/NOTIFY does not scale

Updated at:

April 20, 2026

Written By:

Elliot Levin

Table of Contents

At Recall.ai, we run an unusual workload. We record millions of hours of meetings every month. Each of these meetings generates a large amount of data we need to reliably capture and analyze. Some of that data is video, some of it is audio and some of it is structured data – transcription, events and metadata.

The structured data gets written to our Postgres database by tens of thousands of simultaneous writers. Each of these writers is a “meeting bot”, which joins a video call and captures the data in real-time.

We love Postgres and it lives at the heart of our service! But this extremely concurrent, write-heavy workload resulted in a stalled-out Postgres. This is the story of what happened, how we ended up discovering a bottleneck in the LISTEN/NOTIFY feature of Postgres (the event notifier that runs based on triggers when something changes in a row), and what we ended up doing about it.

Update: Fixed in Postgres core

This commit has eliminated the bottleneck in the postgres core. Credit to Joel Jacobson and the core postgres contributors for resolving this.

TL;DR

When a NOTIFY query is issued during a transaction, it acquires a global lock on the entire database ( ref) during the commit phase of the transaction, effectively serializing all commits. Under many concurrent writers, this results in immense load and major downtime. Don’t use LISTEN/NOTIFY if you want your database to scale to many writers.

How we discovered it

Between the dates 2025-03-19 to 2025-03-22 our core Postgres database experienced three periods of downtime. Each of these downtimes had a similar set of symptoms:

Massive spikes in database load and active awaiting sessions
Database query throughput drops massively
Database CPU, disk I/O and network traffic all plummeted

Why would increased load on the database result in a drastic drop in CPU and I/O? This was extremely surprising to us, and lead us to our first clue. We suspected increased lock contention but we didn’t have enough information to confirm if this was the root cause or a symptom given that load spikes so suddenly. Our first change was to enable log_lock_waits so we could identify the queries, connections and locks being contested.

After extended investigation we correlated the increased load with increased requests to our update endpoint, specifically our update bot endpoint, which triggered a NOTIFY query that notifies a running bot that its configuration has been updated. After combing through gigabytes of logs we spotted a flood of interesting log lines:

12025-03-2215:16:59 UTC:10.3.206.139(34877):postgres@meeting_api:[45081]:DETAIL: Process holding the lock:20841. Wait queue:17474,20898,1931,20926,35855,14865,20846,31717,36912,20927,45081,61596,44128,17462,4576,35852,35940,35854,35864,37057,26765,37041,35872,18380,44271,35802,35840,33384,44279,62275,35440,35948,35804,45084,45147,32250,35898,20932,35862,35859,56845,44233,17448,26745,35883,35865,26753,20809,1929,14858,36904,17452,35892,36910,32798,20892,35899,36901,45137,45105,36835,36897,35901,32036,35856,45171,35941,37046,45160,33013,17458,35874,44092,35893,18356,37055,35942,1939,38662,45079,35889,35839,37058,36900,35910,37056,35951,35907,36908,26767,25575,32037,35868,44278,45188,45120,61287,35863,20817,45062,37048,45151,45179,35939,35849,45161,44220,37054,32265,45119,37059,32264,26239,32822,36894,44224,44121,28308,36906,18367,36914,26821,37047,35894,44210,57960,37042,25564,45136,35908,35866,20895,35882,45208,45138,17475,35871,20857,35953,44108,45215,42698,44280,26782,35847,44253,25571,20906,61366,35944,45172,45221,45228,45223,20896,33015,45193,26740,1873,44288,61619,44234,26744,35295,17464,45220,35838,35845,35900,36963,1883,35848,35895,61368,45227,1919,45069,31716,45243,20814,35850,20156,36889,18207,35952,35765,45277,35801,36896,20802,35877,45348,35873,33264,44125,36895,26211,20907,20851,45187,25561,45238,45066,33263,61274,33261,45371,20161,61410,32545,6762,40503,38660,45242,20937,20936,20928,44187,42696,35766,61414,17442,36911,45121,45307,35897,25566,25574,45417,33807,35881,45183,37052,45386,45408,45381,45368,33223,45150,35902,44256,45311,45294,35943,45310,44042,37050,32796,45364,45256,45387,45330,26743,35886,45332,44199,45293,45331,45329,45291,45312,38658,45366,45363,44043,45184,35905,33260,37043,7160,45435,45323,6750,45288,35906,45336,17453,42695,45333,25565,45545,45321,33810,45326,45320,44184,45395,20859,44293,33014,36837,35945,45442,1943,45423,45396,42700,12431,36892,45181,45400,44174,35767,45401,35904,36832,12082,45425,17469,45278,45511,20940,45585.

22025-03-2215:16:59 UTC:10.3.206.139(34877):postgres@meeting_api:[45081]:STATEMENT: COMMIT

32025-03-2215:16:59 UTC:10.3.161.44(22163):postgres@meeting_api:[20841]:LOG: process 20841 acquired AccessExclusiveLock on object 0 of class 1262 of database 0 after 1015.921 ms

42025-03-2215:16:59 UTC:10.3.161.44(22163):postgres@meeting_api:[20841]:STATEMENT: COMMIT

52025-03-2215:16:59 UTC:10.3.254.68(52497):postgres@meeting_api:[61596]:LOG: process 61596 still waiting for AccessExclusiveLock on object 0 of class 1262 of database 0 after 1000.192 ms

62025-03-2215:16:59 UTC:10.3.254.68(52497):postgres@meeting_api:[61596]:DETAIL: Process holding the lock:20841. Wait queue:17474,20898,1931,20926,35855,14865,20846,31717,36912,20927,45081,61596,44128,17462,4576,35852,35940,35854,35864,37057,26765,37041,35872,18380,44271,35802,35840,33384,44279,62275,35440,35948,35804,45084,45147,32250,35898,20932,35862,35859,56845,44233,17448,26745,35883,35865,26753,20809,1929,14858,36904,17452,35892,36910,32798,20892,35899,36901,45137,45105,36835,36897,35901,32036,35856,45171,35941,37046,45160,33013,17458,35874,44092,35893,18356,37055,35942,1939,38662,45079,35889,35839,37058,36900,35910,37056,35951,35907,36908,26767,25575,32037,35868,44278,45188,45120,61287,35863,20817,45062,37048,45151,45179,35939,35849,45161,44220,37054,32265,45119,37059,32264,26239,32822,36894,44224,44121,28308,36906,18367,36914,26821,37047,35894,44210,57960,37042,25564,45136,35908,35866,20895,35882,45208,45138,17475,35871,20857,35953,44108,45215,42698,44280,26782,35847,44253,25571,20906,61366,35944,45172,45221,45228,45223,20896,33015,45193,26740,1873,44288,61619,44234,26744,35295,17464,45220,35838,35845,35900,36963,1883,35848,35895,61368,45227,1919,45069,31716,45243,20814,35850,20156,36889,18207,35952,35765,45277,35801,36896,20802,35877,45348,35873,33264,44125,36895,26211,20907,20851,45187,25561,45238,45066,33263,61274,33261,45371,20161,61410,32545,6762,40503,38660,45242,20937,20936,20928,44187,42696,35766,61414,17442,36911,45121,45307,35897,25566,25574,45417,33807,35881,45183,37052,45386,45408,45381,45368,33223,45150,35902,44256,45311,45294,35943,45310,44042,37050,32796,45364,45256,45387,45330,26743,35886,45332,44199,45293,45331,45329,45291,45312,38658,45366,45363,44043,45184,35905,33260,37043,7160,45435,45323,6750,45288,35906,45336,17453,42695,45333,25565,45545,45321,33810,45326,45320,44184,45395,20859,44293,33014,36837,35945,45442,1943,45423,45396,42700,12431,36892,45181,45400,44174,35767,45401,35904,36832,12082,45425,17469,45278,45511,20940,45585.

72025-03-2215:16:59 UTC:10.3.254.68(52497):postgres@meeting_api:[61596]:STATEMENT: COMMIT

82025-03-2215:16:59 UTC:10.3.147.201(10551):postgres@meeting_api:[44128]:LOG: process 44128 still waiting for AccessExclusiveLock on object 0 of class 1262 of database 0 after 1000.136 ms

92025-03-2215:16:59 UTC:10.3.147.201(10551):postgres@meeting_api:[44128]:DETAIL: Process holding the lock:20841. Wait queue:17474,20898,1931,20926,35855,14865,20846,31717,36912,20927,45081,61596,44128,17462,4576,35852,35940,35854,35864,37057,26765,37041,35872,18380,44271,35802,35840,33384,44279,62275,35440,35948,35804,45084,45147,32250,35898,20932,35862,35859,56845,44233,17448,26745,35883,35865,26753,20809,1929,14858,36904,17452,35892,36910,32798,20892,35899,36901,45137,45105,36835,36897,35901,32036,35856,45171,35941,37046,45160,33013,17458,35874,44092,35893,18356,37055,35942,1939,38662,45079,35889,35839,37058,36900,35910,37056,35951,35907,36908,26767,25575,32037,35868,44278,45188,45120,61287,35863,20817,45062,37048,45151,45179,35939,35849,45161,44220,37054,32265,45119,37059,32264,26239,32822,36894,44224,44121,28308,36906,18367,36914,26821,37047,35894,44210,57960,37042,25564,45136,35908,35866,20895,35882,45208,45138,17475,35871,20857,35953,44108,45215,42698,44280,26782,35847,44253,25571,20906,61366,35944,45172,45221,45228,45223,20896,33015,45193,26740,1873,44288,61619,44234,26744,35295,17464,45220,35838,35845,35900,36963,1883,35848,35895,61368,45227,1919,45069,31716,45243,20814,35850,20156,36889,18207,35952,35765,45277,35801,36896,20802,35877,45348,35873,33264,44125,36895,26211,20907,20851,45187,25561,45238,45066,33263,61274,33261,45371,20161,61410,32545,6762,40503,38660,45242,20937,20936,20928,44187,42696,35766,61414,17442,36911,45121,45307,35897,25566,25574,45417,33807,35881,45183,37052,45386,45408,45381,45368,33223,45150,35902,44256,45311,45294,35943,45310,44042,37050,32796,45364,45256,45387,45330,26743,35886,45332,44199,45293,45331,45329,45291,45312,38658,45366,45363,44043,45184,35905,33260,37043,7160,45435,45323,6750,45288,35906,45336,17453,42695,45333,25565,45545,45321,33810,45326,45320,44184,45395,20859,44293,33014,36837,35945,45442,1943,45423,45396,42700,12431,36892,45181,45400,44174,35767,45401,35904,36832,12082,45425,17469,45278,45511,20940,45585.

Specifically AccessExclusiveLock on object 0 of class 1262 of database 0 stood out. What is object 0 of class 1262? That is not table, row or anything familiar. We found a similar report of this behavior on a 12 year old thread in the postgres mailing list. A response from Tom Lane in this thread led us to an extremely surprising finding in the postgres source code:

1/*

2 \* Serialize writers by acquiring a special lock that we hold till

3 \* after commit. This ensures that queue entries appear in commit

4 \* order, and in particular that there are never uncommitted queue

5 \* entries ahead of committed ones, so an uncommitted transaction

6 \* can't block delivery of deliverable notifications.

7 *

8 \* We use a heavyweight lock so that it'll automatically be released

9 \* after either commit or abort. This also allows deadlocks to be

10 \* detected, though really a deadlock shouldn't be possible here.

11 *

12 \* The lock is on "database 0", which is pretty ugly but it doesn't

13 \* seem worth inventing a special locktag category just for this.

14 \* (Historical note: before PG 9.0, a similar lock on "database 0" was

15 \* used by the flatfiles mechanism.)

16 */

17 LockSharedObject(DatabaseRelationId, InvalidOid, 0,

18 AccessExclusiveLock);

... See all 18 lines

After reading through the surrounding code and a call sites, we found that this lock is acquired during COMMIT queries when a transaction has previously issued a NOTIFY. It makes sense that notifications are only sent after the transaction has committed, so that’s why the code which triggers the notifications is found in PreCommit_Notify. This is a global lock on the entire database (or to be pedantic, a global lock on all databases within the postgres instance). This lock effectively ensures that only a single COMMIT query can be handled at a time. Under a heavy multi-writer scenario, this turns out to be very sad news.

This made it to the HN Front Page -- Check out the discussion

Reproducing the problem

To confirm our hypothesis that the global database lock was indeed the cause, we simulated load tests on Postgres with and without the LISTEN/NOTIFY code. The results confirm that the global database lock drastically increases lock contention in the database, often stalling it.

With `NOTIFY`

During these periods of “high load” with the LISTEN/NOTIFY code, the database’s CPU, and I/O load actually plummet, suggesting that the database is indeed bottlenecked by a globally exclusive mutex.

Without `NOTIFY`

Eliminating LISTEN/NOTIFY means the database is able to make full utilization of all allocated CPU cores. This means it can make quick progress through the spike in load and recover without intervention.

Remediation

We concluded that having a single global mutex within COMMIT queries was unacceptable. We decided to migrate away from LISTEN/NOTIFY in favor of tracking this logic at the application layer. Given we had only a single (but critical) codepath relying on it, the migration took under a day to ship. Since then, our Postgres has been ticking along smoothly.

We are hiring engineers! Join us in building the future of engineering at Recall.ai We are hiring for a number of roles, apply here to join our close-knit team.

Written By:

Elliot Levin

Table of Contents