Exit code 3 after adding Diesel r2d2 on Rocket

[AcrylicShrimp]

Hi all! I have an issue with the r2d2 module (I think).

My rocket app is exiting with code 3, without any message or log on Windows 10. I found that this is occurred by the r2d2 module of the diesel.

The detailed problem is described in rwf2/Rocket#1931 and rwf2/Rocket#1931 (comment).

Is there any work-around for this issue?

Thank you!

[weiznich]

I'm not aware of this issue, so I fear you need to provide much more information especially the following information:

• Exact diesel, postgres, libpq, rustc, r2d2 versions
• A example to reproduce the issue. It would be helpful to have something that does not depend on rocket as it makes testing much easier for us. Otherwise this requires us to understand how rocket works and I'm not sure I'm able to invest that time currently
• Anything that indicates that this is a diesel specific issue and not a r2d2, rocket or general rustc issue, especially as the corresponding code in diesel is really simple and does not have platform specific variants.
• Anything that may help to reproduce this issue

(Another related note: As I do not have access to a windows 10 machine I cannot just take random code and try to reproduce platform specific issues.)

EDIT:

Another thing that may be useful to know is whether libpq was compiled in a thread safe mode or not. I fear in the second case this "behaviour" is expected.

[weiznich]

I don't know my libpq is thread safe or not. How do I check it? I just downloaded the installer from the official website.

Can you provide a link for the download?

I don't know my libpq is thread safe or not. How do I check it? I just downloaded the installer from the official website.

See https://www.postgresql.org/docs/current/libpq-threading.html for details.

I'm sorry bothering you if this is not caused by the diesel or the r2d2 module in the diesel. If you think so, I'll report it on the r2d2 instead.

It's totally fine to ask those question, but please understand that I need to ask for details to be able to understand the problem and that I cannot promise any solution.

[weiznich]

Another thing I would try is to see if it's possible to "reproduce" this without r2d2 and rocket. The following minimal test program just opens up a number of connection from different threads at the same time. If it's a racing condition in diesel (or libpq) the issue should show up here as well. If that's not the case we know at least that the connection implementation should be fine, which means this is likely a pool issue then.

Can you try running the following program and report back? You likely need to adjust db_url to your environment.

extern crate diesel;

use diesel::prelude::*;
use std::thread;
use std::sync::{Arc, Barrier};

fn main() {

    const THREAD_COUNT: usize = 10;

    let db_url = "postgres://localhost/diesel_test";

    let b = Arc::new(Barrier::new(THREAD_COUNT));
    let mut handles = Vec::new();
    for i in 0..THREAD_COUNT {
        let b = Arc::clone(&b);

        let h = thread::spawn(move || {
            b.wait();

            let _ = PgConnection::establish(db_url).unwrap();

            println!("Successfully initialized connection {}", i);
        });

        handles.push(h);
    }

    for h in handles {
        h.join().unwrap();
    }
}

[CDouv]

Hello weiznich,

I came across this thread having the exact same issue as AcrylicShrimp, except I was using actix-web instead of Rocket. I was able to reproduce the same exit code: 3 error on my Windows 10 machine rustc 1.56.1 using the sample code that you provided

Let me know if you need any additional info on the error.

[weiznich]

As I do not have access to a windows 10 machine to reproduce this I've added the test program as test case to our test suite: #2951. I expected to reproduce the failure on our CI, but that didn't work as expected 😞 This means I likely need more help to investigate this issue.

Let me summarize what I think to know about this problem:

• It's not related to rocket, actix-web, r2d2 or diesel r2d2 manager implementation, as we can reproduce the problem without all of those.
• It seems to be related to threading.
• It only happens on windows (10?), but not always (see CI).

To move forward on this we likely need to collect more information. It would be helpful if you @CDouv or @AcrylicShrimp could:

• Try running the test suite for the linked PR (Simply checkout that branch + set DATABASE_URL to some test database + run cargo test --features "postgres" inside of diesel_tests) and report back if the issue happens there.
• Provide more information about your libpq version. Ideally this would include a download location and information about if libpq is thread safe or not.

I expect this to be somehow related to libpq as PgConnection::establish really does not much. (For reference the corresponding code for PgConnection::establish and for RawConnection::establish.) As you can see we basically call only into libpq to create a new connection there. In my opinion that leaves only a few options:

• Diesel did miss something important there, but then it would be strange that this happens only for windows builds
• It's a bug in libpq, which I personally would be surprised by
• It's due to libpq be configured and build in a not threadsafe mode. In this case it would simply mean that this version of libpq is not compatible with diesel, but we should maybe check that as part of the connection setup?

[AcrylicShrimp]

Sorry for this late response! I tested the PR and got some information for the libpq.

About the PR

I cannot test your PR properly. the test halts after random successful logs. This problem occurs even if the PR you added is stripped out.

test annotations::derive_identifiable_with_composite_pk ... ok
test annotations::derive_identifiable_with_non_standard_pk ... ok
test associations::associations_can_be_grouped_multiple_levels_deep ... ok
test associations::self_referencing_associations ... ok
test associations::custom_foreign_key ... ok
test annotations::derive_insertable_with_option_for_not_null_field_with_default ... ok
test associations::grouping_associations_maintains_ordering ... ok
test annotations::association_where_parent_and_child_have_underscores ... ok
test boxed_queries::boxed_queries_can_differ_conditionally ... ok
test boxed_queries::boxed_queries_can_use_borrowed_data ... ok
test associations::one_to_many_returns_query_source_for_association ... ok
test annotations::nested_queryable_derives ... ok
error: test failed, to rerun pass '--test integration_tests'

As you can see, the cargo shows nothing about the failure.

About the libpq

First of all, here's the download link: https://content-www.enterprisedb.com/downloads/postgres-postgresql-downloads

I wrote a small program that checks for the libpq. See: https://github.com/AcrylicShrimp/libpq-test

This program prints 1 which indicates that the libpq is thread safe.

Thanks for your hard working! Please let me know if you need any helps!

[weiznich]

I cannot test your PR properly. the test halts after random successful logs. This problem occurs even if the PR you added is stripped out.

Can you try running the tests with --test-threads=1 set or by only running the added test?

First of all, here's the download link: https://content-www.enterprisedb.com/downloads/postgres-postgresql-downloads

👍 Will have a look at that.

I wrote a small program that checks for the libpq. See: https://github.com/AcrylicShrimp/libpq-test

This program prints 1 which indicates that the libpq is thread safe.

That leaves two possible causes on my list 🤔 I will need some time to think about a way to verify that it is one or the other.

[hut8]

@clemens-msupply I did see that. I remember getting a similar stacktrace once, I think.
git log REL_13_5..REL_14_1 .\src\interfaces\libpq\fe-misc.c shows a lot of changes to that file. Not sure if they're exactly in that function. However, I'm curious about this memory corruption thing you mention. I have been running cargo run at least 5 times between each git bisect because it only happens sometimes, but I think that when the build contains the error, I can usually get it to crash with either 1 or 2 attempts.
Does this mean that another thread corrupted the memory, then exited, then the crash happened? Any pointers would be helpful while I continue bisecting. Roughly 10 bisections to go.

Oh, in case I didn't make this clear, the libintl-9.dll file that's shipped with 12, 13 and 14 for Windows by EnterpriseDB is exactly the same file. So no bug was introduced during that time in that module.

[hut8]

git bisect is my newest friend. Found it, I think.
postgres/postgres@52a1022

[weiznich]

Thanks for running the bisect 👍 This commit seems to have at least some connection to the error message mentioned on the postgres mailing list, so it seems plausible that this causes the issue.

I would suggest the following next steps:

• Report the findings to upstream, maybe that helps someone there to figure out which part of the patch is broken
• Try to build libpq + diesel with asan enabled and see if that returns something meaningful.

[weiznich]

To add a few additional thoughts here:

@clemens-msupply mentioned on the mailing list

I had a look at the memory of the crashed process and the second arg for
libintl_dcigettext points to a string in memory: 'connection to server
at "%s" (%s), port %s failed:'

which is the same what my app prints when it is not crashing, e.g.
'Failed to initialize connection 1: connection to server at "localhost"
(::1), port 5432 failed: fe_sendauth: no password supplied'

Combined with the linked patch that seems to indicate an issue with getting the host name. So its probably either
postgres/postgres@52a1022#diff-8d819454e061b9d4cdae9c8922ded05753a629d70f2ac1de1d4f6d5a4aeb7f68R1717 or postgres/postgres@52a1022#diff-8d819454e061b9d4cdae9c8922ded05753a629d70f2ac1de1d4f6d5a4aeb7f68R2506 thats causing this issue? Might be even worth to try to change both locations back to use strlen(host_addr) > 0.

[hut8]

Here's today's debugging session: https://gist.github.com/hut8/3b25e6a581a600589bdc62644734de18
This was done on the revision I identified as problematic.

Unfortunately, I can't make much of this. Earlier, I pointed out that the already_bound variable was true and thought this was absurd, because none of the function should run if it's true. But, with a more rested mind, I realized that of course it should be true when it crashed, it was just set that way a few lines above. It just wouldn't get run again. My mistake.

I will report this on the mailing list with this additional info, and refer to @clemens-msupply 's thread.

[hut8]

I finally found the actual problem and a real solution.

The real problem

Yes, the problem is in gettext/libintl. It turns out that in Postgres, according to one of the maintainers from EnterpriseDB:

PG 14.1 (and 13.5) used gettext-0.19.8. The PG <13.5 used v0.20.1 and the reason was that there was a performance issue reported with that version (see here). So it was decided to bundle the lower version 0.19.8 and at the same time continue to debug the performance issue seen with the higher version of gettext.

It is almost impossible to generate a PDB file for gettext/libintl because it must be built with mingw which doesn't produce those. With more effort than I would like to make, there is a tool developed by the D language community that can do this: https://github.com/rainers/cv2pdb

The libintl version that is distributed with EnterpriseDB Postgres is named libintl-9.dll. You would think that this is newer than libintl-8.dll. But you would be wrong. See this explanation. This really threw me off! It turns out, therefore, that libintl-8.dll contains newer code than libintl-9.dll

The only code that really calls abort() anywhere in the stacktrace has to do with threading macros. So it's something to do with that.

There is this one commit that caught my eye:

 #ifdef _LIBC
       __libc_rwlock_fini (domain->conversions_lock);
+#else
+      gl_rwlock_destroy (domain->conversions_lock);
 #endif
       goto invalid;
     }

This code is only present in tags v0.20, v0.20.1, v0.20.2, and v0.21. So I grabbed some new binaries that would contain this code from github (see the solution) and verified that the versions that contain this patch do not have the issue. The ones that do not contain the patch do have the issue.

For some comic relief, here is real gem I found in the source:

#ifdef __GNUC__
# define ZERO 0
#else
# define ZERO 1
#endif

Solution

1. Download gettext0.21-iconv1.16-shared-64 from http://mlocati.github.io/articles/gettext-iconv-windows.html
2. Extract the ZIP. It will contain, inter alia, libintl-8.dll which is newer than libintl-9.dll
3. Rename libintl-8.dll to libintl-9.dll
4. Place your new libintl-9.dll in the same directory as your executable (e.g., target\debug) to get the fixed libintl to link correctly. If you just copy it and don't rename it, it will link to the broken libintl-9.dll in the Postgres install directory.

I asked if we could expect a non-broken libintl DLL in upcoming versions (after they figure out the performance regression). I don't know yet, but at least we have a fix.

[xxgunnery]

This worked, thank you. Discovering this kind of solution myself would be so far out of my reach.

This is a comment towards Rust in general, more of a vent. I don't blame the maintainers of Diesel for this. There seem to be all sorts of strange errors that pop up with Rust libraries. Some of the most time consuming problems I've ever solved in computers in general have occurred in my couple months developing in the Rust ecosystem. These are all problems that are completely unrelated to my code. Getting discouraged with Rust for problems like these... would be great if I could have a different mindset I guess. I'm wondering if anyone else has had similar experiences as I explain and what they do in general to solve problems like this...

[weiznich]

@xxgunnery I want to clarify one thing here: This issue is not related to rust at all. Its an issue with two C libraries that happens independently of rust or diesel. We somehow just manage to trigger it.

[jfmenard]

Thank you @hut8 for the solution, works great! I just deployed Postgres 16 and it still ship the the wrong lib.
The issue with copying the renamed dll in the target\debug folder is that it is lost each time we run cargo clean. I replaced it in the Postgresql bin folder instead, now it works all the time.

[YenHarvey]

2024 they still havn't fixed it😂

[Kratosgado]

this worked, I think the better option is placing it the postgres bin in the installation folder, after changing the name and copying it.

[YenHarvey]

这是来自动回复邮件。你好，我将在稍后尽快给你回复。