Repository navigation
7.4.0 Linux native shared libraries in module not loaded using the RID directory #20740
Description
Prerequisites
- Write a descriptive title.
- Make sure you are able to repro it on the latest released version
- Search the existing issues.
- Refer to the FAQ.
- Refer to Differences between Windows PowerShell 5.1 and PowerShell.
Steps to reproduce
I have powershell module that uses native shared libraries packaged in the module.
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/System.Data.SQLite.dll
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/osx-x64/SQLite.Interop.dll.dylib
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/linux-arm/SQLite.Interop.dll.so
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/linux-x64/SQLite.Interop.dll.so
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/linux-arm64/SQLite.Interop.dll.so
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/win-x86/SQLite.Interop.dll.dll
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/osx-arm64/SQLite.Interop.dll.dylib
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/win-arm/SQLite.Interop.dll.dll
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/win-arm64/SQLite.Interop.dll.dll
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/SQLiteConnection.dll
/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/win-x64/SQLite.Interop.dll.dll
7.3.10 finds and loads these shared libraries.
7.4.0 does not look for them on Linux under the RID directory
Expected behavior
3463: file=/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/SQLite.Interop.dll.so [0]; dynamically loaded by /opt/microsoft/powershell/7/libcoreclr.so [0]
3463: file=/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/libSQLite.Interop.dll.so [0]; dynamically loaded by /opt/microsoft/powershell/7/libcoreclr.so [0]
3463: opening file=/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/linux-x64/SQLite.Interop.dll.so [0]; direct_opencount=25
3463: symbol=sqlite3_set_authorizer; lookup in file=/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/linux-x64/SQLite.Interop.dll.so [0]
3463: binding file /home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/linux-x64/SQLite.Interop.dll.so [0] to /home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/linux-x64/SQLite.Interop.dll.so [0]: normal symbol `sqlite3_set_authorizer'Actual behavior
3223: file=/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/SQLite.Interop.dll.so [0]; dynamically loaded by /opt/microsoft/powershell/7/libcoreclr.so [0]
3223: file=/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/libSQLite.Interop.dll.so [0]; dynamically loaded by /opt/microsoft/powershell/7/libcoreclr.so [0]
3223: file=/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/SQLite.Interop.dll.so [0]; dynamically loaded by /opt/microsoft/powershell/7/libcoreclr.so [0]
3223: file=/home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/libSQLite.Interop.dll.so [0]; dynamically loaded by /opt/microsoft/powershell/7/libcoreclr.so [0]
| /home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/SQLite.Interop.dll.so: cannot open shared object file: No such file or directory
| /home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/libSQLite.Interop.dll.so: cannot open shared object file: No such file or directoryError details
| Unable to load shared library 'SQLite.Interop.dll' or one of its dependencies. In order to help diagnose loading
| problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment
| variable: /opt/microsoft/powershell/7/SQLite.Interop.dll.so: cannot open shared object file: No such file or
| directory
| /home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/SQLite.Interop.dll.so:
| cannot open shared object file: No such file or directory /opt/microsoft/powershell/7/libSQLite.Interop.dll.so:
| cannot open shared object file: No such file or directory
| /home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/libSQLite.Interop.dll.so: cannot open shared object file: No such file or directory /opt/microsoft/powershell/7/SQLite.Interop.dll: cannot open shared object file: No such file or directory /home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/SQLite.Interop.dll: cannot open shared object file: No such file or directory /opt/microsoft/powershell/7/libSQLite.Interop.dll: cannot open shared object file: No such file or directory /home/bythesea/.local/share/powershell/Modules/rhubarb-geek-nz.SQLiteConnection/1.0.118.0/libSQLite.Interop.dll: cannot open shared object file: No such file or directory
Environment data
Name Value
---- -----
PSVersion 7.4.0
PSEdition Core
GitCommitId 7.4.0
OS Ubuntu 22.04.3 LTS
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0Visuals
Important detail -
Running the test.ps1 from within a PowerShell 7.4.0 session works, but running "test.ps1" directly from a bash shell (using the shebang) doesn't.
Activity
237dmitry commentedon Nov 21, 2023
Maybe try to export
LD_LIBRARY_PATHto set path to home shared libraries? In~/.profile:Or do similar in $profile:
bording commentedon Dec 12, 2023
rhubarb-geek-nz The folder structure you have listed seems a bit unexpected to me. If the
dep.jsonfiles are being used at all to the platform-specific dependencies, then all of thoseRIDfolders should be inside aruntimesfolder.The deployed folder structure should match the layout from the NuGet package, with the
nativesubfolders, etc:bording commentedon Dec 12, 2023
It really should be. If it's not, then I'd argue PowerShell is doing something wrong.
iSazonov commentedon Dec 13, 2023
Please update the OP with the repro fact.
iSazonov commentedon Dec 13, 2023
Can you dump env variables in the script and in console. And compare with 7.3 version.
iSazonov commentedon Dec 13, 2023
rhubarb-geek-nz I mean to put
gci env:in the test script and see results.iSazonov commentedon Dec 13, 2023
rhubarb-geek-nz Thanks for additional information! I don't see difference in envs too. Perhaps you could debug the broken scenario. You need to look the method
internal static IntPtr NativeDllHandler(Assembly assembly, string libraryName).iSazonov commentedon Dec 13, 2023
daxian-dbw commentedon Dec 13, 2023
rhubarb-geek-nz, you mentioned the following in the PR description:
Can you please double check if the repro steps work as expected on v7.3.10?
I'm asking because there is no change that we know of in the native library resolver between v7.3.x and v7.4.0, so if it's true that the reported issue doesn't happen on v7.3.10, then the root cause must be in somewhere else instead of the native library resolver.
iSazonov commentedon Dec 13, 2023
Michael Klement (@mklement0) Have you any ideas why shebang scenario doesn't work?
iSazonov commentedon Dec 13, 2023
rhubarb-geek-nz I still wonder that you reported
SQLite.Interop.dll.soandSQLite.Interop.dll.dll- why the names have two file extensions?iSazonov commentedon Dec 13, 2023
https://nuget.info/packages/Stub.System.Data.SQLite.Core.NetStandard/1.0.118.0
SQLLite distributes all native assemblies with
dllextention! Steve Lee (@SteveL-MSFT) we shouldn't change extension in related PR.iSazonov commentedon Dec 14, 2023
rhubarb-geek-nz I know only one package - SQLite - where dll extension is used instead of standard platform extension. I believe it is very edge case (but it's in the law.). So obviously current pwsh behavior is a bug we should fix without a fallback to old behavior. I realize this breaks your module, but you can easily fix it by using two versions of the libraries with old and new names. And I'm surprised you didn't report this problem early, but just used a workaround.
iSazonov commentedon Dec 14, 2023
Thanks for the reminder. I was sure it was supposed to be fixed in SQLite. I still think so now. Obviously it generates unpredictable behavior in other applications. It's always a bad thing to deviate from de facto standards without justifiable need. (Although the analysis showed that we should fix the pwsh code too.)
daxian-dbw commentedon Mar 19, 2024
This issue is actually a duplicate of #21319, only that this one surfaced it with native library loading while the other one is GAC assembly loading.
The root cause is that the native lib resolver and GAC assembly resolvers are not yet registered when running
Import-Modulebypwsh -c. It's an unexpected regression from the PR #18356, specifically this line of change inExternalScriptInfo.We depend on
ClrFacadeto be referenced early at startup, but seemingly unrelated change could break that assumption.microsoft-github-policy-service commentedon Mar 21, 2024
📣 Hey rhubarb-geek-nz, how did we do? We would love to hear your feedback with the link below! 🗣️
🔗 https://aka.ms/PSRepoFeedback