As part of an internal project last year we came across the need to shim a library not accessible at the time through Wine (specifically, NVENC or nvEncodeAPI64.dll). In the process of setting this up, it was noted that there is something of a lack of explanation for how this process works, at least from start to finish. There is definitely documentation to be found for those willing to put in the time and effort to look, but many places only explain part of the larger process. Certainly, all of the tools used here have extensive documentation available, but each is presented (understandably) independently without explaining how they fit together.
Contents
- Introduction to Shimming
- Tools
- Spec files and Stdcall APIs
- Initialization
- Proxy Functions
- Compilation
- Conclusion
Introduction to Shimming
For anyone unfamiliar with the term, shimming is the process of masking a library on a system by creating another library (the eponymous “shim”) with the same filename (possibly renaming the original) and with the same exported set of symbols to be used in its place. There are a variety of use-cases for shims both legitimate and illegitimate. They’re of particular importance to the Wine project where they present an interface for Windows binaries to access functionality outside of the Wine environment. Elaborate shims such as DXVK present an API (in this case, DirectX) that isn’t actually available on the host system by translating all usage of the API to a locally available equivalent (Vulkan) without the user being the wiser.
DXVK actually has the benefit of being an entirely Windows-based shim. Though the process is not officially supported you can actually use DXVK on a Windows machine to have DirectX 11 games run using Vulkan. This is only possible because Vulkan provides its own equivalent to GetProcAddress which presents the same interface on every platform. Therefore when DXVK is running in a Wine environment, it accesses Vulkan the same way it would on Windows.
Wine internally functions largely by providing shims for much of the Windows API, including Vulkan, by calling into equivalent functions on the host machine. In our case, we wanted access to NVENC which did not (at time of writing) have a shim associated with it. These Wine shims are a special case in that they present an ABI in the WINAPI format while actually being unix-type binaries running on the host system. The next section covers the development tools the Wine project has developed for doing this.
Tools
The main tools used in this article are the Winelib development suite. On Ubuntu these are packaged separately from Wine itself, other distros like Arch will include the toolset with the regular Wine package:
-
winegccis a slightly modified GCC compiler for making binaries with a Windows- (or at least Wine-) compatible ABI. There is a correspondingwineg++for building C++ but I’ll stick to C for simplicity. -
winedumphas a couple of different uses. For our purposes, its useful to callwinedump specto grab a list of exported symbols from a Windows DLL. It can also however be used for simple demangling of C++ symbols and dumping different parts of a Windows binary such as file headers and debug info.
Much of the documentation for this process can be found in the Winelib User’s Guide. Though it lacks a concise walkthrough of the entire process of making DLL shim from start to finish, this is a great resource if you want to dig deeper into any particular step.
Spec files and Stdcall APIs
Vital for the process is creating a spec file to create a second set of API symbols to be loaded via WINAPI as opposed to the unix process. If you have access to the DLL you are shimming, you can use the winedump program to create a partial spec file, listing the symbols that are already exposed by the existing DLL. This partial spec file can then be modified to accompany a shim. Alternatively, making one from scratch isn’t very hard as long as you know which function symbols need to be exposed. They’re just plain text and the format is quite straightforward.
Here is an example of a final spec file to use upon compilation:
@ stdcall ExampleAPIGetMaxSupportedVersion(ptr) ExampleAPIGetMaxSupportedVersionProxy
@ stdcall ExampleAPICreateInstance(ptr) ExampleAPICreateInstanceProxy
This closely mirrors the contents of the spec file we used for shimming NVENC, as in this case all the pointers to the remaining functions in the API are in the instance initialized by the second function. To expand on the components of each line:
-
The
@is simply a standin for numbering the symbol, to be automatically assigned by the linker later. -
The
stdcallinstructs Winelib to expose the symbol using thestdcallconvention format, you can use any calling convention but this is the one used by the DLL that we’re shimming so it needs to match. Otherwise they can’t be loaded usingGetProcAddress. -
This is the signature for the function being proxied (eg.
ExampleAPICreateInstance(ptr)in the example above). This includes its list of arguments, but keep in mind this is not actual C types but a more generic set provided by the specfile format. -
Last is the name of the proxy symbol that will be called in place of the aformentioned signature (eg.
ExampleAPICreateInstanceProxyin the example above). The original symbol is not actually reserved for use anywhere in the resulting binary so you can comfortably call into functions with the same symbol name loaded by the corresponding native unix library you’re shimming.
Initialization
You can use the optional DllMain entrypoint function to do global/one-time initialization for simpler setups. (For those needing this on Linux so libraries, an equivalent feature also exists). Commonly you will need to at least do this to dlopen the handle to the underlying library outside of the Wine prefix and then dlsym any functions you need to call into.
Proxy Functions
Each proxy function is used as an interface to the underlying function. At the very least you’ll be calling into the underlying native function or simulating the expected output of doing so, and this is also a really good place for extra diagnostics and logging so you can confirm that the functions you expect to have called are called and with the expected arguments.
Each proxy function’s signature must have the same set of arguments as the function being proxied and then also need to have the correct calling convention macro prepended to their signatures, matching their listings in the spec file. This is necessary for the correct symbols to actually be found when retrieved with GetProcAddress, as the symbol being asked for may have a different name to the function intended to proxy it, as explained up in the example explaining specfiles.
In our case, omitting the __stdcall on functions declared as such in our spec file can either make said functions outright not work, or (worse) cause them to apply the wrong offset to any pointers passed through the shim’s ABI. This can cause problems that are much harder to diagnose or even spot in the first place.
__stdcall NVENCSTATUS NVENCAPI
NvEncodeAPIGetMaxSupportedVersionProxy(uint32_t* version)
{
// This function is expected to be loaded using GetProcAddress and so needs __stdcall
}
__stdcall NVENCSTATUS NVENCAPI
NvEncodeAPICreateInstanceProxy(void *userPointer)
{
// As above
}
static int
nvEncInitLogging()
{
// This is an internal utility function that doesn't need to be accessed by GetProcAddress so it doesn't need __stdcall
}
Digression on logging with WINEDEBUG
To expand a little on diagnostics and debugging, we found it useful to hook into the standard WINEDEBUG environment variable for printing out logging information. This functions as a list of channels combined with the expected minimum level of verbosity (trace, warn, err, etc). The simplest way to use this environment variable is to just parse through the string for the channels relevant to your program/library and use them to selectively control logging options.
There is a simpler way to set up logging through Wine’s own debugging macros, explained in the developer’s guide. In our case though, we already had logging information being printed and just wanted to control it at runtime using environment variables, of which WINEDEBUG was a simple and obvious choice.
Ensure when launching with WINEDEBUG set that you follow the proper format for your specifiers:
- channel specifiers have a
+/-to indicate turning that channel on/off. - channel specifiers are separated with commas
,. - process-specific channel specifiers are separated with colons
:with the initial, environment-wide channel specifiers listed first and per-process overrides afterwards.
Wine itself expects the string to be in this format and it allows you to make enough assumptions to simplify parsing. An example lifted from Wine’s documentation on debug channels:
WINEDEBUG=fixme-all,warn+cursor,+relay
This will turn off all FIXME messages, turn on cursor WARN messages (in addition to ERR and FIXME messages), and turn on all relay messages (API calls).
In our case we wanted to intercept module and loaddll to log information on whether the shim is working correctly. module is more or less a more verbose version of loaddll and we interpreted it as such:
unsigned logLevel = 0;
char* wineDebug;
if (wineDebug = getenv("WINEDEBUG")) {
/* General channels, stop checking if we hit the end or the start of a process name */
char* end = wineDebug + strlen(wineDebug);
char* nextProcess = strchr(wineDebug, ':');
if (nextProcess < end && nextProcess > wineDebug) {
end = nextProcess;
}
for (char* cursor = wineDebug; cursor >= wineDebug && cursor < end; cursor = strchr(cursor, ',') + 1) {
/* Logging class, we only care about trace, we have no warnings while errors we always print anyway */
if (cursor[0] != '-' && cursor[0] != '+') {
if (strncmp(cursor, "trace", strlen("trace")) == 0) {
cursor += strlen("trace");
} else {
continue;
}
}
if (strncmp(cursor + 1, "module", strlen("module")) == 0) {
if (cursor[0] == '+') {
logLevel = 2;
} else if (cursor[0] == '-') {
logLevel = 0;
}
} else if (logLevel < 2 && strncmp(cursor + 1, "loaddll", strlen("loaddll")) == 0) {
if (cursor[0] == '+') {
logLevel = 1;
} else if (cursor[0] == '-') {
logLevel = 0;
}
}
}
/* Process-specific channels */
char* processName = "nvEncodeAPI64.dll:";
char* processChannels = strstr(wineDebug, processName);
if (processChannels) {
char* cursor = processChannels + strlen(processName);
end = wineDebug + strlen(wineDebug);
nextProcess = strchr(cursor, ':');
if (nextProcess < end && nextProcess > cursor) {
end = nextProcess;
}
for (; cursor >= processChannels && cursor < end; cursor = strchr(cursor, ',') + 1) {
// Same loop as above
}
}
}
Compilation
When building, the spec file should be linked as an additional input when making the final binary, similar to if it had been an object file. For example:
winegcc -shared -o nvEncodeAPI64.dll obj/shim/main.o src/shim/nvEncodeAPI64.dll.spec -ldl
After building the shared object you will get an output with a name like libraryName.dll.so. To be detected by Wine, this library needs to be placed in the same location the original DLL would normally be, with the additional .so extension stripped off. Wine documentation implies this last step is unnecessary; however from my experience you do need to do it. Presumably different programs may have different ways of checking whether the library exists in the first place before attempting to load it and so having the additional extension can cause confusion.
Combining with native Windows DLLs
It should be noted that this resulting Wine shim can only interact with Wine/Linux libraries (eg. .so libs). If you need to interact with other Windows DLLs that do not have such shims, then you will need to compile another library as an actual Windows DLL using MSVC or a cross-compiler like MinGW. You can even have this work as a shim in its own right by leaving the .so extension on the above Winelib shim and opening it with its full name from within the Windows DLL, for example:
gNvEncLibHandle = LoadLibraryA("nvEncodeAPI64.dll.so");
if (!gNvEncLibHandle) {
fprintf(stderr, "SPS-NVENC-T: Failed to open nvEncodeAPI64.dll.so\n");
}
With this nested pair of shims you can have access to every library on both sides of the ABI change, both Windows and Wine/Unix.
Conclusion
In review the process is pretty straightforward:
-
Create a
specfile naming all of the functions you want to expose and the Proxy functions to be invoked when they are called. -
Write your proxy functions in the form of a Unix/Wine library.
-
Compile and link with
winegcc, including thespecfile from the first step during the linking stage.
If you require any additional detail on any particular step, you can always consult the Winelib user’s guide, particularly the section on spec files.
Using this we’ve been able to easily provide access to nVidia Hardware Encoding to any given Windows program that needs it even if Wine haven’t gotten around to shimming it themselves.