* Now that we no longer do anything when unloading the overlay DLL,
remove the injection of FreeLibrary. This also drops some undefined
behavior.
** If we inject into rendering (D3Dxx.cpp) we hold a self-reference to
prevent to ever be unloaded. For this case, there is no issue as the
hooks will always exist.
** In case of no rendering-injection, our module can actually be
unloaded. In that case, MyFreeLibrary would call the original function
which in turn would lead to a call to DllMain with DLL_PROCESS_DETACH,
at which point we restore the hooks to their original equivalents in
dllmainProcDetach.
However, afterwards, execution returns to our MyFreeLibrary function,
whichs code is no longer the code we began executing.
** This also leads to the question whether the non-trampoline hooking
ever worked/even works. We restore, call the original (which is already
code that no longer exists) and then inject again.
* When unloading the overlay DLL freeD3D9Hook was called *after* unloading the DLL. This is incorrect.
** For applications that actually use D3D and we inject into, the DLL is
never unloaded because we hold a refernce to it ourselves. So in this case,
the problematic code is never executed.
** When not actually injecting, the DLL will be unloaded, and then the
function freeD3D9Hook is called which is from that DLL.
** As we’re executing code in undefined space which previously held the
function, this may or may not crash.
* Remove the freeD3D9Hook function
** The function freeD3D9Hook just resets some fields to NULL and a flag
to false. As the DLL is unloaded anyway, these are never used again.
Hence, we can just remove it altogether (rather than just calling it
before unloading).
Normally, Mumble itself will terminate the helper processes.
But if Mumble crashes, or is manually killed by the user, it
will not be able to terminate the helper processes itself.
In order to fix this, we create a way for the helpers to know
when their parent process has terminated.
This is implemented by creating an inheritable process handle
in Mumble, and passing its value to the helpers.
The helpers then WaitForSingleObject() on the parent handle, and
exits with status code 0 if it the WaitForSingleObject() call
returns successfully.
When Qt terminates a QProcess via the terminate()
method, it sends a WM_CLOSE message.
However, the overlay helper did not know of WM_CLOSE,
so it would quietly ignore it.
This commit teaches the overlay helper to exit on
WM_CLOSE.
Vtable offsetes is almost all that we store in the shared memory,
and they vary by architecture.
So, don't share the memory between arches. Instead, crate
arch-specific shared memory regions.
This changes Mumble on Windows to run a helper process,
mumble_ol.exe, instead of loading mumble_ol.dll itself.
Prior to this change, Mumble would load mumble_ol.dll and
call PrepareD3D9() and PrepareDXGI() to set up the overlay.
Then, if the overlay was enabled, it would call InstallHooks()
to enable automatic injection of the overlay into new processes.
Similarly, it would call RemoveHooks() to disable automatic
overlay injection when the overlay was disabled in Mmuble.
With this change, Mumble instead runs a helper process called
mumble_ol.exe. This process sets up the overlay (equivalent
to calling PrepareD3D9(), PrepareDXGI() and calling InstallHooks()).
While running, it'll automatically inject the overlay into new
processes, just like Mumble itself did previously. On normal exit,
the helper process calls RemoveHooks() to ensure automatic overlay
injection is disabled.
To enable the overlay, Mumble starts the helper process.
To disable the overlay, Mumble terminates the helper process.
If the helper process dies when it is supposed to be running,
Mumble restarts it to ensure that overlay injection keeps
working as intended.
This change is the first part of enabling both an x86 and an x64
overlay to be active in Mumble at the same time. Since we cannot
load a 32-bit DLL into a 64-bit process (or vice versa), we need
a helper process for each architecture to reach our goal.
Note however that this commit in itself does not make it possible
for Mumble to run both an x86 and an x64 overlay at the same time.
This will come later.
This commit adds MinHook as a 3rd party
dependency and adds an alternative HardHook
implementation that makes use of MinHook.
This new MinHook-based HardHook implementation
allows us to provide an overlay for Mumble on
Windows x64.
The x64 overlay hasn't seen much testing in
real-world x64 games, except some minor testing
for World of Warcraft running in x64 mode, where
it works just fine.
There seems to be a compatibility with the Uplay
overlay, which causes Far Cry 4 to crash at the
"Press any key to continue" screen that is shown
just after launching the game. However,
Assassin's Creed: Unity works fine, so it might
just be a Far Cry 4 issue.
The x64 overlay also seems to interoperate with
the Steam overlay just fine.
I think this is a good starting point for the
feature. Let us get it into snapshots and let
us try to squash any addition bugs we find.
This changeset makes the following modifications:
1. The overlay DLL is changed to always consider the built-in blacklist,
even if a blacklist is present in the registry.
For Mumble users who have been running with the old blacklist behavior,
this means that we'll be doing some duplicate checking for blacklist
entries, but I don't think this matters in practice.
2. Settings.cpp is changed to not do anything with overlay_blacklist.h.
Effectively, this means that qslBlacklist now represents the items
the user has added to the blacklist, and not the combination of both
an outdated built-in list and the user's own entries.
3. OverlayConfig.cpp is changed to always show all entries from
overlay_blacklist.h. Entries from the built-in list are 'disabled',
so they can't be interacted with.
For more information, see PR #1461
createSharedDataMap can fail to map the shared data used by other
functionality in the overlay dll. In this case pointers including
"sd" remain NULLd. With the recent overlay changes the modified
shared data structures changed, triggering this case when running
different versions. This crashed Mumble on startup due to an
unchecked dereference of the "sd" pointer.
Also when extracting createSharedDataMap during refactoring behavior
was changed. The overlay kept injecting itself even if the mapping
failed. This also addresses this by making createSharedDataMap return
a bool indicating success or failure.
* Make cpp-local functions static so they are local file/translation unit
scope.
* Remove unused variables uiAudioCount and bVideoHooked
* Comment extern declarations, where there definitions are located.
* Improve constant naming JUMPOP_OFFSET => JMP_OP_SIZE.
* Remove extern declaration from method definitions.
* Adjust type declaration formatting to typical formatting.
* Use voidFunc typedef to pass around function address pointers.
** C++-style casts instead of C-style raw casts.
* Other minor formatting (/whitespace) adjustments
* Fix log text.
* Improve log text.
* Consistent, improved varnaming in d3d9.cpp on address calculation;
naming fn and base.
* Separate DXGI logic from D3D10 into a separate file dxgi.cpp
* Structure code and logic, introduce additional functions
* Introduce constants
* Code commenting
* d3d9.cpp:
** Access devMap via std::find rather than operator[]
** Introduce class Stash to temporarily set variable value and revert on
destruction
** Move logic to function findOriginalDevice
** Use widestring for modulename (consistency)
** Implement Hook for Direct3DCreate9Ex
** Move function IsFnInModule to common lib.cpp/.h
* opengl.cpp:
** Add whitespace to format code
** Fix logging scope prefix
** Remove static global variable
Extend to handle library freeing.
Codeformatting and -cleanup, robustness, and make debugoutput more consistent.
* Indroduce hook for freeing loaded DLLs.
This separates logic between loading and freeing, leading to adjustments to
the recently introduced checks for freed DLLs.
* In HardHook::reset() do a clean and complete reset rather than minimal.
* Memvarinitialisation in HardHook
* Remove logically unused variable bPresenting in d3d9.cpp (always false)
* In d3d10.cpp prefix debugoutput with D3D10 consistently; no more DXGI
which is ambiguous with a d3d11 file that also uses DXGI.
* Consistently use ods instead of fods in the overlay files.
Not in HardHook yet, as that class is used in the Mumble client as well atm
* Fix forwarded return value types (LONG to ULONG)
* TODOs for hook-call-logic in multiple places
* Commenting, formatting and scope / order adjustments
* Introduce variables with constant values (replacing magic/undescriptive constants/numbers)
This is confirmed to fix the missing overlay in Guild Wars 2.
When D3D was used by a process thus loading the DLL, but then unloaded we did not inject on subsequent loading of the D3D DLL (in the same exe). Guild Wars 2 did un- and reload the DLL when switching from their launcher to the game.
This commit introduces HardHook::reset() and checks for inactive hardhooks for a newly loaded DLL.
The changes were submitted on sourceforge
https://sourceforge.net/p/mumble/bugs/909/#bf9e
We were getting unbalanced AddRef/Releases on Direct3D9 apps running on Windows 8 that ran
fine on Windows 7. Windows 8 seems to be doing somthing funky with the internal ref count
of IDirect3DDevice9.
For now, piggyback on the orignal ref count methods to be able to determine when to release
ourselves. And cross our fingers that we won't be getting useless ref counts from some other
overlay.
This is a band-aid solution until we find something better.
* uninitialized membervars,
* AudioOutput.cpp: fix delete on array to delete[]
* OSS.cpp: close file descriptor in false data case
* OverlayEditorScene.cpp: rm duplicate logic
* fix ifndef to match usage of declared variable
* member var initializations
* check for null (ds in d3d9 as some lines above),
* lower scope of var decl.,
* swap bufsize check and array dereference so check is before! deref,
* initialize member vars in constr.
Factor fods and its base function out to ods.h/cpp and use it in HardHook as well as lib functionality. This also fixes previous divergence between the two ods functions.
The amount of stack allocated for reading the black/whitelist values from the registry caused a stack overflow when closing solutions in vs2010. The fix is to switch to dynamic allocation on the heap for this variable. This fix also contains minor refactoring for better readability and adds a bit of additional debug output.
