BASeCamp Programming Blog

And now for something completely different. By which I mean, A post smaller shorter then 1000 words I would have never expected this to occur; in fact by posting this I am throwing away a lot of work on other post drafts that I have yet to actually publish because I think they are “too short”. Although, come to think of it, I’m equally likely to insert a nonsense story at the end of this in order to add some fiber.

In any case, recently I’ve been rather heavily adding youtube videos. Almost all of them are gameplay; many of them are minecraft; some are doom; etc. They (almost) all share one thing in common – commentary.

Now, Although I haven’t actually gotten any messages (being that I have very few subscribers and so forth) on this, but I have seen people provide advice for making “let’s play’s” with advice something like “never talk about stuff unrelated to the video” obviously, I cannot apply this to my since that pretty damn well defines what one of my videos is. Only in one of my videos will I ramble about fighting cazadors in fallout 3, and somehow not clue in at the time that Cazadors are in New Vegas and not Fallout 3. Another suggestion I read about is that the sound of the keyboard/mouse/controller shouldn’t be heard. I don’t think it really matters, to be frank. In fact one could say it adds an extra dimension to the affair. Of course if you have a nice heavy keytronic keyboard (the mechanical switch type… drools ) then those will be pretty loud and if you are typing anything you will almost certainly find that nobody can hear a damn thing you say.

Also: I’m well aware that my attempts at humourous lyrics to go with the otherwise wordless music in the games has a tendency to fail. These things aren’t planned; if I could plan the lyrics they would be a lot more smooth; it’s the “on-the-fly” nature of it that makes it entertaining, dammit. Same goes for my youtube videos demonstrating progress on any of my programming projects.

922 total views, no views today

There are a lot of people, forum posters and the like, vouching for any number of “addon” programs that allow you to install DirectX 10 on your Windows XP PC, even though DX10 is a Vista+ feature.

There are torrents that claim to have DX10, but half of them are just a few Vista DirectX DLLs zipped up. Instructions say something like “copy to your windows folder, now u have directx10 and your games run faster”

I even found one that claimed to allow you to play, say, Crysis and Ultra High (which usually required DX10) and, when you ran DXdiag after installation, it said DirectX 10.1, so there is NO way its fake!

I say it’s time to truly examine this: normally I would dismiss it off-hand for being stupid, but what the heck, I have a few minutes to spare, may as well give it a proper debunking.

The first step, was, of course to find it. Google is not hard to use, of course. It led me to this page:

http://www.techmixer.com/download-directx-10-for-windows-xp/

yet another blog, it would appear.

In any case, after a bit of running around (the things I do for a blog post) I finally got the ZIP file for this nonsense. I of course don’t have XP installed on a machine so I used a trusty VMWare install, and a XP install that I had sitting about.

I ran the installer, but, while I was running it, I was running Process Monitor to see exactly what it was fiddling with. Once the install was done, I had a logfile from that that I opened on this machine (outside the VM) for inspection. Some interesting entries. First, it explicitly set the version number like so:
RegSetValue HKLM\SOFTWARE\Microsoft\DirectX\Version 4.10.0.5515

This is notable because even my Win7 install identifies my version as 4.09.00.0904; amazingly, this tiny installer was not only able to install a version of DX10 that had a later version then DX11, but it was also able to do so using a tiny 20 MB installer. the technical sycophant might be impressed, but I’m not. Of course the real reason is that it had to explicitly change the version, otherwise, the new version of DXDiag it installed (which is one of the few things it really does copy over) won’t translate that new version number into DirectX10. I also used the results to gather exactly what files it was installing; this is what I found being installed:

C:\WINDOWS\system32\d3d10.dll
C:\WINDOWS\system32\d3d10core.dll
C:\WINDOWS\system32\kernel32new.dll
C:\WINDOWS\system32\msvcrtnew.dll
C:\WINDOWS\system32\ntdllnew.dll
C:\WINDOWS\system32\d3dx10.dll
C:\WINDOWS\system32\d3dx10_33.dll
C:\WINDOWS\system32\d3dx10_34.dll
C:\WINDOWS\system32\d3dx10_35.dll
C:\WINDOWS\system32\d3dx10_36.dll
C:\WINDOWS\system32\d3dx10_37.dll
C:\WINDOWS\system32\d3dx9_33.dll
C:\WINDOWS\system32\d3dx9_34.dll
C:\WINDOWS\system32\d3dx9_35.dll
C:\WINDOWS\system32\d3dx9_36.dll
C:\WINDOWS\system32\d3dx9_37.dll
C:\WINDOWS\system32\dwmapi.dll
C:\WINDOWS\system32\dxgi.dll
C:\Program Files\Common Files\dx.reg
C:\Documents and Settings\Administrator\My Documents\crysis.rtf
C:\Documents and Settings\Administrator\My Documents\msvcrt(DEBUG).dll
C:\WINDOWS\system32\dxcpl.exe
C:\WINDOWS\system32\DXErr.exe

After the install it prompted me to start DXdiag. presumably so it could show off that yes indeed the version number is different. Any idiot can change the version number, though. It’s a simple registry value.

So, anyway, playing the part of such a naive user, I started DXdiag, which of course complained about any number of errors (probably related to the lack of hardware acceleration for most of DX in the VM though) In any case, once started, I was of course greeted to the new version number:

DX10 on XP. yeah right.

Amazing a quick install, and I’m now running DX10! and it was only like a 40 meg file, unlike DX9, so not only did I upgrade from DX 8.1 (included with XP) to DX 10, but I did so without downloading hundreds of megabytes. How very wonderful this all was, so happy and whimsical. So I tried a few small DX programs. they had to run with the software renderer because of VMWare but oh well.

They all crashed, with various DLL entry point not found errors.

So, I installed DX 9.0c. the errors went away. Curious. why would a DX10 demo application need DX9 installed?

Either way, I decided the various D3d10 files, specifically those with “new” on the end, bore very close examination, namely with dumpbin. I copied them out of the VM to a temporary folder for examination with the trusty dumpbin utility. I started with d3d10.dll, despite my curiousity with regards to the “new” files. So I ran a good ol /EXPORTS on the file with dumpbin, something which has never failed on me before. (at least with a valid windows image). Needless to say, there is a first time for everything:

Mysterious. So, I decided to try an /IMPORTS, see what this dll is importing from:

 
Microsoft (R) COFF/PE Dumper Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file d3d10.dll

File Type: DLL

  Section contains the following imports:

  Header contains the following bound import information:
    Bound to d3d10core.dll  [4549BCBE]  Thu Nov 02 02:39:10 2006
    Bound to msvcrtnew.dll  [4549BD61]  Thu Nov 02 02:41:53 2006
    Bound to ll  [4549BD80]  Thu Nov 02 02:42:24 2006
      Contained forwarders bound to ll  [4549BDC9]  Thu Nov 02 02:43:37 2006
    Bound to LL  [4549BCD2]  Thu Nov 02 02:39:30 2006
    Bound to ll  [4549BD27]  Thu Nov 02 02:40:55 2006
    Bound to ll  [4549BCD3]  Thu Nov 02 02:39:31 2006

  Summary

        F000 .data
        A000 .reloc
        1000 .rsrc
       E7000 .text
 

there’s d3d10core.dll as well as msvcrtnew.dll; I used a few other features of dumpbin to try to examine the d3d10.dll file. It was clear that the file was corrupted on purpose; various fields in the header were screwed around so that it was difficult to dump the headers properly. Essentially, the file was for all intents and purposes corrupted. In any case, after examining it with sysinternals “strings” utility, it became clear that it had an embedded shader processor, and a lot of other string data inside, as well as a list of the exports (the ones that dumpbin refused to list because the export table pointer was invalid); they seemed to simply be DX10 function names.

At this point, I turned to the import table to further examine how exactly calls to this D3d10 dll made it to the graphics pipeline. d3d10core seemed like a good start.

Yet again, I hit a stumbling block as I tried to view the exports. Why they did this on purpose, I do not know, but clearly there is something they don’t want people like me to know about this dll, and I’m going to find out what it is. Checking the import table, it was empty. Which, at least for something like this, is impossible. So I once again turned to the strings utility to try to make some sense out of the files contents. through some effort, I was able to find all the dll names referenced in the file; KERNEL32.DLL,USER32.DLL, and dxgi.dll, as well as that msvcrtnew.dll that was referenced also by the other file. dxgi, being that it was one of the files “distributed” with this set, seemed like the next place to go looking.

I was, to my surprise, able to run dumpbin /exports on the file without a error about the pointer to the export table being out of range:

 
Microsoft (R) COFF/PE Dumper Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file dxgi.dll

File Type: DLL

  Section contains the following exports for dxgi.dll

    00000000 characteristics
    4549B462 time date stamp Thu Nov 02 02:03:30 2006
        0.00 version
           1 ordinal base
           7 number of functions
           7 number of names

    ordinal hint RVA      name

          2    0 000050B7 CreateDXGIFactory
          3    1 00012889 DXGID3D10CreateDevice
          4    2 00010087 DXGID3D10CreateLayeredDevice
          5    3 0000FF84 DXGID3D10GetLayeredDeviceSize
          6    4 000126F5 DXGID3D10RegisterLayers
          1    5 0000FDC2 DXGIDumpJournal
          7    6 0001FCBE DXGIReportAdapterConfiguration

  Summary

        6000 .data
        2000 .reloc
        1000 .rsrc
       26000 .text
 

clearly, yet again, several DX10 oriented function names are being exported. I also noticed that I have a dxgi.dll in my windows folder; presumably because I actually have DX10/11. So I did a quick compare. for the most part, it seemed that the dxgi from the XP DX10 set was exporting a subset of the functions from the Win7 install, but since win7 is DX11 there wasn’t really a way to know. Aside from, of course, taking the dxgi from Vista and seeing what it said. Which is, of course, what I did; and I ran a dumpbin /exports on it as well:

 
Microsoft (R) COFF/PE Dumper Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file fromvista\dxgi.dll

File Type: DLL

  Section contains the following exports for dxgi.dll

    00000000 characteristics
    4549B462 time date stamp Thu Nov 02 02:03:30 2006
        0.00 version
           1 ordinal base
           7 number of functions
           7 number of names

    ordinal hint RVA      name

          2    0 000050B7 CreateDXGIFactory
          3    1 00012889 DXGID3D10CreateDevice
          4    2 00010087 DXGID3D10CreateLayeredDevice
          5    3 0000FF84 DXGID3D10GetLayeredDeviceSize
          6    4 000126F5 DXGID3D10RegisterLayers
          1    5 0000FDC2 DXGIDumpJournal
          7    6 0001FCBE DXGIReportAdapterConfiguration

  Summary

        6000 .data
        2000 .reloc
        1000 .rsrc
       26000 .text
 

I’m seeing double! I exclaimed. Now curious, I ran a quick fc…

 
D:\dx10files>fc fromvista\dxgi.dll .\dxgi.dll
Comparing files FROMVISTA\dxgi.dll and .\DXGI.DLL
FC: no differences encountered
 

the dxgi.dll they provide in their “DX10 for XP” pack, is, in fact, the DXGI from a base install of Vista. This is the RTM version of Vista, by the way, not SP1 or SP2. This explained why the file was dumpbin /exports-able, since it was MS produced and not created by whomever was responsible for the others. So far, we have the following; programs link to d3d10.dll, which in turn links to d3d10core, which links to dxgi, which links to a number of Vista/7 only functions. Examining the assembly, however, it becomes clear that when dxgi.dll is being loaded by d3d10core.dll, it forcibly changes the import table of dxgi.dll to use the various “new” dlls; for example, while dxgi.dll imports functions from ntdll, the import table is updated on the fly so that the calls to ntdll don’t go to the XP ntdll.dll (where the functions required aren’t present) but rather to “ntdllnew.dll” which was in fact, ntdll.dll from Windows Vista RTM; additionally, so was dwmapi.dll. So, I dug a little deeper. ntdll, do my understanding, was responsible for almost any User-mode to kernel mode context switch; additionally, ntdll.dll is mapped into the process address space when the application starts; so the game trying to use this special D3d10.dll (most likely by accident) would in fact be mapping not only the XP ntdll.dll into it’s process (at startup) but also a second ntdll (the windows Vista ntdll) implicitly when it loads the d3d10 dll. Normally, you cannot map two files with the same name into the same process. thus, the creator of this little sideshow renamed the Vista ntdll.dll to ntdllnew.dll; this is also the reason for the fixups to the import table of dxgi when it’s loaded. (so that instead of importing functions and calling functions in ntdll (on the system) it is instead calling ntdllnew.dll which d3d10core.dll loads on it’s own)

Somehow, at least, I’m guessing (I haven’t actually seen it in action) this little rube-goldberg device allows games (well, not actually, I’ll get to that in a second) to load and call into d3d10.dll on windows XP.

kernel32new.dll is also a hacked version. One of it’s imports however comes from d3d9.dll; so it seems we have d3d10.dll calling d3d10core.dll which in turn loads dxgi.dll from windows vista, hacks it’s import table to call into the Vista ntdllnew.dll rather then the xp ntdll.dll that it would try to load (and fail to work), additionally, imports to kernel32.dll are fixed up to kernel32new.dll (since the dll is loaded in process this affects the running game/application as well) which in turn will perform it’s own special processing on directX related stuff to call d3d9.dll for some reason, although why the d3d9 functions would be being called from kernel32.dll is beyond me.

The other various dlls are for the most part duplicates of those files from a RTM vista install. Given all of this, we can conclude that the arrangement isn’t to actually add DX10 to your machine but rather to emulate it; rather then actually add dx10 capabilities, it is merely giving programs the ability to access the dx10 dlls as they normally would and then at some point later translating it into a call to the native DX api; it seems that it expected DX9 to be installed as well since the entire thing fell apart when I first tried it with the standard 8.1 that XP comes with.

Conclusion

So the question now is, “who uses it”?

The answer? technical sycophants whose understanding of dll import/export tables is far below what they think, and who consider themselves “extreme gamers” that stick to 7 year old operating systems for reasons they pull out of their ass. MS didn’t “artificially” limit DX10 to Vista and higher in some sort of heavy handed move to get gamers on Vista. They did so because making it backward compatible and testing all the various supported scenarios in that instance would be infeasibly arduous. Additionally, they probably realized you’re a bunch of tards who complain about the smallest thing anyway and that you’re not worth the time to please. But it’s ok, since this other fellow has totally hacked together a solution that let’s you play your games using the DX10 setting that simply calls into these amusingly obfuscated dlls that are trying very hard to hide the fact that they are really a DX9 wrapper for DX10. (The standard DX10 libraries don’t have any DX9 references embedded. I particularly find it interesting that they would [i] purposely [/i] obfuscate the export table of several of the files. Do the DX10 SDK samples run? Yes. Do they use “DX10″ no. And they don’t run very well, either. You aren’t going to be using DX10 hardware on windows XP. You might be able to play DX10 games really slow via their cleverly disguised DX10->DX9 interface translator, but it seems rather pointless. It’s like putting the body kit of a porsche on a Chevette. It’s still a bloody chevette.

EDITS:

Found a MSDN blog about why DX10 wasn’t created/released for XP:

http://blogs.msdn.com/b/ptaylor/archive/2007/02/14/why-dx10-wasnt-created-on-xp-and-why-it-isnt-in-xp.aspx

Given the new features in the driver model and hardware ( with GPU task switching, GPU memory management and more ) all of which require kernel support – hoisting a driver layer like that on XP is rewriting it to be Vista. FWIW, the MS hw developer page has the graphics logo requirements and it explicitly mentions these GPU features as being required. They are essentially hidden features that API programmers and end-users never see.

Basically, even creating a Translation layer would be a waste of time, which is sort of proven in a way by the fact that the “Alky” project has been dead for ages and barely supported running even the most basic DX10 SDK samples.

Also:

http://blogs.msdn.com/b/ptaylor/archive/2007/06/28/dx10-on-xp-round-2.aspx

1,448 total views, 8 views today

As with most of my projects, progress is usually slow because it’s something I do “when I feel like it” rather then on a schedule. Because of this I often have a lot of “design-time” where I don’t actually write code but idly wonder about what I could do to make it better.

One consideration I recently decided on was supporting the packaging of the various files — images, sounds, etc — into a zip file. This came about as a result of my experience (easy enough) of using gzipStream to make the size of the serialized LevelSet data a tad smaller (the default levelset, which is 5 levels, still weighs in at about 145KB; most of this is because it stores a lot of PointF structures from the PathedMovingBlock that I recently added, but I won’t get into that. In any case, I considered at the same time- why not allow the loading of files from a zip?

Of course, this is hardly a new idea. I never thought it was. placing the application/game data into one monolithic (and easier managed file-system wise) file is ancient; games have been doing it for ages. In either case, since I had decided to use ZIP (rather then, say, define my own package file like I did with my Visual Basic BCFile project) I had to be able to work with zip files.

Thankfully, there are a lot of free solutions to read,extract, and otherwise deal with zip files from .NET. after some searching, I decided on dotnetzip

. The reasons I decided on it were simple: it’s easy to use, and it supports both extracting as well as creating zips. I’d rather learn one library rather then several to perform the various tasks in the future. The first step was to decide how it was all going to be done.

A little overview of how I had it arranged in the code may help. The Sounds and Images are managed by classes called cNewSoundManager and ImageManager respectively. recent changes have caused me to rewrite the SoundManager to create the “New” SoundManager, which supports different adapter classes for using various sound libraries (of which BASS.NET has become my top choice).

Thankfully, during the creation of Both ImageManager and SoundManager, I made it able to accept a number of paths in their constructors via a overload accepting an array of Strings. In fact, it was the “low-level” implementation for loading; the other overrides simply transformed their data into an array and called it, so I knew that worked. Although my ImageManager could probably be modified to load files from a stream (the ZipFile class can retrieve a inputstream for the files in a zip) the SoundManager could not feasibly do so; many sound libraries will only load from files, and since most of them are really wrappers around even lower level libraries, I couldn’t optimistically assume that I would always be able to convert a filestream into something I could use; I realized that the “Driver” classes could always take the passed in stream and create a file, but that sort of redirects the issue. Instead, I decided to leave it as-is (loading from files) and instead make it so that during the bootstrap phase (where it loads images and sounds) it also extracts the contents of any zip files that are in the application data folder as well.

I chose to change the extension I would use to bbp (Base Block Package) so that the game won’t get confused or screwed up if a standard ZIP happens to be in the same folder. The first question however is where I would extract these files too; obviously it was a temporary operation, so I opted to choose the system Temp folder, which is easily obtained in .NET via Directory.GetTempPath(). I then decided that as I enumerated each zip, they would be extracted not to this temp folder but to another folder that would be created beneath it; this way, the files in each ZIP file can have conflicting names, and still extract properly. (although at this time that will cause both ImageManager and SoundManager to throw a fit, I decided it best to not add a third class that didn’t understand that files in different folders can have different names). The next problem was easy; I simply took all the various folder names and added them to the arrays I passed to the SoundManager and ImageManager constructor. Easy as pie.

Now I needed to make sure that when the program terminated, the files were deleted. During startup it detected if the special temp folder existed and deleted it, but it would be ideal if that folder could also be deleted when the program is closed normally. the problem here is that I was initializing all of this in a static constructor (well, not really, since apparently Mono didn’t like me using static constructors for this, but it was still a static routine). There is, however, no Static “Destructor” that I could use. So I opted to create a small private class implementing IDisposable that I could create as a static variable and initialize with the name of the temporary folder to delete; the Dispose() method would then delete it, easy as pie.

However, upon testing, I encountered an error; apparently, the handle was still open in the dispose routine. After a little digging, it was clear that the problem was at least partially as a result of the Image.FromFile() method, which apparently caches that it was taken from a file as well as the filename and will keep it open as long as the image is around; since I couldn’t always be sure that the temporary class would be disposed after the ImageManager (and therefore the various Images it holds) it was difficult to make sure they were closed.

However, I decided to change my use of the FromFile() method to something that won’t pass the Image class any filename data at all; that way the Image class couldn’t possibly hold the file open, as long as I close it properly myself.

To do so, I replaced the code:

With:

And so far, it’s worked a treat.

418 total views, no views today