Okay, this is probably a long post but it is something I should have done before ;P

As some people know pspsdk has had limited prx building for a while now, now with a major overhaul and integration into the normal build system I felt it was time to explain a bit about how to build prxes in pspsdk, some things to bear in mind and some information on using the tools.

For people who are too lazy to read a long post, this is a summary of the _improvements_ in the new build system.

- Now links with the pspsdk crt0. You no longer need to manually setup your initial thread or parse the arguments. Just place a main function in your source code and it will just work. If you don't want an initial thread (e.g. in a pure export library) then you can specify the PSP_NO_CREATE_MAIN_THREAD() in your source, main will then be called from the module loader thread, just don't sleep in it .
- No longer need to specify a different template makefile to build a prx, just set BUILD_PRX to 1 in your makefile and it will do everything necessary.
- psp-prxgen now removes undefined re-locations, this was a bug which should have been fixed long ago ;P
- No longer need to write an export file if you are not interested exporting anything other than the default module_start and module_info.
- Sort of related, added a PSP_MAIN_THREAD_NAME(name) definition so you can change the default thread name. Not really required but makes it easier to see what modules are what in the list.
- Oh and malloc should now work, you will need to set PSP_HEAP_SIZE_KB(size_kb) though as prxes have a default heap of 64kb if you need to use more than that. It should be noted on this though that if you are running the prx from a _normal_ app which also uses malloc you dont actually have any memory left for the prx to allocate.

A BIG NOTE: Building prxes does not mean you can make encrypted ones which will work on anything, it will only permit you make plain text modules, and you might need to [You must be registered and logged in to see this link.] some kernel patches to get them to load (e.g. pspSdkInstallNoPlainModuleCheckPatch).

-= And so we [You must be registered and logged in to see this link.] =-

First off to use some of this you need to update both pspsdk and newlib to the very latest versions.

For a simple example how do you build a prx in pspsdk? If you have got a pretty basic application (ideally a user mode one) you can build a prx by simply specifying BUILD_PRX=1 in your makefile.
Rebuild and if all went well you will get a .elf and a .prx file. The .elf should only be used for extracting debugging information, it almost certainly wont run on [You must be registered and logged in to see this link.]. Copy your prx to your psp and run using your favourite method (this is possibly more complicated than it sounds of course ).

To make a kernel mode prx just specify 0x1000 in your PSP_MODULE_INFO attribute. The sdk will worry about the rest for you.

Before going further some other useful new makefile definitions.

USE_KERNEL_LIBS=[1,0] - Only link in kernel mode libraries. If you are building a kernel mode prx this is highly recommended. You can also use it for normal apps but it is inadvisable. The reason for this is the default build mode will link in user libraries before kernel libraries where it can, this allows you to run 100% native kernel mode, eliminating the syscalls for abit extra speed. It should be noted however that you can use user mode libs as well, but you will need to specify them manually.

USE_KERNEL_LIBC=[1,0] - This is a counterpart to USE_PSPSDK_LIBC, it will use the kernel's very limited libc instead of using newlib or pspsdk libc. Use this only if you are making very limited usage of libc and want to go for size.

PRX_EXPORTS=file.exp - This specifies the exports file to use for your prx. It must have an exp extension for the default build rules to work. NOTE: This has superseded the old method where you specified it in the OBJS. For the file format see below on the section about psp-build-exports.

-= The tools =-

There are two tools in pspsdk in support of prxes. One you should never need to worry about but might as well explain it as I go.

psp-prxgen - This is the guts of the prx generation, without it you are screwed. It takes a specially linked elf file (you cannot run it on an elf built using the normal method) and converts it to a valid prx file. That's it, the make system will handle it for you, it would only be called outside of this in order to diagnose problems (run with the -v flag to dump alot of debug output).

psp-build-exports - This tool takes a text file and builds a .c file containing code to establish the exports for the prx. Even if you are not exporting any functions it is still important because standard executables need at least two exports in order to run.

The text file consists of a list of commands which are read sequentially in order to make your exports, the testprx sample has a simple one to demonstrate usage, but I will reproduce it here.

Quote:

PSP_BEGIN_EXPORTS PSP_EXPORT_START(syslib, 0, 0x8000) PSP_EXPORT_FUNC(module_start) PSP_EXPORT_VAR(module_info) PSP_EXPORT_END PSP_EXPORT_START(MyLib, 0, 0x0001) PSP_EXPORT_FUNC(getModuleInfo) PSP_EXPORT_END PSP_END_EXPORTS

This should be placed in a file named something like exports.exp, and referenced in your makefile with the PRX_EXPORTS directive. If you only want to export module_start and module_info you can use the default exports table which will get linked in if you don't specify PRX_EXPORTS.

A description of the commands.

- PSP_BEGIN_EXPORTS - This just starts the export tables, it is really just a place marker.
- PSP_EXPORT_START(name, ver, attributes) - Begin an export library with the specified name, version and attributes. When defining the default export the prx does not use a name (it sets the pointer to 0) so the made up name of syslib is used for this as a place marker. The version is major/minor arrangement, one in each byte of this 16bit value. It is important as you can use this to ensure you can only link to a certain library version. For our purposes it isn't really important. The attributes are not quite clear, 0x8000 is a special case for syslib. For exporting a set of functions from a library to be used by another prx (i.e. user to user) of the same type specify 0x0001. For exporting functions from a kernel prx to be used by user and kernel mode applications through a syscall gateway specify 0x4001.
- PSP_EXPORT_FUNC(name) - Export a function. The SHA1 nid will be autogenerated from the name you give it.
- PSP_EXPORT_VAR(name) - Same as above but for a variable.
- PSP_EXPORT_FUNC_NID(name, nid) - Export a function with a specific name and SHA1 hash value. This is only needed if you either want to export a function with a randomish hash value or you don't know the
real name of the function to generate the _proper_ nid.
- PSP_EXPORT_VAR_NID(name, nid) - Same as above but for variables.
- PSP_EXPORT_FUNC_HASH(name)/PSP_EXPORT_VAR_HASH(name) - Synonyms for PSP_EXPORT_FUNC/VAR, there due to legacy
- PSP_EXPORT_END - End the current export library.
- PSP_END_EXPORTS - Place holder for the end of all the exports.

Specifying syslib is absolutely mandatory, else your prx wont work (however you can add additional exported functions such as module_stop which is called when your module is being shut down). The rest can be added as needed, there is obviously a limit to the number of exports but not a practical one.

While in normal operation you shouldn't need to call psp-build-exports manually there is one important time when you do. There would be no point in exporting functions from your module if nothing can import it. This is where the --build-stubs and --build-stubs-new options come into play. By running the tool with one or other of these options and providing the name of the export text file each of the libraries in the file (barring syslib) will be written to a file of the form libname.S. These can then be included in your project which needs the functions imported. The difference between them is --build-stubs generates the old style assembly files which only have to be included in your project, --build-stubs-new generates the new style files which need to be specially built to get them to work, the benefit to using this is along with psp-fixup-imports it eliminates unused functions from the modules imports (saving space). Unless you are a glutton for punishment or you are adding these to pspsdk (where the usage of this form is close the mandatory) it probably isn't worth the effort

Over the last week I've been attempting to create, load and use my own prx with little success. This is some notes which after discussions with qubitz on IRC seem to be required.

The situation is that I have a user memory prx which is to be called from a user memory thread in an application which has been compiled to allow kernel access on startup. After lots of trial and error this is what we discovered:

1. You must call pspSdkInstallNoPlainModuleCheckPatch from the kernel thread before attempting to laod your module from the memory stick.

2. The load module function requires the full path to the PRX.

3. You can load and start your module in either the kernel thread or user thread.

4. To access your PRX functions you must manually call pspSdkFixupImports. This function can only be called in a kernel thread. This makes point 3 invalid, as you must then load and start your prx in a kernel thread. I'm very curious as to why this needs a kernel thread, and it needs more investigation.

4. As noted above the PSP_HEAP_SIZE_KB is required to get things using malloc working.

After all this there is still problems with bus errors, likely from malloc returning 0. I'm yet to try this specific recipe myself, but it seems getting the memory allocation right is still tricky and will also need further investigation.

A few things I'd like to understand better.

1. In what situations is pspSdkFixImports required? Why doesn't the load or start module fix these imports?

2. Why does pspSdkFixImports need to be in a kernel thread? My intention was to eventually allow LuaPlayer to load PRXs, however this might be more difficult if I can't search and call the PRX exports.

3. Why is it that even when you set PSP_HEAP_SIZE_KB in both the PRX and the application that you can continue to get 0 from malloc? Is there any restrictions on which libc is used, etc?

Hope that helps anyone having troubles with PRXs, and hopefully we can sort out the rest of the issues.


Okay let me try and answer some questions.

Quote:

1. In what situations is pspSdkFixImports required? Why doesn't the load or start module fix these imports?

The automatic fixup in the kernel has a couple of restrictions, the main one though being that it will not link up module exports for two different mode prxes unless you set the 0x4000 flag in the libraries attributes (and then this only allows a user mode app to link to a kernel mode export). This I guess is to prevent accidently passing kernel addresses to a user mode app or to stop kernel mode functions being directly linked to (with a j address instruction) from a user mode app (as it could not jump to the address). The typical example of this is the net libs, they are user mode modules so the functions will not get auto linked into a kernel mode program without manually fixing them up.

Quote:

2. Why does pspSdkFixImports need to be in a kernel thread? My intention was to eventually allow LuaPlayer to load PRXs, however this might be more difficult if I can't search and call the PRX exports.

In order that SdkFixImports can work it calls sceKernelFindModuleByName which is a kernel mode only function. You could rewrite it so it scans memory and trys to find heuristically the module export table but that is considerably more annoying to implement correctly. If I recall there was at some point code which did this in the sdk which was removed as it was unreliable.

Quote:

3. Why is it that even when you set PSP_HEAP_SIZE_KB in both the PRX and the application that you can continue to get 0 from malloc? Is there any restrictions on which libc is used, etc?

Well I don't really know, if you are going to use a malloc implementation I would probably recommend using the one in newlib (along with its sbrk). Think the one in pspsdk libc is still broken (or at least not been updated since the overhaul of the prx builder). It all depends on how you are testing your stuff. One thing to bear in mind is the libc init call (in crt0) uses malloc to strdup some strings (which is kindof annoying if you ask me ) so if you were testing by trying to malloc all your memory in one chunk it is no doubt going to fail.

Based on what I guess is what you are trying to do with lua let me make a few suggestions

1. Make luaplayer a user mode prx only and bootstrap it into memory. This will ensure you have maximum compatibility for linking libraries in (such as net etc.) The bootstrap could operate in a similar fashion to psplink by loading a kernel mode module into memory which could also contain a number of key kernel exports (with 0x4001 as its attribute) which would allow access to sio, adddrv, kernel loadmodule etc. It is worth noting that if you provide a function which is called through the syscall gateway then set the k1 reg to 0 before calling the _real_ kernel function it will operate as if it was a kernel mode function (not a limited usermode one).

2. Don't bother trying to auto import loadable lua prx modules, on the main user mode luaplayer export a library with function calls to register and deregister a library set. Then get the loaded module to call that during its initialisation, you might have race condition issues I guess but it would be more generic. It is worth nothing this is how irx exports worked.



To use the newlib libc and malloc you must remove any USE_KERNEL_LIBC or USE_PSPSDK_LIBC from your makefile. The default is that newlib libc will be linked.

The concept of bootstrapping is to make your application a usermode PRX. Make the main elf/eboot a kernel mode application. You keep the bootstrap short and perform only those functions which require kernel model. Your application will then be correctly in user mode and you have less or the kernel/user mode pains.







Well; I've been playing with PRXs some more, and these are my findings...

1- Most of the problems I had the last time are gone! (malloc / crt0 linkage / etc)!

2- Loading a usermode PRX from a usermode thread in a kernel mode application is NOT the same as loading a usermode PRX from a 100% usermode application. This explains why using a bootstrap works.
I have a kernel mode bootstrap that loads my app -- which is now a usermode prx. Then the app loads other usermode prx's no problem.
There is no need to call pspSdkFixImports or anything. BUT, pspSdkInstallNoPlainModuleCheckPath NEEDS to be called (in the bootstrap).

3- Both my main app(prx) and the other prx are linked against newlib. I found that malloc works fine now from both; but other stdio functions fail from the child prx. Functions like fopen() fail, and generate an exception that points to newlib's findfp.c funcion, or refill.c.
I ended up exporting some functions from the main prx to the child, and this seems to have solved some issues. But I wonder if there is something else I need to do..
*Do I need to initialize newlib or newlib's stdio in the PRX?

4- I tried exporting variables, but when I add an entry to my .exp file like:
PSP_EXPORT_VAR(var_name)
Then when I run psp-build-exports, the .S file has no reference whatsoever to my variable...
*Am I missing something?


OK Now I have a better grasp on what's going on, and why the theory of "if it works on the parent should work on the child" fails in my case.
These are just facts I have discovered, and are purely symptomatic (don't really understand the causes completely, hopefully you can help).

-When a PRX is loaded/started, main() runs in its own thread. In this thread everything I've tried works fine. newlib included.
The "parent prx" runs everything from this main, and so everything works fine.
-Now, when the parent PRX calls an exported function from the child PRX, this function is executed in the parent's thread, but the data (data/bss segments?) seems to not be shared as they would be if the function resided in the same module. And so in this environment, newlib doesn't work.

Digging some deeper, it seems that the main problem with newlib's stdio in particular, is a global variable "_global_impure_ptr" which is dereferenced from most stdio's functions (for some reentry purpose).
This variable is declared in newlib's reent/impure.c (defined in sys/reent.h) as:

Code:

static struct _reent __ATTRIBUTE_IMPURE_DATA__ impure_data = _REENT_INIT (impure_data); struct _reent *__ATTRIBUTE_IMPURE_PTR__ _impure_ptr = &impure_data; struct _reent *_CONST __ATTRIBUTE_IMPURE_PTR__ _global_impure_ptr = &impure_data;

If the address of impure_data(or what's pointed to by _global_impure_ptr) is logged from the prx main(), it appears correct. But once logged from an exported function, it is garbage:

This is from the parent PRX:

Code:

09:07:43.253:ScreenHandler/ScreenHandler.cpp@160<20>: In PSPRadioPRX: _global_impure_ptr=0x088CD340, _impure_ptr=0x088CD340

This is logged from the child's PRX's exported function:

Code:

09:07:43.757:main.cpp@47<20>: In TextUI- Before: _global_impure_ptr=0x00000040, _impure_ptr=0x00000020

I have a work-around that seems to work for me right now, but it is kind of a hack, and hope to be able to remove it once this is all figured out.
My work around involves declaring a local impure_data structure in the prx, and then making _impure_ptr and _global_impure_ptr point to it instead of the default one. Once this is done, all calls to the prx that need stdio work fine.

Code:

#ifndef __ATTRIBUTE_IMPURE_DATA__ #define __ATTRIBUTE_IMPURE_DATA__ #endif static struct _reent __ATTRIBUTE_IMPURE_DATA__ my_impure_data = _REENT_INIT (my_impure_data); extern struct _reent *_global_impure_ptr __ATTRIBUTE_IMPURE_PTR__; void FixNewLibStdio() { ModuleLog(LOG_LOWLEVEL, "In TextUI- Before: _global_impure_ptr=%p, _impure_ptr=%p", _global_impure_ptr, _impure_ptr); _impure_ptr = &my_impure_data; _global_impure_ptr = &my_impure_data; ModuleLog(LOG_LOWLEVEL, "In TextUI- After: _global_impure_ptr=%p, _impure_ptr=%p", _global_impure_ptr, _impure_ptr); } IPSPRadio_UI *ModuleStartUI() { ModuleLog(LOG_LOWLEVEL, "ModuleStartUI()"); FixNewLibStdio(); return new CTextUI(); }

ModuleStartUI() is my exported function. It is called once, it 'fixes' stdio, and then returns the object I use for successive calls.

This code generates this log entries:

Code:

09:07:43.618:main.cpp@55<20>: ModuleStartUI() 09:07:43.757:main.cpp@47<20>: In TextUI- Before: _global_impure_ptr=0x00000040, _impure_ptr=0x00000020 09:07:43.895:main.cpp@50<20>: In TextUI- After: _global_impure_ptr=0x08D2E9B8, _impure_ptr=0x08D2E9B8 09:07:43.905:TextUI.cpp@55<10>: CtextUI: Constructor start 09:07:44.055:TextUI.cpp@71<10>: CtextUI: Constructor end.

The only problem with this approach is that I'm redifining _global_impure_ptr, as it's originally defined as _CONST (read only) in reent.h.. To get around that, I use a local reent.h without this restriction (this makes this an even bigger hack).

I also found that sometimes, if PSP_HEAP_SIZE_KB is not used in the PRX, the _global_impure_ptr pointer points to, apparently, valid data for a while, and the app may run a little further before crashing. (and you are of course limited to the default of 64KB or heap)
I'm using PSP_HEAP_SIZE_KB(2048); right now and get garbage assigned initially.

So all in all, I'm happy that I finally got my child prx running correctly. But there's gotta be a better way of doing this... Or maybe I'm just doing something completely wrong? Don't know. Hopefully you can enlighten me.

As far as I can tell gcc is totally ignoring the G0 switch which is supposed to disable use of the gp register, however if you disassemble the resulting code you will see it is still using it. So the reason it is breaking is fairly simple, when it gets impure_ptr it references it via the gp register which is different for a different module and thus returns garbage.

As for a fix well, I have looked around in gcc, it certainly seems that it is setting the G flag correctly to 0, and there seems to be a check for the size, however there is also a second place where the symbol could be marked as gp relative so it might actually be that which is causing it, bypassing the G flag entirely :/

As a quick and dirty fix you could probably wrap any exported functions with code to set the gp register to the value in the module itself, e.g.

Code:

void* setgp(void* gp) { unsigned int oldgp; asm( "move %0, $gp\n" "move $gp, %1\n" : "=r"(oldgp) : "r"(gp) ); return oldgp; } extern void _gp; void modexport1(void) { void* oldgp; oldgp = setgp(&_gp); /* Do code */ setgp(oldgp); }

Ugh my fingers hurt now ask me any questions if you do not under stand..

What's a Prx file

wtf

A .PRX file is what coders/ programmers use to make PSP mods/cheatdevices/etc. Just like .exe is for microsoft .prx is for psp's. Its really not hard to understand. Its quite easy when you understand the basics
The post explains it but no one wants to read ALL of that...