GStreamer: GstFilters to be (hopefully) merged into gst-plugins-base

Today is the day of the Gstreamer Conference 2010 in Cambridge. Unfortunately, I couldn’t attend, but I thought I’d share a little something about the things I’ve done on GStreamer, as part of my work for Collabora.

If you remember my last post about Fsu, I talked about how you could use the FsuFilterManager and FsuFilter  classes to create some really cool GStreamer applications that can modify the pipeline dynamically, using a minimum amount of code and a very easy to use API. There was a lot of interest to this during last GUADEC and I decided to move the code from Farsight into Gstreamer itself. The FsuFilter* objects are independent of Farsight, and can be useful to a lot of people, so there was no reason to keep them there.

I have ported the code from Farsight into gst-plugins-base and renamed the FsuFilter* classes into GstFilter* and made it into a libgstfilters library. I opened a bug report on Gnome’s bugzilla asking for my branch to be merged into gst-plugins-base. I would appreciate it if everyone interested in this could have a look, review the code if possible, mainly review the API and try to use it. I’d like to make sure that the API is stable, feature complete and easy to use for everyone, so if you have any complaints about it, feel free to comment on the bugzilla so I can fix it and hopefully get cleaner/better code merged upstream soon!

You can find the built gtk-doc of the GstFilters library here.

Update: To clear any possible misunderstandings, the GstFilters library hasn’t been accepted yet into gst-plugins-base. So far, I’ve only opened a bug report and hopefully, after it gets reviewed, it might get accepted into -base (or not). I’d just like to get people to comment on the API and help me improve it.

Thanks.

PS3: Why I don’t like the Hermes payload

First things first, the title says “why I don’t like the hermes Payload” so this has nothing to do with Hermes himself. I don’t know him, I never spoke to him, so I don’t know what kind of person he is and so I have no opinion on him personally. Now, I want to make some things clear, I’ve seen a lot of people criticize me for ‘bashing Hermes’, and many people seem to think that I’m trying to say “I’m better than him” or something. Also, it looks like I created some confusion with my comments from my previous blog post. So I want to apologize and make sure there is no confusion anymore : When I said that the hermes payload is ‘dangerous’, people misunderstood me.. no it is not specifically dangerous for your PS3, it won’t brick it or anything, the only ‘danger’ there is, is that it could (in some situations) crash… then you’d need to reboot, that’s it.. so don’t freak out about his work being harmful or anything, because as far as I know, it’s not! Some people also told me “give credit where its due”, and I want to do it, I’ve always given credit to people, every time I achieved something, I gave credit to those who helped me achieve it. I’m not looking for fame here (if I did, I would have announced PL3’s release 3 weeks ago when I created it) I’m just having some fun in my free time doing something that I like. Hermes did contribute some nice things, and I appreciate what he did, mainly he figured out how to fix the controller issues with some games, that was something very difficult to fix and I’m surprised at how fast he came up with a solution and it was a smart solution and all I can say is “good job”. The other stuff he did in his payload, I don’t like that much, and that’s what I want to cover in this post.. I may have been ‘harsh’, but I don’t see the point in trying to be diplomatic, I’m a programmer, not a politician. I don’t like his code, and I speak my mind, I’m being honest, and I’m not trying to criticize him without any reason, as far as I know, I’ve stayed respectful and that’s all that matters to me. To those who don’t want to know about all the technical details, let me ‘conclude’ here by saying that if the hermes payload works for you, then good, use it, I’m not telling people to stop using it, I’m not saying that PL3 works better either, maybe his payload works better in some situations, maybe not, but overall, the user’s choice should always be “whatever works for you”. The PL3 initiative is about having a standard repository for payloads, and having a common code base for everyone to work on, so in the future, PL3 might evolve faster and have more features, or maybe it won’t, the thing to note is that it’s better for payload developers to base their work off PL3. But again, this is meaningless for most users, apart from maybe clearing up the confusion about all these payloads and nobody knows which one to use. Also, I talked about PL3, which is a common repository for contributors to work on, people seem to have nicknamed it “kakaroto’s payload” or “kakaroto’s pl3”, but I never said it was my payload, PL3 is PL3, it’s not all my work, and if you look at the commit log, you will see that I’m not the only contributor to it. PL3 itself integrates patches and solutions provided by Hermes, Waninkoko and Mathieulh, I improved some of their patches to make sure it works better for non-3.41 firmwares, but it’s still credited to be their work. PL3 is not my payload, PL3 is a payload repository for everyone. Also, PL3 as a project is a repository containing multiple payloads (default one, development one, dump_lv2, dump_elfs, etc..). PL3 is not perfect, nothing in the world is perfect, so it might have bugs, it might not work for some people, who knows what might happen. But I never said that it was perfect, so people should stop thinking that I said that. It’s written more cleanly, it’s better in terms of the infrastructure behind it, but that’s the only thing I can vouch for. Also for those complaining about me adding a donate button to the blog post, I don’t see how that’s relevant, I’m not begging people for money (and I haven’t received any donations in the last ~3 weeks just so you know). If you don’t want to donate, then don’t, no reason to bitch about it. I’ve put the button there so that people who appreciate the work and want to donate something have a way of doing it. I asked for donations before because I needed to buy a PS3 for development, I already raised enough money to buy it, so I don’t need any more donations, so I’m not asking people for donation money anymore, as simple as that. Anyways, here are the more detailed/technical explanations on the reasons why I don’t like his payload : First, the code is not clean, it’s unmaintainable. The fact that he gives his source code in .rar files instead of a git fork is the biggest issue I have with it. And yes, that does not matter for users, it only matters for developers.  The problem with his method of delivery is that you have no way of knowing what he based his code off, so it’s hard to figure out what he changed.. also, when you find out his base, and do a diff, you get a huge diff for everything he did, all in one shot, and then you have to reverse engineer it to understand what he patched. That’s complicated and annoying for developers! For those who follow my twitter, you can see how many commits I do, I always like having “small commits” because each commit becomes independent, self explanatory and easy to review. It also makes things easier to integrate, if you want a specific thing, you just merge/cherry-pick that single commit, instead of copy/pasting code, and editing it to remove the clutter. The  other reason I like git is that if he used it and I merge a commit from him, then the code stays credited to his name in the commit log, it allows me to have his code without ‘taking ownership’ of his work, it allows everyone to be credited for what they did, and I think it’s the first thing to have for an open source and community project. The reason why I said his code could crash is because his payload got too big and couldn’t fit anymore in the allocated memory we have in the kernel (1296 bytes), so he decided to just move the code to a random position (0x7fff000 I think). This means that his payload will work as long as no application, game or kernel allocates memory which ‘randomly’ ends up in that area.. if it does, then the payload would get overwritten and the kernel will crash. The proper way of doing it (PL3 does that) is to allocate memory during the initialization of the payload, copy the functions we want in that memory that we own, and write those functions to be position independent so they would work no matter where they are placed in RAM. Another reason is the way his syscall8 works, I tried to read his assembly and reverse engineer it, and I seriously was lost and couldn’t understand what was happening.. there are no comments (you’ll notice that my payload has a comment on almost every instruction), so how can I integrate his syscall if i don’t even know what it does… if at least it was on git, I could see the commit messages and understand what each chunk of code did, but he doesn’t use git, so… The way he fixed the controller issue was also not very good, he patched two offsets to jump to a function that decides on some kind of enum on what response to return and you controlled that with his own system call 8.. why do something like that? it makes the fix dependent on people using this new syscall, and it’s useless when you can just patch it directly to return the right value. I also didn’t like the fact that his code became a mess that is 3.41 independent, and it would have taken a huge amount of work just to try to make it work again on 3.15. I already spent time cleaning up the payloads and making them work for older firmwares, so why fork and write code that doesn’t integrate that, it just makes collaboration harder. There’s also the whole syscall 35 versus 36 issue, but that has nothing to do with his payload since I added sc35 after he released his payload. It’s not about his payload being bad because it doesn’t support it, it’s simply about PL3 having a ‘superior’ (if I may say so) system call. What it means for users? nothing at the moment, maybe it will be used for doing fancier stuff later on, maybe you can map a game to your bluray and a different game to /app_home, that could be useful for users, but for now, it’s simply more flexible and cleaner code. There are many other small things that I didn’t like, but it mostly just summarizes to “the code is not clean and it’s unmaintainable” and “he doesn’t use git”. Like I said, if you don’t care about that, then I see no reason for you not to use his payload. It doesn’t mean either that he’s not skilled, it simply means that he may lack experience in code sharing and experience in open source. But that doesn’t make his work any less valuable. I hope this clears things up a bit. I criticized his work, said what I thought of it and people over reacted, I wanted to make sure people didn’t  misunderstand me, and didn’t think I don’t respect Hermes for what he’s done already. Everything else is just drama and people trying to get attention. If this post stirs up even more trouble, then so be it, I don’t think I have much more to say. I said what I think, people should take it or leave it. I do not however tolerate people insulting me for no reason at all. So please, criticize me all you want, just stay respectful. Thank you, KaKaRoTo

PS3: The payload mess…

Hi all,

I see a lot of people asking me some questions and I notice a lot of ignorance in the net about the different payload and the latest PL3 payload. So I want to make things clear..
First of all, people should stop talking/requesting/using the hermes v3 payload, I don’t like his work, and the payload is not good, it might crash the system in some cases, it’s not written properly, and hermes doesn’t even seem to understand how git works.
Also, PL3 already includes (for some time now) all the good stuff from hermes, it already supports installing game updates, or running games without a disc, anything else that Hermes added is useless and dangerous could crash in some situations (requiring a reboot).

Some might have seen my tweets about my new payload being released, and many are asking me what is the difference between my payload and what is already available.
PL3 doesn’t support syscall 36 anymore, for multiple reasons, first, it was bad code, it was mapping a path to a single hardcoded value (/dev_bdvd or /app_home or /dev_flash or whatever is hardcoded in the payload) which means that, since we (the PSGroove and PSFreedom developers) don’t want to support running backups, all the official payloads weren’t working with the backup manager without being patched first. The syscall 35 I added in my payload is more generic though, it is the proper way of doing things. You can map any path to another other new path, the prototype looks like this :

  syscall_35 (char *old_path, char *new_path);

This means that the payload doesn’t need to have a hardcoded /dev_bdvd path in it, or have extra code for mapping /app_home to something else.. or having syscall 36 change both /dev_bdvd and /app_home breaking homebrew when using a discless mode with a backup manager. You also don’t need a special payload to run the ‘firmware usb loader’.. It all just works because the choice of the path mapping is given to the homebrew applications themselves. This means that the backup managers will just map /dev_bdvd to what they want and they will work by default on my payload, there will be no need for a patched version of the payload to make them work.
This however means that the backup managers that depend on syscall 36 will stop working. For now Gaia Manager is the only backup manager available that is compatible with my payload. But I’m sure more will be ported to use syscall 35.
People need to understand that this new syscall 35 has to become the new standard, this is what all the payloads should use, nothing else, and this is what everyone should start using, not the old, crappy, backup-manager specific, PSJailbreak written, syscall 36.

We need to have some form of standardization for all these payloads, I’m tired of seeing about 100 different payloads floating on the internet, it doesn’t make sense. I always believed in a single payload that works for everyone, and that’s why I created PL3, that’s why it’s a project independent of PSFreedom (and PSGroove has been ported to it) and that’s where all the efforts should go. Also, by using PL3, you automatically gain support, and all the same features, for whatever previous firmwares PL3 already supports (3.01, 3.10, 3.15 and 3.41).

I have just recently seen this new payload that everyone is so happy about that includes “all the good things from 3 worlds”, the one created by Rancid, which includes the stuff from hermes, waninkoko and Mathieulh… and I was shocked to see how much people were happy about this.. people don’t really seem to understand that this wasn’t necessary at all? PL3 has had all those patches for a while now, so why did Rancid even bother making this payload that includes the patches from hermes, waninkoko and Mathieulh? Why would you spend your time doing something that already is available!

This blog post is meant to stop all this ignorance and let people know that they don’t need to look for a special payload, just use PL3 and you’ll get everything you need. It is also meant to explain to everyone what is different about my payload.

On a side, I have received a P3Hub device, kindly donated to me by the people from r4king.com, and I have now tried PSGroove for the first time! I’ve also created a fork of jevinskie’s port of PSGroove which is now improved and updated to support the latest PL3 version. This means that the PL3 payload is available for everyone, those using PSFreedom as well as those using PSGroove, so there is no excuse now on not using it or relying on badly written payloads developed by people who barely know how to code (yes, using winrar instead of git is a good indication of that).

Update: I forgot to rant about peek&poke!!! So let’s do it now… well, the default payload in PL3 has peek and poke disabled, and for a simple reason : Nobody needs them! and more importantly they are misued! I’ve look at the code of the different backup managers, and it looks like all of them use poke to patch the memory to ‘fix something’ because they think that it’s their job to do it.. no it’s not! If you have a working patch, then submit it to PL3 and if people complain, tell them “use the proper payload”, don’t try to take advantage of peek&poke to go and modify the kernel’s instructions! The reason is simple.. you are a homebrew app that does X, then do X, leave the kernel patching to the payloads! Just as PL3 doesn’t map /dev_bdvd to /dev_usb000/I.Like.This.Game/ and locks it out! Also, I’m on firmware 3.15, so when you decide to poke and patch the kernel with a hardcoded offset, you’re just screwing up my kernel because the offset is firmware dependent! it’s not the same depending on the firmware you use, and I don’t want you playing with it. So.. peek&poke are really not useful to anybody, they are not even available on a normal linux pc, so why would you want them in your default payload, right?! The only people who should use a payload with those syscalls enabled are real developers, people who want to analyze and patch the kernel on the fly while they are doing some development of, maybe, a kernel driver! That’s it. Anyways, that’s enough ranting from me for today!

P.s: In my branch of PSGroove, I wrote a script that build the .hex file for every supported device (from the README) for every supported firmware. You can find all the hex files here : PSGroove+PL3 hex files

Update: Thanks to evilsperm, I’ve updated the archive with hex files for these devices : Blackcat, Xplain, Olimex, UsbTinyMkII, Bentio and OpenKubus.
Update 2: Some people reported crashes with my payload when running backups with installed updates. I figured out the cause and fixed it now in git. The hex files above have also been updated

Thanks for reading.
KaKaRoTo

PSFreedom now supports firmware 3.01, 3.10 and 3.15

Hi,

I’ve got some great news for those of you who have not updated your PS3 firmware! I have just succeeded in adding Firmware 3.01 support into PSFreedom. I’ve pushed the latest code to github and you can now download the source and compile PSFreedom for 3.01.
For now, you will need to edit config.h and change the FIRMWARE_3_41 into FIRMWARE_3_01, then recompile. However, I will soon add support for dynamically choosing the target firmware version by simply doing a :
echo 3.01 > /proc/psfreedom/fw_version

I will soon add support for firmware 3.10 and 3.15, so be patient, and you will be rewarded. I would like to thank Klutsh as well as Philippe Hug who helped me achieve this port to 3.01.
The new payload changes are available in the PL3 github and any project/port that is also using PL3 should automatically gain support for the 3.01 firmware.
You will also be able to enjoy some new ‘tools’ in PL3 that will allow you to dump the LV2 kernel as well as the decrypted ELF files of the XMB and other configuration files it uses. The ethernet dumping is also now compatible with PS3 Slim models.

Update:
Philhug and I have worked together recently to make PL3 compatible with 3.15, and it is now done, working and ready for you to use. I have just pushed the latest changes to github, so just update both PSFreedom and PL3, and define FIRMWARE_3_15 in PSFreedom’s config.h and recompile. You will then be able to enjoy your unrestricted PS3 on 3.15 firmwares. Enjoy!

Update 2:
I have just added support for firmware 3.10 to PL3. You can get it by upgrading to the latest git version of PL3. There are however some changes in there that might break PSFreedom, so wait until I update PSFreedom tomorrow to be compatible with the latest PL3 changes!
I have also added a HOWTO file that explains the steps required to port PSFreedom to an exploitable firmware. Enjoy

I would like to thank, again, those who have donated. For the others, you can still donate, if you appreciate the work I’ve done.

Enjoy!
KaKaRoTo