farooqkz

Farooq's KaiOS related experiences started with a hackable banana

Matrix logo

As of 2021-06-19 72 USD out of 200 USD has been reached.

#matrix #voip #voicecall

The story

It had been a very long time which I wanted to start developing a Matrix client with voice call support for KaiOS. Before that I had tried developing a Jitsi Meet app for KaiOS letting KaiOS users have group voice chat with their feature smartphones, which failed due to their ToS.

What is Matrix?

Matrix is an open and federated protocol for chatting which supports End to End Encryption, 1 to 1 voice and video calls(support for group calls is experimental). Is it also possible to integrate other chat applications like XMPP, Telegram, Discord, IRC, etc into Matrix using bridges.

What am I going to do?

I want to create a Matrix call only app for KaiOS letting users of this platform to use 1 to 1 calls, especially voice call. I need at least 350 US dollars to work on it as a job about 15 hours a week and to get a Nokia 800 Tough which costs me about 120 dollars 90 dollars in my country.

Note that I've got a 8110(aka the bananaphone) but its keypad is almost dead.

Donators' name as well as the amount they've donated shall be written in Credits/About tab of the app visible to everyone who uses the app. If you want your name to be written you MUST email me your crypto public key before making the transaction

hapu [at] riseup [dot] net

I already have written a prototype of a Matrix app which let's you login and see a list of your “DMs” and start a voice call as well. However this prototype needs to be rewritten, most parts at least if not everything.

prototype screenshot

License of the software

The Matrix app source code will be available to all for free on some site like Github under GPLv3. It will be available on BananaHackers store as well.

Stages

So I split the development in the following stages:

Stage 1 -once 200 dollars is reached, summer vacation has started and I have bought a Nokia 800 Tough- will be an app with the following features:

  • Login with username and password is possible only.
  • You could see a list of DMs
  • Invites are auto-accepted
  • Possible to start a voice call with someone or receive voice call as long as the app's running
  • The app will be extensible letting us add other Matrix features later easily

Stage 2 -once 300 dollars is reached and stage 1 has been done, includes push notification for incoming calls enabling users to receive calls even if the Matrix app is not running. This requires some server software as well as some modification in the KaiOS app. Also invites are no longer auto-accepted.

Finally Stage 3 adds video call functionality once 350 dollars is reached and also tries to improve the UI as well as performance.

Limitations

Due to memory hungry design of matrix-js-sdk, users might not be able to use the app with their account if they have joined large rooms.

Another thing is that I won't be sending the app to KaiStore due to their foolish rules for Voice over IP apps.

Tech stack

Tech stack for the KaiOS app, most likely, will be either ReactJS or InfernoJS as well as Adrian's KaiUI. And for the push notification server software it would be either Python or Rust which is preferred.

Entering the campaign

So if you want to enter the campaign, send cryptocurrency to one of the addresses below and remember even small amounts would help: EDIT: The campaign has ended, check the same blog for status of the project.

BTC: 1Lv3FRzqcG7GhLcAMrHDwXAN6dMbfLu78g
BCH: 1AGk1t3oCxeTqYX9kAUJHFULdP32D4d6d6

Check how much was raised here for Bitcoin and here for Bitcoin cash.

You don't have crypto? No problem you could buy some from places like Coinbase or Kraken or other exchanges.

According to commits on kaiscr NotABug Git repository about 9 months after I had done my first commit in the repository, I started another serious research for finding a better and high performance way for mirroring the screen of a KaiOS device. Current kailive.py script works by taking multiple screenshots from the screen using kaiscr.py and can go up to 10-12 FPS. However most of times, the FPS rate is under 10 and forces high resource usage to the device making many heavy apps slow if you want to run the app and mirror the screen at the same time.

My guess was that, the problem is with encoding raw image to PNG on the device for each frame because screenshot file type is PNG. According to what fabrice said in #b2g:mozilla.org(A Matrix room), it should also be able to give JPG instead of PNG but that doesn't solve the problem.

If I was correct, I should be able to achieve good performance by somehow taking raw screenshots. My early researches, shortly after I had done with kailive.py, was more focused on finding a low level way using Android libraries or even the Linux kernel to read raw pixels from GPU or some buffer somewhere. But I had no such a low level experience so I failed finding a way. I tried reading from /dev/graphics/fb0, the Framebuffer, using fbcat, too. But the result was a blank image(all black) with the resolution equal to screen's resolution. And I had other evidences leading me to conclude the fact that Framebuffer is not used for displaying stuff on these devices, like Framebuffer is slow and my bananaphone is capable of running high FPS games(even 30 is high for FPS as far as I know).

I had left this for a long time without any further research, till one week ago(or two). This time my approach was researching a higher level of the system, Gecko. Fortunately after “grepping” and tracing the source code of Gecko for 8110(available on Nokia website at the time of writing this post) I had found how does the System app take screenshots and output them as PNG.

It draws the whole window on a canvas, saves the content of that canvas to PNG and sends it through debugger-socket. Soon after that I found a way to export raw RGB data rather than PNG and sent it to a TCP server listening on my laptop. But did I succeed? The answer is No!

First I tried 30 fps but neither the device was fast nor I got the wanted FPS rate. The same goes for 20 fps... This means that my guess was simply wrong. The only thing which I have not tried is captureStream() but I have no hope it will work as expected.

I have written a short report with code snippet in the wiki which you can read here: https://wiki.bananahackers.net/en/development/screen-mirroring#taking-screenshots-without-encoding-and-decoding-to-or-from-png