OpenPandora: Development toolchains, SDK’s and build platforms (Part 2).

This is the 2nd of the 2 simple guides I have put together to help you through the process of getting a working development environment and basic SDK going for the OpenPandora allowing you to build applications, compile up code, all of that neat stuff really.

Note: These guides and toolchain/SDK packages do not constitute an ‘official’ OpenPandora SDK or anything of the sort, they are just what I have put together during the development process in the hope somebody may find it useful.

There will be things missing libs and odd littke things that did not end up in the SDK package etc. at first and this article and the toolchains will evolve as time goes on.

Consider this 2nd part of the guide a work in progress.

With the OpenPandora you have 2 primary options for code development…

• Native (on device) software development

Want to write code for the OpenPandora on the OpenPandora, have a little read of this. Not the way I would recommend doing normal development but handy sometimes.

The is covered in part 1 of this article.

• Cross compiling

The most common method of code development for a device such as the OpenPandora. Requires access to a regular PC to using for building code.

The is covered by this part of the article.

I’ll aim to outline how you can start to develop code using either setup, and provide a really simple test app to prove your setup is working.

This is mainly aimed at C, C++ and Assembler developers who are familiar with GCC, build tools and Linux in general.

Cross compiling:

Cross compiling is by far and away the most common way of building code for small/embedded systems (the term is relative) like the OpenPandora.

Cross compiling allows you to make use of the capacities of a much bigger systems to compile code that will then run on the target device.

It also frees you from limits imposed by building on the device directly such as running out of memory or storage (both of these are used en-mass by larger application compiles).

Note: The entire OpenPandora OS is built using OpenEmbedded and OpenEmbedded cross compiles everything (in my case on 32bit and 64bit Linux boxs). None of the OS code is actually natively compiled on the device its self.

What’s included in the toolchain and SDK?

The toolchain included in the package below is the same toolchain used to build the OpenPandora OS images (all the toolchain scripts do is package it up neatly).

The initial SDK contains the toolchain and development libraries for most (it should really be all) of the libraries that are installed on the OpenPandora OS including the libraries/headers for the PowerVR SGX core’s OpenGL features.

This is something I am trying to make more slick so if you notice missing libraries or bugs please do let me know.

Setting up the environment:

The 1st thing to consider here is that I am only providing toolchains for recent 32bit and 64bit Linux distributions (with gLibC 2.11 >).

Windows Users:

There are options for Windows users if you want to use a cross compiler targeting ARMv7 such as CodeSourcery providing you know how to add the libraries you need, but it’s not something I plan to cover in this article.

That said, these toolchains are perfectly happy inside a Linux virtual machine so assuming your using something with an x86 or x64 CPU you should be able to get them going.

You will have to look elsewhere for information on setting up a virtual machine but I know you can easily get recent Ubuntu versions that run well in VMWare player.

Download links:

32bit Linux: Toolchain & SDK

64bit Linux: Toolchain & SDK (To follow, I need to get my 64bit build box back)

Installing the toolchain and initial SDK:

Download the appropriate toolchain and SDK for your platform.

You will then need to install it. You should start by untar’ing the components to the root folder (/) of your Linux setup.

Note: Adjust the command to suit the file you have actually downloaded (32 or 64bit etc.). Depending on the distribution and your security setup you may need to pop ‘sudo’ in front of the command below.

tar -C / -xjf 20100611-i686-linux-armv7a-linux-gnueabi-toolchain-openpandora.tar.bz2

Once this operation is finished you will find a new directory /usr/local/angstrom/arm/ and it contains the environment-setup script to setup various paths to allow the toolchain to work and tools to be found by the system.

Making sure you can access the toolchain:

Depending on the distribution and your security setup you may need to give your user explicit access to the new /usr/local/angstrom/arm folder you have just created before you can run anything. I recommend doing this as there is no reason to run cross-compiles as root (or sudo’ed etc.).

The following command would give your normal user full control of the files on Ubuntu.

You can adapt to suit your security needs/setup as appropriate. Obviously replace <your-name>.<your-name> with your username (and don’t forget the dot).

sudo chown -R <your-name>.<your-name> /usr/local/angstrom/arm

Testing the toolchain:

Before using the the toolchain you will need to source the environment. Don’t forget the preceding . to make sure the variables get added to the calling shell, you will need to do this once for every terminal you want to use to compile code.

. /usr/local/angstrom/arm/environment-setup

With the environment setup your almost ready to go.

For testing of the cross compiler we will use exactly the same test app we used in part 1 to test the native compiler.

It is a REALLY simple test app that uses SDL to display a bitmap image.

This will show your toolchain and basic SDL are working. It’s not fit for any purpose what so ever and I don’t suggest you actually build code this way either (use a makefile ).

• Download silly test app.

Compile it by opening a terminal window. Running the toolchain environment script (see above). Then change into the directory you extracted the test app to and issuing the following command.

arm-angstrom-linux-gnueabi-gcc -Wall DisplayImage.c -o DisplayImage -l SDL

Next up you can run the app.

To do this you will need to copy DisplayImage and test.bmp onto your OpenPandora.

Then you can open a terminal, change the the folder you copied it to and run.

./DisplayImage

Note: If you connecting to the OpenPandora remotely (i.e. over SSH) to run the app you will need to run sudo ./DisplayImage or SDL won’t be able to claim the screen.

It will show an image for 10 seconds then exit.

Adding additional development libraries to the toolchain:

One of the big plus points to using a cross compiler built using OpenEmbedded is that support for adding development libraries from the Ångström repositories is baked in. The toolchain ships with a version of OPKG (Ångström’s package manager) pre-configured to look for libraries in the feed.

The main difference from using ‘opkg’ on the device is that you use ‘opkg-target’ on your build system to install libraries into the ‘target’ (i.e. OpenPandora) environment.

The 1st thing you will want to do is update the repository feed data.

Make sure you have correctly run the environment-setup script first then run…

opkg-target update

All you then need to do is find the development library in the Ångström repositories and just install it with a simple command.

The sample below will (re)install the development libraries and headers for libSDL 1.2 into the toolchain.

opkg-target install libsdl-1.2-dev 

Of course, you are also free to compile up any libraries or support apps you may need if they are not in the repository.

With a little luck you should now be able to develop applications for the OpenPandora using whatever method you desire.

All the usual things you would expect with a cross-compiler should work fine, including passing custom CC/CXX etc. to configure/autotools scripts to have them work everything out.

Building the toolchain yourself

If the downloads above are unsuitable for your setup (or you just have to buiold everthing yourself you can always build the toolchain and SDK from OpenEmbedded metadata).

You will shortly be able to find the OpenEmbedded recipes used to create these toolchains in the OpenPandora overlay GIT, if you already have an OpenEmbedded environment setup you will be able to bake your own version of the toolchain & SDK with a simple…

bitbake meta-toolchain-pandora

You will then be able to pickup the .tar.bz2 from the deploy/glibc/sdk folder in your build tree and install it as documented above.

Known Issues

This is the first release I have done of this toolchain & SDK package, I fully expect there to be some issues.

I already know there are some issues with support scripts,like the appropriate target sdl-config (incorrect paths). I am working on fixing up stuff like this but I wanted to get the toolchain out ‘as is’ and update things in a few days as people seem very keen to get there hands on it.

If you notice other issues do please let me know (or better still send patches).

Anyway, once you have struggled thougth all the waffle above you should have a working toolchain and SDK to begin development for the OpenPandora. I’ll leave it up to the reader to decide what they want to develop .

Regards,

John

OpenPandora: Development toolchains, SDK’s and build platforms (Part 1).

This is the 1st of 2 posts that I hope will form simple guides to help you through the process of getting a working development environment and basic SDK setup for the OpenPandora allowing you to build applications, compile up code, all of that neat stuff really.

Note: These guides and toolchain/SDK packages do not constitute any ‘official’ OpenPandora SDK or anything of the sort, they are just what I have put together during the development process in the hope somebody may find it useful.

Some of the information in these articles has already been covered, I just got tired of covering the same ground helping people so I figured I could refer them here so everyone benefits.

With the OpenPandora you have 2 primary options for code development…

• Native (on device) software development

Want to write code for the OpenPandora on the OpenPandora, have a little read of this. Not the way I would recommend doing normal development but handy sometimes.

The is covered by this part of the article.

• Cross compiling

The most common method of code development for a device such as the OpenPandora. Requires access to a regular PC to using for building code.

The is covered in part 2 of this article.

I’ll aim to outline how you can start to develop code using either setup, and provide a really simple test app to prove your setup is working.

This is mainly aimed at C, C++ and Assembler developers who are familiar with GCC, build tools and Linux in general.

Native (on device) software development:

It is possible to easily get a development environment running on your OpenPandora, if this sort of thing floats your boat.

Developing code natively on the OpenPandora comes with a few caveats, don’t expect to be able to build huge suite applications like XOrg, you will run out of RAM and storage long before your compile is complete and even if you setup lots of swap and such on the SD you may well go grey haired waiting for the compile to actually finish .

That said, if your not trying to build 500,000 lines of code in one go you may well find native development quite viable for your needs. And it’s imposable to deny there is a certain ‘geek cool’ to writing some code, building it and running it on the go .

Setting up the environment:

The first thing you will want to do is check you have enough free space on the NAND to install all the development tools.

Assuming you have not installed lots of extra packages to the NAND you should be fine but it is prudent to check 1st.

Run the command below in a terminal.

df -h

Looking at the results, they should look something like this

Filesystem                Size      Used Available Use% Mounted on
ubi0:rootfs             455.1M    325.7M    129.3M  72% /

The important figure here is Available, my OpenPandora has 129M available on the NAND (it may well be more with compression, that is a worst case estimate). You can expect the install of the development tools on the NAND to eat something in the region of 35-40M.

Assuming you have enough free space, the next step is to install some packages from the Ångström repositories. You will need a network connection to do this.

First off, make sure your list of packages is up to date.

sudo opkg update

Once that is done you want to install some tools.

sudo opkg install gcc gcc-symlinks make binutils-dev cpp cpp-symlinks g++ g++-symlinks libstdc++-dev

This will install the development essentials onto the NAND of your unit, you will actually install more then just the packages listed above as dependencies will be dragged in (it’s safe it ignore any warnings about “unsatisfied recommendation for” concerning various locale packages).

At this stage it would be prudent to check the amount of space left on the NAND again.

Filesystem                Size      Used Available Use% Mounted on
ubi0:rootfs             455.1M    357.5M     97.6M  79% /

Ok, so about 32.5M taken installing the core development tools, scary but not all that bad.

Assuming that all goes well you will now want to add some development libraries to your NAND so you can actually build stuff.

sudo opkg install libsdl-1.2-dev libsdl-image-1.2-dev libsdl-gfx-dev libsdl-net-1.2-dev libsdl-ttf-2.0-dev libgles-omap3-dev

That command will install a minimal set of development bits, SDL and the headers/libraries for the OpenGLes hardware. It will also pull in dependencies.

After all this is setup lets check the NAND space once again.

Filesystem                Size      Used Available Use% Mounted on
ubi0:rootfs             455.1M    361.2M     93.9M  79% /

If you need more libraries or support tools I am sure you can install them from the Ångström repositories or compile them yourself.

Once you have this all installed your ready to go…

Testing the toolchain:

Ok, this is just a REALLY simple test app that uses SDL to display a bitmap image. Nothing special but it will show your toolchain and basic SDL are working. It’s not fit for any purpose what so ever and I don’t suggest you actually build code this way either (use a makefile ).

• Download silly test app.

I can’t claim to be the author of this sample code, it’s been sat on my HDD for years and I can’t remember who wrote it .

Extract this somewhere on your OpenPandora (SD, home folder, where ever suits you).

Compile it by opening a terminal window, changing into the directory you extracted the test app to and issuing the following command.

gcc -Wall DisplayImage.c -o DisplayImage –l SDL

Next up you can test the app. It will show an image for 10 seconds then exit.

./DisplayImage

Note: If you connecting to the OpenPandora remotely (i.e. over SSH) to run the app you will need to run sudo ./DisplayImage or SDL won’t be able to claim the screen.

With a little luck your now all set for native development.

Part 2 will cover installing and setting up a cross compiling environment so you can build code on your PC to target the OpenPandora.

Regards,

John

OpenPandora: Building your own ROOTFS image (Part 1).

Updated: 7th June to expand the setup scripts section and tell people to get them from GIT.

This is the 1st of 2 articles I will be publishing on getting started with building the Linux distribution installed on the OpenPandora, from source code, from scratch.

Note: This is not a guide to using OpenEmbedded or bitbake, or writing package recipes. It’s just a guide to getting an OpenPandora build setup running.

This 1st article will cover the basic setup of OpenEmbedded environment and the building of our Ångström distribution derived ROOTFS, the 2nd will detail how you can run the resulting images on your OpenPandora from SD cards, or if you really want to, the NAND.

I also plan to write a separate related article will show how you can setup toolchains and SDK’s to help you develop applications for (and natively on) the OpenPandora using some of the strengths of OpenEmbedded to make this easier.

As I write these posts in my own time I can’t promise to do anything quickly. Nor can I make any promises about how accurate they may be . As always, feedback is welcome.

Most application developers/porters will only be interested in the related article on setting up SDK’s and toolchains, these 2 are really for the hardened Linux hackers who want to mess about with the ROOTFS/distribution development.

Health (and sanity) warning: Building the entire ROOTFS from scratch is pretty technical in nature, if you have no desire to rebuild the ROOTFS or hack with the inner workings of you’re OpenPandora this is probably not for you.

You can’t really damage anything on your OpenPandora but if your not familiar with compiling your own apps, kernels etc. and fixing things when they don’t work this will present a VERY steep learning curve. You also have the potential to make a mess of your build host if you are not careful with the setup.

I’ll focus showing how you can build a ‘one <> one’ version of the 1st official release (GIT tag: “Release-2010-05/1”). 

Once you have that going feel free to build the tips of the metadata GIT’s if you want to work with the latest and greatest or get stuck in modifying the metadata to suit whatever purpose you may have in mind.

Contents:

Closed source prerequisites
Host OS for building the ROOTFS
Important GIT repositories
Setting up your environment for OpenEmbedded
Building the OpenPandora ROOTFS images using OpenEmbedded

Closed source prerequisites:

Before we start you will need to collect up the few closed source components used in the ROOTFS, thankfully Texas Instruments and Imagination Technologies at least make getting the closed source binaries and SDK for the PowerVR SGX quite easy these days.

PowerVR SGX Binaries and SDK:

You will need to download version 3.01.00.02 of the Ti Graphics SDK (OMAP35x_Graphics_SDK_setuplinux_3_01_00_02.bin) from http://software-dl.ti.com/dsps/forms/export.html?prod_no=/OMAP35x_Graphics_SDK_setuplinux_3_01_00_02.bin

You will have to create a Ti account to download the software and agree to a license/export agreement but once you have done this Ti normally instantly grant access to the download. After you download this store it in a safe place, you will need to drop it into a folder in the build system later on.

The other closed source components are the firmware that is run on the WL1251 wireless chip and Bluetooth chip (the drivers themselves are open source and in mainline Linux). However our build system will automatically downloads these closed source firmware binary firmwares and places them into the appropriate place in the ROOTFS.

Host OS for building the ROOTFS:

Note: all my OpenEmbedded build boxes are 32bit Ubuntu Linux 10.04 servers with PAE enabled kernels, building on most other Linux distributions should be (and is) simple enough however some small adjustments may need to be made to suit your distribution. The OpenEmbedded and Ångström documentation can help with this.

You will need to install a number of packages onto your host PC before you start building OpenEmbedded images.

Try the command below, this should get most of the major prerequisite packages installed that you will need to start building using OpenEmbedded on a Ubuntu/Debian derived distribution.

sudo apt-get install ccache sed wget cvs subversion git-core coreutils unzip texi2html texinfo libsdl1.2-dev docbook-utils gawk python-pysqlite2 diffstat help2man libxml2-utils xmlto python-psyco

Note: The OpenEmbedded Required Software page also details this very well and is a must read.

One last thing to consider if your using a Ubuntu derivative is that they use dash as the default shell not bash.

This can cause a few issues with some of the packages we want to try and build, it’s not a problem with the build system per-say but issues with the individual packages assuming bash hacks in whatever /bin/sh is on the system.

Still, the fix is pretty easy, just set your default shell to bash using the command below.

sudo dpkg-reconfigure dash

Important GIT repositories (GITWeb links):

All the metadata (and custom source code etc.) is held on a publically accessible GIT server (git.openpandora.org).

For the purposes of building a working ROOTFS the following GIT repositories are of interest.

• http://git.openpandora.org/cgi-bin/gitweb.cgi?p=pandora-kernel.git;a=summary 

Important remote branches are pandora-27-omap1 (Our ‘default’ kernel as of now) and linux-omap (synced with top of the mainline Linux-OMAP tree).

• http://git.openpandora.org/cgi-bin/gitweb.cgi?p=openpandora.oe.git;a=summary

The important remote branch is master. This is our OpenEmbedded overlay (i.e. custom OpenPandora specific stuff we lay on top of OpenEmbedded/Angstrom to build our images).

• http://git.openpandora.org/cgi-bin/gitweb.cgi?p=openembedded.git;a=summary

The important remote branch is op.openembedded.dev, this is a snap shot from mainline OpenEmbedded with our queued patches stuffed on top. This serves as our feed tree for getting stuff into mainline OpenEmbedded/Angstrom (with a few clearly marked exceptions, look for checkins with HACK: in the commit message, these won’t be going to mainline ).

Should you with to manually clone any of the OpenPandora GIT repositories for whatever reason you can by issuing.

git clone git://git.openpandora.org/repository.name.git destination.folder

But if you just plan to build an image you have no need to do this as the setup script will take care of it for you.

Setting up your build environment for OpenEmbedded:

Using OpenEmbedded to build images is actually a pretty simple affair if done right however often steps are missed or done in the wrong order leading to all manner of spurious issues and resulting pain.

Security Warning: Never build your images as the ‘root’ user, there is no need to use a root account and doing so is a really bad idea, just make sure your normal user has control of the folder containing your OpenEmbedded environment.

To make the process as simple as possible I have knocked up a few simple support scripts that can just be installed to an appropriate place and run, hopefully setting up the environment correctly as they go.

These scripts are pretty crude and lack robust error checking and such so should be run with a note of caution. The idea is to improve them as we go.

Note: You will need LOTS of space to undertake a complete build of the ROOTFS, my typical build folders are often in excess of 100Gbyte . Make sure you have the space before you start .

1st off you need to pick somewhere on your system to construct the OpenEmbedded environment, this should be a path with no symlinks above it, a rebound mount point is fine however.

The user you want to use for builds will need control of this folder.

Open a terminal and change into the above folder that you wish to become the top level for your OpenEmbedded environment.

Get a copy of the setup scripts by running…

git clone git://git.openpandora.org/pandora-oe-environment.git .

Once you have the files checked out run the environment script (you must run it with . ./op-env-oe.sh so it goes into the calling shells environment).

. ./op-env-oe.sh

Now you have all the environment variables setup you can run the script that initialises the environment (creates folders, checks out the GIT’s etc.).

./initial-setup.sh

This will take a while to run as it creates directories, clones the necessary GIT trees and populates the environment. Now is a good time for the 1st coffee .

The next step is to switch from the tip of our GIT trees to the “Release-2010-05/1” tag.

This will ensure you are working with exactly the same metadata as the 1st release.

You can skip this step if you want to work with the latest and greatest code.

./use-release-2010-05-1.sh

Note: If you want to you can later switch back to the tip of the GIT trees (development metadata) using ./use-tip.sh.

Be careful if you have uncommitted changes in your local GIT tree as the scripts will try and ‘git stash’ them but I suspect that may not be what you want.

Now your environment is all setup (assuming the scripts worked ok ) your build folder should look like the breakdown below, more folders will be created as soon as you run a build (folders for sources, package staging and temp data for example).

top of build area \

  bitbake \ < Bitbake lives here.

  build \

     conf \

        local.conf < This is the file that tells OpenEmbedded about our setup.

  metadata \

     openpandora.oe.git \    < Checkout of master from openpandora.oe.git

     openembedded.git \     < Checkout of op.openembedded.dev from

                                               openembedded.git

     user.collection \     < Area for any user created package recipes.

  tmp \ < The temp area that all the building is done under.

The next step is to copy in the copy the PowerVR SDK you downloaded earlier into the right place in the build folder.

You need to copy OMAP35x_Graphics_SDK_setuplinux_3_01_00_02.bin into metadata/openembedded.git/recipes/powervr-drivers (so it ends up in the same folder as libgles-omap3_3.01.00.02.bb).

The command below should do it if you run it from the folder you downloaded the SDK into.

cp ./OMAP35x_Graphics_SDK_setuplinux_3_01_00_02.bin ${OEBRANCH}/recipes/powervr-drivers

Now you want to do a simple test to check everything is working as expected.

From the top of your build directory run…

bitbake nano

After a fair period of time (it will download and build a version of GCC to target the OpenPandora GCC so it can then use that GCC to build NANO ) you should have a successful build of the NANO text editing package.

Note: You can check this by looking in tmp/angstrom.5/deploy/glibc/ipk/armv7a to see the NANO ipk package (nano_2.0.9-r0.5_armv7a.ipk).

The ‘angstrom.5’ part of the path comes from the concatenation of angstrom and it’s internal distribution revision (currently 5).

Your now ready to move on to building something a little more complex…

Building the OpenPandora ROOTFS images using OpenEmbedded.

Now you have a suitable environment setup you can move on to building your 1st image.

Every time you open a new shell/terminal you will need to call the environment setup script to create the necessary environment variables, the rest of the setup is already done.

. ./op-env-oe.sh

You have the full Angstrom and OpenEmbedded metadata at your disposal but for the purposes in hand there are a 2 main types of OpenPandora image stored in our OpenEmbedded overlay and 2 additional ones showcasing other Desktop environments.

You can see if a package is in a given image by looking at the corresponding recipe task file (found in metadata/openpandora.oe.git/recipes/tasks).

They are as follows (image name – task used).

• pandora-core-image – task-pandora-core

This image contains all of the core packages for the OpenPandora but lacks any GUI/X Server etc. – It is lightweight (reasonably) and obviously CLI. Ideal if your working on hardware drivers etc. and don’t need a GUI layered on top. It also features things like libPND baked in.

• pandora-xfce-image – task-pandora-xfce

This is the default OpenPandora image that ships on the NAND. It contains everything in the core image above and adds XOrg, Xfce and MiniMenu as the GUI elements. It also contains a myriad of GUI applications, scripts and the like as you would expect.

• pandora-desktop-image – task-pandora-desktop

This is an Enlightenment (E17) based variant of the Xfce image above, it’s not had much love for a little while but should broadly be comparable to the Xfce image if you swapped out Xfce for E17 . I am hoping this image will get a little love ‘post release’ to bring it in line with the Xfce image.

• pandora-gui-image – task-pandora-gui

This is a Matchbox based variant of the Xfce image above, it’s not had much love for a little while but should broadly be comparable to the Xfce image if you swapped out Xfce for Matchbox . This image feels very ‘PDA’ like and less ‘pocket Linux system’.

pandora-desktop-image and pandora-gui-image are not really images intended for end users but rather then remove them from the tree they have been left so interested parities can mess about with them.

Remember that building any of these images from scratch is going to take several hours even on the fastest of boxes.

Building the OpenPandora release image is now is as simple as running

bitbake pandora-xfce-image

from the top level folder of your newly created build tree. Make sure you have run the environment setup script first or it wont find your metadata.

If all goes well just go away and have a coffee (or 20) and check every now and then to see how things are going.

It’s going to take a good few hours to build everything and there may be the odd snags along the way, if you run into troubles drop me a line and I’ll see if I can help.

Note: Once the build finishes you can check the output in tmp/angstrom.5/deploy/glibc/images/omap3-pandora, with a little luck you should have a version of the pandora-xfce-image image ready to go. A tar.bz2 (ideal for extracting to an EXT 2/3 SD card) and a .ubifs.img (ready to flash to the NAND).

I’ll cover what you should do to get your newly generated ROOTFS running on the OpenPandora in the next article.

Known Issues

OpenEmbedded is a complex beast at the best of times and can often throw up issues when you least expect things. As long as your building on recent, fairly standard, Debian based (so Ubuntu et. all) type distributions you should not run into much trouble.

I can’t speak for distributions like SuSE, Arch, Fedora etc. as I don’t use them.

One known issue that seems to crop up (especially with the release-2010-05-1 metadata, I have a fix lined up for the trunk) is that the build will fail complaining about an issue with the ‘encodings’ package.

Should you see that issue the fix is very simple.

bitbake -cclean encodings

Then you just need to rerun whatever command you had run before (don’t worry, it won’t run all the steps again) and everything should be just fine.

The encodings issue is caused by mixed up dependences and cleaning it out and letting OpenEmbedded rebuild it sorts out those issues.

If you notice other issues do please let me know (or better still send patches).

Regards,

John

ScummVM: Unofficial test “1.1.1” release for the OpenPandora.

This is just a quick post to let people know I have uploaded a test build of ScummVM for the OpenPandora. For those that care, the belated GP2X Wiz 1.1.1 releases will follow when I get back from a few days away at some point next week.

Note: Please don’t mirror or hotlink preview/test/alpha etc. releases or put them on download services but rather, direct people to this page. This helps me ensure that users always have the most recent versions and stops me being swamped with requests for help with ancient test versions.

Nothing especially exotic about the code as it stands (it’s mainly a plugin enabled ARMv7 optimised build of ScummVM packaged up as a PND file) but it works quite well and at least a fair selection of games are very playable.

I am only releasing this so people have some ScummVM goodness for their new OpenPandora consoles (you can stop emailing me now ). It’s largely untested and has been hanging around on my build box for a few days. All 1.1.1 game engines should be working but I have only tested SCUMM, Tinsel (Discworld) and SAGA, and played a little bit of The Curse of Monkey Island .

It does not represent what the final backend will look like and a lot of the OpenPandora specific graphic and input handling is disabled in this release (nothing is mapped to the DPad and game buttons/triggers for example). It does support the keyboard, nubs and touchscreen fine however (for the most part) and most regular ScummVM keyboard shortcuts should work fine.

My plan is to commit the changes to the ScummVM codebase as soon as I have things cleaned up a bit and make the backend official (and manage releases accordingly) but as I am now away for the next few days I thought I would do a quick test release 1st so people have something to mess about with it on there new toy.

To install just download and unzip the file and copy the resulting scummvm-openpandora-2010-05-26.pnd to the following folder on your SD card (or USB2 stick):

/pandora/menu –> This will cause the ScummVM app icon to show up in the Xfce menu sorted by category.
/pandora/desktop –> This will cause the ScummVM app to appear as an icon on the Desktop of Xfce.

Either folder will also ensure the application icon shows up in MiniMenu.

To run, just select the icon as normal, the ScummVM GUI will start up and you can add games as you would normally do with any other ScummVM release (including Mass Add).

Note: On the OpenPandora /media is the root mount folder for removable storage devices. Your SD cards are known by the driver name (MMC) and the card number so the root of your 1st SD card is normally /media/mmcblk0p1, 2nd SD card is normally /media/mmcblk1p1. When you 1st add games you will need to select the SD card you are using (or a folder on your USB2 stick).

Downloads:

• ScummVM for the OpenPandora – Test 1.1.1 release – 2010-05-26

Feedback:

Any feedback, please leave comments or drop me a mail.

Regards,

John Willis

OpenPandora: Booting, NAND layouts and general firmware fluff.

One of the common questions I get now and then on IRC (#OpenPandora on FreeNode) or email concerns how the OpenPandora booting and firmware is setup, what technologies are used in it’s setup and how we try and prevent things like accidental bricking and the like.

With this in mind I thought I would knock up a quick post detailing how these things are setup as a starting point to expand on.

I’ll start off with a technical overview of how the system boots and then detail how this will work for normal end users (and that includes ‘most’ developers).

There are 3 main sections to this post.

• Booting the OpenPandora: The ‘low level’ process.

Only of interest to hacker types .

• Booting the OpenPandora: End user booting and the boot menu.

Normal users who want to play with alternative firmware, boot there own firmware and such look here.

• The stock firmware NAND layout.

Want to know a little about how is the stock firmware organised on the NAND? What setup is used? etc. – read this.

Booting the OpenPandora: The ‘low level’ process.

This OMAP3 features a multi-stage boot process that starts with a hardware boot ROM (burnt into the SoC and totally unchangeable) and ends with the booting of some appropriate code that has full control of the system.

Stage 0: The OMAP3 hardware boot ROM.

One of the good points of the OMAP3 SoC’s is that they feature a robust hardware boot ROM that can be configured to act in several consistent, pre-determined, ways.

The boot ROM is limited, all it can do is get up to 64k of code into internal SRAM and pass control to it. This is however enough to offer a pretty flexible (and ‘brick-proof’) setup at the lowest level.

We utilize this to accomplish the initial factory flashing of units and provide our ‘final backstop’ should a unit get corrupted. Should the NAND of a unit be totally trashed you can use this hardware boot ROM to boot a correctly configured SD card in slot 1 and restore the entire system to a working state.

We have configured the hardware boot ROM to try booting as follows:

Power on normally:
       NAND –> USB Client –> UART (Serial) –> SD

Power on holding START button: 
       USB Client –> UART (Serial) –> SD –> NAND

Other OMAP3 boards may do this in a different order. It’s a characteristic of the board design.

This is called the ‘peripheral boot mode’ (and order) of the OpenPandora.

Please note that we are talking about the HARDWARE boot ROM not the software layer on top at this stage, this is an important distinction to make as most end users, and developers, will never need to have any dealings with the hardware boot ROM and it’s modes.

What the hardware boot ROM does for each type of ‘boot’ is summarised below.

• NAND Boot

The ROM looks for hashed startup code (normally a version of X-Loader hashed with ‘SignGP.c’) in the 1st sector of the NAND. If the sector is blank or corrupt the ROM will check the next 3 sectors before exiting. Once a good sector is found, the ROM transfers the contents to SRAM and transfers control to it.

• USB Client Boot

A vendor ID (Ti) and device ID (the model of the SoC) are sent out to the USB client port and should suitable software on the receiving end interpret and process this information the boot ROM will read data from the USB client port and transfer the data to SRAM. Control is passed to the start of SRAM if no errors are detected. The code transferred over USB is normally an unhashed version of X-Loader.

• UART (Serial) Boot

A special ID is written out to the serial port (on the OpenPandora EXT connection) and if it gets a response within a short period of time, the ROM will will read from the serial port and transfer the data to SRAM. Control is passed to the start of SRAM if no errors are detected. The code transferred over serial is normally an unhashed version of X-Loader.

• SD Card Boot

The ROM looks for an SD/MMC Card on the first MMC controller. If a card is found, the ROM then looks for the first FAT32 partition within the partition table and ensures that the ‘bootable’ flag is set on that partition.

If a suitable partition is found the root directory is scanned for a hashed file called ‘MLO’ (Normally a version of X-Loader hashed with ‘SignGP.c’) is read. Assuming all is well with the file, it is transferred into SRAM and transfers control to it.

Now we move on to the remaining stages of the boot process.

Stage 1: MLO (X-Loader)

Once the peripheral boot has taken place, by whatever means, you will be left with a copy of X-Loader running on the device. X-Loader is really just a massively cut down version of U-Boot designed to fit in the limited initial SRAM on the OMAP3.

Once it is running it will do some very specific checks, turn on a few extra components and jump to trying to load U-Boot for stage 2.

Stage 2: U-Boot

I won’t dwell on U-Boot here, it is a very well documented and well used bootloader used on lots of embedded platforms.

We have an OpenPandora specific U-Boot tree that we use to keep our platform specific patches in, most of our changes have long since gone into the mainline U-Boot trees.

U-Boot’s role is to setup most of the peripherals and load the kernel that will take control of the system. It also, in our case, provides a boot menu and a few additional tweaks to help developers.

Stage 3: Linux Kernel

This is the final stage of the boot process, control is handed from U-Boot to the Linux kernel and it is then up to Linux to get the system booted and running.

Booting the OpenPandora: End user booting and the boot menu.

So, having outlined the technical info on the OpenPandora boot system I am now going to mention that, all being well, most end users will never have any call to try and understand it.

As long as your unit has X-Loader and U-Boot correctly running from the NAND (as all units do out of the factory) you can take advantage of quick booting different OS’es from SD cards and tweaking the boot process without any need to create customised ‘fiddly’ bootable SD’s or compiling hashed copies of X-Loader and U-Boot.

In order to try and keep end users from having to get familiar with the quirks of the OMAP3 hardware boot ROM we (well Gražvydas Ignotas really) have implemented a number of features into the version of U-Boot that is loaded onto every unit.

One of the handier patches we carry in our U-Boot tree is the addition of a simple boot menu. This can be accessed by holding the right trigger when turning on the unit if needed.

The boot menu will present you with a number of options that can be used to control how the OpenPandora boots, the default is to boot the NAND as normal. You can also boot off SD’s, launch a U-Boot console over serial, and if we get the time, a console over USB serial. From within the menu the DPad and ‘B’ on the game buttons are used to select options.

Should you wish you can also bypass the bootmenu all together and boot your kernel (and ROOTFS) etc. off SD cards without any intervention by adding a suitable autoboot.txt boot script to the 1st FAT or EXT2/3 partition of SD1.

This is ideal to run alternative OS’es on the device without any intervention. It is expected that alternative distributions will come with these scripts pre-configured and ready to go, we will document this in more detail soon. If you want to try several OS’es on one SD that is even possible using the boot menu and scripts.

Note: Boot scripts.

The OpenPandora version of U-Boot supports a number of boot scripts.

• Automatically run.

autoboot.txt: This is a text file containing the raw commands you wish U-Boot to run. No error checking or checksums.

When running U-Boot will look for the presence of this script on 1st partition of SD1 providing that partition is FAT or EXT2/3.

• Run from the boot menu (Hold Right Trigger when turning on).

boot.txt: This is a text file with the raw commands you want U-Boot to run. No error checking or checksums.

boot.scr: This file contains commands in the U-Boot binary format with a header and checksum.

When running the menu U-Boot will look for the presence of these scripts on any partition of SD1 providing that partition is FAT or EXT2/3.

Want to try Ubuntu or Neuvoo just setup an appropriate SD card and script, pop it in the OpenPandora and turn it on, want to go back to the stock OS just turn off, eject the card and turn on again. Nice and simple. Just remember not to eject the card when the system is running from it .

The addition of the bootmenu and script support make hacking about on the OpenPandora and booting your own OS’es off SD etc. about as easy as it can be. Our plan is to offer ‘SD’ versions of the stock OS for download for people who want to hack about without messing with the copy installed to the NAND.

The stock firmware NAND layout.

We spent a discordantly large amount of time playing with the NAND layout to support a balance of simplicity, robustness and ease of hackability.

Eventually we settled on using UBI and UBIFS wherever possible as it has proven reliable, robust and has performed well to date.

The upshot of this is that the stock firmware on the OpenPandora is split into 2 volumes on the NAND, both are UBI and UBIFS volumes, one is small (10MB) containing the kernel uImage (or kernel’s if you so desire) and the other is a large volume that takes up the rest of the NAND with the ROOTFS.

Why did we go for 2 UBIFS volumes? Simple really, mounting UBI volumes from U-Boot is a linear process, the bigger the volume the longer it takes to mount. If we mount the big ROOTFS volume directly it takes around 8 seconds, by using a separate smaller volume to hold just the kernel we get this part down to less then a second.

Storing the kernel uImage on a UBI volume allows people to play with kernel updates and modifications without resorting to flashing the kernel to RAW NAND every time (you can even symlink kernels within the UBIFS volume so try new versions and roll back if bad etc. etc.). Believe me, when your playing with the NAND layout you come to appreciate things like this.

For those that may be interested the large volume is running with in kernel LZO compression and the images are pre-made with a combination of ZLib and LZO compression to try and achieve a good combination of compression and performance (-x favor_lzo -X 20 is what we settled on for our mkubifs if there are any UBIFS geeks out there).

Note: We experimented with a number of rather more exotic file system setups and for a long while we used a combination of SquashFS as a read only root and a UBIFS read/write area joined using AUFS to provide the union root file system.

This would have given us some very nice things like the ability to totally factory reset without reflashing etc. but the cost in terms of configuration and lost NAND space when doing updates (you have the original file in the R/O SquashFS volume and a complete new copy in the R/W UBIFS area) led to this being dropped in favor of the above layout. Not to mention that the big UBIFS ROOTFS volume is a little faster.

I suspect this post may lead to more questions than it answers so please fire away via the comments if you want to and I will try and update the post.

Regards,

John

A quick video demo of the Pandora using the PowerVR SGX core.

Whilst I have had PowerVR support working on the Pandora for some time now with the recent public release of the OMAP3 3.0.0.6 PowerVR SDK I now feel comfortable showing some demos using the 3D core.

It is also worth mentioning that from today all the Angstrom file system images available to Pandora developers will also include the drivers for the PowerVR SGX (taken from the above SDK) so expect to see other developers pick this up and run with it (I hope). I’ll admit I am quite looking forward to people getting stuck into really giving the core and the drivers a good workout.

As I am making this post it also seems worthwhile to give a little background on PowerVR support on the Pandora as it has frequently been a source of consternation.

A little while ago TI released the 3.0.0.5 drivers for the OMAP3 PowerVR core and this lead to posts like this one that unfortunately created some misunderstandings.

The 3.0.0.5 drivers TI released were basically old even at the time of release and required all manor of patching to even work on the Pandora/Beagle et al. They had been build for a 2.6.22 kernel paired with an old user space. Whilst they could be made to work on the Pandora they required dropping back to a quick port we did of TI’s 2.6.22 reference kernel and this was hardly an attractive prospect when everything else was fitting nicely with the 2.6.27 kernel we have lined up for the 1st release (don’t worry, we also keep our kernel current and stuck firmly to mainline Linux-OMAP).

I think the confusion at the time came from the fact that 3D did work on the Pandora with our 2.6.27 kernel but required a combination of drivers that were, at the time, not distributable (even to other Pandora devs). With the 3.0.0.6 public release this all becomes a rather moot point.

That is about all there is to it really. The PowerVR works well and we are working to sort the few remaining little snags (default clocks, wrappers etc.) in plenty of time for the release so you will be able to enjoy 3D on your freshly unpacked Pandora.

I’ll apologize in advance for the really crappy video quality but all I can get my hands on is an old digicam that happens to also produce AVI’s . I am sure Ed will produce some decent high res videos of the demos for the official page soon enough.

Oh, and ignore the slight flicker in the bottom right, I had an optical mouse connected when I did the demo that likes to ‘twitch’.

‘Mobile Broadband’ on the Pandora?

One of the little things I have been asked several times is can people use Mobile Broadband (GPRS/UMTS/HSDPA etc.) USB adaptors on the Pandora (in Europe we are lucky enough to have easy and fairly cheap access to cellular broadband tariffs).

While the real answer to this is ‘most probably’ with some caveats about Linux support for your chosen adaptor, USB 2 device support etc. etc. I decided to try and get my own adaptor working on my Pandora development board and this is a post about that exercise.

It is worth remembering that I am messing with the my own development images and this should not be considered anything ‘official’, it is just me having a hack about for the hell of it.

What adaptor am I using?

For this exercise I am using my Huawei E220. It has been upgraded with the 7.2Mbps firmware (version 11.117.09.04.00).

Under Windows this will appear as a USB CDROM (well CDFS on a USB Mass Storage device for the pedantic) and then prompt you to install the included software to drive the modem side of the adaptor. It is all actually quite tidy to be honest and makes use of the device pretty painless on Windows.

How is it connected?

Unfortunately my USB stick is a USB 1.1 full speed (USB OHCI) device so connecting it straight to the Pandora gets us nowhere. Remember the USB host socket on the Pandora is an EHCI (i.e. USB2 high speed) host so it needs a hub inline to step down to USB 1.1 speeds.

So out comes a USB hub and into a hub port goes the USB stick.

What happened?

Well, the first time I plugged it in (as you might imagine) not a lot happened other then the USB stack spitting out some info about the device and installing the mass storage side of things.

usb 1-2.4: new full speed USB device using ehci-omap and address 7

usb 1-2.4: configuration #1 chosen from 1 choice

scsi4 : SCSI emulation for USB Mass Storage devices

usb 1-2.4: New USB device found, idVendor=12d1, idProduct=1003

usb 1-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0

usb 1-2.4: Product: HUAWEI Mobile

usb 1-2.4: Manufacturer: HUAWEI Technologies

usb-storage: device found at 7

usb-storage: waiting for device to settle before scanning

usb 1-2.4: USB disconnect, address 7

usb 1-2.4: new full speed USB device using ehci-omap and address 8

usb 1-2.4: configuration #1 chosen from 1 choice

usb-storage: probe of 1-2.4:1.0 failed with error -5

usb-storage: probe of 1-2.4:1.1 failed with error -5

scsi7 : SCSI emulation for USB Mass Storage devices

usb 1-2.4: New USB device found, idVendor=12d1, idProduct=1003

usb 1-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0

usb 1-2.4: Product: HUAWEI Mobile

usb 1-2.4: Manufacturer: HUAWEI Technologies

usb-storage: device found at 8

usb-storage: waiting for device to settle before scanning

scsi 7:0:0:0: CD-ROM            HUAWEI   Mass Storage     2.31 PQ: 0 ANSI: 2

sr0: scsi-1 drive

sr 7:0:0:0: Attached scsi CD-ROM sr0

sr 7:0:0:0: Attached scsi generic sg0 type 5

usb-storage: device scan complete

Ok, a quick check of the shipped kernel modules shows that I forgot to build that into the image so off I trek back to my OpenEmbedded build system and after a quick edit to the defconfig for our kernel recipe and a bump of the kernel package revision I now have a nice usbserial.ko included in the image.

After deploying the new image I plugged it in and things got a whole lot more interesting. A couple of /dev/ttyUSB devices had been created, just what every modem connection needs.

usb 1-2.4: new full speed USB device using ehci-omap and address 4

usb 1-2.4: configuration #1 chosen from 1 choice

scsi1 : SCSI emulation for USB Mass Storage devices

usb 1-2.4: New USB device found, idVendor=12d1, idProduct=1003

usb 1-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0

usb 1-2.4: Product: HUAWEI Mobile

usb 1-2.4: Manufacturer: HUAWEI Technologies

usb-storage: device found at 4

usb-storage: waiting for device to settle before scanning

usb 1-2.4: USB disconnect, address 4

usb 1-2.4: new full speed USB device using ehci-omap and address 5

usb 1-2.4: configuration #1 chosen from 1 choice

usb-storage: probe of 1-2.4:1.0 failed with error -5

usb-storage: probe of 1-2.4:1.1 failed with error -5

scsi4 : SCSI emulation for USB Mass Storage devices

usb 1-2.4: New USB device found, idVendor=12d1, idProduct=1003

usb 1-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0

usb 1-2.4: Product: HUAWEI Mobile

usb 1-2.4: Manufacturer: HUAWEI Technologies

usb-storage: device found at 5

usb-storage: waiting for device to settle before scanning

usbcore: registered new interface driver usbserial

usbserial: USB Serial support registered for generic

usbcore: registered new interface driver usbserial_generic

usbserial: USB Serial Driver core

usbserial: USB Serial support registered for GSM modem (1-port)

option 1-2.4:1.0: GSM modem (1-port) converter detected

usb 1-2.4: GSM modem (1-port) converter now attached to ttyUSB0

option 1-2.4:1.1: GSM modem (1-port) converter detected

usb 1-2.4: GSM modem (1-port) converter now attached to ttyUSB1

usbcore: registered new interface driver option

option: USB Driver for GSM modems: v0.7.2

scsi 4:0:0:0: CD-ROM            HUAWEI   Mass Storage     2.31 PQ: 0 ANSI: 2

scsi 4:0:0:0: Attached scsi generic sg1 type 5

usb-storage: device scan complete

Driver 'sr' needs updating - please use bus_type methods

sr0: scsi-1 drive

Uniform CD-ROM driver Revision: 3.20

sr 4:0:0:0: Attached scsi CD-ROM sr0

ISO 9660 Extensions: Microsoft Joliet Level 1

ISOFS: changing to secondary root

ls /dev/ttyUSB*

/dev/ttyUSB0  /dev/ttyUSB1

The first thing that jumps out at me is why the kernel has helpfully installed the Option driver when this it not an option card ‘option: USB Driver for GSM modems: v0.7.2’. That does not look right but lets give it a try anyway.

After a quick read it seems that if you don’t have any other USB serial devices connected the modem will be assigned to /dev/ttyUSB0 and the user interface device to /dev/ttyUSB1. So far so good.

Now to wrap all this up and see if we can connect to the Internet.

It is worth noting at this point that our images currently have NetworkManager 0.7.0 in them as I have been evaluating that for managing the WiFi connection (along with Connman) so my plan is to see just how good the NetworkManager integration with the underlying adaptor is using the command line tools (I do not have the nm-applet GUI working with 0.7.0 yet).

So I booted up the resulting image on the Pandora, plugged the USB stick in to the hub and waited a few seconds for that to settle and get connected then started to setup the connection.

netm-cli -l

GSM Devices:

ttyUSB0:

        Udi: /org/freedesktop/Hal/devices/usb_device_12d1_1003_noserial_if0_serial_usb_0

        Driver: option

        Capabilities:

                - Supported

        State: Disconnected

        Managed: True

        IP config:

                No IP settings found

Excellent, ok, NetworkManager sees that we have a ttyUSB0 GSM device. Now lets try connecting to the internet with it using my Vodafone SIM.

netm-cli --connect --gsm=ttyUSB0 --user=web --passwd=web --apn=internet --number=*99#

Device ttyUSB0: activating the connection

Device ttyUSB0: created connection '/org/freedesktop/NetworkManager/ActiveConnection/1'

/org/freedesktop/NetworkManager - State: Connecting

Device ttyUSB0: state - Prepare

Device ttyUSB0: state - Config

Device ttyUSB0: state - Need Auth

Device ttyUSB0: state - IP Config

/org/freedesktop/NetworkManager - State: Connected

Device ttyUSB0: state - Activated

/org/freedesktop/NetworkManager/ActiveConnection/1 - State: Activated

Wonderful, NetworkManager has talked to the modem and negotiated a connection to the internet.

Sparking up a Firefox session on the Pandora proves this and shows that it is working just fine. FTP and all the usual IP services are all working as expected. The connection has been up for a good few hours now and seems very stable.

So, what does this all prove. Well Mobile Broadband sticks (at least the Huawei E220) will work on the Pandora, NetworkManager is happy to configure the connection and as soon as I can get the NetworkManager 0.7.0 GUI going this whole process should be ‘plug and play’ just like it is on other major distributions like Ubuntu.

I am not 100% sure at this stage about choosing NetworkManager or Connman for the final images (it is rather dependent on some tweaks that need doing to our TI1251 Wireless driver to properly support Linux Wireless Extensions and further stability/signal tests) but either way NetworkManager will be available in the Pandora repositories and, as proven here, provides a perfectly viable and quick way to connect to Mobile Broadband if you have supported hardware.

John

OpenPandora, OpenEmbedded and Ångström

One of the things I have been asked a few times is what the relationship is between OpenEmbedded, Ångström and the Pandora.

It really is quite simple . To quote Wikipedia “The OpenEmbedded Project (OE for short) is a software framework to create Linux distributions aimed for embedded devices”.

Ångström is a distribution that builds upon OpenEmebdded to collect everything needed for a feature rich Linux kernel based operating system.

We use Ångström as a base to then build the operating system images for the Pandora. There are a number of reasons behind this but some of the top ones include OpenEmbedded’s extreme flexibility, Ångström’s sensible approach to configuration and the ability for us to get fine grain optimisation into the images (packages on the Pandora are ARMv7 where possible and many include patches for Cortex A8 features like the NEON or OMAP specific optimisations).

OpenEmbedded’s overlay approach to the way it handles metadata also allows us a great deal of freedom to tailor the Pandora distribution without compromising the main Ångström/OpenEmebdded metadata removing any need for silly things like forks or additional OpenEmebdded distributions.

When I talk about Ångström on the Pandora I really am talking about the mainline Ångström and not some nasty Pandora specific fork. If we add new features to Ångström our aim is to push them upstream in a sensible way quickly and only provide our overlay with the minimal amount of transitional stuff or things that would never be acceptable in mainline.

Anyway, on the back of all this I recently made a long overdue hello to the Ångström developers mailing list to try and garner some interest and support. If your interested in reading that you can find the mails in the Ångström developers mail archive here.