Broken Onkyo receiver

So my friend had an Onkyo receiver which after a network update broke some of its functions.
Obviously this gave me an excuse to unscrew the receiver, and get inside and see the linux executions inside. I am documenting what I found there, in hope that it may help someone else.

This receiver is pretty old, and is out of warranty. And my friend was not too worried about me breaking it, since it will give him an excuse to go by a new one. The exact model number is TX 515. But I believe the same or similar mechanism / file system is used in most Onkyo receivers released around that time. Since when new these cost a lot, and these file systems are constructed in a very unstable way in general, I do not recommend messing around with these when your receiver is still under warranty.


1. Main processors/components of interest in HDMI board.

Before touching anything I did my usual google searches and found that the service manuals of these units are available, and have good information and schematics. Kudos to the manufacturer to provide that info.

From the service manual and the physical HDMI board, I found that the below components are interesting.
   a. Texas Instruments DA830 DSP
   b. Marvell 88DE2755 VPU

   Both the above SOC has internal ARM (v5 arch), and have serial port access. On further inspection of both boot logs I find that the DA830 boot/filesystem is most useful for basic changes.

On further searching I find that the DA830 is documented well and DaVinci SDK is available for public download. I studied the SDK and felt really confident that there are enough tools available for me to get it back to life, even if in case I find some components have hardware failure.

2. Serial Port

Looking at the board I could identify some candidates for serial port access. But with schematics in hand, it was good to go.

PA3102 connector has DA830 serial port.
      pin3: RX
      pin6: TX
P8030 connector has Marvell VPU serial port
      pin1: +3.3V
      pin2: RX
      pin3: TX
      pin4: GND

3. The issue

By this time I actually almost forgot the issue :) .... welll... it was just an excuse anyway. This TX-515 does not anymore initialize its USB or Network, there by pretty much making those functions totally unusable. He still had the HDMI ports working and have HDMI out etc. But no power in USB ports and no network access.

Some of these boards have soldering issues overtime because of heat. A search find that a lot of such failures are related to the same DSP we were going after. But in this specific case, we had the HDMI components working and the sound output was there. Which means the DSP does infact boot. So there is a good chance that some of the filesystem is corrupted and not getting all the required modules intialized.

By the for folks who are interested in the DSP solder failure issue (which is indicated by no sound when system is cold), there are tons of information available online. So I am not documenting that here. But the second issue my friend had does not had any clearly documented analysis or fix.

4. First thing first, got my USB serial port connected to analize Davinci DSP boot (PA3102 connector).

Find what is happening

5. U-Boot

Uboot is loading properly, and from log I see that the NAND chip used is
NAND device: Manufacturer ID: 0x98, Chip ID: 0x76 (Toshiba NAND 64MiB 3,3V 8-bit)

And it is not a dumped down uboot. I was planning to flash the full uboot if this was a dumped down version of uboot before further debugging. But seems like I don't have to. All the needed bells are there. I can do tftpboot/usbboot etc without even touching the existing file systems. That is promising.

U-Boot > help
?       - alias for 'help'
askenv  - get environment variables from stdin
autoscr - run script from memory
base    - print or set address offset
boot    - boot default, i.e., run 'bootcmd'
bootd   - boot default, i.e., run 'bootcmd'
bootm   - boot application image from memory
bootp- boot image via network using BootP/TFTP protocol
cmp     - memory compare
coninfo - print console devices and information
cp      - memory copy
crc32   - checksum calculation
dhcp- invoke DHCP client to obtain IP/boot params
echo    - echo args to console
exit    - exit script
fatinfo - print information about filesystem
fatload - load binary file from a dos filesystem
fatls   - list files in a directory (default /)
forceenv  - force set environment variables
go      - start application at address 'addr'
help    - print online help
iminfo  - print header information for application image
imxtract- extract a part of a multi-image
itest- return true/false on integer compare
loadb   - load binary file over serial line (kermit mode)
loads   - load S-Record file over serial line
loady   - load binary file over serial line (ymodem mode)
loop    - infinite loop on address range
md      - memory display
mdc     - memory display cyclic
mii     - MII utility commands
mm      - memory modify (auto-incrementing)
mtest   - simple RAM test
mw      - memory write (fill)
mwc     - memory write cyclic
nand    - NAND sub-system
nboot   - boot from NAND device
nfs- boot image via network using NFS protocol
nm      - memory modify (constant address)
ping- send ICMP ECHO_REQUEST to network host
printenv- print environment variables
rarpboot- boot image via network using RARP/TFTP protocol
reset   - Perform RESET of the CPU
run     - run commands in an environment variable
saveenv - save environment variables to persistent storage
saves   - save S-Record file over serial line
setenv  - set environment variables
sleep   - delay execution for some time
test    - minimal test like /bin/sh
tftpboot- boot image via network using TFTP protocol
usb - USB sub-system
usbboot - boot from USB device
version - print monitor version

Now checking the boot parameters I see that the kernel is at offset 0x454000

U-Boot > printenv
baudrate=115200
bootfile="uImage"
verify=n
ethaddr=00:09:b0:cd:a8:7a
bootargs=console=ttyS1,115200n8 root=/dev/mtdblock8 rw quiet lpj=999424 rootfstype=yaffs mem=56M ip=off
setparam=ok
bootcmd=nboot.jffs2 0xc0200000 0 0x454000
autostart=yes
bootdelay=1
dspboot=yes
stdin=serial
stdout=serial
stderr=serial
ver=U-Boot 1.3.3-svn1998 (Mar  5 2012 - 08:10:29)

Environment size: 356/16380 bytes

6. I continued the boot and immediately found the basic issue. The netapp binary keeps crashing, and since it is called with a restart on crash, it keeps trying and crashing.

7. So I went back to uboot. Changed the bootargs temporarily and booted for single user access. This mounted /dev/mtdblock8 as root file system. Checked the service /etc/init.d/netapp

Before further experimenting with netapp, I manually tried loading the important services and module loading followed by netapp, and found that the USB power was up and NET was up. So it was just the netapp application crashing causing the boot to halt and not let other services up.

8. Now let us dig a bit more deep into the netapp service... and try to figure out why is it crashing.. 

If needed I was ready to recompile a new netapp binary.

I see that the netapp binary is called from /opt/onkyo/avr/bin/netapp

But before that it does mount a few filesystems. The most interesting one for us the mounting of /tmp/squashfs followed by a loop mount of /opt

First it mounts a yaffs file system from a variable optfs defined in /etc/boot.properties

This essentially should leads to mounting /dev/mtdblock9 as /mnt/squashfs . Please note that /mnt is linked to /tmp. Followed by loop mounting of /opt using a file in /mnt/squashfs/da83x_rootfs_opt.img

Interestingly enough based on existence of a file /etc/bootchange it mounts a different partition as /mnt/squashfs. /dev/mtdblock10 in this case.

Eventually it executes the netapp binary from 

/opt/onkyo/avr/bin/netapp

In the current system the /etc/bootchange was available and so it was using mtdblock10 partition. From the name I guess this was some kind of a messup during some firmware update process. So out of curiosity I mounted /dev/mtdblock9 followed by loopmount of the img file in there, and see that md5sum of that netapp differs from the one from mtdblock10 loopfile.

Executing that newly found netapp binary seems to not crash.

9.  So the final FIX....

mount /dev/mtdblock8 as writable and delete /etc/bootchange file. and reboot...

Thats it......This seems easier that expected. I was infact a bit disappointed, since there is almost no challenge :) . Too straight forward. Almost too boring to answer to my friend when he ask me what I had to do to fix it.

mount -o remount,rw /
rm -f /etc/bootchange
mount -o remount,ro /

powercycle.


10. Final thoughts...

So just in case if you could not find the correct files in /dev/mtdblock9 you can always get it from downloading the firmware update files from web. Someone else already clearly documented on how to get the file systems extracted from the onkyo firware update files downloaded.

Also you could manually load all the modules, and once USB/NET is available you can also do a clean firmware update to get things back to stable.

Also I was able to compile linux from SDK and so it is possible to replace with custom kernels. But since it is not a device I own, and I don't have any further excuse to mess around with this, I am stopping it here and returning the fully functional receiver back to my friend.

There were 3 binaries from the optware of ONKYO which gained my attention.

1. Audioappe : This is the main audio processing app. You need this to have ANY sound output.

2. spid : This app handles the peripherals communications. You need this for HDMI communications.

3. netapp : This is needed for the higher level communications like Spotify Connect/ Firmware Update communications, DLNA etc. This application is very resource heavy and use a lot of CPU. If you mostly listen to Audio without using the inbuilt NET function of the receiver, it maybe worth to disable this app for most usage. But this binary internally calls the module loader. But modules can be loaded separately via /etc/init.d/modloader function call.

At some point I will create an HTML api to disable netapp and still load modules like USB driver etc.
So when you need spotify/DNLA/fwupdate functions you can just re-enable the netapp via the HTML. I will do that later and when I do it I will publish the details in another page.

I believe this disabling of netapp, would help with fixing the overheating issues of the DA830 soc, since it reduces the cpu load drastically.

USBBooting Belkin F7D7301 (incomplete)

===================================
Warning: Please note that this is something I started documenting almost a year back, and then lost interest to make it complete for properly publishing to be usable for most. Whatever I documented are still applicable, and I have a working USB boot. But I do not plan to make the missing pieces of links complete on this topic. Publishing this as it is, in hope that it could be usable for somebody who is thinking in the same terms.

Warning: Not for non-tech users.
===================================


I have published earlier how to apply some custom patches and get OpenWrt working for Belkin F7D7301. I found a lot of interest from folks in updating the firmware, and due to the specific custom headers required for these TRX files, I believe it is a good idea to not update these TRXs often, if you are a guy experimenting with different kernels and versions. If you dont touch the trx, and only worry about the raw linux for the updates, you also dont need most of the custom patches addressing the trx stuff, and as long as you have got the networking drivers compiled, you got your router alive. F7D7301 on the heart is anyway a mips, so if you handle the drivers, you open the world for a lot of experimentation via USB and NFS or NBD roots.

Another big advantage for the folks who like to live on the edge are, in the case of a wrong kernel or firware update, going back is as simple as taking your usb drive out and changing the boot file.

Please note that most of the concepts mentioned here can be applied to other embedded devices or routers as well.

We have three options for getting the usb boot working.

a) Use a modified CFE to allow usbfs boot.
     Advantage: Uses least amount of flash/mtd, so if you are going to use USB loading, you dont depend on anything else in the flash other than CFE. You could still leave your stock firmware in flash or even have Tomoto/DDWrt in flash.
      Disadvantage: You mess up CFE, you brick the router really bad.

b) Use a second level boot loader using methods like kexec. Load this second level boot loader from CFE.  This options stands exactly between a and b. Uses and depends a bit more on flash than a custom CFE, but a lot less than a full OpenWrt Image on flash. You could still have stock firmware in flash, but might need some customizations.

c) Have a regular root squashfs image in flash and use a custom /etc/preinit script, to do the kexec loading.
     Advantage: Least risky for a regular user. Even in case of usb boot failure, you could boot the regular openwrt image from flash and business will be as before any usb booting.
    Disadvantage: You still depend on your squashfs image. So a flash corruption in this area still could break the USB boot. Also you read a full linux image from flash and mounts a full squashfs image from flash. Which means a lot of dependability on the flash, even though eventually you are keeping your experimental boot images in USB drive. But this is probably OK for most users.

I have tried all the above 3 methods, but found that option c is the least risky one and will be the most transparent one for any openwrt user.

The custom preinit script in squashfs.

If you are familiar with the openwrt boot process, you know the importance of /etc/preinit. This is the first script runs after mounting squashfs. I moved the /etc/preinit from squashfs to /etc/preinit.00, and put a custom init script in that place, so that I can do the USB booting or Network booting ealiest possible stage.  Here are the steps done by custom preinit.

1. Loads the required modules for USB and basic Networking.
2. mount tmpfs for all the work.
3. Initialize IP and try getting a boot script from an http server on LAN side.
4. If that succeds execute that init script (will get to the details of this init below)
5. If no network boot script try mounting USB read only using vfat or ext4.
6. See whether  USB has the boot init script at location /boot/
7. If it finds the boot script in USB, copy that over to tmpfs
8. unmount USB and execute that boot script (will get to details f this in below)
9. If no boot script was found from network or usb, then go proceed with squashfs preinit.
10. boot the openwrt in regular way via squshfs+jffs2.

Please note that you cannot modify the custom preinit script, since it is in squashfs filesystem, which is read only. But since you are loading your main boot script from either network or from USB you could modify them and experiment all you want without breaking the router.

Boot scripts for USB boot and Network boot

Note that you are not just looking to have your root file system in USB or Network, you also want your kernel to be in USB or Network.That way you dont depend on the version of kernel you have in your MTD/Flash of the router. The boot script is the one achiving this, and below are the functions.

1. Mount your USB or Network block device.
2. use kexec to load your new custom kernel from USB or network mount. (details of the customizations needed in kernel are explained below).
3. umount your USB or network device.
4. execute the custom kernel.

Now your custom kernel will take care of other stuff.

The custom Kernel Image for USB boot and Network boot

The custome Kernel Image does use a bigger initramfs. So the image is bigger. But it doesnt steal much memory, once you mount your root file system from USB, since the switch_root function does a good job of cleaning the initramfs from memory. Below are the functions of custom initramfs.

1. Initiramfs has a static busybox for doing all the required functions it need to do.
2. Load All the required modules for mouting your rootfs. For network root it will be the network drivers and nbd or nfs drivers. For USB root it will be your usb drivers. Also you need file system driversto be loaded. It could be vfat ext4 etc. You may also want loop driver.
3. Mount USB or NBD block device. If you keep the root files in a loopimage, mount that as well.
4. Now since the root is mounted, SwitchRoot to your root, and thereby cleaning up the memory used by initiramfs.

The customizations of preinit in USB root image.

I added some customizations in the USB/NBD root image preinit, so that I wont corrupt the jffs2 in flash. Also I chose to not use an overlay, since the USB root image is writable. So I commented out the boot hooks for jffs2 check and mount.

Your options.

1. You can just not use any of it and do something more useful :)
2. You can just flash the trx I compiled and have a regular openwrt image.
3. Following step 2, copy the usb-kernel and usb-root I compiled to your USB and have a USB boot.
4. Follow step 2 and 3. And then compile your own usb-kernel and usb-root.

Enough talk... Show me the example of USB Boot.

 1. Follow the steps here and flash the trx image.
 2. Format your USB drive to ext4/ext3/ext2 or vfat.
 3. Create a directory boot in your USB drive.
 4. Download and copy the below files to /boot/ of USB drive.
       a) Customized Kernel elf
       b) Cutomized root loop image
       c) USB boot script
 5. Unmount your USB drive from computer, and attach it to your F7D7301 router.
 6. You should be able to boot to USB loop image when you power cycle the router.

Update2 : OpenWrt on a Belkin Share Max N300 (F7D3301 or F7D7301)

Please refer to my old blog post for the details on the modifications. This is just an update from the earlier patches, and some of the content in earlier blog apply here as well.

As of rev@34995 (Jan 2013), the bgmac driver is officially supported in OpenWrt. This means now we dont have to apply George's patches to get the bgmac support anymore.

We still need the patches for
1. LZMA loader
2. Belkin custom TRX magic
3. Diag
4. Netconfig scripts.

I am posting a cleaner updated patch below, which applied perfectly on top of rev@35445. It should work without much pain on most other recent trunk versions as well.


Below are the exact steps to produce a trx file, which can be flashed via CFE.
1. Checkout openwrt (a specific version)
svn checkout --revision=35445 svn://svn.openwrt.org/openwrt/trunk
Other versions may work, but I have not tested any other version for these patches.

2.  Apply My custom patches
    Download from here: belkin.7301.r35445.patch
    cd trunk   
    patch -p1< belkin.7301.r35445.patch

3. make menuconfig    
    select Target System Broadcom BCM947xx/953xx
    select Target Profile Broadcom SoC, bgmac Ethernet, BCM43xx Wifi wl
    make

You will have the trx in the usual place after a successful compile.
Look for the file: bin/brcm47xx/openwrt-brcm47xx-f7d3301-squashfs.trx


Please note the below points if you flash the above trx.
1. default IP address at LAN side is 192.168.2.1
2. telnet can be accessed at default IP,



--------------------------------
Please note that I am trying to submit this patch to openwrt officially and get it supported. But it seems like there might be some work to clean it up for the likings of openwrt. Any of you folks are familiar with the process, please feel free to take the patch and submit it.
--------------------------------

Update : OpenWrt on a Belkin Share Max N300 (F7D3301 or F7D7301)

Please refer to my old blog post for the details on the modifications. This is just an update from the earlier patches, and some of the content in earlier blog apply here as well.

Below are the exact steps to produce a trx file, which can be flashed via CFE.
1. Checkout openwrt (a specific version)
svn checkout --revision=33760 svn://svn.openwrt.org/openwrt/trunk
Other versions may work, but I have not tested any other version for these patches.

2.  Apply My custom patches and George's patches
    Download from here: 
000-openwrt4716-TARGET_brcm4716-clone-brcm47xx.patch
001-openwrt4716-TARGET_brcm4716.patch
002-openwrt4716-TARGET_brcm4716-deps.patch
004-f7d7301.r33760.patch


    patch -p1 < 000-openwrt4716-TARGET_brcm4716-clone-brcm47xx.patch
    patch -p1 < 001-openwrt4716-TARGET_brcm4716.patch
    patch -p1 < 002-openwrt4716-TARGET_brcm4716-deps.patch
    patch -p1 < 004-f7d7301.r33760.patch

3. make menuconfig     select Target System -> Broadcom BCM94716
     select Target Profile -> Belkin PlayMax F7D4301
     make

You will have the trx in the usual place after a successful compile.
Look for the file: bin/brcm4716/openwrt-brcm4716-f7d3301-squashfs.trx

For reference one trx I compiled is here : openwrt-brcm4716-f7d7301-squashfs-r33760.trx

Please note the below points if you flash the above trx.
1. default IP address at LAN side is 192.168.2.1
2. telnet can be accessed at default IP,
3. I have added a custom pre-init script to enable me do USB Booting. But it should not affect anything, if you dont do USB boot, except for a few more seconds boot time. I will add the details on how to do USB booting and publish the USB kernel modifications in another section.
For reference the custom preinit is here: /etc/preinit
4. For additional reference, the contents of the root file system : root-brcm4716.tar.gz

Suggestions are welcome, and I will try to learn from your suggestions. But please note that I dont have much time to spend on keeping the trx upto date with openwrt releases, or to add/delete packages or modules.

Yate-4.2 on Openwrt with Google Voice support

==================================
Update: August 22, 2012
Yate 4.2 is already out and so I updated the below instructions and updated the makefile. This time I also took some time to find the exact dependancies and added that in the makefile.


1. Goto your openwrt compile environment base directory.
2. mkdir package/yate4
3. Download Makefile.yate4.2 to package/yate4/Makefile
----
Note: if you dont have compiled openwrt in this directory, you might have to build the toolchain before the next step.
make tools/install
make toolchain/install
----
4. make package/yate4/compile
5. make package/yate4/install

==================================

Yate-4.1 on Openwrt with Google Voice support

==================================

Update: August 22, 2012
Yate 4.2 is already out and so I updated the below instructions and adde a new Makefile. See the related blog on Yate 4.2 at the link below:

Yate-4.2 on Openwrt with Google Voice support

==================================
Puiblished: June 2, 2012

I had been using Yate4 for sometime and it is an excellent telephony application. As of this writing openwrt trunk only includes yate-3.2 and this limits some of the advancements in the application. I like a lot of things about yate, even though it may not be categorized as a replacement for asterisk. For a lot of folks, the excellent Google Voice support in yate, makes it a must have item in their embedded device running openwrt.

I will explain how to make yate 4.1working in openwrt. There are help pages available in openwrt for
1. how to create custom feeds
2. how to create a package
And they both explain a lot of detail on how you can achive getting custom feeds in clean way. So I am not planning to put a lot of effort on explaining those, and will focus on the core part, the makefile.

Also I will show you how to create an ipk file which you can install on your openwrt platform. The cleaner way of doing is create a custom feed and put yate4 package with the makefile in the custom feed.

Here is the makefile you can use for this purpose.
Makefile.yate

The steps to compile your own ipk is below. The pre-requisite for this is that you already have setup your openwrt compile environment, and know how to compile openwrt from source.

1. Goto your openwrt compile environment base directory.
2. mkdir package/yate
3. Download Makefile.yate to package/yate/Makefile
----
Note: if you dont have compiled openwrt in this directory, you might have to build the toolchain before the next step.
make tools/install
make toolchain/install
----
4. make package/yate/compile
5. make package/yate/install

You will have the yate4 ipk file under bin/packages
Now you can use opkg install yate4.ipk to install this package to your openwrt platform.

How to make google voice working. I was going to explain this detail here, and was documenting how I made it working. But I realized that now there is an excellent documenation in the official yate webpage.
Here is the link:  Yate4-ConnectingToGoogleVoice

Follow that link to make google voice working.

Note: For a handful of folks, who dont have an openwrt compile environment, and want to have an ipk, please post a comment with the exact router model you have (to know the cpu) and your kernel version. If I already have an ipk compatable for the model, I will share it with you. (please dont expect a reply if I dont have the ipk for you.)

OpenWrt on a Belkin Share Max N300 (F7D3301 or F7D7301)

Please see the update on thistopic at below link:
=====================================

UPDATE : OpenWrt on a Belkin Share Max N300 (F7D3301 or F7D7301)

=====================================


OpenWrt on a Belkin Share Max N300 (F7D3301 or F7D7301)

It was not a difficult job. Had to write a couple of patches to get it working. But a lot of work had been done by "George Kashperko" for Asus-RT-N16 . I took his patches, and applied some Belkin Specific patches.

The main modifications I made are:

1. Belkin trx headers: For some reason Belkin chose to use a custom header for trx. So modified the Makefiles to generate trx files with this custom header. Without this custom header Belkin CFE wont allow flashing the trx. And I preferred to not touch the CFE.

2. LZMA loader: Due to above modification, now I had to add some more code in the lzma decompressor and loader, to recognize the belkin headers. Without this the lzma loader will load, but wont boot, since it doest understand the new trx format.

3. Bridge setups. The virtual lan setup files needed some modification because of changes in port connections.

4. Diag and LED/GPIOs. Needed some modification to recognize the correct gpio pins and power/diag/usb leds.

Below are the exact steps to produce a trx file, which can be flashed via CFE.

1. Checkout openwrt (a specific version)
svn checkout --revision=30776 svn://svn.openwrt.org/openwrt/trunk

2.  Apply George's patches
    Download from here: 
000-openwrt4716-TARGET_brcm4716-clone-brcm47xx.patch
001-openwrt4716-TARGET_brcm4716.patch
002-openwrt4716-TARGET_brcm4716-deps.patch

    patch -p1 < 000-openwrt4716-TARGET_brcm4716-clone-brcm47xx.patch
    patch -p1 < 001-openwrt4716-TARGET_brcm4716.patch
    patch -p1 < 002-openwrt4716-TARGET_brcm4716-deps.patch

3. Apply my patch
   Download from here: 003-belkin-f7d3301.patch
   patch -p0 < 003-belkin-f7d3301.patch

4. make menuconfig
     select Target System -> Broadcom BCM94716
     select Target Profile -> Belkin PlayMax F7D4301
     make

You will have the trx in the usual place after a successful compile.
Look for the file: bin/brcm4716/openwrt-brcm4716-f7d3301-squashfs.trx

For reference one trx I compiled is here : openwrt-brcm4716-f7d7301-squashfs.trx
This is for reference ONLY. Even though I can see many users finding this good enough, I suggest you compile your own trx, when you see your needs are different. If you flash the above trx, you will have telnet access at 192.168.1.1 to your router.

You can install packages from openwrt trunk. But for that you have to addthe below lines to /etc/opkg.conf . Please dont install kernel modules from trunk snapshot, that may lead to kernel instability. But other packages should work without much issue. But as always, if something doesnt work from trunk snapshot, try compile that yourself using proper menuconfig.
#-------------

arch all 100
arch brcm4716 200
arch brcm47xx 300

#-------------

For details on opkg look here : opkg reference


Have fun...