matlab shlib cancer

Ah Matlab.  Some things are made easier by this language/platform... it's got it's large share of rough edges though.  But today I'm not going to complain about issues like race conditions in the ui libraries somehow existing on top of swing, a toolkit, one of the first, developed to be safe against such issues.  No, today I'm going to complain about how Matlab includes the following libraries:

• Boost
• OpenCV
• libtiff, libpng, libjpeg, openjpeg
• wxwidgets
• libhdf5, netCDF
• OpenSceneGraph
• ITK
• FFTW3
• expat
• QT (Qt4)
• protobuf 

Here's the problem though.  They don't distribute headers or any kind of SDK.  They also patch the softwares and don't release the diffs causing all kinds of breakage.  They end up making it impossible to use these libraries in mex files.  But they are using more of these libraries and others like them as time goes on and breaking more and more mex files and codebases as time goes forward. Why are they including everything under the sun and not providing a way to use it?  What do they have to gain by not shipping headers? Do they think this is sustainable?  How frustrating and greedy!

Please mathworks, include the freaking headers, make these libraries continue to be usable in mex files.

And update your copy of HDF5.

linux WTFs

Long have I been noticing things in linux that just don't make sense. I've not always submitted bug reports for these things and instead quickly noted them down so I could fix them later, some rainy day.  Many could involve big long discussions or otherwise warrant an RFC. Others I just need to notify the maintainer or otherwise submit a patch.  Here is my unordered list as of today:

1. filesystem does not support epoll/poll - only select - breaks lots of software such as epoll reactors
2. select and poll use different backends in the kernel
3. no way to dequeue a randomly sized network packet without doing 1+ MSG_PEEK reads until success, and then it's messy because another msgrecv is needed to dequeue those bytes but this does a second copy! The "discarding read" is only supported with sock_dgram and socket_seqpacket (when not SCTP). An approach with SCTP and socket_seqpacket also dequeues x bytes at a time. For all protocols, the size of the packet/message is in the packet/protocol headers... pass it up!  Let the application decide of the packet is of reasonable size to be acceptable more directly.
4. no way to know of crc failures in sockets api
5. CLOCK_MONOTONIC_RAW is what you really mean - CLOCK_MONOTONIC is not monotonic
6. timerfd doesn't support CLOCK_MONOTONIC_RAW
7. pthread blocking calls are non cancelable/interruptable. They should be like poll and return EINTR or some such, instead utilizing software becomes prone to deadlock.  Posix is stupid if it disagrees.
8. No way to get kernel buffers / physical memory in userspace. something close to malloc in use. Integration to write etc also needed. Idea is to have DMAable memory so there's no copying being done.  The idea is also to be integrated with the ecosystem and not so much a stand alone kernel module. You might quickly spout mmap, but it doesn't solve everything...  For instance, there's no way to hand off one buffer to say the write system call and guarantee there's no copying going on.  Perhaps the buffer came from another device's DMA memory so it is something that shouldn't be allocated from the write device either.
9. opening a file can have large latency, even for O_NONBLOCK
10. No separation between UART driver and console stack - they are tightly coupled together.  And ancient.
11. No reliable way of allocating a large chunk of memory from kernel. Think image processing situations from a camera... big chunks are needed.
12. libgomp: active and default consume too much cpu! passive also sucks... needs to use syscalls or something to minimize latency and cpu consumption.  Say with eventfd.
13. GCC packed struct fields: can't take the address of them with const &? They have an address usually... how is this wrong (except for when bit fields are present). How about an unpacked packed and packed version for free so we don't need to write both.

not quite wtf or gnu:

• qlz or lz4 for realtime usable compression (much better than lzo, zfs has included lz4).  Mostly looking at kernel inclusion, especially for btrfs, although these days I only care about zfs on linux.
• openssh: no modularization for key authentication - ie one could tie into an sql or otherwise external authenticator/key database

To blog someday:

• headaches with rs485 or transceivers after rs232? make sure to reset and lower rts...
• how I got 900 MB per second with no latency through the XFS filesystem
• how to survive sigbus in a multithreaded environment - say from mmap'd files
• the glory that is ppoll/epoll & how eventfd kicks pthread condition's ass
• the awesomeness that is pc104e? for io needs at least
• the situation with framegrabbing and poll... rather its absense and buffer management.
• how to implement a camera framebuffer management ie with doubly linked lists
• matlab with numtraits and create_scalar

2012 Ford focus failings

Wait this is a software related blog, right? Yep, I'm not so much a car guy. I wanted functionality in a good package that's cost effective.... and something sporty (I'm still young, kind of). So I got myself a 2012 Ford Focus with the Titanium trim level. Overall great car, I think. But as a software developer I am WTFing at sync and the nav system. This thing is a buggy POS. Speech recognition seems to crash after an hour at the wheel and now I've learned that the nav system can partially crash as well and report stale information. Touch input can also die, combined with the computer rebooting itself and performing checks (like filesystem recovery), though I've had to walk out of the car to get the freaking thing to fully shutoff and reboot (if there's a button, please tell me where I can find it). The hardware specs are also crap. For 800 dollars you end up with a nice screen but no real hardware backing it. 46 megabytes of ram and some weak processor my Evo 4g (snapdragon based) phone outperforms or TI OMAP3+powervr by far exceeds in cost and performance. This is why the system lags (though the software probably isn't optimal either from the rush job I heard it got). Tomtom would've also hooked up something decent for you on fairly robust and time tested gps systems. I can kind of understand problems in speech rec though having worked with nuance's speech rec in robotics, I can only imagine memory related issues are only more fun in a windows embedded environment, for which I hear windows automotive but interpret as windows ce6. 46 freaking megabytes of ram doesn't help either. Actually it's something like 460 megabytes, I was off by a factor of 10. This actually makes the software even worse, however.

Why such a crap system, ford? Last time I checked it was 2012, not 2002, right? Why didn't you work up some carrier grade linux system built for reliability & deterministic performance. Or how about QNX / greenhills? The military / aerospace knows not to fool around on crap like windows automotive, but in something I use everyday capable of killing me and others? I don't see why such reliable systems are not the basis of your automobiles' human interface computer. Bullshit here causes distractions and car accidents. Liability warnings can be understood in the world we live in but it is downright evil to ship such a risky system and stupid considering its probably not all that cost effective.

So here's how you make a successful auto-system you can keep building on and update so you can keep people enjoying your system while also getting lockin and presenting a real and competitive selling point. By the way, you may have noticed that window's lockin for desktop means nothing to an ipad or smartphones as this is not the look/feel environment or user experience these devices provide so there's no outright reason to consider it as a platform for the car's interface system.

Make it capable of real featurewise updates that smartphone users would be capable of updating themselves, not just bug fixes. Ship it with decent competitive hardware so featurewise updates and software remain tangable. As a particular focus, keep tight integration going with the mobile world.

Keep feature "backports" limited to probably 3 and maybe up to 5 years of vehicles (cost limiting, not everything is feasible for backport, esp if resource heavy or 3d).

Get an inhouse talented software development team and keep them. A team of 5-10 is probably good enough. Consider that this system is an asset to your company and a selling point. Redirecting to outside sources for upgrades will only hurt user experience and consistency/compatibilities in the long term.

Keep the hardware similar and compatible on 3-5 year intervals, keep an eye towards ARM architecture for it's cost competitive nature so you can ride the tablet/smartphone technology wave. Don't disregard improvements to the tecnology - keep the platforms reasonably up to date so newer vehicles have more software capability.

Build your own linux variant (if done, please keep it relatively in-sync with it's FOSS dependancies), maybe with the help of montavista or some such experienced entity.

Consider some existing systems as leveragable work: intel/nokia's meego would form a good basis of stable quality software.

You may consider android though you will need to do something about the garbage collection nature with regards to real time / deterministic operation but you get to ride the "app" wave of android for free. Perhaps your core apps would not be dalvik vm apps with some other real time touches to get things working well. Beware of the flux of this route though. It's got a selling point of android/apps but it may have more technical hurdles, especially if you constrain your developers to android app programming.

You may also consolidate more cabin systems into such a computer, further reducing some costs and using some proprietary but internal standard for inter-device communication to allow for more of a one size fits all hardware platform. Do make it future proof however.

xfx radeonhd 6950 / 6970 on linux

Well, my 6870 seemed to have a little bit of problems with red dots and spazing out. Was reminiscent of old laptops heating up or some other problems with ati graphics cards I've had in the past. So I put in for a return on the card and after checking out the specs decided to get a 6950 instead. It was awesome that you can flash the vbios of the 6950 to be a 6970 and that the cards are physically the same. This really is awesome. So I saved about 100 bucks (after the extra 60 to upgrade from the 6870 to the 6950) , loaded up freedos w/ unetbootin and my spare usb flash drive, and used atiflash to save and write the new 6970 bios. Again XFX on account of a nice warranty package.

And it works, perfectly fine. It's much snappier overall with many windows so either that extra graphics ram or those 90 something texture units make a lot of difference.

So as below, I loaded up catalyst 10.12, ran the nm/sed script to remove the unsupported hardware logo (AMD remove that crap sooner and make aticonfig more robust! Official drivers would also be nice!) . I tested alot of the OpenCL programs in the ati stream sdk too and saw impressive numbers.

I like your hardware and am a pure linux user... AMD... please keep improving things (esp on flickery video on compositing - I understand this probably isn't entirely your fault! Oh btw, 4 xv ports, hah - scale up like nvidia, this card in reality supports much more than 4). It'd be more encouraging if I didn't have to run x to run opencl on the gpu too :-). FOSS integration with these same technologies would also be great on your products. kthxbye

just cmake it

CMake is awesome, sure it is a little weird at times but vs all competition it's wiping the floor in maintaining complex builds over many platforms. The code can be really clean in comparison with some things I've seen in the past too. Well anyway, here's a few build scripts I'd like to contribute:

• A 32bit software on a 64bit linux install configuration handler This, by default, automatically detects if the build environment is not 32 bit and adjusts accordingly when included. The automatic behavior is disableable by setting NO_AUTO_X86_32 before include.
  

  macro(setx86_32)
  set(CMAKE_C_FLAGS "-m32 ${CMAKE_C_FLAGS}")
  set(CMAKE_CXX_FLAGS  "-m32 ${CMAKE_CXX_FLAGS}")
  set(LINK_FLAGS "-m32 ${LINK_FLAGS}")
  set_property(GLOBAL PROPERTY FIND_LIBRARY_USE_LIB64_PATHS FALSE)
  set(QT_LIBRARY_DIR /usr/lib)
  endmacro()
  
  if(NOT NO_AUTO_X86_32 AND NOT CMAKE_SIZEOF_VOID_P EQUAL 4)
  setx86_32()
  endif()
  

  
  

• An XPlane plugin cmake file . Include and use pretty much like add_library / install_target.
  

  set(XPLANE_LINKER_FLAGS "-nodefaultlibs")
  if(NOT DEFINED XPLANE_SDK)
   if(UNIX)
       set(XPLANE_SDK "/opt/xplane sdk")
   endif()
  endif(NOT DEFINED XPLANE_SDK)
  
  if(NOT DEFINED XPLANE_PATH)
   if(UNIX)
       set(XPLANE_PATH "/opt/X-Plane 9/")
   endif()
  endif(NOT DEFINED XPLANE_PATH)
  
  foreach(X Widgets Wrappers XPLM)
   include_directories(${XPLANE_SDK}/CHeaders/${X})
  endforeach()
  
  if(UNIX)
   add_definitions(-DLIN=1)
  endif()
  
  add_definitions(-DXPLM200=1)
  
  macro(add_xplane_plugin TARGET_NAME SRCS)
   add_library(${TARGET_NAME} SHARED ${SRCS})
   set_target_properties(${TARGET_NAME} PROPERTIES LINK_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} ${XPLANE_LINKER_FLAGS}")
   set_target_properties(${TARGET_NAME} PROPERTIES PREFIX "")
   set_target_properties(${TARGET_NAME} PROPERTIES SUFFIX ".xpl")
   if(UNIX)
       set_target_properties(${TARGET_NAME} PROPERTIES OUTPUT_NAME "lin")
   elseif(WINDOWS)
       set_target_properties(${TARGET_NAME} PROPERTIES OUTPUT_NAME "win")
   endif()
  endmacro()
  
  macro(install_xplane_plugin TARGET_NAME PLUGIN_NAME)
   install(TARGETS ${TARGET_NAME} LIBRARY DESTINATION "${XPLANE_PATH}/Resources/plugins/${PLUGIN_NAME}")
  endmacro()
  

  
  

XFX ATI 6870 on linux

I got one and its not too bad but not without quirks. I'm using fglrx 10.11 and while there is nothing in the xorg logs that seems to be wrong, only amdcccle works for configuration. That is, aticonfig refuses to work saying:

aticonfig: No supported adapters detected

Most stuff works fine but it can be oddly slow with moving windows or, with compositing enabled, flickery in video (opengl and xv). Gtk doesn't seem to play too well with it either as updates don't occur correctly all the time within such windows. Still, things are working better than my old eVGA 8800 GTS. Oh, btw - what made me choose XFX over eVGA? Warranty.

What's funny though is amdcccle works and aticonfig doesn't and I get that unsupported hardware overlay (which, thanks to some hackery with objdump and sed, has been removed). I'm happy that on an unsupported driver stuff works as much and stably as it does but I think they should've delivered a bit more support given its already been a few months since these cards were released. Anyway, I voted with my wallet and I hope ati will be a better ride in this day and age than nvidia. I'm all for the reduced heat and power usage of the Radeon HD cards. I'm also all for the opensource drivers so I don't have to deal with so many of the quirks and instabilities that proprietary drivers offer. I also hate having to update something out-of-band with linux distros and only partially supported. For such an expensive part of the desktop, we sure do have to deal with alot of bullcrap. Such things aren't put up with on CPUs which has always puzzled me.

OpenCL here I come!

Qt sucks at option parsing

Qt doesn't handle parsing program options (ie --foo=1) very well. But don't worry, boost program options can save the day! The problem is how to hook up boost::program_options to Qt's arguments after it has parsed Qt specific options.

#include <boost/program_options.hpp>
std::string qstr2str(const QString &qstr){
   return qstr.toStdString();
}

int main(int argc, char *argv[]){
QApplication application(argc, argv);
namespace po = boost::program_options;
po::options_description clspec("Options");
clspec.add_options()
("help", "see available options")
("arg1,a", po::value(&arg1)->default_value("banana"), "the arg1 option")
("arg2,b", po::value(&arg2)->default_value(false), "the arg2 option")
;

QStringList qargs = application.arguments();
std::vector args(qargs.size());
std::transform(qargs.begin(), qargs.end(), args.begin(), qstr2str);

po::variables_map vmap;
po::basic_command_line_parser parser(args);
parser.options(clspec).style(0).extra_parser(po::ext_parser());
po::store(parser.run(), vmap);
po::notify(vmap);
}

And with that you have parsed the options from Qt after it's had it's way with them