Expandable/Collapsable ListView items in WinRT Xaml

This post contains a lot of references to specific behaviors from SnooStream but the general principle involved is the same now matter where you're doing it. You can see the all of the source here https://github.com/hippiehunter/Baconography/tree/appx or you can see the specific xaml in question for the email conversation like view here https://github.com/hippiehunter/Baconography/blob/appx/SnooStream/SnooStream.Shared/View/Controls/SelfActivityView.xaml.

Expander controls can be used to show a preview of a control that when clicked on expands out to show a potentially large number of more specific items. A particular location where this is used is the built in email client. A conversation with multiple items will show as a single item in the list until you click on the preview item, expanding out to show all of the items in the conversation. Unlike Windows Phone 8.0 there aren't any readily available controls that can be directly used as an Expander. Even if there was one available, my experience in the past with the ExpanderControl from the WindowsPhone toolkit is that it doesn't do well when placed inside a ListView. The performance is quite bad because changing the size of a visible ListItem causes a pretty complete reevaluation of the layout for all of the visible ListItems. Quite frequently even if you can live with the performance issues, on collapse of the Expander, you will just see a big blank space leftover. Now thatwe know what ListView doesnt like us doing, what can we do differently to achieve the same effect? Well, virtualized controls like ListView love to add and remove items, and they even have a user controllable add/delete animation. So the solution we're using in SnooStream is to remove items from the list rather than collapse them, and add new items into the list rather than expand from one ListItem. So far it has delivered all of the behaviors we're looking for without any performance issues. You can see an example of this in action in two places, the SelfActivityView and the CommentsView. SelfActivityView uses it in very much the same style as the built in email client as that is the user interaction we're trying to mimic. CommentsView is considerably more complicated because it starts out fully expanded and has the potential for very deep nesting. It also has to deal with items being added on the fly when users “LoadMore” or load context from a comment set that was only partially loaded.

Really the most important idea to take away for use in other source bases, is your ViewModel needs to understand how to add and remove the children from the ObservableCollection that is bound to your ListView, separately to its own internal representation of the items that it wants to present. So When a user expands on of the group header items, the ViewModel can add all of the groups child items into the actual bound ObservableCollection, and remove them again when the user no longer wants that group header to be expanded. The meat of this flattening can be seen here https://github.com/hippiehunter/Baconography/blob/appx/SnooStreamCore/ViewModel/SelfStreamViewModel.cs.

Working with WinRT DateTime in C++

DateTime in managed languages has a very comprehensive set of methods that make working with it very easy, C++/CX not so much. Some answers on stackoverflow suggest using ATL and all kinds of other very complicated solutions. Here is my entry into this fight, if I can convert DateTime -> std::chrono::seconds and std::chrono::seconds -> DateTime then I can use the entirely reasonable set of functions that surrounds std::chrono.

DateTime toFileTime(std::chrono::seconds sec)
{
    long long unixTime = sec.count();
    DateTime result;
    result.UniversalTime = unixTime * 10000000ULL + 116444736000000000LL;
    return result;
}

std::chrono::seconds toDuration(DateTime dt)
{
    return std::chrono::seconds((dt.UniversalTime - 116444736000000000LL) / 10000000ULL);
}

Now this is assuming that seconds is a duration from Unix epoch, you can easily get the current distance from epoch using time(nullptr) or std::chrono::system_clock::now().time_since_epoch().count()

Animated navigable ultra high resolution images with Direct2D and the Nokia Image SDK

This post is a mix of rationale, history and a sprinkling of technical details. Here is the Github link

The original problem that caused me to write this library was I needed to be able to display large GIF images on windows phone 8.0 and windows store 8.0. Neither platform has native support for GIFs but both platforms are supported by ImageTools. ImageTools was created to support Silverlight and its written entirely in C#. There are a few problems with ImageTools that became apparent rather quickly. First it was only able to load about 90% of the GIFs that we saw on /r/gifs. Next up it is a CPU hog, it pegs the CPU anytime a GIF is visible. When you load a GIF using ImageTools you must wait until its fully loaded from the network before you can start displaying anything. Finally ImageTools has a voracious appetite for memory because it fully loads everything prior to display and keeps all frames fully in memory.

The first version of GifRenderer for Windows Store solved two of those problems, CPU hog and comparability. The first decision was easy, it was going to need to be implemented in C++ and fortunately both windows phone 8 and windows store 8 had excellent interop between C++ and .Net. I decided to use Windows Imaging Component (WIC) because it has easily the best support for GIFs available anywhere. Difficulties involved in creating the first version were mostly centered around learning Direct3D/XAML interop and frame timing. It turns out that the specified delay(in units of 1/100th of a second) in many GIFs cannot be actually followed. GifRenderer defaults to 10 in case the extension block is not present, is corrupted or is less than 2. The reason for this default is simple, there are lots of bad GIF encoders that have existed throughout time and this particular method of defaulting goes all the way back to netscape navigator, so a surprising amount of gifs still rely on this behavior. XAML interop was achieved using SurfaceImageSource.

Once we started to work on the Windows Phone 8 version of Baconography I realized two components GifRenderer was relying on were not available. WIC was unsupported and so was Direct2D. This meant everything would need to be rewritten to support the platform. Looking around for an alternative to WIC I found GIFLIB. Because of its extensive use in other open source software for both encoding and decoding it has extremely high levels of compatibility with GIFs found out in the wild. GIFLIB also compiled fine on Windows Store, so I knew if things were ever unified between Windows Phone and Store, it would be easy to bring the Windows Phone implementation in to support both platforms. XAML interop was achieved using DrawingSurface. It was considerably easier to implement than SurfaceImageSource because it handled syncing with the XAML render rate.

Need arose for significant changes when an incredible set of GIFs showed up on reddit, titled NYC etiquette. Most GIFs do not actually take advantage of the compression mechanisms that are present in the format, so you usually see full image frames that take quite a bit of space for any reasonably long GIF. The NYC etiquette GIFs were completely different, they had taken advantage of every last trick in the GIF playbook from sliding update windows to very limited but dynamic color palet selection. This lead to a 700k GIF that when fully decoded occupied several hundred megabytes of memory. The correct path forward was readily apparent, frames would need to be deinterlaced/decoded but not actually constructed into full frames until the moment they were to be displayed. This dropped memory consumption for the NYC etiquette GIF set below 15 megabytes and aside from some difficulty in implementing frame disposal methods in an efficient manner, this version would stand for quite a while.

Once we started implementing SnooStream as a Windows Universal App, I needed to implement GifRenderer as a Universal App compatible library. VirtualSurfaceImageSource was the XAML interop method of choice, because it offered the flexability of SurfaceImageSource but as long as you invalidate your frame area, they handle synching rendering to the XAML render rate. While working on GifRenderer I decided it would be nice to add support for incremental GIF loading, the same way a web browser would have displayed it as the frames were loaded. GIFLIB isnt really suited to incremental loading but I tried anyway. Essentially checkpoints were put into the loader loop so that if an error was encountered, it could roll back the state of the loader and try again when more data was available. This method worked ok, but there were memory leaks somewhere that I was never able to track down. The biggest problem was that GIFLIB was originally written a long time ago and its written in very stateful C style. Memory allocation and operations of the GIF frames collection was therefore very intertwined. So I started to re-implement it using mostly modern C++ style, the actual frame load operation state was separated from the rest of the GIF, and frames could be added/removed without impacting anything other than the collection of loaded frames itself. Many times people think re-implementing something in modern C++ style is going to have an inferior memory profile to something that was written in C back when a megabyte was a lot of memory. However in this case where GifRenderer was loading all of the frames anyway, the biggest difference was memory locality. More of the underlying structures were implemented to not live on the heap and the allocation performance of the STL containers is actually quite a bit better than the prior allocation mechanism, realloc. So at this point we have a fully BSD licensed ultra high performance, low memory, incrementally animated GIF renderer that is easy to plug into anything that takes an ImageSource in Windows Universal XAML Apps.

Now that GIFs were out of the way, we still had a problem with static images that are super large. This was a little bit less of a problem on Windows Phone 8 because the max texture size is 2048x2048 so there was an absolute cap at least on the texture memory being used. The actual JPEG decoding process was however not bounded so quite a few of the crashes we experienced on windows phone in baconography were caused by JPEG related OOM. So, moving to Windows Universal where the maximum texture size is something like 16k x 16k and remembering that we're already doing most of the hard work to display images ourselves, it wasn't much of a leap to say, lets make our own ImageControl that loads just the right amount of a JPEG for the current zoom level and why not take advantage of the RAJPEG technology Nokia has given us all access to through the Nokia Image SDK. That last part is really what makes all of this possible, normally if you ask WIC to load a JPEG for you or even load a BitmapImageSource, you are going to have to pay the memory budget for loading the entire image even if you only want 1/100 of the pixels in the image. RAJPEG makes it possible to crop, and re-size with a tiny fraction of the memory otherwise required. Creating a zoomable image control is by no means a new idea. The winner of the Nokia Imaging Wiki Competition 2013Q3. did exactly that, but it was for Windows Phone 8 and it had quite a few rough edges in terms of API usage and its memory profile while much better than the default one, was still less than ideal.

So, at this point we know we need a new Nokia Image SDK powered zoom-able image control, and we also know that Windows 8.1 and by extension Windows Universal Apps added support for super large virtualized surfaces using VirtualSurfaceImageSource.But using this technique brings more to the table than just offering a convenient way to interface with a ScrollViewer. Because its fundamentally implemented using the same DirectManipulation Techniques that a ScrollViewer uses, you get amazing compositor performance and no nasty tearing from areas you've not gotten around to on the render thread. This article is already a bit too long so the most important takeaway from this is DirectManipulation is the amazing technology that makes touch interaction smooth on WindowsPhone 8.1.

In my first try at this, I tried just creating the VirtualSurfaceImageSource at full size then only created textures large enough to render the currently visible portion of the image (with scaling to a max image size for full zoom-out). This fell pretty flat on its face. After a lot of testing is became apparent that despite what Windows.System.MemoryManager.AppMemoryUsage and the memory profiler were telling me, VirtualSurfaceImageSource does not take kindly to actually rendering ultra large surfaces, it really wants you to only render a small portion at a time. The symptoms on this were pretty unpleasant, rather than just being a normally reported OOM, the process would just get terminated without any message whatsoever. Figuring out that this was just a simple OOM on the compositor thread took several very long nights and a lot of cursing. So with my first attempt burning in a flaming wreckage, I decided to read as much as I could about VirtualSurfaceImageSource as I could. It turns out there really isn't that much, Microsoft has a sample for making a magazine reader but that doesn't really deal with changing the detail of an image as the user zooms in and out. So I kept digging, and found out ScrollViewer was implemented using DirectManipulation, then I even found a sample of using DirectManipulation to make an image viewer, It didn't take long to realize that actually using DirectManipulation directly in your app was not allowed for Windows Universal Apps. But some of the technique still stuck, and I started looking around at the VirtualSurfaceImageSource documentation again. This time I found what I needed, in a presentation at build 2013 it was mentioned that VirtualSurfaceImageSource is very good at handling rapid changes in its surface size. So I had a path set out for me. If I could get the DirectManipulation events from the ScrollViewer then I would be able to change the VirtualSurfaceImageSource size when the user does a zoom operation. At that point it was pretty simple to re-implement the crop/re-size logic I had from the first try to respond to system provided invalidation events. Now, all bundled up into one place we have a memory friendly image control that works with super large images and GIFs at the same time. Its all licensed under a 3 clause BSD license so you're free to use it in any app, open source or not.

Playing HoN with Xorg Edgers and open source ATI drivers

Let start off with my setup, I'm running x64 Ubuntu 12.04 with Xorg Edgers. Ive got a Radeon 5850, and  a Phenom II X6 1100T.

If you just try and run the installed HoN using this setup you will see the screen go to funny colored blocks and pop back having failed to run. If you run it from the command prompt you can see this message  "LibGL error: failed to load driver: r600".  So set LIBGL_DEBUG=verbose and try again.

warning: The VAD has been replaced by a hack pending a complete rewrite

libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/tls/r600_dri.so

libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/r600_dri.so

libGL error: dlopen /usr/lib/x86_64-linux-gnu/dri/r600_dri.so failed (/home/hippiehunter/HoN/libs-x86_64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/lib/x86_64-linux-gnu/dri/r600_dri.so))

libGL error: unable to load driver: r600_dri.so

libGL error: driver pointer missing

libGL error: failed to load driver: r600

libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/tls/swrast_dri.so

libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so

libGL error: dlopen /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so failed (/home/hippiehunter/HoN/libs-x86_64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so))

libGL error: unable to load driver: swrast_dri.so

libGL error: failed to load driver: swrast

K2 - Fatal Error: ARB_vertex_buffer_object not available.

 alright, that gives us something to work with, it looks like the C++ runtime libraries that were linked into Xorg Edgers are a higher version then the ones shipped with HoN. Easy fix go delete ~/HoN/libs-x86_64/libstdc++.so.6. Problem solved, HoN now runs at an acceptable framerate and doesnt seem to have any graphical glitches or crashes.

Using C++11 lambdas with boost::spirit::qi semantic actions

here is a micro example

 typedef rule<Iterator, Label*(), space_type> label_rule_type;  
 label = lit(':') > symbol[[&](string& name, typename label_rule_type::context_type& context)   
          {   
           boost::fusion::at_c<0>(context.attributes) = _ast->addLabel(name);  
          }];  

the most important part here is the typedef and its context_type. If you just want to use C++11 lambdas to do very simple things with your passed attribute things are very easy, but if you want access to the locals or the qi::_val, you'll have to use the context parameter. context is a very templated instance of boost::spirit::context that gives you access to two boost::fusion sequences; attributes and locals.

Vector arg to map conj must be a pair

One of my least favorite things about working in clojure is the error messages. Often you will end up with an error that makes perfect sense if you're looking at the problem but is nearly useless when trying to find it. Here is one example that I ran into today and wasn't able to find a direct solution by searching the error message it manifestsed itself to me in leiningen but its really a clojure error message. Sometimes when using leiningen you'll run into "Vector arg to map conj must be a pair". Its caused when you try to pass more then 2 elements as a vector instead of as a map.

Example in code -
(conj {} [:something "something" :dark "side"]) -- Fails
(conj {} [:something "something"]) -- Succeeds

The second conj succeeds and reveals why this can be masked, passing a vector with 2 elements into the conj of a map will succeed.

Example when using lein-js

(defproject myexample "1.0.0-SNAPSHOT"

  :js { :prod-options [:process-closure-primitives true

                                 :compilation-level :advanced-optimizations]})

should be

(defproject myexample "1.0.0-SNAPSHOT"

  :js { :prod-options {:process-closure-primitives true

                                 :compilation-level :advanced-optimizations}})