Quantcast
Channel: Lucian's VBlog
Viewing all 79 articles
Browse latest View live

How to call UWP APIs from a desktop VB/C# app

$
0
0

Latest update:

I've put this functionality into a NuGet package. To use UWP APIs from your >=.NET45 desktop app, simply add a NuGet reference:

 

I've also moved the technical information from this blog over to the package's GitHub project site:

 

Out of date: first steps

Add these references to your project, by right-clicking on your project in Solution Explorer and doing References > AddReference > Browse. This minimum needed to get going.

  1. c:\Program Files (x86)\Windows Kits\10\References\Windows.Foundation.FoundationContract\1.0.0.0\Windows.Foundation.FoundationContract.winmd
  2. c:\Program Files (x86)\Windows Kits\10\References\Windows.Foundation.UniversalApiContract\1.0.0.0\Windows.Foundation.UniversalApiContract.winmd
  3. c:\Program Files (x86)\Windows Kits\10\UnionMetadata\Facade\Windows.WinMD
  4. c:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5\System.Runtime.WindowsRuntime.dll

The complete set of Desktop WinRT APIs

Here is a MSDN list of WinRT APIs that can be called from desktop apps (but I believe this list is incomplete).

 

Just the four references is enough to get you 90% of the WinRT APIs you'll need in practice.

But to go the full mile, I think the best way would be to package it up as a NuGet package according to the instructions below. I'll explore that idea later. I'd want to get the other UWP facades, get something that works with newer UWP SDKs than just 10240, also include APIs from the Desktop Extension SDK, include the remaining .NET interop DLLs, and get all the default namespace-imports.

Until I or someone else produces that NuGet package, here below is a detailed explanation:

Explanation

To find the UWP WinRT APIs themselves, look for example at this file:

C:\Program Files (x86)\Windows Kits\10\Platforms\UAP\10.0.10240.0\Platform.xml

This Platform.xml file conveys which WinRT APIs are available in "version 10240" of the UWP SDK. (This was the version that came out at VS2015 RTM. Newer versions will come out periodically). The Platform.xml file works by being an index into a set of other winmd files. It looks like this:

<ApiContract name="Windows.Foundation.FoundationContract" version="1.0.0.0" />
<ApiContract name="Windows.Foundation.UniversalApiContract" version="1.0.0.0" />
<ApiContract name="Windows.Networking.Connectivity.WwanContract" version="1.0.0.0" />

Each line denote a .winmd file in the "C:\Program Files (x86)\Windows Kits\10\References directory". That's how I picked up exact filenames and pathnames to add as references. (I should really have added a reference to the WwanContract.winmd file as well, but couldn't be bothered)

These three winmd files make up the WinRT APIs available across all Win10 devices. You might additionally want to use the additional WinRT APIs that are part of Desktop. Look at this file:

C:\Program Files (x86)\Windows Kits\10\Extension SDKs\WindowsDesktop\10.0.10240.0\SDKManifest.xml

It has a load more <ApiContract> directives, pointing to additional .winmd files you should reference from the "C:\Program Files (x86)\Windows Kits\10\References" directory. I haven't bothered to do that.

 

To interop with DLLs or WINMDs build for Win8.1 or WindowsPhone8.1, the issue is that those old DLLs and WinMDs used to think that all WinRT types were in a file called "windows.winmd". But they're not anymore! Instead they're in files called Windows.Foundation.FoundationContract.winmd and the like. The solution is a façade winmd, which is basically a load of type-forwarders from windows.winmd to the new winmd files.

c:\Program Files (x86)\Windows Kits\10\UnionMetadata\Facade\Windows.WinMD

 

To use "await" on WinRT types, this and other interop between .NET and WinRT are provided by the following DLL. Note that it's an 8.1-era DLL, and therefore only works in conjunction with "facade\windows.winmd" above. (There are several other interop assemblies but they're not so crucial).

c:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5\System.Runtime.WindowsRuntime.dll

If you fail to add both these two assemblies, then you'll get error messages like this:

  • BC36930: 'Await' requires that the type 'IAsyncAction' have a suitable GetAwaiter method
  • CS0012: The type 'IAsyncAction' is defined in an assembly that is not referenced. You must add a reference to assembly 'Windows, Version=255.255.255.0, Culture=neutral, PublicKeyToken=null, ContentType=WindowsRuntime'.
  • CS4036: 'IAsyncAction' does not contain a definition for 'GetAwaiter' and no extension method 'GetAwaiter' accept a first argument of type 'IAsyncAction' could be found (are you missing a using directive for 'System'?)

 

Example code (VB)

Here's an example console app in VB which uses UWP WinRT APIs:

Imports System.IO

Module Module1
    Sub Main()
        MainAsync().GetAwaiter().GetResult()
    End Sub

    Async Function MainAsync() As Task
        ' basics
        Dim folder = Windows.Storage.KnownFolders.DocumentsLibrary
        Dim opts As New Windows.Storage.Search.QueryOptions(
                              Windows.Storage.Search.CommonFileQuery.OrderByName, {".txt"})
        Dim files = Await folder.CreateFileQueryWithOptions(opts).GetFilesAsync(0, 20)
        For Each file In files
            Console.WriteLine(IO.Path.GetFileName(file.Path))
        Next

        ' streams
        Using reader = New IO.StreamReader(Await files.First.OpenStreamForReadAsync())
            Dim txt = Await reader.ReadToEndAsync()
            Console.WriteLine(txt.Substring(0, 100))
        End Using

        ' pictures
        Dim pics = Await Windows.Storage.KnownFolders.PicturesLibrary.GetFilesAsync(
                         Windows.Storage.Search.CommonFileQuery.OrderBySearchRank, 0, 1)
        Using stream = Await pics.First.OpenReadAsync()
            Dim decoder = Await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(stream)
            Console.WriteLine(decoder.OrientedPixelWidth & "x" & decoder.OrientedPixelHeight)
        End Using

        ' httpclient
        Dim client As New Net.Http.HttpClient()
        Dim html = Await client.GetStringAsync("http://www.microsoft.com")
        Console.WriteLine(html.Substring(0, 100))

    End Function
End Module

 

 

Example code (C#)

Here's an example console app in C# which uses UWP WinRT APIs:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;

class Program
{
    static void Main()
    {
        MainAsync().GetAwaiter().GetResult();
    }

    static async Task MainAsync()
    {
        // basics
        var folder = Windows.Storage.KnownFolders.DocumentsLibrary;
        var opts = new Windows.Storage.Search.QueryOptions(Windows.Storage.Search.CommonFileQuery.OrderByName, new[] { ".txt" });
        var files = await folder.CreateFileQueryWithOptions(opts).GetFilesAsync(0, 20);
        foreach (var file in files)
        {
            Console.WriteLine(System.IO.Path.GetFileName(file.Path));
        }

        // streams
        using (var reader = new System.IO.StreamReader(await files.First().OpenStreamForReadAsync()))
        {
            var txt = await reader.ReadToEndAsync();
            Console.WriteLine(txt.Substring(0, 100));
        }

        // pictures
        var pics = await Windows.Storage.KnownFolders.PicturesLibrary.GetFilesAsync(
                               Windows.Storage.Search.CommonFileQuery.OrderBySearchRank, 0, 1);
        using (var stream = await pics.First().OpenReadAsync())
        {
            var decoder = await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(stream);
            Console.WriteLine($"{decoder.OrientedPixelWidth} x  {decoder.OrientedPixelHeight}");
        }

        // httpclient
        var client = new System.Net.Http.HttpClient();
        var html = await client.GetStringAsync("http://www.microsoft.com");
        Console.WriteLine(html.Substring(0, 100));
    }
}

 


OpenCV: unsure how to proceed

$
0
0

OpenCV http://opencv.org/ is the standard open-source library that everyone uses for image detection. It's written in C++.

I want to use it from my .NET UWP apps, but I'm struggling to figure out how. If anyone has solid guidance on how to proceed, I'd love to know! This blog post is just a documentation of my first tentative investigations. (Also eventually I want to use OpenCV from my cross-platform .NET Core apps as well, so I can write an ASP.NET5 webservice running on Linux that does face detection).

I think there should ideally be three layers of NuGet packages:

  1. OpenCV.dlls - there should just be a base NuGet package whose only job is to carry the native C++ dlls. It needs to carry the three flavors, x86 + x64 + ARM. It should use the new features in NuGet3 so it can carry all three flavors of dll in a single NuGet package, and at build-time the project.json should pick out the right binaries.
  • There's a catch. The normal "opencv_world300.dll" is huge at 30mb and includes absolutely all OpenCV functionality. Typically my store apps will only need a small subset of that functionality. I see that the "OpenCV.NET" NuGet package instead delivers it split up into 19 separate DLLs, each of them about 1-2mb. It would be best if you could get the DLLs also through a load of smaller NuGet packages, one by one, to pick out which of those binaries you need.
  • Note that a C++ DLL that you built for an earlier version of Windows Store or Windows Phone will not work in UWP. That's because even though UWP supports all the same APIs that were there in the earlier WinRT platforms, it divides them up into different "api-sets", so the binaries are incompatible.
  • Ideally this NuGet package should also carry the native C++ dlls for Linux and for OSX, so it can be used on other .NET Core platforms. I don't know how native dlls are carried on those platforms
     
  • OpenCV.interop - there should be an intermediate NuGet package whose job is solely to contain p/invoke declarations for those DLLs without any fancy wrapping.
    • Maybe it would also include header files, for consumption by C++
    • I don't know how p/invoke declarations work for cross-platform .NET Core.
    • Why decouple this from the raw DLLs? solely to enable the "pick-and-choose" idea above. Maybe it's not worth it. Maybe the "OpenCV.dlls" and "OpenCV.interop" should be combined.
       
  • OpenCV.Net - there should be a higher-level set of wrapper classes, more .NET friendly.
    • Maybe this package need never be written, if everyone is happy using the unwrapped versions of the APIs.
    • Maybe there will be multiple competing versions of this package, with different API designs, some more friendly than others.

     

    OpenCV itself 

    http://opencv.org/ - the main website has a "download binaries for Windows" link. This includes x64+x86 builds of "opencv_world300.dll" (32mb). I don't know if they're built for UWP but I suspect not. It has two 8.1-era Phone+Store sample apps but they didn't build out of the box. It has a directory called "sources\platforms\winrt" which has instructions for building on WinRT, but these instructions don't include x64, and I suspect they date from before UWP.

    https://github.com/Itseez/opencv - the main GitHub repository for OpenCV

    https://msopentech.com/blog/2015/03/06/opencv-enabled-for-windows-runtime-family/ - has pre-UWP instructions on how to build binaries from the main GitHub repository using CMake

    https://www.nuget.org/packages/OpenCV/ - this NuGet package with 18k downloads claims to contain "binaries and build targets". This isn't true. It contains C++ header files, and it contains a .targets file which (based on build parameters) pick up the OpenCV native DLLs that it assumes you got from elsewhere and copied into a particular location. It only does this for x86 and x64; not ARM.

    There are two old sites - https://github.com/Microsoft/opencv was the old staging repo where WinRT support was first added to the main OpenCV GitHub, and http://code.opencv.org/projects/opencv/wiki/WindowsRT had old instructions on how to build for WinRT. I think both links are out of date.

    Analysis: I suspect there are no "ready-to-go" binaries here for UWP, nor any useful NuGet packages. If I want to use OpenCV from source, I'll have to learn the thing's build system, configure it to build UWP, and built it myself. I'll have to write a OpenCV.dlls NuGet package from scratch myself. I don't know whether the work I do here can be submitted back into OpenCV's daily builds, or if I'll have to keep it separate.

     

    OpenCV Extension SDK 

    http://channel9.msdn.com/Events/Build/2015/3-82 - this video discusses an "OpenCV for Windows Store" Extension SDK which contains native binaries for UWP apps. However, the Extension SDK has never been publically released. (Also, ExtensionSDKs were used in the past because prior versions of NuGet weren't good at selecting x64/x86/ARM binaries based on build-time configuration. That limitation has gone away, and now NuGet is a much better option than ExtensionSDK).

    Analysis: The author of the video has moved on to a different team. I've contacted his former colleagues to see if I can get anywhere with this. I don't know if it will be a single huge 30mb "world" dll, or a series of smaller ones.

     

    OpenCVSharp

    This NuGet package contains a 30mb "world" binary for x86+x64 (not ARM) that come from here. It also contains C# wrappers that are as close as possible to the raw C API. This is the most widely used package, with 32k downloads total.

    I'm confused, because the NuGet package has that huge DLL, but then the project's install.ps1 seems to have them split up into separate pieces.

    The NuGet package only installs onto desktop .NET Framework 4.0 and higher -- not .NETCore, and not UWP.

    As for the raw native binaries themselves, I don't know. But given the desktop focus, and the lack of ARM, I strongly suspect they won't work on UWP.

    https://github.com/shimat/opencvsharp - this is the main repository.

    The OpenCVSharp NuGet is actually offered in four flavors:

    1. https://www.nuget.org/packages/OpenCvSharp3-AnyCPU/ - the main one for OpenCV3 which was released in 2015; most recent
    2. https://www.nuget.org/packages/OpenCvSharp3-WithoutDll/ - as above, but missing the "world.dll" (so you'll pick it up yourself elsewhere)
    3. https://www.nuget.org/packages/OpenCvSharp-AnyCPU/ - like [1] but for OpenCV 2.4
    4. https://www.nuget.org/packages/OpenCvSharp-WithoutDll/ - like [2] but for OpenCV 2.4

    There's also this NuGet package https://www.nuget.org/packages/OpenCVSharp/ but it doesn't seem to be related.

    Analysis: this project seems to provide the correct "interop p/invoke" layer. But I think it won't be a good source of binaries for UWP.

     

    OpenCV.NET

    This NuGet package has a nicer .NET-style set of wrapper classes. I don't understand how/where the NuGet package picks up its raw binaries from, but it appears to get them somehow! The binaries seem to be x86 only (not x64, not ARM) and are split into ~15 smaller DLLs. The package has been downloaded 8k times. The most recent update was 9 months ago.

    The NuGet package only works on desktop .NET Framework 4.0 and higher (not UWP).

    https://www.nuget.org/packages/OpenCV.Net/

    Analysis: I think it'd be ideal to split out the "interop p/inoke" layer from the wrapper classes. I'm interested to know whether the interop declarations for "split-into-separate-dlls" that this project uses is different from the interop declarations for "world.dll". In any case, this project isn't a suitable source for raw binaries.

     

    Emgu

    This project does provide ARM and WinRT for 8.1-era. However I haven't read anything to indicate support for UWP. Its NuGet package was last updated 20 months ago and has 6k downloads.

    This project requires a commercial license for ARM and WinRT.

    http://www.emgu.com/wiki/index.php/Main_Page

    https://www.nuget.org/packages/VVVV.EmguCV/

    Analysis: this isn't in the OSS spirit that I like about .NET Core, so I haven't pursued it.

     
     

    Stack Overflow 

    http://stackoverflow.com/questions/14065370/using-opencv-in-a-windows-store-app-with-visual-c - this seems to be the most valuable StackOverflow resource. It has Raman Sharma (author of the "OpenCV for Windows Store Apps" ExtensionSDK) give some answers on how he built OpenCV for WinRT.

     

    OpenCV: building UWP binaries

    $
    0
    0

    [This blog article documents my continuing mission to figure out how to use OpenCV in UWP apps...]

     

    How to build OpenCV native dlls for UWP 

    OpenCV is on GitHub. There are three relevant forks/branches, as of writing on November 2015:

       [1] https://github.com/itseez/opencv

         +- [2] fork https://github.com/Microsoft/opencv

              +- [3] branch https://github.com/Microsoft/opencv/tree/vs2015-samples

    Repository [1] is the master. In it, the native OpenCV binaries are built for all different platforms using CMake. As far as I can tell, this repository only has support via CMake for building Windows Store and Windows Phone 8.0 and 8.1 apps. It doesn't seem to have UWP. And it also requires you to install some particular old versions of VS, as per its readme.txt.

    Fork [2] doesn't seem to bring anything new.

    Branch [3] contains Visual Studio .sln files from which you can build UWP versions of all the native OpenCV binaries: x86, x64 and ARM, Debug and Release. The Release builds actually were a bit wonky, but I submitted PR #69 to make them build right. So: if you clone this repository, select this branch, set an environment variable, and open up vs2015/WS/10.0/ARM/OpenCV.sln, or x86/OpenCV.sln, or x64/OpenCV.sln, then you can successfully build UWP binaries for OpenCV. This will produce 15 binaries for each architecture+build combination.

     

    How to use OpenCV native dlls for UWP

    I tried out the OpenCV native dlls in a C++ UWP app, and they worked fine.

    More interestingly, the dlls all together add up to about 30mb, but I was able to use just three of them that I needed and they worked fine. This satisfies one of my central requirements, to be able to deploy just a cut-down version of OpenCV with my store app, to keep size down.

    I did File>New>C++>Windows>UWP blank app. In Solution Explorer I added three DLLs as "content" files in the root directory of my project so they would be copied into the resulting appx, adjacent to the .exe:

    opencv_core300d.dll
    opencv_igmcodecs300d.dll
    opencv_igmproc300d.dll

    In Solution Explorer I added three import-libraries. By default, when you add LIBs to a C++ project, they are taken as import libraries:

    opencv_core300d.lib
    opencv_igmcodecs300d.lib
    opencv_igmproc300d.lib

    In Project Properties I added an include directory to my C++ project options

    vs2015/WS/10.0/include

    In my mainpage C++ file, I added a bunch of #include and using statements:

    #include <robuffer.h>
    #include <opencv2\imgcodecs.hpp>
    #include <opencv2\imgproc.hpp>

    using namespace Windows::UI::Xaml::Media::Imaging;
    using namespace Windows::Storage::Streams;
    using namespace Microsoft::WRL;

    I added a content file "demo.jpg" in Solution Explorer, and in MainPage.xaml I added an Image control named "image1", and added these lines in OnNavigatedTo in my mainpage C++ file:

    cv::Mat bmp1 = cv::imread("demo.jpg");
    cv::Mat bmp2 = cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);
    cv::cvtColor(bmp1, bmp2, CV_BGR2BGRA);
    WriteableBitmap^ wbmp = ref new WriteableBitmap(bmp2.cols, bmp2.rows);
    IBuffer^ buffer = wbmp->PixelBuffer;
    unsigned char* dstPixels;
    ComPtr<IBufferByteAccess> pBufferByteAccess;
    ComPtr<IInspectable> pBuffer((IInspectable*)buffer);
    pBuffer.As(&pBufferByteAccess);
    pBufferByteAccess->Buffer(&dstPixels);
    memcpy(dstPixels, bmp2.data, bmp2.step.buf[1] * bmp2.cols * bmp2.rows);
    image1->Source = wbmp;

    Result: it all worked. The image displayed correctly.

     

    Next Steps

    I want to package each individual DLL up as a NuGet package, for 15 NuGet packages in total. Each package should include x86+x64+ARM versions of the DLLs, and corresponding import libraries. It should also include the include files pertaining to that DLL. That way, as a C++ author, I will be able to bring in only the particular DLLs that I want. To do this I'll have to figure out the dependencies between the DLLs, and the dependencies between header files, and then express these as NuGet dependencies. (I don't yet know how to create NuGet packages for C++. I'll look to "Win2d.uwp" as the canonical example. I don't know if each package should also include debug builds, but I suspect not. I don't know where PDBs should go.)

    I think that NuGet generation should be done as part of [3] branch vs2015-samples. This is the branch that has .sln files for building all the DLLs, so it's appropriate for it to also include NuGet-package-building. I don't know if this is within the vision of the owners of that branch or not. Ultimately, in the future, I don't know if the owners of Itseez/opencv would be open to their CMake system building the NuGet packages as well?

     

    Beyond that, the major issue is that OpenCV has a C++ API. This is not friendly to pinvoke interop with .NET. OpenCV used to have a plain C API which is friendly, but the C API has been deprecated and is not getting new features. I don't wish to invent a pinvoke-friendly wrapper around the OpenCV C++ API.

    Approach (1): I'd like to test the scenario of a hybrid app, where most of the app is in .NET, and it calls into a C++ winrt component that I author, and this C++ component does all the calls into OpenCV, which it obtains via NuGet.

    Approach (2): OpenCVSharp has a BSD license and already contains a set of 'extern "C"' wrappers around the C++ API. The ideal would be for OpenCVSharp to be able to consume my native UWP dlls for OpenCV. There already is a "dll-free" version of OpenCVSharp, which merely contains the pinvoke wrappers, and expects you to supply the OpenCV DLLs some other way. There are a few unknowns with this approach. I don't know if the OpenCVSharp project has different version of OpenCV (and hence different include files) as compared to the one in branch [3]. I don't know if the native extern "C" assembly that comes with OpenCVSharp will be compatible with the DLLs I build, and the fact that there are 15 of them rather than just 1 big "world" dll. I don't know if it will need to switch to "delay-load" to support the scenario where you only package say 3 of the OpenCV dlls with your store app.

     

    Notes

    I am hugely excited! It was thrilling to see the UWP versions of all the DLLs get built properly. I think I know how to package up DLLs into NuGet better than most folk, and I can do a best-in-class job. I'm excited that I was able to get something working end-to-end at least in C++.

     

     

    OpenCV: how to split it up into components

    $
    0
    0

    [This article documents my ongoing attempts to get OpenCV working in UWP apps]

     

    My goal is to figure out how to package OpenCV up into several individual NuGet packages. For instance, I expect that these files would all be packaged together into a single "OpenCV.native.flann" package. The question is, which other similar NuGet packages would it depend upon?

    • DLL: opencv_flann300.dll
    • import library: opencv_flann300.lib
    • include file: opencv2\flann.hpp
    • include files: opencv2\flann\*.h*

     

    I looked through every DLL that makes up OpenCV and used "dumpbin" to figure out which ones depend on which others. I also looked through all the header files in the include directory to see which ones depend on which others. (I also looked at how the project files in GitHub Microsoft/opencv/tree/vs2015-samples claim to depend upon each other, but that seemed to include superfluous dependencies - ones that aren't reflected in dumpbin or in include files - so I discounted them).

     

    The end result is this dependency graph below. This shows all the NuGet packages that I intended to create. For instance, the "OpenCV.native.flann" package will include all the files listed above, and it will have a NuGet dependency solely upon "OpenCV.native.core". A few packages also need to contain extra header files, beyond those that follow the example of "flann", and they're indicated in the graph below.

     

    (Note: the picture above seems to have cut off the final line of headers that go into the package allheaders. It should also have included "opencv\cv.h, cv.hpp, cvaux.h*").

    It's mysterious that these allheaders headers from OpenCV don't also include the headers for shape and stitching and videostab and winrt. I'm undecided whether to create an "opencv.native.complete" NuGet package that depends upon every single package, or merely the ones suggested by the "opencv.hpp" header file.

     

    PlatformSpecific.Analayzer - to spot when you're using platform-specific UWP APIs

    $
    0
    0

    With the release of 10586, I’ve wanted to know whether an API I just used was newly introduced in 10586 or was already present back in 10240.

    So what I did is write a “Roslyn Analyzer” for VB/C# and put it on NuGet. It detects whether an API I’ve used is platform-specific or version-specific. If so then it puts in a warning squiggle and gives you a quickfix (unless you've already put an ApiInformation.IsTypePresent check, or similar)

    I’d love for folks to try this out and see if you find it helpful!

     

    What I’d like to learn is this:

    1. Overall, does the analyzer add value, not just for your simplistic examples, but also for your advanced/large projects?
    2. The analyzer has its convention for “transitivity” where you can mark an entire method or field as platform-specific. If you use this, does it work out okay?
    3. This kind of analysis can never be perfect, so it falls back on loosey-goosey heuristics. Are they too loosey-goosey? Do you find places where you’d expected a warning but it produced none? Do you find places where it produced an erroneous warning?

    If people really find it useful, then I’ll push for it to be included by default in the File>New>UWP project template. But that’s a big unprecedented step so I’m cautious.

     

    Here are some of the new 10586 APIs you could test it on! …

    OpenCV: first version up on NuGet

    $
    0
    0

    I got OpenCV to work via NuGet! Here are the steps to consume it:

     

    1. File > New > C++ > UWP

     

    2. References > Manage NuGet References > Add OpenCV.UWP.native.imgcodecs

    This package is currently in pre-release, so it only shows up if you check the "Include Pre-release" checkbox.

    It brings in two other dependencies: imgproc and imgcore.

     

    3. Write some code! Here are some starters: add these to your #include and your using sections in MainPage.xaml.cpp:

    #include <robuffer.h>
    #include <opencv2\imgcodecs.hpp>
    #include <opencv2\imgproc.hpp>

    using namespace Windows::UI::Xaml::Media::Imaging;
    using namespace Windows::Storage::Streams;
    using namespace Microsoft::WRL;

    Next add this inside your MainPage constructor, just after InitializeComponents:

    cv::Mat bmp1 = cv::imread("demo.jpg");
    cv::Mat bmp2 = cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);
    cv::cvtColor(bmp1, bmp2, CV_BGR2BGRA);
    WriteableBitmap^ wbmp = ref new WriteableBitmap(bmp2.cols, bmp2.rows);
    IBuffer^ buffer = wbmp->PixelBuffer;
    unsigned char* dstPixels;
    ComPtr<IBufferByteAccess> pBufferByteAccess;
    ComPtr<IInspectable> pBuffer((IInspectable*)buffer);
    pBuffer.As(&pBufferByteAccess);
    pBufferByteAccess->Buffer(&dstPixels);
    memcpy(dstPixels, bmp2.data, bmp2.step.buf[1] * bmp2.cols * bmp2.rows);
    image1->Source = wbmp;

    Next, add an Image to your MainPage.xaml:

    <Image x:Name="image1"/>

    Next find a picture, rename it "demo.jpg", and add it to your project. (By default when you drag a jpg into SolutionExplorer it gets placed in the Assets folder; the code above assumes instead that it's at top level).

     

    4. Run it!

    What's exciting is that this will run on all x86|x64|ARM configurations, and on all Debug|Release. It will pick the correct OpenCV binaries+libraries for the configuration you select.

     

     

    5. Debugging: download the symbols

    I tried to upload the PDBs to symbolsource.org as recommended in the NuGet docs, but symbolsource.org seems to be down or not accepting symbols. So instead, download them from here, and unzip onto your desktop:

    • OpenCV.UWP.native.symbols.zip [145mb; right-click > save-as].
    • Once it's downloaded, right-click > Properties > Unblock and then right-click > ExtractAll

     

    6. Debugging: step into the OpenCV source code

    The steps here are kind of fiddly, so that video might be the easiest way to follow them. It shows every single step explicitly.

    1. The first time you debug, go to Debug > Windows > Modules, sort this list alphabetically by the Name column, look for opencv_*, right-click > LoadSymbolInformation, and point it to the symbol directory you unzipped.
    2. Tools > Options > Debugging > uncheck the "Just My Code" button
    3. Tools > Options > Debugging > "Enable source server support".

    Once that was set up, I wanted to F11 into my call to "cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);". At first when I F11 it just stepped through header files, until it got to this line in mat.inl.hpp:

    create(2, sz, _type);

    This particular function isn't inline in a header file; it resides in the DLL. There's something I don't understand about C++. How come, when I F11 now it just skips over that function? And when I right-click step-into-specific then it says no source is available? but then when I hit F11 a second time then it jumps into the source code proper? Anyway, here's how to make ti work. I did right-click > Step Into Specific > cv::Mat::create. This claims the source isn't available, so I click on view disassembly, to this line:

    00398F48  jmp dword ptr [__imp_cv::Mat::create (03F2490h)]

    If I hit F11 a second time then it downloads the OpenCV source code file automatically from GitHub (using the "enable source server support" feature that you checked earlier, and a clever technique by Hamish Graham to use GitHub itself as the source server). You can close the disassembly now and double-click on the top frame in the callstack. Now you can step through the source code of the DLL.

     

     

     So far I've only put three of the 15 OpenCV modules on NuGet: core, imgproc and imgcodecs. The work I did to generate these NuPkgs is here on GitHub.

     

     

    Win10 apps in .NET – getting started with win2d accelerated graphics

    $
    0
    0

    This is Part 8 of my “VB Win10 Apps” series. (Everything applies to C# as well, of course).

     

    It’s time for some fun graphics! Here are three small apps I wrote using the win2d accelerated-graphics library.

             
    Getting Started         Circles         Explosions

     

    Transcript of key points in the screencast:

    [0:40] To get started, References > Managed Nuget References > add a reference to “Win2d
     

    [1:10] Inside your MainPage.xaml, add these two lines. Unfortunately in VS2015 RC the XAML designer just shows an error instead of the CanvasControl. This will be fixed in VS2015 RTM.

    xmlns:canvas=”using:Microsoft.Graphics.Canvas.UI.Xaml”

    <canvas:CanvasControl x:Name=”canvas1″ ClearColor=”Black”/> 

     

    [2:05] Win2d naturally works with System.Numerics.Vector and Matrix types, and also with single-width floating points (“Single” in VB, “float” in C#). This can get a bit awkward because most of the .NET framework is more at home with “Double”. You’ll have to figure out the cleanest dividing line between the two. In the app we’re writing, the cleanest point was the random number generator.

     

    [4:05] There’s a neat trick to do animation. We override the Draw method, and at the end of it call “canvas1.Invalidate”. This causes our Canvas_Draw method to be invoked once a frame, 60 frames a second. This makes it a handy place to update the animation as well as draw stuff.

    Sub Draw(sender As CanvasControl, e As CanvasDrawEventArgs) Handles canvas1.Draw
        Update()
        e.DrawingSession.DrawLine(pos(1), pos(2), Colors.Azure)
        canvas1.Invalidate()
    End Sub

    Pro tip: the CanvasAnimatedControl is specifically designed for animations such as these. It specifically has an Update event which it fires on a background thread so as not to hinder responsiveness of the UI thread.

    Pro tip: if you want all your drawing commands to be transformed by a view matrix – for scaling, rotation, offsets or others – use the CanvasDrawingSession.Transform property.

     

    [6:15] There’s a useful feature in VB called “Static Dim”. It’s similar to “static” in C++, but unrelated to “static” in C#. You can declare a local variable to be a “Static Dim”. Its initializer is run only once, the first time the method is invoked. All subsequent invocations re-use the existing variable. (Under the hood, the compiler basically promotes it to be a field of the containing type, with appropriate thread-safe initialization).

    Function Rnd(min As Double, max As Double) As Single
        Static Dim r As New Random
        Return CSng(min + (max – min) * r.NextDouble())
    End Function

     

    [8:15] How to do to bitmaps? The win2d type is called Microsoft.Graphics.Canvas.CanvasBitmap. You have to load it (along with all your other win2d resource initialization) in response to CanvasControl.CreateResources event. The deal is that the graphics accelerator might be reset (such as remote desktop, screen sharing, mirrorcast), and all your resources lost, so you have to cope with the possibility that Win2d will call your CreateResources handler off its own accord to re-establish those resources: your handler has to be idempotent, a fancy word for “no harm if it’s called subsequent times”.

    Pro tip: in the video I set the ball’s velocities inside CreateResources. That’s not the best place, since things like screen sharing will make the ball’s velocities get reset. It would have been better to leave them set in OnNavigatedTo.

     

    [11:30] There’s a particular technique that Win2d uses if your CreateResources handler is asynchronous (e.g. for loading bitmaps): the asynchronous work you do has to be wrapped up in a call to CanvasCreateResourcesEventArgs.TrackAsyncAction. Here’s an example…

    Sub CreateResources(sender As CanvasControl, e As CanvasCreateResourcesEventArgs) Handles canvas1.CreateResources
        Dim loadTask =
            Async Function()
                bitmap = Await CanvasBitmap.LoadAsync(canvas1, “explosion.png”)
            End Function

        e.TrackAsyncAction(loadTask().AsAsyncAction)
    End Sub

     

    [14:30] Please remember to check that your app works on Mobile emulator and also in Release build.

     

    [14:50] When your UWP app runs on mobile, the size of its MainPage (and other pages) doesn’t normally include the “Status Bar” area at the top (with wifi strength indicator, battery indicator and clock). For these graphics apps, what we’ll do is extend our MainPage to include the full size of the mobile screen including that Status-Bar area, and we’ll leave the indicators to be superimposed on top of our graphics. Here’s the magic code to do that, as the first line of your App.OnLaunched event handler:

    ApplicationView.GetForCurrentView().SetDesiredBoundsMode(ApplicationViewBoundsMode.UseCoreWindow)

     

     

    Writing a NuGet package for VS2015 RTM and UWP

    $
    0
    0

    [I’ve removed this old article, and posted a revised+edited version here]

     


    Using project.json for other project types

    $
    0
    0

    If you use VS2015 to create a new UWP project, there are three interesting things in the References node:

    1. Analyzers – this node lets you use so-called “Roslyn analyzers”, which allows anyone to add extra warnings and error messages and quick-fixes into the IDE. Read more about it here: https://msdn.microsoft.com/en-us/magazine/dn879356.aspx
    2. project.json – this is used by UWP projects instead of “packages.config” to manage the NuGet packages you’ve installed. It brings many advantages. Read about them here: http://blog.nuget.org/20150729/Introducing-nuget-uwp.html
    3. Microsoft.NETCore.UniversalWindowsPlatform – in UWP apps, even the .NET framework and runtime themselves are delivered via this NuGet package.

     

    Folks have asked: “Can I get the benefits of project.json in my other (non-UWP) projects?” The answer is technically yes, but this area isn’t finished yet. But it’s interesting to learn about it.

    NOTE: For all of this, you need to have the Win10 Tools installed into VS. That’s because project.json was introduced into VS in order to support Win10 development. Instructions on how to add the Win10 tools are here.

    NOTE: Everything in this post is “unsupported”. That doesn’t mean “impossible”. What it means is that it wasn’t one of the scenarios that was forefront in mind while the feature was being developed, and it hasn’t been tested in all kinds of scenarios you’ll want to use it, and there are probably pieces missing.

    A .NET46 console app using project.json

    (1) Do File > New > Project > VB > Windows > ClassicDesktop > Console Application, and choose “.NET 4.6″ in the dropdown at the top of the new-project dialog.

    (2) Next, add a new file “project.json” to your project, with this content:

    {
      “dependencies”: {
        “Newtonsoft.Json”: “7.0.1”
      },
      “frameworks”: {
        “net46″: { }
      },
      “runtimes”: {
        “win-anycpu”: { }
      }
    }

    (3) Next, unload+reload your project and rebuild.

    You’ll observe a number of things:

    • The References node shows Newtonsoft.Json as a NuGet reference. That indicates that it’s being picked up via project.json, not via the traditional packages.config way.
    • One of the nice things about project.json is that to add a new package, it’s enough merely to edit the project.json file and add a dependency. You don’t need to go via the Manage NuGet Packages > Install dialog, nor the NuGet console.
    • The .NET runtime+framework are not delivered via NuGet in this console app. That only happens in UWP and ASP.NET 5 projects.

     

    A class library using project.json

    If you create a new Portable Class Library that targets .NET4.6, Windows10 and ASP.NET 5, or a subset of them, then it will use project.json by default.

    But what to do if you want to gain the benefits of project.json, and yet still need your PortableClassLibrary to target .NET4.5?

    One approach is to create an ASP.NET 5 class library project (also called xproj). This uses project.json by default.

    Another approach is to create a Portable Class Library, and then add in the project.json file manually, similar to how we did with the console app. As I said, project.json isn’t finished yet. Here’s what you can do, and how to work around the “not finished yet” part…

     

    (1) File > New > Project > VB > Windows > Classic Desktop > either a normal ClassLibrary targeting .NET4.5, or a Portable Class Library targeting whatever you want.

    (2) Add this project.json file, unload then reload, then build. This will make your class library now use project.json instead of packages.config:

    {
      “dependencies”: {
        “Newtonsoft.Json”: “7.0.1”
      },
      “frameworks”: {
        “net45″: { }
      },
      “runtimes”: {
        “win-anycpu”: { }
      }
    }

    (3) Now create a .NET45 application, e.g. a ConsoleApp. And add a Project-to-Project reference to your class library.

    You can build your library and the consuming app, and see that they all compile correctly.

    What’s not finished yet: if you try to run the result, you’ll see that Json.Net (which was referenced by the class library) is needed by the app for it to run. And it’s failing to pick up this fact. So you have to work around it in one of three ways…

    1. You can make the Console App itself use project.json. Then it will pick up the transitive dependencies correctly from its class library
    2. You can make the Console App itself have a NuGet reference to Json.Net, so the library gets pulled in after all.
    3. Instead of having your library has just a Class Library project, you could package it up as a NuGet package, one which pulls in Json.Net as a dependency via its .nuspec file. When you add a reference to the NuGet package from your class library, it will pull in Json.Net.

     

    Writing a NuGet package for VS2015 RTM [repost]

    $
    0
    0

    There are some exciting changes coming to NuGet in VS2015 RTM, under the name NuGet 3.1, also known as “project.json“. You should read about them on the NuGet team blog [link].

    How should you prepare for this as a NuGet package author? This article explains.

     

    Disclaimer: I’m not on the NuGet team, and this post is my non-authoritative personal opinion

    Motivation

    As explained on the NuGet team blog, project.json is a new way for Visual Studio projects to consume NuGet packages. It brings several benefits, which is why NuGet package consumers will like it, and why NuGet package authors will have to support it too:

    • Your .vbproj/.csproj no longer gets polluted with NuGet artifacts. They are kept completely separate.
    • You can change your app target platform, and change debug/release and x86/x64/arm/anycpu, and NuGet packages will now pick this up immediately — before you had to uninstall+reinstall them.
    • You can have two different solutions in two different directories that include the same project (this is particularly useful when you’re working across two different repositories)
    • Your SolutionExplorer>References node looks cleaner because it only includes the packages you’ve actually installed, rather than all their dependencies too.
    • Uninstalling NuGet packages is easier, again because you only have to uninstall the ones you installed rather than all their dependencies.
    • Packages are cached globally (on a per-user per-machine) basis rather than being downloaded and unzipped into every single solution that uses them.
    • File > New and Managed NuGet Packages > Install both become faster.
    • You get more precise control over NuGet package upgrades, and version mismatches.

    The intent of project.json is this: “Installing a NuGet package is nothing more than adding a line to the project.json file; at build-time the libs and .targets from that NuGet package get consumed by msbuild.”

    Project.json – for now – will be used in the following kinds of projects in VS2015. Look for this list to grow longer as new project types are introduced which will also use project.json:

    • In ASP.NET 5 (“DNX”) projects
      • Currently still in preview form
    • For Windows 10 Universal app projects (“UWP”)
      • Requires installation of the VS Universal Windows app development tools. If they’re not already installed, you can install them with “Add/Remove Programs > VisualStudio2015 > Modify”.
    • For modern Portable Class Library (“PCL”) development
      • “Modern PCLs” are ones whose targets are limited to amongst ASP.NET Core 5.0 (“DNX”), Windows 10 (“UWP”) and .NET46.
      • Again this requires installation of the VS universal windows tools.
    • For any other project type,
      • You can delete packages.config and add a project.json file manually, to take advantage of its benefits [instructions].
      • This path isn’t officially supported. It mostly works, except that if you have a P2P (Project-to-project) reference from a packages.config project to a project.json library. In that case, any NuGet dependencies in project.json will fail to be copied to the consuming project.
      • VS universal windows tools are required.

    Master table of all the various .NET frameworks

    There have been many different versions of the .NET framework over the years. Key points:

    1. As a NuGet package author you have to be aware of the different versions to write your NuGet packages effectively.
    2. Microsoft has gotten on top of the problem with a solution called the “dotnet” TxM, described below (TxM stands for “Target x Moniker” and refers to the codes in the table below). The project.json approach was introduced in part to support dotnet.

    Various codes, such as “net45″ and “win81″ and “dotnet” are an essential part of your NuGet package authoring. It is important to understand why they’re available, and what they’re compatible with.

    The following table is based upon that in the NuGet team blog, but contains additional color codes you’ll find useful when authoring your class libraries.

    Backwards-compatibility works within a row, but there’s no compatibility across different rows. For instance, a library or NuGet package designed for “net40″ will work in “net45″ but might not work (and NuGet will refuse to add a reference) in “win81″. That’s because it might depend on some .NET APIs that are present in net40 but absent in win81.

    As explained in the NuGet team blog , Microsoft’s solution is the “dotnet” TxM. You won’t author NuGet package “for a particular platform or set of platforms” any longer. Instead you will author your NuGet package for “dotnet“, and you will call out which particular APIs your package depends upon. This way your package will install fine on any platform, past present or future, which can offer those APIs.

    For instance, if your code uses List<T> internally, then you’ll author your NuGet package with a dependency on Microsoft’s System.Collections NuGet package since that’s where List<T> is implemented. Your package will install fine on every platform that System.Collections can be installed onto. (Microsoft will author their System.Collections package with diligence, and will be quick to make sure it’s updated for all new platforms…) Let’s look at some interesting platforms…

    • .NET46. This is the framework that’s installed as part of the Windows Desktop+Server operating system. If you write an app which targets this, and you add a reference to the System.Collections NuGet package, all it says is ” nothing to do here since System.Collections is already part of the OS platform which this code will run on“. 
    • UAP10.0 (Windows 10 Universal Apps) and DNXCORE50 (ASP.Net 5 Core). These platforms expect apps to carry their .NET runtime and framework with them, app-locally. If you write an app which targets either of these platforms, and you add a reference to the System.Collections NuGet package, then it copies the actual DLL which implements List<T> right there into your application’s “bin” directory, adjacent to your EXE.

    Here’s a subtle but practical question… Why is uap10.0 in the same row as win81? Let’s walk through these platforms: The win81 platform is guaranteed to provide lots of .NET APIs since they’re part of the operating system. When you do File>New>Universal>Blank to create a new uap10.0 app then it creates a project which includes all those APIs (and more) app-locally via NuGet references. The app author is at liberty to delete those NuGet references, resulting in something with fewer .NET APIs that win81. So how can the table honestly say that uap10.0 is in the same row and bigger than win81?

    Well, uap10.0 is in the same row as win81 simply because that reflects the rules that NuGet uses for judging compatibility. As for the possibility that an app might delete their default NuGet references from their app? You as a NuGet package author must defend against this by specifying dependencies on everything you depend upon.
    It seems likely all of Microsoft’s future .NET platforms will operate the same way as UAP10.0 and DNXCORE50: they won’t expect the .NET framework to be part of the operating system, they will instead expect it to be delivered app-locally via NuGet packages, and the way to target them will be to author your NuGet packages with “dotnet”.

    Just how large are we talking? The .NET runtime and framework together add up to ~20mb, which seems a lot. On platforms that use .NET Native, this strips away everything that you’re not using and gets it down to ~6mb. On platforms that don’t, you can probably expect disk-deduplication or other techniques to reduce the footprint to something manageable.
    Color code for the above table:

     

    Red: These are older platforms that are not recommended for future development. When you code for these then your DLL generally references “mscorlib” (which makes it hard to know which APIs you’re using and impossible to use on new platforms). And/or these platforms lack modern language functionality such as async/await.

     

    Yellow: by all means support them. When you code for these then your DLL references “contracts” from which you can easily figure out which APIs (i.e. which NuGet packages) your library depends upon. These platforms provide a modern base level of functionality.

     

     

    Green: these platforms are the future. If you write a DLL which targets only green platforms, then it will be built from the ground up with NuGet dependencies.

     

     

    Green with red border: these platforms don’t assume that .NET is part of the platform. Everything — the .NET runtime, Microsoft’s .NET framework libraries, third-party libraries –all are referenced as NuGet packages and deployed app-locally.

    *

    “win” and “windows” can be used interchangeably.
    “wp” and “windowsphone” can be used interchangeably.
    “net” can be omitted; it is the default

    A worked example

    Goal: I wish to write NuGet package which can be used by as many project types as possible. However, I’m only going to bother making it work on >=.NET45 project types, i.e. ones where I can use modern features like Async/Await and CallerMemberInfo without fuss. Also, my package uses Json.Net internally.

    NuGet is hugely expressive. There’s a many goals you can achieve, and a large number of ways to accomplish any one goal. I can’t explain everything. So what I’ll do is pick just one goal, present my solution, explain my rationale for it, and describe the under-the-hood algorithms that NuGet uses. From that you’ll be able to figure out how to accomplish your own goals.

    Step 1: Create the DLL

    Do File > New > ClassLibrary (Portable) and pick the following targets. This is sometimes known as a “Profile 259 PCL” [ see this webpage for why].

    Rationale: As NuGet package authors we start by thinking about all the platforms that we want our package to be used by. I referred to the Master Table earlier in this document and for each row (i.e. each possible target platform), I picked the oldest version that was still yellow. That gives me the widest reach, without bogging myself down into the difficult-and-niche areas.

    Exception: I have to unselect the ASP.NET Core 5.0 checkbox. This is counter-intuitive since my DLL will still run fine on ASP.NET 5, thanks to being in a NuGet package with “dependencies”. It’s just a point-in-time bug that will be fixed in the first VS2015 update.

    Note: the bottom three “Xamarin” rows appear in the dialog because I’ve installed the Xamarin tools into VS. But even if you haven’t installed the Xamarin tools, no matter: these three checkboxes are placebos, since the other checkmarks implicitly imply that my PCL will also work on Xamarin.
    Portable Class Library. I want PCLs to be able to consume my NuGet package. The deal is: given that I create my ClassLibrary1.dll with the targets above, then my ClassLibrary1.dll can be consumed by any PCL whose targets are the same or more relaxed. A more relaxed set of targets is where either some rows are unchecked, or they’re switched to higher versions. (Thus, to make my library maximally consumable, that’s why I checked as many targets as I could, and picked versions as low as possible).
    Testing that my PCL can be consumed by the different project types. This step isn’t strictly necessary but it’s useful. And indeed this is how I debug my PCLs.

    For all project types that I want to consume my NuPkg, I can actually first try having them reference my PCL directly — either with a Project-to-Project “P2P” reference (Reference > AddReference > Solution > ClassLibrary1) or with a raw file reference (References > AddReference > Browse).

    There are a few known limitations:

    • You can’t P2P reference your Profile259 PCL from a Modern PCL that has ASP.NET Core checked. (A “Profile259 PCL” is one whose targets are as picked at the top of this section; a Modern PCL is one whose targets are limited to amongst .NET46, Win10, ASP.NET Core). This is a point-in-time bug that will be fixed in the first VS2015 update. As a workaround you can still do a raw file reference. And once you turn your PCL into a NuGet package then it will work fine. Another workaround, for testing purposes, is to just go to your consuming project’s targets and remove ASP.NET Core and then the P2P reference will work.
    • You can’t P2P reference your Profile259 PCL from a “DNX” project (i.e. ASP.Net 5). And you can’t even add a raw file reference because the “Browse” button doesn’t work in DNX projects. Don’t worry. Once you turn your PCL into a NuGet package then it will work fine.

     

    Test in .NET Native. I urge everyone to test their libraries from a Win10 application (“UWP”) in Release mode. That’s especially true if your library uses reflection or serialization in any way.

    “UWP Release mode” is the first .NET platform to use Ahead-Of-Time “AOT” compilation: it compiles the application and its libraries into native machine code; later, when the app is then deployed and run on customer machines, it no longer even uses the CLR! (As you can imagine, native machine code has a harder time with reflection and serialization than does the CLR). I wouldn’t be surprised to see AOT on more .NET platforms in future. That’s why I’d prefer to validate my library right now while I’m first architecting it, rather than having to work around issues after-the-fact.

    If your library runs into problems, there’s a technique called ” rd.xml files” by which your library can provide hints to the AOT compiler. If you discover you need these, then the rd.xml file should be deployed in your NuGet package. Best way to do this is to set the build action for your rd.xml to be “embedded resource” – more details here.

    Step 2: plan the NuPkg directory structure

    My NuPkg will have the following structure. Note that there are two identical copies of ClassLibrary1.dll in it.

    WorkedExample.1.0.0.0.nupkg
    +– WorkedExample.nuspec
    +– lib
    +– dotnet
    |    +– ClassLibrary1.dll
    +– portable-net45+win8+wpa81+wp8
    +– ClassLibrary1.dll

    Rationale . This requires an in-depth explanation. I want to start by saying that the “lib\dotnet” subdirectory is the way of the future. We recommend using the dotnet folder structure as a best-practice for all packages that expect to target multiple platforms. The only reason I have an additional subdirectory in here is so that my NuGet can be consumed by traditional PCLs, which can’t consume the “lib\dotnet” subdirectory. This table shows which directories are used by which project types:

    Directory

    Used by these project types

    <no match; fails to install>

    Any platform < .NET45

    lib\portable-net45+win8+wpa81+wp8

    Traditional PCLs.

    lib\dotnet

    Everything else (e.g. UWP apps, .NET Framework apps, Xamarin apps, Modern PCLs)

    NuGet algorithm . How can you figure out which lib\subdirectories you need? And how can you figure out which project types will use which lib\subdirectory? Here I’ll spell out the algorithm that NuGet uses. When you have a project which wants to consume a NuGet package, NuGet will look at your project type, and look at the name of all the lib\subdirectories, and figure out which single subdirectory is the “best match”. The rules for best-match are complicated, and depend crucially on the Master Table earlier in this article.

    • If your project isn’t a portable class library, then it uses these tie-breakers in order to search for the best match…
      1. Look for an “exact match platform”. For instance, if your project is “uap10.0″ and there is a subdirectory called “lib\uap10.0″ then use it.
      2. Failing that, look for an earlier version of the platform or for the unnumbered code for that platform (i.e. in the Master Table, look in the same row, but further to the left)
        • Example: if the project is .NET45, and there’s no “lib\net45″ subdirectory, then look for “lib\net40″, failing that “lib\net35″ and so on.
        • Example: if the project is UWP and there’s no “lib\uap10.0″ subdirectory, then look for “lib\uap”, failing that “lib\win81″ or “lib\netcore451″, and so on.
      3. Failing that, and if your project is >=45, then look for “lib\dotnet”
      4. Failing that, look for all subdirectories of the form “lib\portable-xyz” where your project type is amongst those “xyz” as per rules 1..4. Amongst all of those portable-* candidates, there’s a rule for picking the best one, but I don’t know it.
      5. Failing that, packages.config used to look just in the root “lib\” folder, but that functionality is no longer supported in packages.json. If your nupkg used to put DLLs directly in the root “lib\” folder then it has to change.
    • If your project is a PCL,
      1. If your project is a “modern PCL” and there exists a subdirectory “lib\dotnet”, then use it. (A “modern PCL” is one which targets some or all of .NET46, Win10 and ASP.NetCore, and which targets nothing other than those three.)
      2. Failing that, calculate your project’s targets. In the case of a traditional PCL, the targets are those listed in the PCL targets dialog. In the case of a modern PCL, I don’t know what the targets are. Then look for a lib subdirectory that is an exact match for your project’s targets.
      3. Failing that, look for the best-match “lib\portable-*” subdirectory. I don’t know the rules for “best-match” in this case.

    Attentive readers might spot that I could have got away with just a single subdirectory, called “portable-net45+win8+wpa81+wp8+dnxcore50″. That’s possible, but not best-practice. Best practice is to embrace the “lib\dotnet” folder. The reason for this approach is to make it easier on you the package author. It’s so that your NuGet package will continue to work when Microsoft or Xamarin or anyone else introduces a new platform that isn’t net/win/wpa/wp/dnxcore, but yet still does understand lib\dotnet.

    Step 3: Craft the .nuspec dependencies

    I create a .nuspec file as follows. This .nuspec file should be part of your project. We will discuss the <dependencies> section below…

    <?xml version=1.0?>
    <package>
    <metadata>
    <id>SampleNuGetPackage</id>
    <version>1.0.0.2</version>
    <title>Sample NuGet Package</title>
    <authors>Lucian Wischik</authors>
    <owners>Lucian Wischik</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>This is a demo portable .NET class library</description>
    <summary>It is referencable by all >=.NET45 clients, including WindowsStore and Phone</summary>
    <dependencies>
    <group targetFramework =net>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    </group>
    <group targetFramework =win>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    </group>
    <group targetFramework =wp>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    </group>
    <group targetFramework =wpa>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    </group>
    <group targetFramework =MonoAndroid>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    </group>
    <group targetFramework =MonoTouch>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    </group>
    <group targetFramework =Xamarin.iOS>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    </group>
    <group targetFramework =portable-net45+win8+wpa81+wp8>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    </group>
    <group targetFramework =uap>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    <dependency id=System.Runtime version=4.0.0/>
    <dependency id=System.Resources.ResourceManager version=4.0.0/>
    <dependency id=System.Collections version=4.0.0/>
    <dependency id=System.Linq version=4.0.0/>
    </group>
    <group targetFramework =dotnet>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    <dependency id=System.Runtime version=4.0.0/>
    <dependency id=System.Resources.ResourceManager version=4.0.0/>
    <dependency id=System.Collections version=4.0.0/>
    <dependency id=System.Linq version=4.0.0/>
    </group>
    </dependencies>
    </metadata>
    </package>

    Goal. My DLL depends on certain other DLLs. I must ensure that when an app references my NuGet package then it also transitively references certain other NuGet packages that contain those other DLLs. The exact set of DLLs that I depend upon varies by platform. I don’t want to cause an app to take on more NuGet dependencies than it strictly needs. (This is particularly important for traditional packages.config projects, where each additional NuGet dependency looks poor in Solution Explorer > References, and is awkward to uninstall via Manage NuGet References).
    There’s a tool to auto-generate the dependencies . Oren Novotny has written a great tool called NuSpec.ReferenceGenerator. It’s a NuGet package. You add it do your class-library project, and automatically generates the correct dependencies in the .nuspec file within your project.

    In the rest of this section I’m going to explain how to do stuff manually instead of using the tool. This is merely so you can understand what’s going on under-the-hood, or in case the tool doesn’t work for whatever reason. Let’s first take stock of what we know…

    Source code . My library project’s source code is listed below. The code is obviously using Json.Net, and Linq, and List<T>, and String.Format.

    Public Class Class1
    Public Shared Function f() As String     Dim s = “[“”hello””,””world””]”
    Dim x = Newtonsoft.Json.JsonConvert.DeserializeObject(Of List(Of String))(s)
    Return $”{x.First} I am here”
    End Function

    End Class

     

    Packages.config . My library project’s “packages.config” file is below. My library obviously depends upon Json.Net.

    <?xml version=1.0 encoding=utf-8?>
    <packages>
    <package id=Newtonsoft.Json version=7.0.1 />
    </packages>

    Assembly manifest. I built my library, and opened the resulting DLL in the “ildasm” tool (which comes with Visual Studio 2015). The MANIFEST section looks like this. This provides a list of DLLs which it depends upon.

    .assembly extern System.Runtime {.ver 4:0:0:0 }

    .assembly extern System.Resources.ResourceManager { .ver 4:0:0:0 }

    .assembly extern System.Collections { .ver 4:0:0:0 }

    .assembly extern Newtonsoft.Json { .ver 7:0:0:0 }

    .assembly extern System.Linq { .ver 4:0:0:0 }

     

    From this I can produce the list of NuGet packages that my library depends upon:

    1. Newtonsoft.Json version 7.0.1
      1. This comes straight from my library project’s “packages.config”
      2. Note that 7.0.1 is the version of the Newtonsoft.Json NuGet package which I depend upon and I read this version number from packages.config; the number { .ver 7:0:0:0 } doesn’t refer to a NuGet package version; it is instead the assembly version of the file Newtonsoft.Json.dll which is contained within that NuGet package.
      3. It’s good to write the exact NuGet package version number you depend upon (rather than just writing an earlier number like 7.0.0). This way, NuGet package restore can avoid an extra search.
    2. System.Runtime version 4.0.0
      System.Resources.ResourceManager version 4.0.0
      System.Collections version 4.0.0
      System.Linq version 4.0.0
      1. For the .NET framework, Microsoft adopted a simple 1:1 mapping between DLL names, and the NuGet packages that contain them.
      2. (This 1:1 mapping only works because the PCL targets none of the red platforms in the Master Table. Those red platforms simply use mscorlib as a reference, which wouldn’t provide any clues as to which NuGet packages are needed).
      3. I had no idea which NuGet package versions to use. So I just did File>New>UWP as a throwaway example of a modern platform, built it, then opened the “project.lock.json” file. From this I was able to find some typical NuGet package version numbers for each of these NuGet packages.

    Second, craft platform-specific .nuspec dependencies

    Now that I have this full list of NuGet package dependencies, I could simply write it into the .nuspec file as below.

    <dependencies>
    <dependency id=Newtonsoft.Json version=7.0.1 />
    <dependency id=System.Runtime version=4.0.0/>
    <dependency id=System.Resources.ResourceManager version=4.0.0/>
    <dependency id=System.Collections version=4.0.0/>
    <dependency id=System.Linq version=4.0.0/>
    </dependencies>

    This approach would certainly work. But I think it’s inelegant because anyone who references my NuGet package from one of the traditional platforms like .NET45 will get those additional NuGet references. They don’t even need them.


    NuGet algorithm . The way I solved this problem was through use of nuspec dependency groups. The .nuspec <dependency> section can have either a flat list of dependencies (as above) or a list of groups of dependencies (as at the start of this section), each one tagged with a “targetFramework”.

    In the case where the .nuspec <dependency> section has a list of groups, then it picks a single best-match group for the current project. It does this using exactly the same algorithm as it used to pick a single best-match lib\subdirectory for the current project.

    If you consult the Master Table, and work it through with the NuGet algorithm for picking best match, then here’s how my selection of dependency groups will be used by different project types:

    Dependency group targetFramework

    Used by these project types

    net

    net45, net451, net452, net46

    win

    win8, win81,

    wp

    wp8, wp81

    wpa

    wpa81

    MonoAndroid
    MonoTouch
    Xamarin.iOS

    MonoAndroid10

    MonoTouch10
    Xamarin.iOS10

    uap

    uap10.0

    dotnet

    dnxcore, modern PCLs

    portable-net45+win8+wpa81+wp8

    traditional PCLs

     

    Rationale . For the modern project types with a red border around them (uap10 and dnxcore50), plus for any conjectured future project types (likely based on dotnet and deploying the framework app-locally), then I need to spell out all of the dependencies including the framework ones:

    • [Full dependencies] Newtonsoft.Json, System.Runtime, System.Resources.ResourceManager, System.Collections, System.Linq

    But for all other project types, I can safely assume that the framework dependencies are already satisfied by the platform/OS, so all I want to do is spell out the extra ones:

    • [Brief dependencies] Newtonsoft.Json
    • (Pedantically, you were supposed to add <frameworkAssembly> directives to your .nuspec just in case the user had manually deleted important framework references from their References node. But no one ever used these directives correctly, and it wasn’t worth doing.)

    The only way I could figure out how to do this, going by the Master Table and the NuGet best-match algorithm, was to spell out every single traditional dependency group and also spell out uap10.0 explicitly. In a nutshell, the reason it ends up like this is (1) I need to specify full dependencies for dotnet; (2) dotnet matches every traditional platform, but the traditional platforms should only have brief dependencies, so each one must be explicitly exempted; (3) uap10.0 the traditional “win” platform, but needs full dependencies, so it has to be exempted once again.

    (If anyone can see an easier way to achieve this, please let me know!)
    The lazy approach

    The work to craft a proper .nuspec <dependencies> section is a bit awkward. I want to explicitly rule out some tempting “lazy” alternatives, because they don’t give a good experience for the consumer of your NuGet package…

    1. I might omit the <dependencies> section entirely. But then if my NuGet package is referenced by a project which doesn’t also reference Newtonsoft.Json, then it will fail at runtime with a DllLoadException.
    2. I might merely put into <dependencies> the items in my library project’s packages.config, in this case Newtonsoft.Json. This will run fine on all traditional platforms. It will likely run fine on uap10.0 unless the app author has gone out of their way to delete the default NuGet references, or unless Microsoft in future decides to trim down the set of default NuGet references for a faster leaner system. It will likely fail on dnxcore50, since those projects have just a tiny set of default NuGet references. It will likely fail on future platforms, since they’re most likely to follow the lead of dnxcore50.
    3. I might merely put into the <dependencies> those items in my library project’s packages.config, plus the single catch-all “Microsoft.NETCore” . The Microsoft.NETCore package represents all of the default NuGet references. This approach will run fine on uap10. It will run fine on dnxcore50 and on future platforms, but it will bloat them due to including “the whole world and the kitchen sink” while they aim to be lean. It will work on traditional platforms, but will look really terrible since 100+ NuGet references will be added to them.
    4. I might abandon the complicated “dependency groups”, and merely use a flat list of dependencies . As discussed above, this will work on all platforms. On traditional platforms, however, it adds superfluous NuGet references to traditional platforms — in this case just 4 superfluous references plus their transitive dependencies, but in more realistic libraries it might add 50 or more.

    Step 4: produce the .nupkg file as part of your build

    It’s best to make the .nupkg be automatically generated as part of my PCL project’s build. This section shows how.

    First, from my PCL project, I add a NuGet reference to the “NuGet.CommandLine” package.

    Second, I unload my project, edit the .vbproj/.csproj, and add these lines near the end:

    <Target Name=AfterBuild>
    <Copy SkipUnchangedFiles=true
    SourceFiles=bin\$(Configuration)\ClassLibrary1.dll
    DestinationFiles=bin\$(Configuration)\nupkg\lib\dotnet \ClassLibrary1.dll />
    <Copy SkipUnchangedFiles=true
    SourceFiles=bin\$(Configuration)\ClassLibrary1.xml
    DestinationFiles=bin\$(Configuration)\nupkg\lib\dotnet \ClassLibrary1.xml />
    <Copy SkipUnchangedFiles=true
    SourceFiles=bin\$(Configuration)\ClassLibrary1.pdb
    DestinationFiles=bin\$(Configuration)\nupkg\lib\dotnet \ClassLibrary1.pdb />
    <Copy SkipUnchangedFiles=true
    SourceFiles=bin\$(Configuration)\ClassLibrary1.dll
    DestinationFiles=bin\$(Configuration)\nupkg\lib\portable- net45+win8+wpa81+wp8\ClassLibrary1.dll />
    <Copy SkipUnchangedFiles=true
    SourceFiles=bin\$(Configuration)\ClassLibrary1.xml
    DestinationFiles=bin\$(Configuration)\nupkg\lib\portable- net45+win8+wpa81+wp8\ClassLibrary1.xml />
    <Copy SkipUnchangedFiles=true
    SourceFiles=bin\$(Configuration)\ClassLibrary1.pdb
    DestinationFiles=bin\$(Configuration)\nupkg\lib\portable- net45+win8+wpa81+wp8\ClassLibrary1.pdb />
    <Copy SkipUnchangedFiles=true
    SourceFiles=SampleNugetPackage.nuspec
    DestinationFiles=bin\$(Configuration)\nupkg \SampleNugetPackage.nuspec />
    <Exec Command=..\..\..\..\packages \NuGet.CommandLine.2.8.6\tools\NuGet.exe pack SampleNugetPackage.nuspec -OutputDirectory ..
    WorkingDirectory=bin\ $(Configuration)\nupkg
    Outputs=bin\ $(Configuration)\*.nupkg />
    </Target>

    Rationale . After the project has been built, these msbuild steps create a “bin\Debug\nupkg” directory which contains the .nuspec and contains the “lib\” directory structure we discussed earlier. Then it runs the “nuget.exe” command-line tool to turn it into a .nupkg and sticks it into your bin\Debug directory. Or Release directory, if you’re building in Release mode.

    I’m not a great expert on msbuild. These msbuild steps have a few shortcomings, which I’m sure an msbuild expert can point out J But they’re the best I’ve got.

    The easiest way to test your .nupkg is Tools > Options > NuGet > PackageSources, and add a new source which points straight to your project’s bin\debug directory. Then you can easily add a NuGet reference to it from any other app.

    Special Circumstances

    There are a number of advanced NuGet features. Some of them no longer work with project.json. I’ve colored these back-compat breaks in red. Some of them are also discussion on the NuGet team blog.

    “Content” no longer supported by project.json. A few NuGet packages use “content”. These are files such as .cs or .png which, in traditional packages.config, got added as source files to your project when you install a NuGet package. You were even able to modify these files once they’ve been added. (Content files are no longer supported by project.json and are simply ignored. You must either rewrite your package to avoid it, or join in the discussion with the NuGet team about how content should work.

    “Lib\files” no longer supported by project.json. It used to be you could include files directly in the root of the lib\ directory, and if no appropriate lib\subdirectory was found, then NuGet would pick files straight from the root. This is no longer supported by project.json and such files are simply ignored. You should consult the Master Table to figure out which lib\subdirectory (or subdirectories) are more appropriate now.

    “Build\files” no longer supported by project.json. Likewise it used to be you could include files directly in the root of the build\ directory. This is not currently supported by project.json. You must either rewrite your package to avoid it, again by consulting the Master Table, or follow the NuGet team’s progress on fixing this bug.

    “install.ps1″ no longer supported by project.json. It used to be you could have an arbitrary powershell script be invoked when the package is added or removed. This no longer makes sense in the world where NuGet packages are resolved by msbuild, and so install.ps1 and uninstall.ps1 are now simply ignored. You must figure out alternatives. A really powerful alternative is .targets files in a build\subdirectory.

    Analyzers have changed since VS2015 RC. An “Analyzer” is a mechanism by which a NuGet package can provide additional warning messages, error messages and quick-fixes inside the VS2015 code editor for VB/C#. You get started with an analyzer by installing the .NET Compiler Platform SDK and doing File>New>Extensibility>Analyzer. How do Analyzer NuGet packages install? In packages.config projects, they install via “install1.ps”. In project.json projects, they install by convention that they’re inside an “analyzers\dotnet” directory (or “analyzers\dotnet\vb” or “analyzers\dotnet\cs” if the analyzer is VB or C# specific). Note that “analyzers\dotnet” here doesn’t follow the same rules as “lib\subdirectory“… it should be “dotnet” exactly. (This rule isn’t currently enforced, but it might be in future). If you created an analyzer prior to VS2015 RTM (release date July 29th 2015) then I suspect your analyzer NuGet package is out of date. You should do File>New>Extensibility>Analyzer once again and copy+paste your old code.

    mscorlib-based projects. If you want to write a NuGet package which is very portable, including to .NET4-era targets, things get more complicated. (1) When you try to write such a PCL, it is “mscorlib”-based. That means the DLL contains an assembly reference to “mscorlib”. I don’t know how to figure out which NuGet packages this corresponds to. (2) Such an mscorlib-based PCL cannot be referenced by a dnxcore50 project. Therefore, at the bare minimum, you must build two separate DLLs, one for older project types and one for newer ones. (3) You’ll typically want to include references to packages like Microsoft.Bcl and Microsoft.Bcl.Async and Microsoft.Net.Http. I don’t know which of these packages should also be depended upon by modern targets.

    How to call UWP APIs from a desktop VB/C# app

    $
    0
    0

    Latest update:

    I’ve put this functionality into a NuGet package. To use UWP APIs from your >=.NET45 desktop app, simply add a NuGet reference:

     

    I’ve also moved the technical information from this blog over to the package’s GitHub project site:

     

    Out of date: first steps

    Add these references to your project, by right-clicking on your project in Solution Explorer and doing References > AddReference > Browse. This minimum needed to get going.

    1. c:\Program Files (x86)\Windows Kits\10\References\Windows.Foundation.FoundationContract\1.0.0.0\Windows.Foundation.FoundationContract.winmd
    2. c:\Program Files (x86)\Windows Kits\10\References\Windows.Foundation.UniversalApiContract\1.0.0.0\Windows.Foundation.UniversalApiContract.winmd
    3. c:\Program Files (x86)\Windows Kits\10\UnionMetadata\Facade\Windows.WinMD
    4. c:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5\System.Runtime.WindowsRuntime.dll

    The complete set of Desktop WinRT APIs

    Here is a MSDN list of WinRT APIs that can be called from desktop apps (but I believe this list is incomplete).

     

    Just the four references is enough to get you 90% of the WinRT APIs you’ll need in practice.

    But to go the full mile, I think the best way would be to package it up as a NuGet package according to the instructions below. I’ll explore that idea later. I’d want to get the other UWP facades, get something that works with newer UWP SDKs than just 10240, also include APIs from the Desktop Extension SDK, include the remaining .NET interop DLLs, and get all the default namespace-imports.

    Until I or someone else produces that NuGet package, here below is a detailed explanation:

    Explanation

    To find the UWP WinRT APIs themselves, look for example at this file:

    C:\Program Files (x86)\Windows Kits\10\Platforms\UAP\10.0.10240.0\Platform.xml

    This Platform.xml file conveys which WinRT APIs are available in “version 10240″ of the UWP SDK. (This was the version that came out at VS2015 RTM. Newer versions will come out periodically). The Platform.xml file works by being an index into a set of other winmd files. It looks like this:

    <ApiContract name=”Windows.Foundation.FoundationContract” version=”1.0.0.0″ />
    <ApiContract name=”Windows.Foundation.UniversalApiContract” version=”1.0.0.0″ />
    <ApiContract name=”Windows.Networking.Connectivity.WwanContract” version=”1.0.0.0″ />

    Each line denote a .winmd file in the “C:\Program Files (x86)\Windows Kits\10\References directory”. That’s how I picked up exact filenames and pathnames to add as references. (I should really have added a reference to the WwanContract.winmd file as well, but couldn’t be bothered)

    These three winmd files make up the WinRT APIs available across all Win10 devices. You might additionally want to use the additional WinRT APIs that are part of Desktop. Look at this file:

    C:\Program Files (x86)\Windows Kits\10\Extension SDKs\WindowsDesktop\10.0.10240.0\SDKManifest.xml

    It has a load more <ApiContract> directives, pointing to additional .winmd files you should reference from the “C:\Program Files (x86)\Windows Kits\10\References” directory. I haven’t bothered to do that.

     

    To interop with DLLs or WINMDs build for Win8.1 or WindowsPhone8.1, the issue is that those old DLLs and WinMDs used to think that all WinRT types were in a file called “windows.winmd”. But they’re not anymore! Instead they’re in files called Windows.Foundation.FoundationContract.winmd and the like. The solution is a façade winmd, which is basically a load of type-forwarders from windows.winmd to the new winmd files.

    c:\Program Files (x86)\Windows Kits\10\UnionMetadata\Facade\Windows.WinMD

     

    To use “await” on WinRT types, this and other interop between .NET and WinRT are provided by the following DLL. Note that it’s an 8.1-era DLL, and therefore only works in conjunction with “facade\windows.winmd” above. (There are several other interop assemblies but they’re not so crucial).

    c:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5\System.Runtime.WindowsRuntime.dll

    If you fail to add both these two assemblies, then you’ll get error messages like this:

    • BC36930: ‘Await’ requires that the type ‘IAsyncAction’ have a suitable GetAwaiter method
    • CS0012: The type ‘IAsyncAction’ is defined in an assembly that is not referenced. You must add a reference to assembly ‘Windows, Version=255.255.255.0, Culture=neutral, PublicKeyToken=null, ContentType=WindowsRuntime’.
    • CS4036: ‘IAsyncAction’ does not contain a definition for ‘GetAwaiter’ and no extension method ‘GetAwaiter’ accept a first argument of type ‘IAsyncAction’ could be found (are you missing a using directive for ‘System’?)

     

    Example code (VB)

    Here’s an example console app in VB which uses UWP WinRT APIs:

    Imports System.IO

    Module Module1
        Sub Main()
            MainAsync().GetAwaiter().GetResult()
        End Sub

        Async Function MainAsync() As Task
            ‘ basics
            Dim folder = Windows.Storage.KnownFolders.DocumentsLibrary
            Dim opts As New Windows.Storage.Search.QueryOptions(
                                  Windows.Storage.Search.CommonFileQuery.OrderByName, {“.txt”})
            Dim files = Await folder.CreateFileQueryWithOptions(opts).GetFilesAsync(0, 20)
            For Each file In files
                Console.WriteLine(IO.Path.GetFileName(file.Path))
            Next

            ‘ streams
            Using reader = New IO.StreamReader(Await files.First.OpenStreamForReadAsync())
                Dim txt = Await reader.ReadToEndAsync()
                Console.WriteLine(txt.Substring(0, 100))
            End Using

            ‘ pictures
            Dim pics = Await Windows.Storage.KnownFolders.PicturesLibrary.GetFilesAsync(
                             Windows.Storage.Search.CommonFileQuery.OrderBySearchRank, 0, 1)
            Using stream = Await pics.First.OpenReadAsync()
                Dim decoder = Await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(stream)
                Console.WriteLine(decoder.OrientedPixelWidth & “x” & decoder.OrientedPixelHeight)
            End Using

            ‘ httpclient
            Dim client As New Net.Http.HttpClient()
            Dim html = Await client.GetStringAsync(“http://www.microsoft.com“)
            Console.WriteLine(html.Substring(0, 100))

        End Function
    End Module

     

     

    Example code (C#)

    Here’s an example console app in C# which uses UWP WinRT APIs:

    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using System.Threading.Tasks;

    class Program
    {
        static void Main()
        {
            MainAsync().GetAwaiter().GetResult();
        }

        static async Task MainAsync()
        {
            // basics
            var folder = Windows.Storage.KnownFolders.DocumentsLibrary;
            var opts = new Windows.Storage.Search.QueryOptions(Windows.Storage.Search.CommonFileQuery.OrderByName, new[] { “.txt” });
            var files = await folder.CreateFileQueryWithOptions(opts).GetFilesAsync(0, 20);
            foreach (var file in files)
            {
                Console.WriteLine(System.IO.Path.GetFileName(file.Path));
            }

            // streams
            using (var reader = new System.IO.StreamReader(await files.First().OpenStreamForReadAsync()))
            {
                var txt = await reader.ReadToEndAsync();
                Console.WriteLine(txt.Substring(0, 100));
            }

            // pictures
            var pics = await Windows.Storage.KnownFolders.PicturesLibrary.GetFilesAsync(
                                   Windows.Storage.Search.CommonFileQuery.OrderBySearchRank, 0, 1);
            using (var stream = await pics.First().OpenReadAsync())
            {
                var decoder = await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(stream);
                Console.WriteLine($”{decoder.OrientedPixelWidth} x  {decoder.OrientedPixelHeight}”);
            }

            // httpclient
            var client = new System.Net.Http.HttpClient();
            var html = await client.GetStringAsync(“http://www.microsoft.com“);
            Console.WriteLine(html.Substring(0, 100));
        }
    }

     

    OpenCV: unsure how to proceed

    $
    0
    0

    OpenCV http://opencv.org/ is the standard open-source library that everyone uses for image detection. It’s written in C++.

    I want to use it from my .NET UWP apps, but I’m struggling to figure out how. If anyone has solid guidance on how to proceed, I’d love to know! This blog post is just a documentation of my first tentative investigations. (Also eventually I want to use OpenCV from my cross-platform .NET Core apps as well, so I can write an ASP.NET5 webservice running on Linux that does face detection).

    I think there should ideally be three layers of NuGet packages:

    1. OpenCV.dlls – there should just be a base NuGet package whose only job is to carry the native C++ dlls. It needs to carry the three flavors, x86 + x64 + ARM. It should use the new features in NuGet3 so it can carry all three flavors of dll in a single NuGet package, and at build-time the project.json should pick out the right binaries.
    • There’s a catch. The normal “opencv_world300.dll” is huge at 30mb and includes absolutely all OpenCV functionality. Typically my store apps will only need a small subset of that functionality. I see that the “OpenCV.NET” NuGet package instead delivers it split up into 19 separate DLLs, each of them about 1-2mb. It would be best if you could get the DLLs also through a load of smaller NuGet packages, one by one, to pick out which of those binaries you need.
    • Note that a C++ DLL that you built for an earlier version of Windows Store or Windows Phone will not work in UWP. That’s because even though UWP supports all the same APIs that were there in the earlier WinRT platforms, it divides them up into different “api-sets”, so the binaries are incompatible.
    • Ideally this NuGet package should also carry the native C++ dlls for Linux and for OSX, so it can be used on other .NET Core platforms. I don’t know how native dlls are carried on those platforms
       
  • OpenCV.interop – there should be an intermediate NuGet package whose job is solely to contain p/invoke declarations for those DLLs without any fancy wrapping.
    • Maybe it would also include header files, for consumption by C++
    • I don’t know how p/invoke declarations work for cross-platform .NET Core.
    • Why decouple this from the raw DLLs? solely to enable the “pick-and-choose” idea above. Maybe it’s not worth it. Maybe the “OpenCV.dlls” and “OpenCV.interop” should be combined.
       
  • OpenCV.Net – there should be a higher-level set of wrapper classes, more .NET friendly.
    • Maybe this package need never be written, if everyone is happy using the unwrapped versions of the APIs.
    • Maybe there will be multiple competing versions of this package, with different API designs, some more friendly than others.

     

    OpenCV itself 

    http://opencv.org/ – the main website has a “download binaries for Windows” link. This includes x64+x86 builds of “opencv_world300.dll” (32mb). I don’t know if they’re built for UWP but I suspect not. It has two 8.1-era Phone+Store sample apps but they didn’t build out of the box. It has a directory called “sources\platforms\winrt” which has instructions for building on WinRT, but these instructions don’t include x64, and I suspect they date from before UWP.

    https://github.com/Itseez/opencv – the main GitHub repository for OpenCV

    https://msopentech.com/blog/2015/03/06/opencv-enabled-for-windows-runtime-family/ – has pre-UWP instructions on how to build binaries from the main GitHub repository using CMake

    https://www.nuget.org/packages/OpenCV/ – this NuGet package with 18k downloads claims to contain “binaries and build targets”. This isn’t true. It contains C++ header files, and it contains a .targets file which (based on build parameters) pick up the OpenCV native DLLs that it assumes you got from elsewhere and copied into a particular location. It only does this for x86 and x64; not ARM.

    There are two old sites – https://github.com/Microsoft/opencv was the old staging repo where WinRT support was first added to the main OpenCV GitHub, and http://code.opencv.org/projects/opencv/wiki/WindowsRT had old instructions on how to build for WinRT. I think both links are out of date.

    Analysis: I suspect there are no “ready-to-go” binaries here for UWP, nor any useful NuGet packages. If I want to use OpenCV from source, I’ll have to learn the thing’s build system, configure it to build UWP, and built it myself. I’ll have to write a OpenCV.dlls NuGet package from scratch myself. I don’t know whether the work I do here can be submitted back into OpenCV’s daily builds, or if I’ll have to keep it separate.

     

    OpenCV Extension SDK 

    http://channel9.msdn.com/Events/Build/2015/3-82 – this video discusses an “OpenCV for Windows Store” Extension SDK which contains native binaries for UWP apps. However, the Extension SDK has never been publically released. (Also, ExtensionSDKs were used in the past because prior versions of NuGet weren’t good at selecting x64/x86/ARM binaries based on build-time configuration. That limitation has gone away, and now NuGet is a much better option than ExtensionSDK).

    Analysis: The author of the video has moved on to a different team. I’ve contacted his former colleagues to see if I can get anywhere with this. I don’t know if it will be a single huge 30mb “world” dll, or a series of smaller ones.

     

    OpenCVSharp

    This NuGet package contains a 30mb “world” binary for x86+x64 (not ARM) that come from here. It also contains C# wrappers that are as close as possible to the raw C API. This is the most widely used package, with 32k downloads total.

    I’m confused, because the NuGet package has that huge DLL, but then the project’s install.ps1 seems to have them split up into separate pieces.

    The NuGet package only installs onto desktop .NET Framework 4.0 and higher — not .NETCore, and not UWP.

    As for the raw native binaries themselves, I don’t know. But given the desktop focus, and the lack of ARM, I strongly suspect they won’t work on UWP.

    https://github.com/shimat/opencvsharp – this is the main repository.

    The OpenCVSharp NuGet is actually offered in four flavors:

    1. https://www.nuget.org/packages/OpenCvSharp3-AnyCPU/ – the main one for OpenCV3 which was released in 2015; most recent
    2. https://www.nuget.org/packages/OpenCvSharp3-WithoutDll/ – as above, but missing the “world.dll” (so you’ll pick it up yourself elsewhere)
    3. https://www.nuget.org/packages/OpenCvSharp-AnyCPU/ – like [1] but for OpenCV 2.4
    4. https://www.nuget.org/packages/OpenCvSharp-WithoutDll/ – like [2] but for OpenCV 2.4

    There’s also this NuGet package https://www.nuget.org/packages/OpenCVSharp/ but it doesn’t seem to be related.

    Analysis: this project seems to provide the correct “interop p/invoke” layer. But I think it won’t be a good source of binaries for UWP.

     

    OpenCV.NET

    This NuGet package has a nicer .NET-style set of wrapper classes. I don’t understand how/where the NuGet package picks up its raw binaries from, but it appears to get them somehow! The binaries seem to be x86 only (not x64, not ARM) and are split into ~15 smaller DLLs. The package has been downloaded 8k times. The most recent update was 9 months ago.

    The NuGet package only works on desktop .NET Framework 4.0 and higher (not UWP).

    https://www.nuget.org/packages/OpenCV.Net/

    Analysis: I think it’d be ideal to split out the “interop p/inoke” layer from the wrapper classes. I’m interested to know whether the interop declarations for “split-into-separate-dlls” that this project uses is different from the interop declarations for “world.dll”. In any case, this project isn’t a suitable source for raw binaries.

     

    Emgu

    This project does provide ARM and WinRT for 8.1-era. However I haven’t read anything to indicate support for UWP. Its NuGet package was last updated 20 months ago and has 6k downloads.

    This project requires a commercial license for ARM and WinRT.

    http://www.emgu.com/wiki/index.php/Main_Page

    https://www.nuget.org/packages/VVVV.EmguCV/

    Analysis: this isn’t in the OSS spirit that I like about .NET Core, so I haven’t pursued it.

     
     

    Stack Overflow 

    http://stackoverflow.com/questions/14065370/using-opencv-in-a-windows-store-app-with-visual-c – this seems to be the most valuable StackOverflow resource. It has Raman Sharma (author of the “OpenCV for Windows Store Apps” ExtensionSDK) give some answers on how he built OpenCV for WinRT.

     

    OpenCV: building UWP binaries

    $
    0
    0

    [This blog article documents my continuing mission to figure out how to use OpenCV in UWP apps…]

     

    How to build OpenCV native dlls for UWP 

    OpenCV is on GitHub. There are three relevant forks/branches, as of writing on November 2015:

       [1] https://github.com/itseez/opencv

         +- [2] fork https://github.com/Microsoft/opencv

              +- [3] branch https://github.com/Microsoft/opencv/tree/vs2015-samples

    Repository [1] is the master. In it, the native OpenCV binaries are built for all different platforms using CMake. As far as I can tell, this repository only has support via CMake for building Windows Store and Windows Phone 8.0 and 8.1 apps. It doesn’t seem to have UWP. And it also requires you to install some particular old versions of VS, as per its readme.txt.

    Fork [2] doesn’t seem to bring anything new.

    Branch [3] contains Visual Studio .sln files from which you can build UWP versions of all the native OpenCV binaries: x86, x64 and ARM, Debug and Release. The Release builds actually were a bit wonky, but I submitted PR #69 to make them build right. So: if you clone this repository, select this branch, set an environment variable, and open up vs2015/WS/10.0/ARM/OpenCV.sln, or x86/OpenCV.sln, or x64/OpenCV.sln, then you can successfully build UWP binaries for OpenCV. This will produce 15 binaries for each architecture+build combination.

     

    How to use OpenCV native dlls for UWP

    I tried out the OpenCV native dlls in a C++ UWP app, and they worked fine.

    More interestingly, the dlls all together add up to about 30mb, but I was able to use just three of them that I needed and they worked fine. This satisfies one of my central requirements, to be able to deploy just a cut-down version of OpenCV with my store app, to keep size down.

    I did File>New>C++>Windows>UWP blank app. In Solution Explorer I added three DLLs as “content” files in the root directory of my project so they would be copied into the resulting appx, adjacent to the .exe:

    opencv_core300d.dll
    opencv_igmcodecs300d.dll
    opencv_igmproc300d.dll

    In Solution Explorer I added three import-libraries. By default, when you add LIBs to a C++ project, they are taken as import libraries:

    opencv_core300d.lib
    opencv_igmcodecs300d.lib
    opencv_igmproc300d.lib

    In Project Properties I added an include directory to my C++ project options

    vs2015/WS/10.0/include

    In my mainpage C++ file, I added a bunch of #include and using statements:

    #include <robuffer.h>
    #include <opencv2\imgcodecs.hpp>
    #include <opencv2\imgproc.hpp>

    using namespace Windows::UI::Xaml::Media::Imaging;
    using namespace Windows::Storage::Streams;
    using namespace Microsoft::WRL;

    I added a content file “demo.jpg” in Solution Explorer, and in MainPage.xaml I added an Image control named “image1″, and added these lines in OnNavigatedTo in my mainpage C++ file:

    cv::Mat bmp1 = cv::imread(“demo.jpg”);
    cv::Mat bmp2 = cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);
    cv::cvtColor(bmp1, bmp2, CV_BGR2BGRA);
    WriteableBitmap^ wbmp = ref new WriteableBitmap(bmp2.cols, bmp2.rows);
    IBuffer^ buffer = wbmp->PixelBuffer;
    unsigned char* dstPixels;
    ComPtr<IBufferByteAccess> pBufferByteAccess;
    ComPtr<IInspectable> pBuffer((IInspectable*)buffer);
    pBuffer.As(&pBufferByteAccess);
    pBufferByteAccess->Buffer(&dstPixels);
    memcpy(dstPixels, bmp2.data, bmp2.step.buf[1] * bmp2.cols * bmp2.rows);
    image1->Source = wbmp;

    Result: it all worked. The image displayed correctly.

     

    Next Steps

    I want to package each individual DLL up as a NuGet package, for 15 NuGet packages in total. Each package should include x86+x64+ARM versions of the DLLs, and corresponding import libraries. It should also include the include files pertaining to that DLL. That way, as a C++ author, I will be able to bring in only the particular DLLs that I want. To do this I’ll have to figure out the dependencies between the DLLs, and the dependencies between header files, and then express these as NuGet dependencies. (I don’t yet know how to create NuGet packages for C++. I’ll look to “Win2d.uwp” as the canonical example. I don’t know if each package should also include debug builds, but I suspect not. I don’t know where PDBs should go.)

    I think that NuGet generation should be done as part of [3] branch vs2015-samples. This is the branch that has .sln files for building all the DLLs, so it’s appropriate for it to also include NuGet-package-building. I don’t know if this is within the vision of the owners of that branch or not. Ultimately, in the future, I don’t know if the owners of Itseez/opencv would be open to their CMake system building the NuGet packages as well?

     

    Beyond that, the major issue is that OpenCV has a C++ API. This is not friendly to pinvoke interop with .NET. OpenCV used to have a plain C API which is friendly, but the C API has been deprecated and is not getting new features. I don’t wish to invent a pinvoke-friendly wrapper around the OpenCV C++ API.

    Approach (1): I’d like to test the scenario of a hybrid app, where most of the app is in .NET, and it calls into a C++ winrt component that I author, and this C++ component does all the calls into OpenCV, which it obtains via NuGet.

    Approach (2): OpenCVSharp has a BSD license and already contains a set of ‘extern “C”‘ wrappers around the C++ API. The ideal would be for OpenCVSharp to be able to consume my native UWP dlls for OpenCV. There already is a “dll-free” version of OpenCVSharp, which merely contains the pinvoke wrappers, and expects you to supply the OpenCV DLLs some other way. There are a few unknowns with this approach. I don’t know if the OpenCVSharp project has different version of OpenCV (and hence different include files) as compared to the one in branch [3]. I don’t know if the native extern “C” assembly that comes with OpenCVSharp will be compatible with the DLLs I build, and the fact that there are 15 of them rather than just 1 big “world” dll. I don’t know if it will need to switch to “delay-load” to support the scenario where you only package say 3 of the OpenCV dlls with your store app.

     

    Notes

    I am hugely excited! It was thrilling to see the UWP versions of all the DLLs get built properly. I think I know how to package up DLLs into NuGet better than most folk, and I can do a best-in-class job. I’m excited that I was able to get something working end-to-end at least in C++.

     

     

    OpenCV: how to split it up into components

    $
    0
    0

    [This article documents my ongoing attempts to get OpenCV working in UWP apps]

     

    My goal is to figure out how to package OpenCV up into several individual NuGet packages. For instance, I expect that these files would all be packaged together into a single “OpenCV.native.flann” package. The question is, which other similar NuGet packages would it depend upon?

    • DLL: opencv_flann300.dll
    • import library: opencv_flann300.lib
    • include file: opencv2\flann.hpp
    • include files: opencv2\flann\*.h*

     

    I looked through every DLL that makes up OpenCV and used “dumpbin” to figure out which ones depend on which others. I also looked through all the header files in the include directory to see which ones depend on which others. (I also looked at how the project files in GitHub Microsoft/opencv/tree/vs2015-samples claim to depend upon each other, but that seemed to include superfluous dependencies – ones that aren’t reflected in dumpbin or in include files – so I discounted them).

     

    The end result is this dependency graph below. This shows all the NuGet packages that I intended to create. For instance, the “OpenCV.native.flann” package will include all the files listed above, and it will have a NuGet dependency solely upon “OpenCV.native.core”. A few packages also need to contain extra header files, beyond those that follow the example of “flann”, and they’re indicated in the graph below.

     

    (Note: the picture above seems to have cut off the final line of headers that go into the package allheaders. It should also have included “opencv\cv.h, cv.hpp, cvaux.h*”).

    It’s mysterious that these allheaders headers from OpenCV don’t also include the headers for shape and stitching and videostab and winrt. I’m undecided whether to create an “opencv.native.complete” NuGet package that depends upon every single package, or merely the ones suggested by the “opencv.hpp” header file.

     

    PlatformSpecific.Analayzer – to spot when you’re using platform-specific UWP APIs

    $
    0
    0

    With the release of 10586, I’ve wanted to know whether an API I just used was newly introduced in 10586 or was already present back in 10240.

    So what I did is write a “Roslyn Analyzer” for VB/C# and put it on NuGet. It detects whether an API I’ve used is platform-specific or version-specific. If so then it puts in a warning squiggle and gives you a quickfix (unless you’ve already put an ApiInformation.IsTypePresent check, or similar)

    I’d love for folks to try this out and see if you find it helpful!

     

    What I’d like to learn is this:

    1. Overall, does the analyzer add value, not just for your simplistic examples, but also for your advanced/large projects?
    2. The analyzer has its convention for “transitivity” where you can mark an entire method or field as platform-specific. If you use this, does it work out okay?
    3. This kind of analysis can never be perfect, so it falls back on loosey-goosey heuristics. Are they too loosey-goosey? Do you find places where you’d expected a warning but it produced none? Do you find places where it produced an erroneous warning?

    If people really find it useful, then I’ll push for it to be included by default in the File>New>UWP project template. But that’s a big unprecedented step so I’m cautious.

     

    Here are some of the new 10586 APIs you could test it on! …


    OpenCV: first version up on NuGet

    $
    0
    0

    I got OpenCV to work via NuGet! Here are the steps to consume it:

     

    1. File > New > C++ > UWP

     

    2. References > Manage NuGet References > Add OpenCV.UWP.native.imgcodecs

    This package is currently in pre-release, so it only shows up if you check the “Include Pre-release” checkbox.

    It brings in two other dependencies: imgproc and imgcore.

     

    3. Write some code! Here are some starters: add these to your #include and your using sections in MainPage.xaml.cpp:

    #include <robuffer.h>
    #include <opencv2\imgcodecs.hpp>
    #include <opencv2\imgproc.hpp>

    using namespace Windows::UI::Xaml::Media::Imaging;
    using namespace Windows::Storage::Streams;
    using namespace Microsoft::WRL;

    Next add this inside your MainPage constructor, just after InitializeComponents:

    cv::Mat bmp1 = cv::imread(“demo.jpg”);
    cv::Mat bmp2 = cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);
    cv::cvtColor(bmp1, bmp2, CV_BGR2BGRA);
    WriteableBitmap^ wbmp = ref new WriteableBitmap(bmp2.cols, bmp2.rows);
    IBuffer^ buffer = wbmp->PixelBuffer;
    unsigned char* dstPixels;
    ComPtr<IBufferByteAccess> pBufferByteAccess;
    ComPtr<IInspectable> pBuffer((IInspectable*)buffer);
    pBuffer.As(&pBufferByteAccess);
    pBufferByteAccess->Buffer(&dstPixels);
    memcpy(dstPixels, bmp2.data, bmp2.step.buf[1] * bmp2.cols * bmp2.rows);
    image1->Source = wbmp;

    Next, add an Image to your MainPage.xaml:

    <Image x:Name=”image1″/>

    Next find a picture, rename it “demo.jpg”, and add it to your project. (By default when you drag a jpg into SolutionExplorer it gets placed in the Assets folder; the code above assumes instead that it’s at top level).

     

    4. Run it!

    What’s exciting is that this will run on all x86|x64|ARM configurations, and on all Debug|Release. It will pick the correct OpenCV binaries+libraries for the configuration you select.

     

     

    5. Debugging: download the symbols

    I tried to upload the PDBs to symbolsource.org as recommended in the NuGet docs, but symbolsource.org seems to be down or not accepting symbols. So instead, download them from here, and unzip onto your desktop:

    • OpenCV.UWP.native.symbols.zip [145mb; right-click > save-as].
    • Once it’s downloaded, right-click > Properties > Unblock and then right-click > ExtractAll

     

    6. Debugging: step into the OpenCV source code

    The steps here are kind of fiddly, so that video might be the easiest way to follow them. It shows every single step explicitly.

    1. The first time you debug, go to Debug > Windows > Modules, sort this list alphabetically by the Name column, look for opencv_*, right-click > LoadSymbolInformation, and point it to the symbol directory you unzipped.
    2. Tools > Options > Debugging > uncheck the “Just My Code” button
    3. Tools > Options > Debugging > “Enable source server support”.

    Once that was set up, I wanted to F11 into my call to “cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);”. At first when I F11 it just stepped through header files, until it got to this line in mat.inl.hpp:

    create(2, sz, _type);

    This particular function isn’t inline in a header file; it resides in the DLL. There’s something I don’t understand about C++. How come, when I F11 now it just skips over that function? And when I right-click step-into-specific then it says no source is available? but then when I hit F11 a second time then it jumps into the source code proper? Anyway, here’s how to make ti work. I did right-click > Step Into Specific > cv::Mat::create. This claims the source isn’t available, so I click on view disassembly, to this line:

    00398F48  jmp dword ptr [__imp_cv::Mat::create (03F2490h)]

    If I hit F11 a second time then it downloads the OpenCV source code file automatically from GitHub (using the “enable source server support” feature that you checked earlier, and a clever technique by Hamish Graham to use GitHub itself as the source server). You can close the disassembly now and double-click on the top frame in the callstack. Now you can step through the source code of the DLL.

     

     

     So far I’ve only put three of the 15 OpenCV modules on NuGet: core, imgproc and imgcodecs. The work I did to generate these NuPkgs is here on GitHub.

     

     

    Async workflow [how to hibernate async methods, part 2]

    $
    0
    0

    Four years ago I’d written a post “How to hibernate async methods (how to serialize a task)“. I decided to dig it up and improve it. Here’s what I can now write with it:


    await AlphaAsync().RunWithCheckpointing("a.json");
    // This kicks off the async method. But it allows the async method to
    // serialize its state to disk.

    await Checkpoint.Save();
    // This serializes a snapshot of the async method’s current state to disk.
    // It also serializes the async caller who is awaiting it, and so on
    // all the way up the async callstack (up to "RunWithCheckpointing")

    throw new Checkpoint.Defer(Timespan.FromHours(1));
    // I can abort my current method now. A worker will resume
    // it in time, once a condition is met (in this case a delay for one hour)

    Task t = Checkpoint.ResumeFrom("a.json");
    // You can transfer that serialized state across the wire to another machine,
    // who can deserialize it to resume work exactly where it left off.
    // The task "t" here stands for the top-level AlphaAsync().

     

    My idea is to enable a kind of “async workflow” — I’d like to write my long-running processes as async methods, using for loops/while loops and exceptions and recursive functions and all that – things where the current execution state is actually a pretty complicated thing to serialize manually, and where I need all the assistance I can get. The “Checkpoint” class provides that assistance.

     

    Where might it be useful? …

    • You could kick off a long-running method on Azure, which saves its state periodically. If the Azure VM goes down then another VM can just pick up where it left off.
    • You could write an interactive website which prompts the user to click a button, then saves its state to disk and releases all resources. When the user clicks the button on their website, that incoming HTTP message causes the state to be picked up from where it left off.
    • You could write a mobile agent, which can migrate to another machine to be near to some local resources to do the rest of its work.

     

    I don’t know if this idea has legs! It might be completely crazy. It might be that serializing state is too difficult to do reliably.

     

    Source code is here: https://github.com/ljw1004/blog/tree/master/Async/AsyncWorkflow

     

    Please let me know if you have thoughts on this – feasibility, directions, improvements etc. Next time I revisit this topic, I plan to make the code more robust – make it deal with VB, with ConfigureAwait(), with Task.WhenAll and Task.WhenAny, and most importantly make it uses EntityFramework to serialize the current state of async methods in the async callstack. Once that’s done I’ll package it up as a NuGet library.

    Manifesto: a new fluid UI for desktop/LOB data-entry

    $
    0
    0

    I’m fed up with clunky dialog-based UIs.

     

    Here are two apps, the Windows 7 Calculator and Windows 10 People App. How many of you use Calculator by clicking on buttons with the mouse? And how many do free-form typing with the keyboard? answer: everyone prefers the keyboard. I think that free-form typing is the Holy Grail of LOB apps. Dialogs look cluttered and force users to waste their time on UI rather than directly on data entry.

    calculator peopleapp

     

     

    How about medical record software? Whenever I’m at the doctor’s office, I peer over shoulders to see what they’re using. In the consulting room, nurses and doctors usually seem to use free-form text entry to edit a template. The reception/billing system is usually a clunky dialog filled with too many buttons.

    ims   vetinarygate

     

    I want free-form text-based data entry. (As for free-form text-based commanding, that’s already popular. Here are some examples. The final one is an exciting new indie game called Duskers where you control droids by text-based commanding.)

    vscommand  vscode
    monkey game

     

     

    Example of free-form text-based data entry

    Let’s re-imagine the Windows 10 People app. How could it get away from dialogs, and embrace free-form text entry? Here’s my attempt…

    (1) A simple text-entry dialog with grayed out suggestions for what to type:

    pic1

    (2) While you’re typing, heuristics in the editor (e.g. “this line has overtyped the grey Name suggestion”) infer that this is the name. The caption is updated to reflect this live while you type:

    pic2

    (3) When you hit ENTER to commit a line, the editor pretty-prints it. Pretty-printing is important to show what’s been inferred:

    pic3

    (4) While I type the next line, more heuristics in the editor (e.g. “typing what looks like a phone number even though it’s in the Address field”) infer that this is the telephone number:

    pic4

    (5) When you hit ENTER, the editor pretty-prints it. In this case there are several kinds of phone number available, but the editor picked the most likely one and offered a way to change that (using the mouse or hitting Ctrl+Dot):

    pic5

    (6) I haven’t shown any examples, but I’d also like the editor to show an intellisense completion list with suggestions of what it would expect and understand for you to type right now.

     

     

    Manifesto: fluid UIs for text-based data entry

    We should abandon clunky forms with lots of fields.

    We should embrace free-form text entry.

    This requires a smart text editor with natural language processing, heuristics, and most importantly intellisense.

     

     

     

    What’s next? My friend Klaus Löffelman had a great insight. He observed that the same characteristics that make for free-form text entry are also well suited to dictation and voice control. You could use them for Cortana/Siri/Alexa. You’d actually use the same heuristics, NLP and AI in for free-form data entry as you do bots in Skype or Facebook Messenger.

    Update: there was some discussion on reddit. I appreciated this comment from @grauenwolf: “I have built several medical record applications over the years, and I can say without hesitation that there are lots of places where this would have been a perfect fit.

     

    Visual Studio Text Adornment VSIX using Roslyn

    $
    0
    0

    A VSIX is a way to install plugins into Visual Studio. The kind of plugin I’m interested in right now is a text adornment. This lets you superimpose things (text, pictures, interactive controls) right on top of a code editor window.

    It took me a frustrating day figuring out how to get started. This post is to save everyone else the frustration.

     

    Install the Visual Studio SDK

    Its official title is Visual Studio Extensibility Tools. Full instructions are here. In a nutshell, quit VS and then do:

    1. Control Panel > Add Remove Programs > Microsoft Visual Studio 2015 > Modify
    2. Common Tools > Visual Studio Extensibility Tools > Ok

     

    Install Roslyn SDK

    Its official title is .NET Compiler Platform SDK.

    1. Visual Studio > Tools > Extensions and Updates > Online
    2. Search for “.NET Compiler Platform SDK” > install it, and restart VS

     

    Create a new VSIX project

    Within Visual Studio,

    1. File > New > Project > Visual C# > Extensibility > VSIX Project
    2. This has created some files which we no longer need: delete index.html and stylesheet.css from your project

     

    Add references

    Officially the way you’re supposed to add a text adornment to your project is right-click on it > Add > New Item > Visual C# > Extensibility > Editor > Editor Text Adornment, which also adds some references. But I found it generated unhelpful references, so I’m adding references manually instead. In a nutshell:

    1. Project > References > Add Reference > Framework > check System, PresentationCore, PresentationFramework, WindowsBase > OK
    2. Tools > Options > NuGet Package Manager > General > Automatically check for missing packages during build > ensure this button is checked > OK
    3. Right-click on project > Add > New Item > Visual C# Items > Web > JSON file, and call it “project.json”
    4. Edit the project json and replace its contents with what’s below
    5. Right-click on project > Unload Project
    6. Right-click on project > Reload Project
    7. Build > Build Solution. This will produce warnings “Detected package downgrade System.Collections.Immutable from 1.2.0 to 1.1.37” — these warnings are fine, and indeed necessary (see below). But if there are errors about .NETFramework version, that’s because of a mismatch between net46 specified in the project.json “frameworks” section and the .NET version in your Project > Properties > Application > TargetFramework. The minimum target is .net46. You should make sure both are in sync.
    8. Open source.extension.vsixmanifest > Assets > New, type = “MefComponent”, source = “A project in current solution”, and specify the project.
    9. The NuGet version numbers are specific to an update to VS2015: the versions “1.3.x” and “14.3.x” listed here require at least Update3; you should switch to the older “1.2.x” and “14.2.x” for something that will run on Update2.
    {
        "dependencies": {
            "Microsoft.CodeAnalysis": "1.3.0",
            "Microsoft.VisualStudio.LanguageServices": "1.3.0",
            "Microsoft.VisualStudio.Shell.14.0": "14.3.25407",
            "Microsoft.VisualStudio.Text.UI.Wpf": "14.3.25407",
            "System.Collections.Immutable": "1.1.37",
            "System.ComponentModel": "4.0.1",
            "VSSDK.ComponentModelHost": "12.0.4",
            "VSSDK.DTE": "7.0.4"
        },
        "frameworks": {
            "net46": { }
        },
        "runtimes": {
            "win": { }
        }
    }

     

    Note 1: As much as possible I’m using NuGet references instead of the traditional framework references under AddReference > Framework. I just prefer NuGet. I hope it’ll make this project loadable/buildable even on a machine which doesn’t have the Visual Studio SDK installed (but I haven’t yet tried). Two of the packages are third-party ones written by the awesome .NET MVP Sam Harwell, which are basically just NuGet repackagings of parts of the VS SDK.

    Note 2: For NuGet, I’m using the new-fangled project.json instead of packages.config. This is the faster cleaner way of referencing NuGet packages: with it, the References node in Solution Explorer only shows the 8 NuGets I’m referencing rather than their 20+ dependencies; it also means that NuGets are shared amongst all solutions, rather than being downloaded into a per-solution “packages” directory; and it means your project doesn’t break when you change its directory relative to its .sln file. To activate project.json, you have to create the file, then unload your project, then reload your project, then nuget restore. Apparently the roadmap is for project.json to eventually be folded into .csproj, but until then, project.json is the best we can do.

    Note 3: I have an explicit reference to an older version System.Collections.Immutable v1.1.37, which generates warnings every build. Had I removed this reference then it would have picked up the newer System.Collections.Immtuable v1.2.0 which would cause the VSIX to fail. The deal is that (as of VS2015 Update3) the Roslyn in your Visual Studio is still built upon v1.1.37, and if your project referenced a newer version, then the CLR would throw a MethodNotFound exception when it tries to enter a method that calls any of the ImmutableArray-returning Roslyn APIs.

     

    Add code for the Text Adornment

    We’ll add source code for a simple text adornment.

    1. Right-click on your project > Add > Class
    2. Replace its contents with the following.

    I’ve written out the code with as few “usings” as possible, instead using fully-qualified names. That’s to help you understand the code at-a-glance, to see which library each type comes from. Feel free to switch over to “usings” once you know them!

    using System.Linq;
    
    namespace MefRegistration
    {
        using System.ComponentModel.Composition;
        using Microsoft.VisualStudio.Utilities;
        using Microsoft.VisualStudio.Text.Editor;
    
        [Export(typeof(IWpfTextViewCreationListener)), ContentType("text"), TextViewRole(PredefinedTextViewRoles.Document)]
        public sealed class TextAdornment1TextViewCreationListener : IWpfTextViewCreationListener
        {
            // This class will be instantiated the first time a text document is opened. (That's because our VSIX manifest
            // lists this project, i.e. this DLL, and VS scans all such listed DLLs to find all types with the right attributes).
            // The TextViewCreated event will be raised each time a text document tab is created. It won't be
            // raised for subsequent re-activation of an existing document tab.
            public void TextViewCreated(IWpfTextView textView) => new TextAdornment1(textView);
    
    #pragma warning disable CS0169 // C# warning "the field editorAdornmentLayer is never used" -- but it is used, by MEF!
            [Export(typeof(AdornmentLayerDefinition)), Name("TextAdornment1"), Order(After = PredefinedAdornmentLayers.Selection, Before = Microsoft.VisualStudio.Text.Editor.PredefinedAdornmentLayers.Text)]
            private AdornmentLayerDefinition editorAdornmentLayer;
    #pragma warning restore CS0169
        }
    }
    
    
    public sealed class TextAdornment1
    {
        private readonly Microsoft.VisualStudio.Text.Editor.IWpfTextView View;
        private Microsoft.CodeAnalysis.Workspace Workspace;
        private Microsoft.CodeAnalysis.DocumentId DocumentId;
        private System.Windows.Controls.TextBlock Adornment;
    
        public TextAdornment1(Microsoft.VisualStudio.Text.Editor.IWpfTextView view)
        {
            var componentModel = (Microsoft.VisualStudio.ComponentModelHost.IComponentModel)Microsoft.VisualStudio.Shell.Package.GetGlobalService(typeof(Microsoft.VisualStudio.ComponentModelHost.SComponentModel));
            Workspace = componentModel.GetService<Microsoft.VisualStudio.LanguageServices.VisualStudioWorkspace>();
    
            View = view;
            View.LayoutChanged += OnLayoutChanged;
         }
    
    
        internal void OnLayoutChanged(object sender, Microsoft.VisualStudio.Text.Editor.TextViewLayoutChangedEventArgs e)
        {
            // Raised whenever the rendered text displayed in the ITextView changes - whenever the view does a layout
            // (which happens when DisplayTextLineContainingBufferPosition is called or in response to text or classification
            // changes), and also when the view scrolls or when its size changes.
            // Responsible for adding the adornment to any reformatted lines.
    
            // This code overlays the document version on line 0 of the file
            if (DocumentId == null)
            {
                var dte = Microsoft.VisualStudio.Shell.Package.GetGlobalService(typeof(EnvDTE.DTE)) as EnvDTE.DTE;
                var activeDocument = dte?.ActiveDocument; // sometimes we're constructed/invoked before ActiveDocument has been set
                if (activeDocument != null) DocumentId = Workspace.CurrentSolution.GetDocumentIdsWithFilePath(activeDocument.FullName).FirstOrDefault();
            }
    
            if (Adornment == null)
            {
                var line = e.NewOrReformattedLines.SingleOrDefault(l => l.Start.GetContainingLine().LineNumber == 0);
                if (line == null) return;
                var geometry = View.TextViewLines.GetMarkerGeometry(line.Extent);
                if (geometry == null) return;
                Adornment = new System.Windows.Controls.TextBlock { Width = 400, Height = geometry.Bounds.Height, Background = System.Windows.Media.Brushes.Yellow, Opacity = 0.5 };
                System.Windows.Controls.Canvas.SetLeft(Adornment, 300);
                System.Windows.Controls.Canvas.SetTop(Adornment, geometry.Bounds.Top);
                View.GetAdornmentLayer("TextAdornment1").AddAdornment(Microsoft.VisualStudio.Text.Editor.AdornmentPositioningBehavior.TextRelative, line.Extent, null, Adornment, (tag, ui) => Adornment = null);
            }
    
            if (DocumentId != null)
            {
                var document = Workspace.CurrentSolution.GetDocument(DocumentId);
                if (document == null) return;
                Microsoft.CodeAnalysis.VersionStamp version;
                if (!document.TryGetTextVersion(out version)) version = Microsoft.CodeAnalysis.VersionStamp.Default;
                Adornment.Text = version.ToString();
            }
        }
    }

    Note 1: VS text adornments normally only exist while they’re in view on the screen. VS deletes the ones that have scrolled off-screen, and calls LayoutChanged when new lines get exposed so you can create adornments on-demand.

    Note 2: Typical VS text adornments don’t need to be “live”, i.e. they can be created on demand and ignored after that. But I reckon that typical Roslyn-powered adornments will want to display stuff asynchronously. That’s why I save a reference to the adornment that I created — so my code can go back and update it at a later time (so long as it’s still onscreen). If I did async work on a background thread then I’d need View.Dispatcher.Invoke to update the adornment on the UI thread.

     

    Run it!

    Press F5 to run it. It should launch a new “experimental” instance of VS, with your VSIX installed into it. When you open any text document you should see the new adornment in yellow:

    adornment

     

    How did it know how to launch and debug your VSIX?

    1. Project > Properties > Debug > Start External Program = “C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.exe”: when you created your VSIX project, it was set by default to launch a fresh instance of Visual Studio
    2. Project > Properties > Debug > Command Line Arguments = “/rootsuffix Exp”: when VS is launched, it uses an “experimental hive” whose settings and extensions don’t affect your main VS instance.
    3. Project > Properties > VSIX > Deploy VSIX content to experimental instance = “checked”: when you start debugging, VS will install your VSIX into that experimental instance.

    Sometimes (e.g. when you uninstall the VSIX from the experimental instance) that final part won’t work right. I think that the experimental instance gets stuck thinking “this is a VSIX that has been deleted”. The fix for that is to open Windows File Explorer, delete “%userprofile%\appdata\local\Microsoft\VisualStudio\Exp” and “\14.0Exp” and “\roaming\Microsoft\VisualStudio\Exp” and “\14.0Exp”. Then launch Regedit and delete all the HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\14.0Exp* keys.

    What I like is, when I hit F5, for it to automatically load a test solution into that experimental instance of VS:

    1. Project > Properties > Debug > Command Line Arguments = “c:\…\ConsoleApplication1.sln” /rootsuffix Exp

     

    I hope you have fun with Roslyn-powered editor adornments.

     

    Troubleshooting

    If it fails to hit breakpoints in your extension, check that it’s even loaded: when you’ve done F5, hit the pause button, open up Debug > Windows > Modules, and look for the name of your DLL in the list. You might need to load symbols. If it’s absent from the list, verify that you’ve added your project as an asset to source.extension.vsixmanifest.

    If it crashes without even entering your method, this is often caused by assembly version mismatches. For instance, if your DLL was built against version 1.3.0 of Roslyn, but the current instance of Visual Studio is running version 1.2.2, then any method which invokes a Roslyn API will throw a MissingMethodException or TypeLoadException before the method is even entered. You can test this by having a safe top-level method which only does “try {f();} catch (Exception ex) {Debug.WriteLine(ex.Message);}”, and then move the proper body into f(). That way you’ll at least find the MissingMethodException.

    If you’re getting a MissingMethodException or TypeLoadException, check you’ve referenced the right assemblies. To run in VS2015 you need to use the older System.Collections.Immutable v1.1.37. To run in Update2 you need to reference v1.2.x and v14.2.x of the NuGet packages.

     

    Viewing all 79 articles
    Browse latest View live


    Latest Images

    <script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>