Setting up a Microsoft build server

What is needed to build .NET code without installing Visual Studio?

The question seems simple enough, but it is not at all so simple. There’s a lot to install on a blank server, to make this happen, especially if you need to build many different .NET Framework versions and project types. It’s also useful information, if you need to setup virtual build servers on the fly. In my next blog post, I will even show you how to setup a Dockerfile and pipeline, that makes it possible to setup a build server with docker. In this post, I assume that you have access to some kind of “blank” Microsoft server, for instance a Microsoft Windows Server 2016.

But first, we need to establish the requirements of a Microsoft build server.

 

Requirements

The requirements for the server is quit simple:

  • Build .NET and .NET Core code without Visual Studio installed
  • Test the code (in this blog, I will use NUnit, as this is the unit test framework I primarily use)
  • Create a build artifact to publish/deploy somewhere (deployment is not described in this post – build examples will be described in a later post)

 

Components to install

So, what should we install and how? I will show you what to install and how to do it using some Powershell commands.

 

Windows features

First thing is to install some basic Windows features:

These features is used by MSBuild to define how an ASP.NET project is build.

 

MSBuild

MSBuild itself comes wrapped in a package from Microsoft called “Build Tools”. There’s a correlation between releases of Visual Studio and MSBuild versions. The reason is that a Visual Studio project file contains references to targets or target files. These target files defines the build process and thereby describes how MSBuild should build a given project file. All these target files are installed when installing Build Tools.

The Microsoft Build Tools are NOT backwards compatible. Microsoft have changed quit a lot in the format and the content of both the target and project files over the years (new project types have also emerged), so if you need to build older Visual Studio project files, then you have to install the corresponding Microsoft Build Tools versions! In the script below, I have chosen to install the three latest versions.

Notice the lines 3, 7 and 11, where the Build Tools is installed.

The first two lines install packages can be installed with the parameters:

  • “/Silent” (run the install in the background and don’t ask for user input)
  • “/Full” (everything is installed).

 

The latest Build Tools package (MSBuild 17 – last line in the script above), has all new parameters:

  • As the package contains a lot of components you wouldn’t normally need (mostly C++ stuff), you can choose only to install the MSBuildTools and the WebBuildTools component by using the “–add” parameter. If you need additional components, here’s a full overview: https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools
    • MSBuildTools contains all the basic stuff you need, including MSBuild and target files, to build .NET projects.
    • WebBuildTools is important, if you need to build web projects (ASP.NET), as it contains the target files referred in those project files.
  • “–passive” is the same as “/Silent”
  • “–norestart” is what it is – don’t restart after installation (we will install a lot of stuff as well in the final install script).

 

.NET Framework Developer/Targeting packs

The .NET Frameworks does not ship with the Microsoft Build Tools packages, so we need to install them seperately. Again, the .NET Framework packages are not backwards compatible. All the .NET Framework versions that you would like to build, needs to be installed on the build server.

We will only be installing Developer/Targeting packs, as these also contain the .NET Framework itself – no need to install the framework files twice. Apart from the .NET Framework itself, the developer packs also contains the .NET SDKs. If they are not installed, you will get some weird build errors when trying to run MSBuild for some projects.

Parameters for all three packages:

  • “/q” as in “Silent”, meaning that it won’t ask for user input during installation.
  • “/norestart”, again the server will not restart after installation.

 

If you have solutions/projects that target older .NET Frameworks, these should of course be installed as well.

 

Optional: Windows SDK

If you need to build WPF applications, then it is important to install the Windows SDK package.

 

.NET Core

This is the new hot stuff in the .NET community – .NET code that builds and runs on nearly all environments, including on Linux based systems. A lot of the new project types that ships with Visual Studio 2017, is build with the .NET Core SDK, also including the new cross-platform and -API .NET Standard projects.

As we will only build and test the code, the .NET Core SDK is what we need. The only thing we need to do is downloading and unpack it to a folder on the file system.

 

NuGet

No build server without NuGet. Just download and unpack to a folder on the file system. This makes it possible to restore (download), pack and add NuGet packages.

 

Optional: NUnit

Download your preferred unit test framework – mine is NUnit.

 

Web Targets

If you need to build ASP.NET (or other Visual Studio web projects), you need the Web.targets package(s). Again again, the package is not backwards compatible, so you need to install the ones you need to build – the versions correlate with the MSBuild versions. 14 should be sufficient for both MSBuild 14 and 17.

Notice two things in the above script:

  1. I “navigate” to the correct MSBuild folder prior to installation and then move the files installed to the correct folder.
  2. We use NuGet to install the packages.

 

System variables

The last thing we need to do on the server, is to set some environment variables, including the PATH variable.

The PATH variable now contains the paths for .NET Core, MSBuild and NuGet. Read more about the PATH environment variable here.

 

We also need to rewrite the MSBuildSDKsPath, to tell how to build .NET Core projects with MSBuild. In this way, we use the .NET Core SDK to build our Visual Studio 2017 projects. We do this by pointing the MSBuildSDKsPath environment variable to where we unpacked the .NET Core SDK. The reason why we do this, is covered in a MSBuild issue at Github.

 

The final install script!

That’s it! This is all that needs to be installed, to create a Microsoft build server.

The full Powershell script for installing your own build server is available below.

 

Next step…

In my next blog posts, I will show you:

  • How you can use the above build server installation to setup a docker build server image.
  • Create a Powershell build script, to build and test your code, and create a deployable artifact.

Indexing files in Solr using Tika

After installing the latest Solr release I noticed that the schema.xml file (the XML-file that holds the information about how Solr should index and search your data) already was setup for something called Apache Tika that can be used to index all sorts of documents and other files that holds all kinds of metadata (audio-files for instance).

See this post to learn how to install Solr on Windows: Installing Apache Solr on Windows

The great thing about Tika is that you don’t need to do anything in order to make it work – Tika does everything for you – or almost everything. You can also set it up so that it suits your needs just by adding a config-XML file in the Solr directory and a reference in the Solr-config XML-file. I tried to get it to work, but found that quit difficult also because there wasn’t much help to get out on Google. Because of all my problems getting it to work I created this post on StackOverflow. Before I got a reply on the subject though, I found the solution myself burried deep inside some forum posts about SolrNet.

This how you do in order to get it to work inside Solr using SolrNet as client.

Install Solr using the link from my earlier post above or something similiar – the main thing is that you install it using the release from Solr that is already build.
Create a new folder called “lib” inside your Solr-install folder.
Copy the apache-solr-cell-3.4.0.jar file from the “dist”-folder from the Solr zip-file to the newly created “lib”-folder the folder where you installed Solr.
Copy the content of contrib\extraction\lib from the Solr-zip to the same newly created “lib”-folder the folder where you installed Solr.
Now Tika is installed in Solr! Remember to go to http://localhost:8080/solr and confirm that it is installed correctly.

To use it in a .NET client application you can use the newest release of SolrNet (currently the 0.4.0.2002 beta version release) and add the DLLs to your .NET project (all of them – seriously!). This is an example of how to use it in C#:

The response will hold all the metadata that has been extracted from the file using Apache Tika. The ExtractParameters is given a FileStream object and an ID for the Solr index (here just “doc1” – can be anything as long as it is unique). The ExtractOnly property can be set to true if you don’t wan’t to get Tika to index the data, but only wan’t it to extract the metadata from the file that is sent. The file is streamed to the Solr API using HTTP POST. You can read more about that here: http://wiki.apache.org/solr/ExtractingRequestHandler

In the above code the data sent to Solr is indexed in the last line, where the data is committed to Solr. If you would like Solr to index and commit the files when sent to the service you can set the AutoCommit property to true inside the initiation of ExtractParameters:

Because the commit is done everytime you send a new file to the Solr-API you can search during the indexing and, of course, you don’t need to call the solr.Commit() method after indexing.

You need a request handler inside your solrconfig.xml (inside {your-solr-install-path}/conf) to make Solr understand the request from the client. Below is an example of how the solrconfig.xml looks when you haven’t changed anything after install of Solr. See this for further information about configuring Tika inside Solr: http://wiki.apache.org/solr/ExtractingRequestHandler

Your Solr schema.xml file (inside {your-solr-install-path}/conf) needs some fields in order to index the metadata from the files you send to Solr. You can provide the fields you need and index/store/ the metadata as is required for the files you need to index. This is the fields that Solr is installed with:

The fields above is a fits-all scenario, so with this you can both index audio and document files.

Supported formats in Tika can be found here:
http://tika.apache.org/1.0/formats.html

Installing Apache Solr on Windows

Apache Solr is a Java-based enterprise search platform build on top of the Apache Lucene search engine (the two of them are now merged). It makes all the great search engine features available through a RESTful API (HTTP/XML and JSON): indexing, full-text search, hit highlighting, faceted search, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search. The best part is that it’s open source and free for all.

 

I have got a lot of know-how about this great tool both for home and business purpose and would like to share, in this blog-post, how you can install Solr on a Windows system (Windows 6 / 7 / 2008 (R2) Server). This guide is written because I had a hard time finding a guide on this subject out there on the web. So here we go.

 

First of all, you need to install a webserver that can run Java-servlets. I use the Apache Tomcat webserver. Download the latest Tomcat server (the MSI installer is perfect for this – Binary Distributions -> Core -> 32-bit/64-bit Windows Service Installer) and install it on your system: http://tomcat.apache.org/download-70.cgi (right now the latest version is 7.0) After this is installed, check that it is correctly installed and running (go to http://localhost:8080/).

 

After you have checked that it is running correctly, go to the directory where you installed Tomcat and then open the server.xml file in the conf folder (conf\server.xml). Inside this you then add this attribute to the first Connector XML-tag (Server -> Service -> Connector): URIEncoding=”UTF-8″.

 

Download and unzip the latest version of Solr into a temporary folder on your system – could be something like ”C:/temp/solr” (I have experienced some problems running version 3.5 on Tomcat – use the 3.4 version for now): http://www.apache.org/dyn/closer.cgi/lucene/solr/

 

Create a folder on your file system where you would like Solr to be installed. Copy the content from the ”C:\temp\solr\example\solr” folder into the folder you just created.

 

Stop the Tomcat service. If you installed using the MSI installer you can do this by going to the Tomcat folder inside All Programs in the start menu and click on ”Configure Tomcat” (you might need to do this by right-clicking on it and choose to ”Run as administrator”). Keep the Tomcat configuration window open after you have stopped the service. We are going to use it later.

 

Copy the *solr*.war file from ”C:\temp\solr\dist” to the webapps folder inside your Tomcat installation folder. The .war file is called apache-solr-3.4.0.war for instance when you have the 3.4 version of Solr. When the file is copied, rename it to ”solr.war”.

 

Now we need to configure Tomcat so that it recognizes the Solr install folder that you created earlier. This is done by adding a Java Option: Open the Tomcat configuration window mentioned earlier and then go to the ”Java” tab. Here you have a ”Java Options” textbox with alot of lines in it. On the bottom of this textbox add the line ”-Dsolr.solr.home={solr-install-folder}”, where {solr-install-folder} is the path to your Solr install folder.

 

In the Tomcat configuration window, start the Tomcat service again. After starting the service, try to open a web-browser and navigate to this site (the local Solr administration site): http://localhost:8080/solr/admin. If the site starts nicely, Solr has been installed on your system.

CSV parser

I’m currently working on a project where I needed a C# console application that was able to read through a Excel CSV (Comma Separated Values) file.

Basically the CSV file format is just a txt file with rows and each column is then separated by a comma (surprise!) or a semicolon. Besides a comma the data in each column can optionally be ”framed” by quotation marks.

Therefore i started out with the following code, just as I would read through a normal txt file:

This is, as you can see, really straight forward. First of all I declare an object of a StreamReader in a using statement. Using the object ”readFile” I am able then to navigate the file. The using statement is important as this will do the cleanup for me, by calling StreamReader.Dispose(), when the statement finishes. I always wrap this kind of code in a try…catch because when you work with files, errors just occasionally happen.

Now, to read the data from the CSV file I add the following lines of code inside the using statement:

It just declares a new List that can hold an array of strings and the line and row variables is needed when traversing through the file. I then use the readFile object to call the ReadLine() method of the StreamReader class in a while loop. When there is no more lines in the file the line variable will be null. Inside the while loop I use the string.Split() method to split the line into an array of strings (my columns) and I then add this array to my List object (parsedData).

The problem then was that I didn’t know exactly what encoding the file would be in. What to do then? I settled on a solution where I tell the StreamReader what encoding the file probably has and it will then open it in that encoding. This can be done by adding a parameter when calling the constructor on the StreamReader class like this:
using (StreamReader readFile = new StreamReader(path, encoding))

Finally all this can be wrapped in a nice method. I also added a check to be sure that the file I want to parse is actually available. But there you go:

WCF-services in a Windows service

Some time ago I ran into one of those problems with no obvious solution. I was on a project where I needed to use a WCF-service for a Silverlight solution. First I started out by making a service that was hosted on the IIS. That was working fine with a connection to a database, but the service was also going to open up some physical files in a specific folder on the Windows server where it was hosted. This could not be done as the service didn’t have access permissions to a folder outside of those under the service. So what to do then?

I searched for a solution to the problem and one that I found was to use Windows impersonation in the service. This “simulates” a user logged in on the server with the rights given to that user. For me this wasn’t an optimal solution for a number of reasons, first of all because it didn’t seem very secure. I quickly started to search for another way to cope with the problem.

The solution I came up with was this: I realized that as I had administrator rights to the server I could host my WCF-services in a Windows service and install it as such. In this way you can run the WCF-service outside of the IIS and run multiple services in the same Windows service as well. Another great thing about it is that if you install it right (as I will show below) you can get it to have access to the file system and it will run as a service under Windows. The example below shows you how the constructor in a Windows service can look like:

What happens here is that I provide the name the service has (in Windows this will be the name of the service) and if it should use the application log under Windows to log in. I also tell it that it should log automatically to the application event log when something happens with the service and I tell it that it’s okay to stop, shutdown and pause and continue among others.

The code example below shows the Main-method of the service. As with any other Windows application this has to be provided as the starting point of the application.

Here I make the service run. Remember the try-catch block because if something bad happens when the service is initialized this will not make everything crash. Also notice the inheritance from the ServiceBase-class. This is what makes our class a service and is needed when we’re going to install it later also and make it run. To make the service do something in certain situations before Windows service events is fired, when it starts, stops, continues, pauses or shuts down (if the server it resides under for instance shuts down), you can override the OnStart, OnStop, OnContinue, OnPause or OnShutdown methods respectively. Up until know I haven’t shown how I combine the Windows and WCF-services, but my next code examples shows just that. What you need first of all is to make your WCF-services start when the Windows service starts. This is done by overriding the OnStart method as mentioned above and then inside this hosting the WCF-services in some service hosts and open them. I learned that a good way to be able to control every service host with the WCF-service inside is to have it declared as variables inside the class. An example on this is provided below.

In my OnStart method I do the following:

First of all for every WCF-service I find out whether it has been initialized before. If it has, I close it so I don’t get an exception thrown when I try to open a service that is already open. Afterwards I declare the service host given the type of WCF-service that it should open and open it. As with everything else I pack it all inside a try-catch block to prevent the Windows service from crashing.

Finally you need to provide the normal configuration settings for your WCF-services inside an App.config file in the Windows service class library. The App.config file can look like the one below:

There you have it! The service works fine like this and if you have provided the correct base address the service can be called from there (remember that what you provide as address in the baseAddress property as the full address for the service – so when you call it you don’t need the .svc prefix). Tip: You can provide multiple endpoints if you need multiple bindings.

The service worked great until I started to test it with the Silverlight application and realized that the service, of course, needed a clientaccesspolicy.xml file to have the right access rights for the WCF-service, but where do you put this when you don’t have your service on the IIS and you can’t just put the access policy file in the root directory? The solution for this is to understand how the Silverlight application asks for the clientaccesspolicy.xml file when calling the WCF-service. When a Silverlight application calls the service in another domain it automatically assumes that the clientaccesspolicy file is located in the root of the domain where the service resides (the base address provided in the configuration file without the service name). If it isn’t there it gives you the standard 404 error (”File not found” or something like that) when you try to call the service from the application. So if you can ”broadcast” a clientaccesspolicy.xml file on the root address this is the solution, but how to do that? The solution is to add a new WCF-service that, by using the HTTP GET-protocol stream the xml-file as a message. The interface for the new service should look like below. The UriTemplate-property of the WebGetAttribute tells what the URI should be. In our example it is the name of the clientaccesspolicy file.

Then, in the implementation of the service interface, you just open and read the clientaccesspolicy.xml file as a stream, load it into a StringReader, add this to a XmlReader, and make a new instance System.ServiceModel.Channels.Message, add your Xmlreader object to the message and return it.

There you have it! When you now call your service from a Silverlight application, you will see that it gets the clientaccesspolicy.xml from the service.

After my service is done I can install it using a Windows installer or a basic console application.

Installing a Windows service

To install a service you need to use the ServiceProcessInstaller and the ServiceInstaller classes. These handle the installation of the Windows service as a process with the provided information. The Account-property on the ServiceProcessInstaller is used to tell what security context the service should run under when installed. I use LocalSystem here because I need my service to have access to my file system, but here you have to choose what best fits your situation of course. The username and password properties are used to tell what user the service should run as. Next for the properties of the ServiceInstaller you can specify how and when your service is started with the StartType-property and you can set the service name, display name, a description and then you need to assign the ServiceProcessInstaller instance as parent to the ServiceInstaller instance. Then you need to specify the context in which the service should be installed: the full path to the executable that holds the service as a commandline and a path to a log file if needed. Then you install it by calling the Install-method on the ServiceInstaller instance. Then, because we need to start up the service, we takes control of the installed service with the ServiceController-class and call the Start-method. That’s it! Your service is installed 🙂

 

If you need to uninstall your service to update it for instance (that’s right, the service executable cannot be altered as long as the service is installed), you can use the below code to do just that. It should be quite straight forward. You just take control of the service as in the code above, figure out if the service is running, if it’s running then it has to be stopped before we can uninstall it, wait for it to stop and then gives the need context-information and then we uninstall by calling Uninstall on our ServiceInstaller object.