Author: thoean

Logging an exception and structured message data with ASP.NET 5

Once you started using ASP.NET 5 (aka. ASP.NET vNext), it’s surprisingly difficult to add a message that contains structured data to an exception. The LoggerExtensions either supports a flat message (which is probably good for a lot of cases), but it doesn’t support structured data for more sophisticated analysis.

Workaround

Let’s quickly look into a simple workaround, which is a 2-liner instead of a one-liner:

Solution

The logging extensions only support structured log messages when FormattedLogValues are provided, but don’t provide an interface to do the formatting behind the scenes. This is what I actually want to do:

I have created a pull request to the aspnet/Logging repository, but as of late October 2015, this wasn’t accepted. For the time being, you can simply copy/paste my changes to your project and start using the simplified logging capabilities.

Background

Why am I so obsessed with structured data in messages? Probably simply because I got used to it. I started using Elasticsearch as a logging sink over a year ago, and I mostly look into analyzing fields instead of full text messages. The detailed story is long, and probably worth a few posts, but it’s mostly around analyzing numeric data like performance data, or in the case of errors, by looking into categories of errors (outside of a hard to read error numbers).

Service level tests with the .NET Execution Environment (DNX)

While writing a fairly simple RESTful service with ASP.NET 5, I started to leverage the ease of self-hosting with the .NET Execution Environment (DNX) by executing service-level tests as part of the automated build. I use Powershell for the build script. I first publish the service to a temporary directory, start it in a background thread, and perform the actual tests.

With these few lines of code I am able to validate the service’s health after every commit, and notably running these set of tests as part of a pull-request build before being merged to the main line.

One example where I used such a build script is the xUnit2NUnit web service. The source code can be found on github, and a description in a separate blog post.

Automated build for ASP.NET vNext hangs

I had an automated build with ASP.NET vNext correctly working for a few weeks until it suddenly stopped working. It was hanging with the last output message at:

removing Process DNX_HOME

My powershell script started with something like the following:

Internet search pointed me to an issue on ASP.NET and a few other sources, but double-installation or spaces in usernames weren’t my problem.

Running this set of commands works very nice when an environment is installed, it downloads the dnvm scripts, it lists the currently installed environments, and eventually ensures the required environment gets installed if not already available.

My build server went through some auto-cleanup process and had therefore no environments installed. If that’s the case, dnvm list will start in interactive mode, asking the user “It looks like you don’t have any runtimes installed. Do you want us to install a runtime to get you started?“. That prompt didn’t show up in my automated build output (using Jenkins), so it was hard to troubleshoot, especially since I didn’t expect any interactive mode in the dnvm commands.

Once found, the solution was really simple by just removing dnvm list, or if that’s interesting information as part of the build script, to move it after dnvm install.

Wedding in June 2015

I rarely take the camera to shoot a wedding, but through friends or relatives I sometimes end up in that situation. I’m not an expert in that topic, but I’m still satisfied with the end result.

I printed the album at Albelli with an astonishing quality for the low price.

Learning Spanish

This post is about learning to speak a language using Spanish as an example.

I’m convinced that I did quite well learning German (my mother tongue) while I was not even attending school. I then completely screwed up with English as a foreign language and ended up as a I-can-do-grammar-but-cannot-understand-you guy after more than 10 years of education (to be honest, most others did not better as well).
So I decided to look for a better method when learning another languages – this time I started with Spanish. Eventually I decided to try the Pimsleur Method as my alternative way of learning – in combination of total immersion (see these 10 points to successful language learning). Mainly I liked the approach being a lot closer to how I learned my mother tongue – speaking and understanding comes first, grammar second. Eventually I found a great opportunity to combine my understanding of learning languages and a language school and took some time off to attend an intensive course at Habla Ya in Boquete, Panama.
Below are listed a few resources – some based on the Pimsleur Method, others are some great podcasts to improve understanding of Spanish in daily life.

Pimsleur foreign language program

Best for absolute beginners. The Pimsleur Method from the guy who invented it. Expect to learn a few new words every lesson. These words and phrases are nicely brought up again a few lessons later for reminder (http://www.pimsleur.com/Learn-Spanish).

Learning Spanish Like Crazy

Similar to the Pimsleur method, but a lot more natural speaking. Also it’s not so topic-oriented, but more grammar-oriented. I’d go for that as a nice addition for the Pimsleur, but it also works for total beginners when starting at the first level – definitely a good joice as well (http://www.learningspanishlikecrazy.com/).

Notes in Spanish

The guys from Notes in Spanish have some nice free audio podcasts. Lessons of around 10mins each. Available for beginners, intermediates and advanced speakers (http://www.notesinspanish.com/).

Podclub (from Migros Klubschule)

Nice bi-weekly podcasts from Migros Klubschule. Registration is required, but it’s then free to download all podcasts from Spanish and other languages (http://www.podclub.ch/).

Wedding in August 2012

This time, I was the selected victim of getting married ;-). I was fortunate enough that one of my best friends from university became a professional wedding photographer and created a set of memorable pictures of this weekend.

Nina Hintringer and Philipp Stampfl, acting under the brand “Nina Hintringer Photography“, titled this weekend “Mexican color explosion in Switzerland”. Her blog post and some pictures can be found here.

XP, Scrum and Kanban

There are various processes and philosophies around to manage project, especially software projects, successfully. Managers can choose from a set of traditional approaches, such as Waterfall, or following some kind of agile methodology such as XP, Scrum or Kanban.

The past

Extreme Programming (aka XP) has been very popular around the year 2000. It’s focus lies on describing engineering practices for software engineers like pair programming or test driven development. As they are limited to the software engineering department and were technical details on internal processes there was a big ideological gap to management or the customer. Outsiders understood pair programming as waste and test driven development as a strange approach.

Now

Scrum was mainly introduced to prevent the disconnect between technical people and business owners or customers. Scum does therefore not focus on any technical details, but rather the process. It’s core is set around standardizing communication with the business owner, setting expectations right for all involved people and increasing motivation of developers.
Following the Scrum ideology does not prevent to use ideas from XP. It must rather be seen as an addition. Poeple often refer to this as Scrum-but. Meaning: “I’m doing Scrum, but…”.

The future

XP and Scrum are a great combination to prevent common shortcomings of more traditional project management approaches. However, Scrum has a very fixed process definition, but does not include optimization of the process itself, as the process is taken as a fixed constant.

This is where Kanban comes into place. Kanban is not new, it’s idea is well established in manufacturing processes for years already and part of the Lean Thinking philosophy.

Kanban is about visualization. And it’s about changing the process. Visualization is taken as core instrument to see the current process and it’s bottlenecks. Those bottlenecks (and therefore problems of the process) trigger a constant refinement and improvement of the process itself.

In software development a good way to start with Kanban is visualizing the current work in progress. The team shares one common board, which is split into parts. Each part refers to a status (e.g. awaiting requirements, development, test, rollout) and each status has a limit of work items assigned at any given time. Each work item gets a sticky note, which is placed on the board at it’s current status. If one stage gets too many work items, the team must work in order to resolve the conflict.

Initial thoughts might be, that this process prevents developers to work on many items at the same time and brings down productivity. But it’s not about productivity, it’s about delivery and creating value for the customer. Creating value is usually achieved in constant delivery and a short throughput time. Which itself is achieved by bringing down the total amount of work in progress.

Agile software development

The general recommendation of being agile is… to be agile:

  • Change your process whenever you think it can be improved.
  • Visualize current status to reveal problems.
  • Measure changes to see impact of improvements.
  • Limit work in progress and focus on delivering value instead of increasing productivity.
  • Use a mix of agile methods from XP, Scrum and Kanban and adapt them to match your and your teams needs.

References

The following links should give a good start into agile software development and lean philosophy:

Facebook: Add Social Plugins

Facebook makes it fool proved to add social plugins to your website. Basically a thing done in minutes:

  1. Register your website as an application on facebook. This can be done at http://developers.facebook.com/setup/, the only thing required is a name and the site’s URL.
  2. Place a javascript onto every page you’re going to add social plugins. The javascript can be found at http://developers.facebook.com/docs/reference/javascript/. You only need to add your application-id as it has been created in step 1.
  3. The only thing left is to place a facebook-tag on each of the pages where you want to add the like-box, recommend-box or comment feature. For the like-button it’s as simple as: <fb:like></fb:like>.

Upgrade to JSF 2.x and Webflow 2.1.x

Generally, the upgrade could be easy, but still took me a while to get there. So here’s a summary of my changes to the existing project.

Update References (e.g. with Maven)

First, add the references to JSF 2.0 (in my case I used Apache MyFaces 2.x). With maven it looks like this:

Additionally (I think it might not be necessary) I upgraded to the latest facelets release (version 1.1.15):

Now it’s also time to upgrade the spring versions (you might also need to upgrade further dependencies such as Spring Security, Spring ORM etc. Make sure that at the end you don’t have any 2.x references left of any of your dependencies or sub dependencies:

And eventually the upgrade to Spring Webflow 2.1.x:

This was actually all to get it running again. However, see below some further information on troubleshooting.

Eclipse: JavaServer Faces Project Facet

First I had eclipse running with the Project Facet configured for JavaServer Faces 1.2. However, I was not able to add that for Faces 2.0, thus I removed it completely. Now I take care of the distribution of the faces-jars by myself via dependency to these libraries in the project itself (see maven configurations above).

If you forget the references to the faces libraries (or actually if they are not available to your application server), you’re most probably running into the following exception:
code>javax.faces.FacesException: Undefined component type javax.faces.ViewRoot

Recommended Reading

Secure Data Backup with OpenVPN and TrueCrypt

Note: This article mentions TrueCrypt, which has since been discontinued. The content of this article might still be interesting conceptually, but please use alternative storage options to securely store your files.

 

Backing up data securely, mainly for private persons, is always a hassle. Having an external disk still does not prevent data loss in case of a robbery into the house or while taking the disk and the laptop out of the house at the same time… the usual paranoia.

However, fast internet connections allow backup to internet services (there are tons of online backup solutions available) or to other private persons. Taking the 2nd the issue is usually (1) how do I securely get my data to the other person’s computer and (2) how do I make sure my data are stored securely.

Secure Backup – Tools

My solution uses quite a simple, but very effective approach. Taking OpenVPN as secure connection over internet, having robocopy (or a similar sync-tool) synchronizing the files and storing the backup files in a TrueCrypt container.

OpenVPN – Setup over a broadband connection

Generally, the installation guidelines can be found in the HowTo-Section of the OpenVPN Website. However, sometimes it’s easier to have shorter guidelines as described in the following procedure:

  1. Download OpenVPN (Community Edition) from the OpenVPN Website.
  2. Install the OpenVPN software on both computers.
  3. Decide one computer to be the server, the other the client to connect to the server.
  4. Now we need to create the keys. OpenVPN provides a easy-rsa directory in the installation folder. To do so, call the following .bat files:
    • init-config.bat
    • Now edit vars.bat and create a sub-directory called keys (also make sure the KEY_DIR in vars.bat has the name keys as well).
    • clean-all.bat
    • build-ca.bat (creates the CA certificate)
    • build-dh.bat
    • build-key-server.bat <MachineName> (builds the private key/certificate for the server)
    • build-key <MachineName> (builds the key files for the client, run this for every client you wish to connect to the server)
    • Eventually copy the following files to each of the clients (over a secure channel!): ca.crt, <MachineName>.crt and <MachineName>.key
  5. In the installation folder there’s a directory called sample-config with same example configurations for server or client respectively. Copy the client.ovpn for the client-computer and the server.ovpn for the server-computer to the config directory of the installation folder.
  6. Open the ovpn in a text editor and adjust it. Mainly the values for ca, cert, and key need to be adjusted on both, the server and the client configs.
    Example for ca.crt would be:
    ca “C:\\Program Files\\OpenVPN\\easy-rsa\\keys\\ca.crt”
  7. The server has the setting for dh (adjust this to point to the dh*.pem file created before) and server. Server specifies the VPNs private network. Make sure you use another network than the server or any clients are in (usually 192.168.0.x or 192.168.1.x). So take e.g. 192.168.123.0 with a subnet of 255.255.255.0. A correct value for the server parameter would then be:
    server 192.168.123.0 255.255.255.0
  8. Now access your broadband router (of the server’s computer). You need to configure the routing from the external public port to the computer/port where the OpenVPN server is running. Usually it’s called “virtual server” or “DMZ” in the router settings. There add the public port 1194 (standard) and set the internal computer/port to your server and Port 1194.
  9. The client additionally has the setting for remote. Set the Public-IP and Port of the server. Easiest to find that out is to go to www.whatismyip.com from the server’s computer. In case of dynamic IP-Addresses, this config would need to be adjusted before every connection or a service like dyndns.org can be used.
  10. Now start the server (either the windows service if you’ve installed it as a service; otherwise the openvpn-gui.exe is a handy tool).
  11. Now start the client. It should connect and you’re in the same network as the server (which you should be able to ping). If not, try to test the command “telnet <remote-ip-address> <remote-port>”. If it connects, then the server is running correctly and the error will be in the client-config. If it does not connect, the server is not yet running correctly.

TrueCrypt and Robocopy – simple but efficient backup

As mentioned before, the data are “out of the house”, so storing them securely is highly recommended. There are tons of disk-encryption tools on the market. Due to ease of use and it’s high reputation I’m recommending TrueCrypt (open source software and therefore free to use).

Same is for copying. Robocopy, a tool in the Windows 2003 server tools of Microsoft, is easy and free to use as well.

  1. Download TrueCrypt and Robocopy.
  2. Create a TrueCrypt container using the GUI Wizard on the disk where you’ll store your backup files. Keep the passwords or keyfiles used secretly to you.
  3. Connect to the OpenVPN server as described above and make sure the VPN connection is correctly working.
  4. Mount the network share on the OpenVPN server (or actually any computer that’s in the same VPN) where the TrueCrypt container is located.
  5. Mount the TrueCrypt Container. Note, that this is only mounted locally on your computer and nobody on the other computer is able to view your data! Only the encrypted (and therefore useless) TrueCrypt Container.
  6. Synchronize your local files to the mounted TrueCrypt Container.
    The command I’m usually using is /E and /PURGE. This synchronizes two file systems, so also deletes unnecessary files on the target computer.
    robocopy.exe C:\MyFiles X:\backup\ * /E /PURGE /R:3 /W:3 /NP /LOG:backup.txt
  7. When done, unmount the TrueCrypt Container.
  8. Disconnect from the VPN connection.