Category: Software development | Page 2

Using Nginx on Docker to redirect HTTP to HTTPS

I had a website running using HTTPS behind a load balancer, and didn’t want to bother setting up HTTP as well. Instead, I configured the load balancer to point to a very simple Nginx webserver that does nothing else than redirecting HTTP to HTTPS. So from the application side I only had to take care of HTTPS and could ignore additional configuration. As a nice side-effect, the Nginx redirection is generic so that I only need to run a single instance for all my applications.

Since I don’t need anything else than Nginx on the Docker image, I used Alpine Linux as a base and added Nginx, or more precisely the preconfigured Nginx alpine-stable docker image from https://hub.docker.com/_/nginx/. The Dockerfile looks like the following:

FROM nginx:alpine

EXPOSE 80
COPY nginx.conf /etc/nginx/nginx.conf

FROM nginx:alpine

EXPOSE 80

COPY nginx.conf /etc/nginx/nginx.conf

And the related nginx.conf file, which gets copied when the docker image is created like this:

events {

}
http {
  server {
    listen       80;
    location / {
      return 301 https://$host$request_uri;
    }
  }
}

events {

}

http {

server {

listen 80;

location / {

return 301 https://$host$request_uri;

}

Assuming the Dockerfile and nginx.conf are in the same directory, a simple docker build command creates the docker image which can be loaded into your docker host. Writing a simple script to include this step in your build automation should be fairly trivial, depending on your needs.

LambdaWrap blog post at LifeInVistaprint

Recently, I’ve worked on LambdaWrap as part of deploying an AWS Lambda based microservice. LambdaWrap is an open source ruby gem that allows to easily publish Lambda functions and its associated API Gateway configuration during automated deployment when using rakefiles.

I got the opportunity to write a larger blog post than this note at LiveInVistaprint. The blog post “LambdaWrap, a Ruby GEM for AWS Lambda” is available at http://lifeinvistaprint.com/techblog/lambdawrap-ruby-gem-aws-lambda/.

xUnit2NUnit web service

There are various easy ways to convert one XML file into another XML file, usually through an XSL transformation. Most languages support this with little code. For example, in C# it could be something like this (simplified):

// load the XSL transform
var transform = new XslCompiledTransform();
var xmlReader = new XmlTextReader(streamFromXslFile);
transform.Load(xmlReader);

// load the input XML file
var reader = new StringReader(inputXmlData);
var doc = new XPathDocument(reader);

// create a writer for the destination XML data
var stringWriter = new StringWriter();
var writer = new XmlTextWriter(stringWriter);

// perform the transformation
transform.Transform(doc, null, writer);

// load the XSL transform

var transform = new XslCompiledTransform();

var xmlReader = new XmlTextReader(streamFromXslFile);

transform.Load(xmlReader);

// load the input XML file

var reader = new StringReader(inputXmlData);

var doc = new XPathDocument(reader);

// create a writer for the destination XML data

var stringWriter = new StringWriter();

var writer = new XmlTextWriter(stringWriter);

// perform the transformation

transform.Transform(doc, null, writer);

However, I decided to create a simple web service to achieve a few things. For one, to demonstrate that such repetitive code can be wrapped into a service and accessed from everywhere, for example as part of build scripts. Other reasons include that I wanted to play with ASP.NET 5, experimenting with more complex options than just request-response despite the simple use case and host the service on an Azure Web App.

The current version of the service can be found on github at https://github.com/thoean/xUnit2NUnit. It builds with AppVeyor and as part of the build, I start the web service in the background and run a very basic service level test against it (I’ve written a dedicated blog post on service level testing recently).

Logging an exception and structured message data with ASP.NET 5

Once you started using ASP.NET 5 (aka. ASP.NET vNext), it’s surprisingly difficult to add a message that contains structured data to an exception. The LoggerExtensions either supports a flat message (which is probably good for a lot of cases), but it doesn’t support structured data for more sophisticated analysis.

Workaround

Let’s quickly look into a simple workaround, which is a 2-liner instead of a one-liner:

var values = new FormattedLogValues("Error during operation {Operation}..", myOperation);
logger.LogError(values, ex);

1 2	var values = new FormattedLogValues("Error during operation {Operation}..", myOperation); logger.LogError(values, ex);

Solution

The logging extensions only support structured log messages when FormattedLogValues are provided, but don’t provide an interface to do the formatting behind the scenes. This is what I actually want to do:

logger.LogError(ex, "Error during operation {Operation}..", myOperation);

1	logger.LogError(ex, "Error during operation {Operation}..", myOperation);

I have created a pull request to the aspnet/Logging repository, but as of late October 2015, this wasn’t accepted. For the time being, you can simply copy/paste my changes to your project and start using the simplified logging capabilities.

Background

Why am I so obsessed with structured data in messages? Probably simply because I got used to it. I started using Elasticsearch as a logging sink over a year ago, and I mostly look into analyzing fields instead of full text messages. The detailed story is long, and probably worth a few posts, but it’s mostly around analyzing numeric data like performance data, or in the case of errors, by looking into categories of errors (outside of a hard to read error numbers).

Service level tests with the .NET Execution Environment (DNX)

While writing a fairly simple RESTful service with ASP.NET 5, I started to leverage the ease of self-hosting with the .NET Execution Environment (DNX) by executing service-level tests as part of the automated build. I use Powershell for the build script. I first publish the service to a temporary directory, start it in a background thread, and perform the actual tests.

function Invoke-Build {
    dnu publish .\src\Service\project.json --out .\tmp\service --no-source --configuration Release --quiet
}

function Run-ServiceTests {
    # start the service in the background
    $cwd = Get-Location
    Start-Job -Name MyServiceName -ScriptBlock ({param($path)
        Set-Location -Path $path
        .\tmp\service\web.cmd
    }) -ArgumentList $cwd

    # wait until the service is started; instead of waiting for a random time, ping the service until it starts responding
    Start-Sleep -Seconds 5

    # run the service tests
    dnx -p .\test\ServiceTests\project.json test

    # stop the job and also write the output
    Get-Job -Name LabelPrintingService | Stop-Job
    Get-Job -Name LabelPrintingService | Receive-Job
}

Invoke-Build
Run-ServiceTests

function Invoke-Build {

dnu publish .\src\Service\project.json --out .\tmp\service --no-source --configuration Release --quiet

}

function Run-ServiceTests {

# start the service in the background

$cwd = Get-Location

Start-Job -Name MyServiceName -ScriptBlock ({param($path)

Set-Location -Path $path

.\tmp\service\web.cmd

}) -ArgumentList $cwd

# wait until the service is started; instead of waiting for a random time, ping the service until it starts responding

Start-Sleep -Seconds 5

# run the service tests

dnx -p .\test\ServiceTests\project.json test

# stop the job and also write the output

Get-Job -Name LabelPrintingService | Stop-Job

Get-Job -Name LabelPrintingService | Receive-Job

}

Invoke-Build

Run-ServiceTests

With these few lines of code I am able to validate the service’s health after every commit, and notably running these set of tests as part of a pull-request build before being merged to the main line.

One example where I used such a build script is the xUnit2NUnit web service. The source code can be found on github, and a description in a separate blog post.

Automated build for ASP.NET vNext hangs

I had an automated build with ASP.NET vNext correctly working for a few weeks until it suddenly stopped working. It was hanging with the last output message at:

removing Process DNX_HOME

My powershell script started with something like the following:

&{$Branch='dev';iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/aspnet/Home/dev/dnvminstall.ps1'))}
dnvm list
dnvm install 1.0.0-beta7 -arch x64 -runtime clr

&{$Branch='dev';iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/aspnet/Home/dev/dnvminstall.ps1'))}

dnvm list

dnvm install 1.0.0-beta7 -arch x64 -runtime clr

Internet search pointed me to an issue on ASP.NET and a few other sources, but double-installation or spaces in usernames weren’t my problem.

Running this set of commands works very nice when an environment is installed, it downloads the dnvm scripts, it lists the currently installed environments, and eventually ensures the required environment gets installed if not already available.

My build server went through some auto-cleanup process and had therefore no environments installed. If that’s the case, dnvm list will start in interactive mode, asking the user “It looks like you don’t have any runtimes installed. Do you want us to install a runtime to get you started?“. That prompt didn’t show up in my automated build output (using Jenkins), so it was hard to troubleshoot, especially since I didn’t expect any interactive mode in the dnvm commands.

Once found, the solution was really simple by just removing dnvm list, or if that’s interesting information as part of the build script, to move it after dnvm install.

Facebook: Add Social Plugins

Facebook makes it fool proved to add social plugins to your website. Basically a thing done in minutes:

Register your website as an application on facebook. This can be done at http://developers.facebook.com/setup/, the only thing required is a name and the site’s URL.
Place a javascript onto every page you’re going to add social plugins. The javascript can be found at http://developers.facebook.com/docs/reference/javascript/. You only need to add your application-id as it has been created in step 1.
The only thing left is to place a facebook-tag on each of the pages where you want to add the like-box, recommend-box or comment feature. For the like-button it’s as simple as: <fb:like></fb:like>.

Upgrade to JSF 2.x and Webflow 2.1.x

Generally, the upgrade could be easy, but still took me a while to get there. So here’s a summary of my changes to the existing project.

Update References (e.g. with Maven)

First, add the references to JSF 2.0 (in my case I used Apache MyFaces 2.x). With maven it looks like this:

<dependency>
  <groupId>org.apache.myfaces.core</groupId>
  <artifactId>myfaces-api</artifactId>
  <version>2.0.0</version>
</dependency>
<dependency>
  <groupId>org.apache.myfaces.core</groupId>
  <artifactId>myfaces-impl</artifactId>
  <version>2.0.0</version>
</dependency>

<groupId>org.apache.myfaces.core</groupId>

<artifactId>myfaces-api</artifactId>

</dependency>

<groupId>org.apache.myfaces.core</groupId>

<artifactId>myfaces-impl</artifactId>

</dependency>

Additionally (I think it might not be necessary) I upgraded to the latest facelets release (version 1.1.15):

<dependency>
  <groupId>com.sun.facelets</groupId>
  <artifactId>jsf-facelets</artifactId>
  <version>1.1.15</version>
</dependency>

<groupId>com.sun.facelets</groupId>

<artifactId>jsf-facelets</artifactId>

</dependency>

Now it’s also time to upgrade the spring versions (you might also need to upgrade further dependencies such as Spring Security, Spring ORM etc. Make sure that at the end you don’t have any 2.x references left of any of your dependencies or sub dependencies:

<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>org.springframework.web</artifactId>
  <version>3.0.4.RELEASE</version>
</dependency>

<groupId>org.springframework</groupId>

<artifactId>org.springframework.web</artifactId>

<version>3.0.4.RELEASE</version>

</dependency>

And eventually the upgrade to Spring Webflow 2.1.x:

<dependency>
  <groupId>org.springframework.webflow</groupId>
  <artifactId>org.springframework.faces</artifactId>
  <version>2.1.1.RELEASE</version>
</dependency>

<groupId>org.springframework.webflow</groupId>

<artifactId>org.springframework.faces</artifactId>

<version>2.1.1.RELEASE</version>

</dependency>

This was actually all to get it running again. However, see below some further information on troubleshooting.

Eclipse: JavaServer Faces Project Facet

First I had eclipse running with the Project Facet configured for JavaServer Faces 1.2. However, I was not able to add that for Faces 2.0, thus I removed it completely. Now I take care of the distribution of the faces-jars by myself via dependency to these libraries in the project itself (see maven configurations above).

If you forget the references to the faces libraries (or actually if they are not available to your application server), you’re most probably running into the following exception:
code>javax.faces.FacesException: Undefined component type javax.faces.ViewRoot

Multiple Authentication Provider with Spring Security

Nowadays, websites need to provide multiple login options such as a custom login, LDAP login, by facebook connect or openID. For this purpose, Spring Security allows to set up multiple authentication providers.

The basic setup is easy, just add additional authentication providers:

<security:authentication-manager alias="authenticationManager">
  <security:authentication-provider user-service-ref="userServiceInternal">
    <security:password-encoder hash="md5" />
  </security:authentication-provider>
  <security:authentication-provider user-service-ref="userServiceDb">
    <security:password-encoder hash="plaintext" />
  </security:authentication-provider>
  <security:authentication-provider ref="authenticationProviderFacebook">
  </security:authentication-provider>
</security:authentication-manager>

<security:authentication-manager alias="authenticationManager">

<security:authentication-provider user-service-ref="userServiceInternal">

<security:password-encoder hash="md5" />

</security:authentication-provider>

<security:authentication-provider user-service-ref="userServiceDb">

<security:password-encoder hash="plaintext" />

</security:authentication-provider>

<security:authentication-provider ref="authenticationProviderFacebook">

</security:authentication-provider>

</security:authentication-manager>

With a custom authentication provider such as the facebook authentication provider, it is sometimes required to setup a custom filter. Reason for this is, that the standard user service checks for the request parameters j_username and j_password. With facebook connect, those parameters are not sent, so some other parameters must be checked, such as an URL or similar.

<security:http auto-config="true">
  <security:form-login
    login-page="/login"
    login-processing-url="/loginProcess"
    default-target-url="/inbox"
    authentication-failure-url="/login?login_error=1" />
  <security:logout
    logout-url="/pages/logout"
    logout-success-url="/logoutSuccess" />
  <security:custom-filter
    before="FORM_LOGIN_FILTER"
    ref="facebookAuthenticationFilter" />
</security:http>

<security:http auto-config="true">

<security:form-login

default-target-url="/inbox"

authentication-failure-url="/login?login_error=1" />

<security:logout

logout-url="/pages/logout"

logout-success-url="/logoutSuccess" />

<security:custom-filter

before="FORM_LOGIN_FILTER"

ref="facebookAuthenticationFilter" />

</security:http>

The facebookAuthenticationFilter is a custom class, best to be extending AbstractAuthenticationProcessingFilter with the defaultFilterProcessUrl set to something like /j_spring_facebook_security_check.

Then change the on-login javascript code of the facebook connect button to redirect to the above specified URL:

function facebook_onlogin() {
  document.location.href = "/j_spring_facebook_security_check";
}

function facebook_onlogin() {

document.location.href = "/j_spring_facebook_security_check";

}

Once the users clicks on the facebook connect button, the typical facebook connect user interface pops up. Upon successful login, the page is redirected to the facebook login URL, from there automatically to the standard target page.

Facebook Connect Login with Spring Security

Kadir Pekel already described a procedure to integrate Facebook Connect in a blog entry. So just a quick summary on my experiences:

Dependency on Facebook Java API

Add a dependency to the Facebook Java API. Easy done with Maven:

<dependency>
  <groupId>com.google.code.facebookapi</groupId>
  <artifactId>facebook-java-api</artifactId>
  <version>2.1.1</version>
</dependency>

<groupId>com.google.code.facebookapi</groupId>

<artifactId>facebook-java-api</artifactId>

</dependency>

Spring Security Facebook

Download the code for spring-security-facebook and either integrate that into your project or build it into an own jar.

Spring Config and HTML

The only missing thing is to configure the Facebook Security code in the Spring configuration files and the according HTML code.

Adjust the Spring Config to have an additional authentication provider.
The Facebook On-Login function must redirect to the configured authentication URL, e.g. /j_spring_security_facebook.

Details on adding these things can be found in my description on Multiple Authentication Providers.

Workaround

Solution

Background

Update References (e.g. with Maven)

Eclipse: JavaServer Faces Project Facet

Recommended Reading

Dependency on Facebook Java API

Spring Security Facebook

Spring Config and HTML