AWS ElasticSearch anonymous access from a VPC

It’s a bit tricky to find how to configure anonymous access to AWS ElasticSearch from a VPC. Especially when you start from this AWS support article and dive deep into configuring IP-based access and policy conditions.

Even if you look at the AWS Console, there are pre-configured access policies. None of them saying “Enable anonymous access” or related hints.

And at the same time it’s too obvious.

On the UI, it’s saying Do not require signing request with IAM credential.

That’s it. Click this. Apply without modification. There are no IP restrictions required or even possible from within the VPC for anonymous access. (Which is different for public AWS ElasticSearch domains.)

And if you’re already using CloudFormation or other forms of scripts, here’s the short version as access policy:

A calm place

Or the importance of operational excellence

When I joined my current team last year, there was a lot of excitement going on. First, the schedule was determined by people outside of the team. Almost everything being built was based on ad-hoc, seemingly urgent support requests. The team was busy rushing features and bug fixes out to get through the backlog, which only kept increasing. Second, the on-call rotation was being passed through the whole team of almost 20 engineers. The weekly rotation hit you roughly twice a year. It meant sleepness nights, waking up multiple times. But it was over soon, and your motivation to fix things dropped to zero – merely hoping the situation will improve until your next turn.

It was time to challenge the status-quo. Persistently asking why we woke up people just to snooze an alert for a few hours. Why we had little ownership. Why the motivation was low to fix the root cause.

I challenged myself to measure my contribution by my very own, subjectively measured “team calmess index”. It started with giving support. Standing behind bold decision to only alert during office hours for many cases. Removing the large team-wide responsibilities and make them specific to a sub-group. And also finding time to fix some of the underlying system’s errors. Within weeks, my subjective “team calmess index” improved a lot. Without a change in technology, without a change in skills. Only the right management support, focus and ownership.

It has become a calm place. As the numbers tell.

Number of incidents per week with 3 month moving average.

Promotion by PowerPoint

Or how to avoid killing people with your slides

Earlier this year I came across the blog post Death by PowerPoint: The slide that killed seven people. Not only dramatic, but sad reality in companies today.

Here’s a typical encounter: At the end of the meeting, the meeting owner asks the fearful question: “Who wants to summarize?” Silence. Avoiding eye contact. After a few silent seconds that feel like minutes, a shy hand goes up. The new employee. Ah! Still innocent.

Two days later, a link to the meeting notes arrive in the inbox. Keeping it there. Unread. Postponing clicking on it until 5 minutes before the follow up meeting. Time passes. On the way walking to the meeting, swiping through it on the phone. Shocked.

The new employee took initiative. By creating something that wasn’t just a report. Something that wasn’t meeting the existing low standards. It was establishing something new. A new way of thinking. Taking initiative by interpreting the meeting. Making suggestions. Drafting solutions. Stating decisions.

The whole “report” ended up as the baseline of the new project. How it was done. What was done. Who works together. After a few minor tweaks, everyone nodded. Key decisions usually taking hours of discussion were taken away in a few slides. By the new employee!

It’s your chance to change culture and influence your team. Make bold and clear statements in executive summaries, team reports and presentations.

Ethics as technical solution

Or how to avoid another GDPR meeting

I recently blogged about Regulations as bad. And how regulations help big corporate. However, also big corporate is being challenged, and the usual solution is to hire a team of lawyers and have them schedule endless meetings across the company.

Regulations are one thing. They demand the absolute minimum with an abolute maximum of bureaucracy. This is an opportunity to take an engineering approach. To find purpose. To do better than demanded. And also to do it right from a technical standpoint.

So, what can an engineer do?

Start with asking the question. Asking the question of the one person who is not at the meeting table. The customer. Your customer. What would she think if you’d discuss how to find a loophole in the regulation instead of taking care of her desire to treat her data with respect and the deserved respect and privacy?

I’m glad we agree. Let’s dive into simple technical solutions: End-to-end encryption (in transit and at rest). This might have been hard a few years ago (remember self-signed certificates in a local intranet?), but in todays world of cloud-computing there are no excuses of not using HTTPS with a free certificate or enable the option for encryption at rest of storage, backups or databases.

Use an external authentication provider to authenticate access to your systems. Don’t re-invent the wheel. Auth0, Facebook Connect, Azure AD and many more have long solved OAuth and SAML authentication and can be connected to directory services for federated logins. Most 3rd party systems support that, and custom services should leverage those systems and just deal the straight forward call of validating JWTs.

Encryption and authentication alone don’t solve the privacy problem, especially if everyone working in your company has access to the data. Tokenizing data containing personal information is moving it however to a next level, especially for central log or analytics systems. Those system often contain the full contact details of a customer. But why? Why do you want to expose your analytics system to data breaches that can be executed with a simple “select email from customers” query? Why isn’t it enough to just store the postal code and country, and potentially a token to assign a useful primary key without personal information to the customer address?

And finally, there are still regulatory requirements you need to meet, such as having a data protection officer assigned. But hopefully that’s just a small administrative overhead to everything you already do. All the things you do much better than regulation demands.

Disclaimer: Above is a simlification trying to make a point. There’s more you can and should do from a security and privacy standpoint. And there are a few more things you must do for aligning with regulatory authorities.

Regulations are bad

Or how they help big corporate. And how to navigate them as a small company.

I don’t think it comes to a surprise to anyone. But regulations are a hot topic recently. They are created with all the good intent. But in the end, they only help the largest corporations, and sometimes destroy the smallest companies.

So, let’s start with a popular example. GDPR. And there’s one very visible side-effect of it – the cookie consent banner that pops up on most websites these days. Adding this banner is relatively straight forward from a software engineering perspective, but it’s either mission impossible or an expensive exercise for small companies. Small companies (outside the tech world) don’t have much idea what a cookie is, what privacy implication it has, but they still run a website. Often leveraging WordPress or Shopify etc. It was easy to setup those websites self-service. But all of a sudden they have to choose between million of plugins that claim to have the best cookie consent banner for their small website. And once they activated the plugin, they won’t know whether it even complies. Or they contract a software engineer, and all their margins of the past month are gone for a few cookies. With all the good intents, the small can only lose.

Let’s move to food. Labels are popular. From sustainable fishing to organic food to socially responsible sourcing. Labels have good intent. They help to identify a standard quickly. Example fair trade. The fair trade price is meant to pay the world market price plus a premium. A premium that should reach the producer, or better the worker at the farm. The deal is, that the premium must first be used to improve work conditions so workers benefit in a way that can easily be audited. Sadly, the premium isn’t a lot, and often the certification process and keeping authorities happy for renewals is more expensive than the premium alone. So for the smaller ones it could actually mean less money for the workers. And the very small can’t even afford the certification in the first place. The small can only lose.

Big corporate love regulations, and their complexity. They have the lobby on their side so regulations are ending up being highly complicated, the lawyers to find the right loophole, and the money to actually work within those regulations. And they have independent lawyers on their side who try to find the small who don’t (or better can’t) comply, simply to make money from them. The small can only lose.

So, what’s your choice? Regulations cover the minimum only. Loopholes included. Then there’s ethics. It’s what should be done. It’s way above regulations. Write them down for yourself as your principles. Make decisions following these principles. Your customers will understand them when being consistent. And they’ll be loyal as long as you stay true to your principles.

Glamour in Business Analytics

Or how mindset and attitude are most critical hiring criteria

If you need access to data, you usually go through some central entity. The department of Global Business Analytics. Their main skill is to be a guard keeper on who is allowed to send data, what data should be sent, how it’s transformed, and who has the right to gain access to data. Additionally, technical complexities around building on-premise data cubes required hiring highly skilled database and server admins.

It’s an exclusive club of professionals. They are in the center of the world. They are looked at in awe. Lines are queuing up every day to get a slice of their data for the next presentation to the CEO.

Business Analytics needs personalities who need this glamour.

There is an alternative way. Supported by technological advancement that make it easier to democratize data. Technologies that are cloud-based, focusing on ease of moving data, and delaying complex data transformation towards the end of the chain. And supported by organizations that believe in democratizing and decentralizing access to data.

It’s a way that removes the glamour from the Central Business Analytics team. It merely becomes an engineering team. Just like any other software engineering team. With technical challenges, interesting client interactions and a publicly visible roadmap to execute on. No glamour, no fuss, just work. Interesting and challenging work.

Embrace technology and start hiring for the right mindset.

The organization that shouldn’t exist

Or how to rethink your next org-chart.

Refactoring code is straight forward. I don’t have hard feelings deleting code. Potentially even enjoy it. This is different with organizations. While organizations must evolve, change and adopt, the speed and flexibility is still limited with regards to people.

So it’s critical that organizations that shouldn’t exists don’t start to exist. And I continue seeing it happening. And then people are surprised of a re-org once someone notices the problem. They are everywhere:

The software engineers maintaining a legacy system that are tasked to also re-invent the future instead of radically under-staffing the legacy system and truly focus on the future.

The DevOps team not believing in the company’s principles by holding on to a central, gate-keeping mindset and related staffing while it could focus on engineering, trust and enabling autonomous teams.

The project management office with a heavyweight, central quarterly planning process involving 10.000s of hours and imposing non-suitable processes to teams instead of working along simple principles and objectives leading to autonomously operating teams.

The system admins that believe in the past and continue hosting servers and building racks for virtual machines instead of embracing the cloud.

The security teams that continue purchasing expensive firewall hardware and worry about USB sticks being put into computers instead of following the more secure and lightweight zero-trust concept.

Embrace change. Be radical. Trust in people. Think of the future.

From pipe dream to a small company

Or: How my wife created The Small Batch Project, a company that’s importing award-winning chocolate into Switzerland.

Let’s start with some context. Seth Godin lays out a concept to revoluzionize school and education. If you care about the next generation, then Stop Stealing Dreams is a must read (or at least a must watch of the 20min long TED talk).

This idea, the fact that everyone can excel, and that the best education can happen online, anytime and independent of location made me and my wife discuss a lot. It was a time when she was looking to quit her day job and start an adventure on her own. And then we found the altMBA, an investment of roughly 4’000 USD and 1 month of intense workshops. Signed up. And it all happened in August 2018.

So, did it help? My wife started the year 2018 with a pipe dream of building a company which would have cost probably more than a million to start, and likely years to execute. Ideas we all have in our head, but never get to execute. Simply, it’s too big to even start. The coaching, the constant feedback of peers and the intense reading mainly simplified “mission impossible”. There are no guidelines, no instructions. Just feedback from peers, constant coaching pushing you to make a leap and convert your dream into an executable project. All of a sudden ideas like “start writing a blog” or “do workshops” were discussed. Zero or close to zero capital investment, easy to test, easy to pivot and also not a full crash in case of a failure.

With that, she started her idea to import award winning bean-to-bar chocolate from around the world to Switzerland, online available at The Small Batch Project. She started with visiting a chocolate fair in September 2018, got a company incorporated and a website based on Shopify up and running in October 2018, had an appearance in a market stand in November 2018, another one in December 2018, her first chocolate tasting event in January 2019, and well, a few month into starting the adventure learned a lot, made interesting connections, received broad support from her friends and got positive and re-inforcing feedback from all corners, including an alumni of the altMBA contacting her with some suggestions to improve her marketing strategy.

So all the altMBA did was simplifying a huge business idea into something actionable? Dare to jump? Dare to take action? Kind of. At least it did the most scary part. There was no point anymore of saying “that’s impossible”. Almost no point of going back. And most of that by following the philosophy of doing something good in this world and making it happen.

With all that, my new job title is “professional chocolate taster”.

Push Git Tag from GitLab Runner

The GitLab Runner defaults to having read access only to the current repository. Git pushes such as tagging a commit won’t work. There are some suggestion out there, such as issue #23894, but I didn’t find anything more straight forward than what I’m writing here.

GitLab Personal Access Tokens are one way of using git pushes. However, they are tied to a user and needs changes in case the user leaves the company or simply changes the team. And if you use a shared “service” user, it’ll consume a license in the Enterprise Licensing model.

GitLab deploy keys are an alternative. This article shows how to use the GitLab deploy keys to push tags to a git repository.

Create and configure deploy keys

The long anwer can be found on the GitLab and SSH keys documentation. The short answer is:

Copy the content of the public key (by default named id_rsa.pub) to your project. It’s located at GitLab Project -> Settings -> Repository -> Deploy Keys. Once added, the SSH keys fingerprint is displayed on the user interface. You can double check for a match by extracting the fingerprint from your private key locally via:

Encrypt deploy keys in your repository

I recommend encrypting the private key in your repository, and decrypt it at runtime. I’m using AWS Key Management Service (KMS), but there are many alternatives available, and also corresponding implementations at different cloud providers. Anyway, here’s how to get the private key encrypted:

Build file

The trick part of the build file, besides decrypting the private key, is configuring the git push URL and comment correctly. See inline comments of the following file:

.gitlab-ci.yml

Eventually, the script needs to be invoked on in the .gitlab-ci.yml file. Additionally, all other stages need to be excluded when tags are pushed to avoid an infinite loop of builds when each build pushes a tag and triggers a build (yes, I did it).

Once figured out, it’s straight forward to create an SSH key, encrypt it, store it in source code, update the git remote push URL and put those components together.

Manage ACM certificates through AWS CloudFormation

Certificate management has historically been fairly manual, costly and often related to trial and error (or long documentation). AWS ACM based certificates removed most of the pain.

ACM offers Email and DNS based validation. Email adds overhead in two ways. First, you need an Email address for the valid host (and you might not have an app.your-company.com Email address, forcing you into setting up AWS SES). Second, you need to regularly re-validate the certificates. DNS based validation removes those hassles and is the recommended way.

Remaining is still its setup. AWS CloudFormation (CF) offers creating a Certificate resource. Attaching DNS validation however isn’t straight forward, and the best way I could find so far was leveraging a Lambda function, which can be inlined in the CF template.

In short, the template creates the following resources:

  1. An IAM role to execute the AWS Lambda function.
  2. An AWS Lambda function that creates and deletes ACM certificates, and returns the created AWS Route53 RecordSet values that must be used for DNS validation.
  3. An AWS Route53 RecordSet matching the ACM certificate’s DNS validation settings.