Amazon CloudHSM
Related to the PKCS#11 testing, I gave Amazon's CloudHSM a spin. Some random notes here about it.
Setting it up
-------------
Setting up CloudHSM is not particularly difficult, just very slow. And has few things that might make live easier if it was actually mentioned somewhere. Your live will be easier if you follow https://docs.aws.amazon.com/cloudhsm/latest/userguide/getting-started.html as well.
To start with, remember that CloudHSM is not particularly cheap. I tried it out for less than 24 hours and it costed
about 7$. Remember to set all retention times to absolute minimum (7 days) and delete all backups once testing is done.
So, setting it up. First, setup some VPC (unless you have one) for access. Do this first to make sure you know which
subnet it got deployed into, making next steps slightly easier.
Then, deploy a CloudHSM cluster from https://console.aws.amazon.com/cloudhsm/home. The deployment is super slow, for me
it took over 10 minutes, so just be very patient. Remember to choose *all* subnets in your AZ.
Once the cluster has been deployed, you need to create a new HSM. Make sure you pick the subnet your vpc lives in.
After long while (again), it will ask you to sign a certificate. Download the request and sign it with a key, see https://docs.aws.amazon.com/cloudhsm/latest/userguide/initialize-cluster.html for how to sign it easily.
I used a slightly different, and easier command sequence.
```
openssl genrsa -out customerCA.key 2048 # add -aes256 if you want to encrypt it
openssl req -new -utf8 -subj '/CN=HSM-CA/' -x509 -days 3652 -key customerCA.key -out customerCA.crt
openssl x509 -req -days 3652 -in
About PKCS#11
-------------
PKCS#11 is an interface designed to allow using hardware tokens for storing cryptographical objects and to perform
cryptographic operations. These are found usually in devices called "Smart Cards", which can range from credit card
form factor to small USB sticks.
I have been working with providing capability to [PowerDNS](https://www.powerdns.com) auth server to allow
signing domains with these tokens. The first iteration code has been there for a while now, but it has some drawbacks
mainly due to the fact that I didn't know how PKCS#11 works, really. Probably still don't, yet I thought it would be
interesting to write down things that I know and have learned from them. Including some quirks along the way.
On top level, PKCS#11 works with drivers that provide an API. You can see a good explanation on how PKCS#11 is supposed to work at https://www.cryptsoft.com/pkcs11doc/v100/group__SEC__7__GENERAL__OVERVIEW.html. If you are unfamiliar with this, it might help to skim it through first.
Error handling
--------------
This is something that needs to be kept in mind. Every token out there will only ever give you pre-defined error codes. These are, for some reason, designed to be very cryptic.
So instead of telling you what's wrong, you get a vague error code about "something" being wrong. Logging all calls can help, so you might catch the error.
This also means that you should always check the return value of everything, otherwise you'll miss the original cause of failure.
Initializing modules
--------------------
First thing to do when you want to use PKCS#11 is to load the engines/modules/drivers for this. Each manufacturer
has usually their own, which is needed in order to actually talk to the cards. There are more than one way to use these
drivers, I have been using [p11-kit](https://p11-glue.github.io/p11-glue/p11-kit.html) for this, it has been
mostly good. You configure the module to it and it's then rather easy to just use them by name.
To configure module for p11-kit, you usually create a name.module file under /usr/share/p11-kit/modules. This is pretty
consistent path on most systems, there is also /etc/pkcs11/modules directory on some RHEL based systems. So here you
create the file, and mostly you need to put in
```
module: /path/to/some/cryptoki/library.so
## for debugging, you can add
#log-calls: yes
```
First step along the way to use a module is to first load them. You can either load a module directly, or you can load
all modules, then look for the name you configured for this. For longest times, I was using `p11_kit_modules_load_and_initialize(int flags)` for initializing all the modules, which has worked so far.
Up until I tried with Amazon AWS CloudHSM, which requires that you actually pass s real arguments for it.
```
CK_INITIALIZE_ARGS args;
memset(&args, 0, sizeof(args));
args.flags = CKF_OS_LOCKING_OK;
```
This was luckily enough to get going, and it seems that there is no huge issues with calling this again.
I do think though that I have to redesign my code once more so that it'll really try to initialize modules only once, in case there is some more exotic tokens there again.
Finding out about tokens
------------------------
For some drivers, tokens are visible right away. For others, you first have to list them all, so that they actually become visible.
If you just try to access them with `C_OpenSession` directly, or anything else, they pretend the token is not there.
So a good idea here is to call `C_GetTokenList` twice, first to find out how many tokens there are, then allocate and call again to get the tokens.
The function accepts a `CK_ULONG count` parameter, which **must** be set to the amount of space you're allocated. On first call it **must** be set to 0, otherwise
some tokens flat out refuse to work, others do not mind.
Now that you have a list of tokens, you can use `C_GetTokenInfo` to find out about. You also find out the real slot IDs that you need to access the tokens, which you can use.
You need the slot ID to open sessions.
Opening a session
-----------------
Techically, tokens are supposed to support read-only and read-write sessions. Some tokens do support these, and some only support read-write sessions. So either you always
open a read-write session, or you try to be smart about it, and maybe try open a read-only and then read-write session, if it fails. I think I'll keep on opening a read-write session always.
There are probably some tokens also out there that support parallel cryptographic operations on a single session, but so far I haven't seen any that does, so, to open a session use:
```
CK_SLOT_ID = number;
CK_SESSION_HANDLE = CK_INVALID_HANDLE;
CK_RV rv = C_OpenSession(slotid, CKF_SERIAL_SESSION|CKF_RW_SESSION, 0, 0, &handle);
if (rv != CKR_OK) {
// deal with it
}
```
Very important note is that you can't always use a session repeatedly. E.g. [Yubico](https://yubico.com) tokens refuse to do more than one signature per session, but there are others too.
Logging in
----------
Before you can actually do much, you need to log in. This is done using `C_Login` call. There is probably no harm in calling it more than once, but you can avoid this. You can use `C_SessionInfo` call to
determine if you have `CKS_RW_USER_FUNCTIONS` or `CKS_RO_USER_FUNCTIONS`, which tells you that you are logged in properly. Remember that your application is supposed to use CKU_USER, not CKU_SO (Security-Officer).
You can generate objects as CKU_USER. CKU_SO mode is for managing the token itself, not the objects.
Finding an object
-----------------
Before you can actually sign anything, you need to find an object to sign with and get a handle for it. This is done with `C_FindObjects` which comes in three parts. Attributes are either
numbers or byte sequences. For almost all cases you only need to specify object class and label or ID.
Once you've found an object with handle, you can sign things with it. Now, the API specifies two kinds of signature APIs, there is a way to do multi-part signing and then you have single-part signing.
Turns out that most newer tokens only support single-part and that's fine for DNS at least.
Signing
-------
Now, before you sign anything, there is one step to do first. Some tokens support hashing and signing, some only support signing. So, what to do? Best option is to always hash first, then sign, so you
don't have to check. You can use the token to sign things, or you can just use whatever tools you have to do the signing. Since hashing isn't so expensive as it used to be in 1990s, you can hash using
local code and avoid a call to the token.
Make sure you carefully choose the correct mechanism for the signing process. For ECDSA, you use CKM_ECDSA, for RSA you use CKM_RSA_PKCS.
Tips
----
Some things I've found so far:
- Some token drivers are *really* picky about others. Do not attempt to use them for wrong slots.
- All the drivers I've seen so far are not very good quality, expect Issues.
- Don't like threads? Well, most of the drivers do. There is an option to say that threads are unacceptable, but I haven't tried that.
- [https://www.opendnssec.org/softhsm/](SoftHSM) will live in your process's memory, including keying material, unless you use proxy driver.
- Do not reuse sessions. Most tokens won't like this.
- Amazon AWS CloudHSM racks up costs pretty fast. I spent day testing on it and it'll already costed like $11 for me.
- Using a virtual machine is fine, although not all drivers like e.g. VMWare's PIV_II emulator.
- If you want to pass the entire token to your VMWare virtual machine, see https://kb.vmware.com/s/article/55789.
- Pulling out your token can make it disappear until you restart your application, despite plugging it back in.
- You can get really weird error codes.
Hopefully this some day helps someone. And probably my code is soon ready enough to be included in PowerDNS to replace the current code.
Some things though that I think I should do with it. In PowerDNS you have the DNSCryptoKey objects. I'm guessing
I need to do some separation there and move the PKCS#11 handling in a way separate that I can cache information
about tokens and such so that I don't have to keep asking things over and over again, except when they fail.
Also I think it might be best idea to cache the public key value during provisioning so I only have to do signing with
the token in the end.
When setting up spam filtering for mailman, it is important to make sure you don't accidentically filter too much.
Things like bounces need to go thru, and you probably don't want to filter out mails for your users.
We have set up rspamd in front of dovecot.org mailman.
As you are not permitted to send mail unless you are subscribed, we had quite a lot of administrative workload, and setting up rspamd reduced fair share of that.
But, since it's a mailing list, we needed to make some adjustments.
First of all, we do not want to accidentically classify our subscribers as spam. Luckily this was almost trivial step.
PKCS#11 for DNS revisited
rspamd with mailman
#!/bin/bash
/usr/lib/mailman/bin/list_members dovecot > /etc/rspamd/local.d/maillist_members.txt
restorecon /etc/rspamd/local.d/maillist_members.txt
# local.d/multimap.conf
MAILLIST_MEMBER {
type = "header";
header = "From";
filter = "email:addr";
map = "file:///etc/rspamd/local.d/maillist_members.txt";
}
# rspamd.conf.local
metric "default" {
symbol "MAILLIST_MEMBER" {
score = -5.0;
description = "Member of a mailing list";
}
}
The next hurdle is to permit bounces to work. Initial attempt was just with multimap.conf and do prefilter that accepts anything sent to bounces.
# local.d/multimap.conf
WHITELIST_RCPT {
type = "rcpt";
filter = "email:user";
map = "file:///etc/rspamd/local.d/bounces.txt";
prefilter = true;
action = "accept";
}
Turns out this was not such a good idea, because if someone sends spam to mailing@list mailing-bounces@list, it will end up matching here and bypasses the spam filter.
A more refined solution was to configure a conditional whitelisting, that permits bounces if there is only one recipient.
# local.d/multimap.conf
WHITELIST_RCPT {
type = "rcpt";
filter = "email:user";
map = "file:///etc/rspamd/local.d/bounces.txt";
}
# local.d/force_actions.conf
rules {
WHITELIST_BOUNCES {
action = "accept";
expression = "WHITELIST_RCPT && RCPT_COUNT_ONE && !FORGED_RECIPIENTS";
}
}
This, while not as efficient as the first one, does the job more carefully, and prevents spam from getting thru.
It would be probably useful to have multimap option that specifies that it only matches for single recipient, not if any of them matches.
Appsuite & Dovecot installation
I thought I'd write up a small guide for setting up Open-Xchange App Suite and Dovecot using free versions.
There are lots of these already, but I thought I'd freshen this up, to latest avavailable versions. But first, Full Disclosure, I work for Open-Xchange (at time of writing). This guide is for Debian Stretch, but with some small changes, it should be appliable to some other OSes.
I also assume you are capable of configuring your MTA yourself, and that you are using virtual user setup.
Feedback is welcome to aki.tuomi at dovecot.fi
Setting things up
Before installation, one needs to set up repositories, since we want to use packaged versions.First we import PGP geys to get things started.
curl https://software.open-xchange.com/oxbuildkey-community.pub | gpg --import curl https://software.open-xchange.com/oxbuildkey.pub | gpg --import curl https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import
Then, export the imported keys into trusted key file.
gpg --export CD6B53ED ED409DA1 DD1A5E9CEED949F0 > /etc/apt/trusted.gpg.d/open-xchange.gpg
Then we setup the repository list.
cat<<EOF > /etc/apt/sources.list.d/open-xchange.list deb http://repo.dovecot.org/ce-2.3-latest/debian/stretch stretch main # this is not a typo, but please use Stretch repos when they become available # 2018-11-18 - Updated to use Stretch repositories deb http://software.open-xchange.com/products/appsuite/stable/appsuiteui/DebianStretch/ / deb http://software.open-xchange.com/products/appsuite/stable/backend/DebianStretch/ / EOF
Now you are ready to install the server. Run the following commands to get the necessary packages. It is recommended to upgrade your system before installation.
apt update apt install mariadb-server mariadb-client apt install open-xchange open-xchange-appsuite open-xchange-smtp \ open-xchange-imap open-xchange-authentication-imap dovecot-imapd \ dovecot-lmtpd dovecot-managesieved dovecot-mysql apache2
Add group and user vmail, create directory /srv/vmail/ and chown it to vmail:vmail, chmod the directory to 0700.
Configuring things
Lets start by configuring dovecot using minimal setup. As stated in the start of this blog, this is for small installations. We use MySQL for user authentication. You can choose to use either Dovecot or App Suite for authentication, both work. You can replace /etc/dovecot/dovecot.conf with this, or find out relevant places in the split config to make doveconf -n look like this.
mail_home=/srv/vmail/%Ld/%Lu
mail_location=sdbox:~/Mail
mail_uid = vmail
mail_gid = vmail
mail_attribute_dict = file:%h/Mail/dovecot-attributes
## this is sometimes needed
#first_valid_uid = uid-of-vmail-user
ssl=yes
# you might want to change these to match whatever you are using
ssl_cert=<cert.pem
ssl_key=<key.pem
# generate this file on system with lots of entropy
# openssl dhparam 4096 (or if you are in hurry, 1024 or 2048)
ssl_dh=<dh.pem
ssl_cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384
ssl_min_protocol = TLSv1
ssl_prefer_server_ciphers = yes
protocols = imap lmtp
# Needed for appsuite
imap_capability = +XDOVECOT
## if you are using dovecot for authentication, use
#passdb {
# driver = mysql
# args = /etc/dovecot/dovecot-mysql-auth.conf
#}
namespace inbox {
inbox = yes
list = yes
location =
mailbox Drafts {
special_use = \Drafts
auto = create
}
mailbox Spam {
special_use = \Junk
auto = create
}
mailbox Sent {
special_use = \Sent
auto = create
}
mailbox Trash {
special_use = \Trash
auto = create
}
prefix =
separator = /
subscriptions = yes
type = private
}
Then restart dovecot and make sure it works.
telnet localhost 143 * OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS AUTH=PLAIN] Dovecot ready. 1 LOGOUT * BYE Logging out 1 OK Logout completed.
Authenticating with MySQL
Create a database with following schema. If you get problems with the primary key, you can either shorten the username and domain field, or you can configure mariadb using following things.
innodb_file_format=Barracuda; innodb_file_per_table=ON; innodb_large_prefix=1;
CREATE DATABASE users ( username VARCHAR(255) NOT NULL, domain VARCHAR(255) NOT NULL, password VARCHAR(255) NOT NULL, PRIMARY KEY(username, domain) ) ROW_FORMAT=DYNAMIC;
Create file /etc/dovecot/dovecot-auth-sql.conf
driver = mysql connect = host=/var/run/mysqld/mysqld.sock dbname=mails user=admin password=pass password_query = SELECT username, domain, password \ FROM users WHERE username = '%n' AND domain = '%d' iterate_query = SELECT username, domain FROM users;
Authenticating with App Suite
This is not supported via direct SQL lookups. You should not do this.
Configuring App Suite
Stop open-xchange service. Then change following settings and start open-xchange. Give it few minutes to start.
server.properties
com.openexchange.forceHTTPS=true com.openexchange.servlet.contentSecurityPolicy="script-src 'self'; object-src 'none'" com.openexchange.connector.networkListenerHost=localhost com.openexchange.server.backendRoute=APP1 com.openexchange.server.knownProxies=127.0.0.1 com.openexchange.rest.services.basic-auth.login=rest-login com.openexchange.rest.services.basic-auth.password=<random string>
Configuring Apache
Put this file into webmail.conf
<VirtualHost *:80>
ServerAdmin webmaster@domain.name
ServerName webmail.domain.name
Redirect permanent / https://webmail.domain.name/
</VirtualHost>
<VirtualHost *:443>
ServerAdmin webmaster@domain.name
ServerName webmail.domain.name
DocumentRoot /var/www/html
<Directory /var/www/html>
Options None
AllowOverride None
Order allow,deny
allow from all
RedirectMatch ^/$ /appsuite/
</Directory>
<Directory /var/www/html/appsuite>
Options None +SymLinksIfOwnerMatch
AllowOverride All
</Directory>
RequestHeader set X-Forwarded-Proto "https"
SSLCertificateFile fullchain.pem
SSLCertificateKeyFile privkey.pem
<IfModule mod_proxy_http.c>
ProxyRequests Off
ProxyStatus On
# When enabled, this option will pass the Host: line from the incoming request to the proxied host.
ProxyPreserveHost On
# Please note that the servlet path to the soap API has changed:
<Location /webservices>
# restrict access to the soap provisioning API
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
# you might add more ip addresses / networks here
# Allow from 192.168 10 172.16
</Location>
# The old path is kept for compatibility reasons
<Location /servlet/axis2/services>
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
</Location>
# Enable the balancer manager mentioned in
# http://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Updating_a_Cluster
<IfModule mod_status.c>
<Location /balancer-manager>
SetHandler balancer-manager
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
</Location>
</IfModule>
<Proxy balancer://oxcluster>
Order deny,allow
Allow from all
# multiple server setups need to have the hostname inserted instead localhost
BalancerMember http://localhost:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
# Enable and maybe add additional hosts running OX here
# BalancerMember http://oxhost2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>
# When specifying additional mappings via the ProxyPass directive be aware that the first matching rule wins. Overlapping urls of
# mappings have to be ordered from longest URL to shortest URL.
#
# Example:
# ProxyPass /ajax balancer://oxcluster_with_100s_timeout/ajax
# ProxyPass /ajax/test balancer://oxcluster_with_200s_timeout/ajax/test
#
# Requests to /ajax/test would have a timeout of 100s instead of 200s
#
# See:
# - http://httpd.apache.org/docs/current/mod/mod_proxy.html#proxypass Ordering ProxyPass Directives
# - http://httpd.apache.org/docs/current/mod/mod_proxy.html#workers Worker Sharing
ProxyPass /ajax balancer://oxcluster/ajax
ProxyPass /appsuite/api balancer://oxcluster/ajax
ProxyPass /drive balancer://oxcluster/drive
ProxyPass /infostore balancer://oxcluster/infostore
ProxyPass /publications balancer://oxcluster/publications
ProxyPass /realtime balancer://oxcluster/realtime
ProxyPass /servlet balancer://oxcluster/servlet
ProxyPass /webservices balancer://oxcluster/webservices
#ProxyPass /documentconverterws balancer://oxcluster_docs/documentconverterws
ProxyPass /usm-json balancer://eas_oxcluster/usm-json
ProxyPass /Microsoft-Server-ActiveSync balancer://eas_oxcluster/Microsoft-Server-ActiveSync
</IfModule>
</virtualHost>
Setting up App Suite
Before you can use your new App Suite installation, it needs to be configured. Begin by ensuring 'hostname -f' works.
Initialize config database
/opt/open-xchange/sbin/initconfigdb --configdb-pass=password --configdb-dbname=ox_configdb -a
Then run following command
# if you want to use dovecot as authentication source, use --mail-login-src mail /opt/open-xchange/sbin/oxinstaller --servername oxserver --mail-login-src login --mail-server-src global --transport-server-src global \ -configdb-user openexchange --configdb-pass password --configdb-dbname ox_configdb --master-pass password --no-license --servermemory 1024 service open-xchange restart
Then we need to register the server.
/opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P password
Then we need to setup container for your users. Each user in the container has same domain name, if you want multiple domains, create multiple containers. You can add alias addresses for your users from other domains.
/opt/open-xchange/sbin/createcontext -c 1 -A oxadminumaster -P password -N domain.name
Open vSwitch
No L2 for you
We started experimenting with UpCloud Ltd. cloud services. Our goal was to move most of our infrastructure there behind a vFirewall, such as pfSense. Unfortunately we found out that even within same region, you can end up in more than one L2, so putting routes, default or any kind, towards your pfSense is not really going to work.
After discussing with UpCloud people they suggested creating overlay network, using e.g. Open vSwitch to it. So, how hard can that be? Well. Turns out it can be quite hard. At least to me, the documentation appears to be, as usual, highly fragmented, partially out of date and conflicting, and everything is done by combining little bits together.
Outline of vSwitch
What does vSwitch do? Well, it's a switch. At most simple case, you have a bridge, where you can attach one or more interfaces. The idea in most setups is to run the switch in your host and put the guest interfaces to your vSwitch bridge along with, say, vxlan tunnel for making logical L2 topology.
So, I setup openvswitch-switch, and found a handy guide on creating vxlan tunnel between two switches. The command is
# ovs-vsctl add-br ovsbr0 # ovs-vsctl add-interface ovsbr0 vxlan0 -- set interface type=vxlan remote_ip=some.other.ip # ip addr add 10.123.0.1/24 dev ovsbr0and then you provide IP for ovsbr0 on both sides, and lo presto, you get virtual L2 network. In correct usage, you would not need necessarly even add an IP for your interface, you could just attach your vnet's or docker0 device to it and it would probably work out just fine.
Our case though is that we can't run switch in host, we need to run it on the guest. Nothing wrong with it, but it's bit more complicated. Well, only because it becomes pretty tedious to manually create all those vxlan tunnels, it's not going to automate at all.
Ryu controller to rescue
Clearly automation is needed. If you look for a OpenFlow controller you'll find some. Not that documentation is very useful, as you are mostly expected to DIY everything. I chose ryu, because it's python. Unfortunately ryu seems to be having some pause in development, since there are pull requests from 2015 unmerged.UPDATE: ryu development seems to be quite active, based on their mailing list.
I started experimenting with simple_switch_13.py and quickly got a controller up and running, but the downside is that it doesn't really do anything useful in my case. What I understood was that I need something that changes configuration on the switch automatically.
Automation for lazy
Armed with grep, guesswork and mostly just staring at the example code for few days, I understood that what I need is a manager. Manager can configure a vSwitch. Manager can be either active (pull) or passive (push) type. Because I don't need to constantly do things, and because it turned out to be a lot easier, I opted for push configuration. Only to find out that ryu's library had a bug. Turns out, the bug is easy to fix by synchronising the copy of Idl component in ryu from ovs, but it took quite long moment to figure out, because even though ryu's Idl inherits the one in ovs, it does not actually call the parent constructor.
That sorted, I started to think how to make the configuration happen when switch comes online. It should happen on it's own, without any provoking, and while you could make a REST API to call, I thought, why? It will connect to the controller, so I should hook to that, somehow, and then automatically setup the VxLAN pair.
After many frustrating hours, I was able to figure this out. So now whenever a vSwitch connects to a controller, it automatically connects to it, and ensures it has VxLAN for the hub, and connects to hub as well, and adds VxLAN pair for the spoke.
Last hurdle
The hub is special case, because it needs modified ryu code to work. But the spokes, they must work without customized scripts as much as possible. This was somewhat hard to figure out, mostly due to lack of documentation, but if you install openvswitch-switch package in Debian, it also installs useful helpers for configuring openvswitch, so it was actually possible to put everything into /etc/network/interfaces file. I also added dhcpd for automating IP distribution.
In the end, I had very good, functional system, that works with debian jessie, at least. You can find all this at https://github.com/cmouse/ryu-auto-vxlan. README.md has the interfaces file described.
New look on site again
A new look on site, and some new tricks. Not using PHP at all on this iteration, because I could not find any use for it here. Site layout made with twitter bootstrap and added a theme for the fun of it to get it dark. Why dark? Because it's easier on my eyes.
Matching text
While C/C++ provides the power of Regular Expressions, sometimes a slightly more simple matching is desired. FnMatch provides * and ? based mask support, but for non-filename use I find it bit suprising, and it is not nil safe. Following code is released to public domain (in countries where public domain is not applicable, CC BY 4.0 is applied).
The code is fashioned after fnmatch, but removes any special cases from it. You might be worried about the recursion there, but it must be evaluated against the SUBJECT and MASK, not just mask. And if it worries you, add some recursion prevention measures there, like counter.
#include <string>
#include <cctype>
class SimpleMatch
{
public:
SimpleMatch(const std::string &mask, bool caseFold = false)
{
this->d_mask = mask;
this->d_fold = caseFold;
}
bool match(std::string::const_iterator mi,
std::string::const_iterator mend,
std::string::const_iterator vi,
std::string::const_iterator vend)
{
for(;;mi++) {
if (mi == mend) {
return vi == vend;
} else if (*mi == '?') {
if (vi == vend) return false;
vi++;
} else if (*mi == '*') {
while(*mi == '*') mi++;
if (mi == d_mask.end()) return true;
while(vi != vend) {
if (match(mi,mend,vi,vend)) return true;
vi++;
}
return false;
} else {
if ((mi == mend && vi != vend)||
(mi != mend && vi == vend)) return false;
if (d_fold) {
if (std::tolower(*mi) != std::tolower(*vi)) return false;
} else {
if (*mi != *vi) return false;
}
vi++;
}
}
}
bool match(const std::string& value) {
return match(d_mask.begin(), d_mask.end(),
value.begin(), value.end());
}
private:
std::string d_mask;
bool d_fold;
};
Managing NLNOG ring keys in git repository
NLNOG ring is a community which provides visibility within other AS to it's members by having each member provide a server with access. This access is done with OpenSSH public keys. If you are a small operator with, say, 1 or 2 keys it is not really difficult to manage these keys. When you are a larger company with tens or hundreds of keys to maintain, it becomes more difficult. To solve this, I wrote this script for our users so that keys are kept in git repository, in gitlab and also in NLNOG ring. You keep one (or more) key per user in a per-user file, which is then concatenated when you update git repository in manage.ring.nlnog.net.
First, create a git repository in manage.ring.nlnog.net. (You can use whatever repo name you want, I used keys. It can also be under some directory)
~$ mkdir keys ~$ cd keys # we need a bare repository for this to work ~/keys$ git init --bare Initialized empty repository in /home/user/keys
Then create hooks/post-update under repository directory with content from https://gist.github.com/cmouse/4dc3c24d5cdcaad0f681. Remember to set executable bit on it.
This script will basically concatenate any *.key files into $HOME/ssh-keys file. Now, if you are using some VCS in your company, you can keep the files there, and have people who understand what they are doing push the keys into ring management.
On your local system, execute following commands to setup your key repository
~$ mkdir keys ~$ cd keys ~/keys$ git init Initialized empty repository in /home/user/keys/.git/ ~/keys$ git remote add ring username@manage.ring.nlnog.net:keys
After this, your workflow is basically
- create user.key with openssh public key(s)
- git add user.key
- git commit user.key
- git push ring master
This will cause keys to be updated on ring and once their puppet runs, you're up to date.
2015-03-23 12:53 Update: added information about how to use git.
Formless buttons in Rails
I wanted to make a simple toggle button inside a form in Rails, and I tried to do it with <%= button_to .. %>, only to find out that nested forms are not accepted. So, I made a small helper script to allow buttons work without a nested form.
$(document).ready(function(){
$(document).delegate(".btn-remote", "click.rails", function(e){
var button = $(this);
if (!$.rails.allowAction(button)) return $.rails.stopEverything(e);
$.rails.handleRemote(button);
return false;
});
});
Fixing ubuntu/debian json
A simple way to get the original code back. For the ones that just want it fixed fast, head to https://cmouse.fi/php-json-orig/ for packages. To build the sources yourself, run
dpkg-buildpackage -b -uc -us
How to make it yourself then?
First, grab yourself php sources. I used 5.5.9 as that is the base they use in ubuntu today. This will change eventually.
Then you make directory /tmp/php-orig-json-1.0-1 and copy everything under php-5.5.9/ext/json/ to it. After this, compress the /tmp/php-orig-json-1.0-1 directory into /tmp/php-json-orig_1.0.orig.tar.xz.
Now, to debian stuff.
Make debian directory, and put following files into it
changelog
php-json-orig (1.0-1) trusty; urgency=medium * Original version of PHP json -- your name <your@email> output of date -R
Note that there are two spaces between email and date. Important.
Then
echo 9 > compat
You want to grab the original copyright message for the json code and toss it into copyright file.
json.ini
; configuration for php json module ; priority=20 extension=json.so
php5-json-orig.dirs
usr/lib/php5 usr/lib/php5/json
php5-json-orig.install
*/php_json.h usr/include/php5/ext/json/
php5-json-orig.php5
mod debian/json.ini
rules
#!/usr/bin/make -f
# This has to be exported to make some magic below work.
export DH_OPTIONS
DEB_HOST_GNU_TYPE ?= $(shell dpkg-architecture -qDEB_HOST_GNU_TYPE)
DEB_BUILD_GNU_TYPE ?= $(shell dpkg-architecture -qDEB_BUILD_GNU_TYPE)
SOURCE_VERSION = $(shell dpkg-parsechangelog | grep ^Version | sed "s/Version: //")
CPPFLAGS:=$(shell dpkg-buildflags --get CPPFLAGS)
CFLAGS:=$(shell dpkg-buildflags --get CFLAGS)
CXXFLAGS:=$(shell dpkg-buildflags --get CXXFLAGS)
LDFLAGS:=$(shell dpkg-buildflags --get LDFLAGS)
# CPPFLAGS not used by makefile
CFLAGS += -O2 -Wall -fno-strict-aliasing $(CPPFLAGS)
ifeq ($(DEB_HOST_GNU_TYPE), $(findstring $(DEB_HOST_GNU_TYPE), ia64-linux-gnu powerpc64-linux-gnu aarch64-linux-gnu))
CFLAGS += -g
else
CFLAGS += -gstabs
endif
build: build-php5-stamp
build-php5-stamp: configure-php5-stamp
dh_testdir
# Add here commands to compile the package.
cd build-php5 && $(MAKE) CFLAGS="$(CFLAGS)" && $(MAKE) test NO_INTERACTION=1
touch build-php5-stamp
configure: configure-php5-stamp
configure-php5-stamp:
dh_testdir
rm -rf build-php5 && mkdir build-php5
cp -r config.m4 config.w32 CREDITS json.c json.dsp JSON_parser.c JSON_parser.h package.xml php_json.h README tests utf8_decode.c utf8_decode.h build-php5
cd build-php5 && phpize && \
CFLAGS="$(CFLAGS)" CPPFLAGS="$(CPPFLAGS)" LDFLAGS="$(LDFLAGS)" \
./configure --build=$(DEB_BUILD_GNU_TYPE) --host=$(DEB_HOST_GNU_TYPE) \
--prefix=/usr \
--with-php-config=/usr/bin/php-config5 \
--disable-rpath \
--disable-static \
--with-json=shared,/usr
touch configure-php5-stamp
clean:
dh_testdir
dh_testroot
rm -f configure-php5-stamp
rm -f build-php5-stamp
rm -f install-stamp
# Add here commands to clean up after the build process.
rm -rf build-php5
dh_clean
install: DH_OPTIONS=
install: build
dh_testdir
dh_testroot
dh_clean -k
dh_installdirs
# Add here commands to install the package into debian/php5-json-orig.
(ext=`/usr/bin/php-config5 --extension-dir`;mkdir -p debian/php5-json-orig/$${ext};install -m 644 -o root -g root build-php5/modules/json.so debian/php5-json-orig/$${ext}/json.so;)
mkdir -p debian/php5-json-orig/etc/php5/mods-available
install -m 644 debian/json.ini debian/php5-json-orig/etc/php5/mods-available/json.ini
touch install-stamp
# Build architecture-independent files here.
binary-indep:
# Build architecture-dependent files here.
binary-arch: DH_OPTIONS=
binary-arch: build install
# Need this version of debhelper for DH_OPTIONS to work.
dh_testdir
dh_testroot
dh_installdebconf
dh_installdocs
dh_installchangelogs
dh_strip
dh_link
dh_compress
dh_fixperms
dh_installdeb
dh_shlibdeps
echo "php:Depends=phpapi-`php-config5 --phpapi`" >> debian/php5-json-orig.substvars
dh_gencontrol
dh_md5sums
dh_builddeb
binary: binary-arch binary-indep
.PHONY: build clean binary-indep binary-arch binary install configure
Now that you have functional debian directory, you can go into /tmp/php-orig-json-1.0_1/ and issue dpkg-buildpackage. Add -uc -us if you don't want signed changes. Then you can install package with dpkg and it should replace the json module provided by ubuntu.
Converting PCKS#12 PFX into Java keystore
You have a key and certificate in PFX format, and would like to get them into a java keystore with least hassle. Here"s how I do it:
#!/bin/sh # (c) Aki Tuomi 2012 # You are free to use it as you like, no warranty or guarantee # base=`basename $1 .pfx` if [ -z "$1" ] || [ -z "$2" ]; then echo "Usage $0 file.pfx alias [password]" exit 1 fi # extract key openssl pkcs12 -nodes -nocerts -in $1 -out $base.key # extract certificate openssl pkcs12 -nodes -nokeys -in $1 -out $base.crt # recode key into pkcs8 openssl pkcs8 -topk8 -nocrypt -in $base.key \ -outform DER -out $base.pk8 # recode certs into pkcs7 openssl crl2pkcs7 -nocrl -certfile $base.crt \ -outform DER -out $base.p7b # just in case rm -f $base.jks # create jks java ImportKey $base.pk8 $base.p7b "$2" $base.jks if [ "$3" != "" ]; then keytool -keypasswd -alias "$2" -keypass changeme \ -new "$3" -keystore $base.jks -storepass changeme keytool -storepasswd -new "$3" -keystore $base.jks \ -storepass changeme fi # cleanup rm -f $base.key $base.crt $base.pk8 $base.p7b
You also need a Java snippet for inserting keys into java keystore, as this is not supported by keytool. https://cmouse.fi/ImportKey.java
Jul. 9th, 2012Getting rid of problem waste in Finland
Recycling is easy, or so they claim here. And complain that people are not recycling properly. Wonder why?
Last weekend, when cleaning up my mother-in-law's cellar, I had to dispose a set of old winter tyres. Simple thing, eh?
There are few ways to get rid of them:
- Take them to Kuusakoski, which is nearest, and open from 8-17 every week day. Handy for anyone working normal day.
- Take them to HSY Sortti-station, open from 8-21, every week day. Better, but not really usable during weekends
This applies to many other types of difficult to dispose waste as well. The places you can take these are open during week, only, and at least the nearest Kuusakoski was so damn difficult to find that it took me 10 minutes riding around before I could figure out how to get there.
Why is it so impossible to do this sensibly, by having few places that are open also on weekends, when people actually need them, and have that place somewhere you can get with public transportation, or by car easily. Now it's been made so difficult that most people opt to leave their stuff in any place they dare and then the victims complain that people do not know how to recycle. They do, but it's just too damn difficult.
Apr. 1st, 2012Creating map cache for Neongeo
Now that I have android phone I decided to give geocaching software Neongeo a go. Works like charm, but I really like having my maps downloaded for faster operations and less waiting for downloads. Not to mention the cost.
Anyways, Neongeo wiki has no instructions, so I decided to write 'em here, and see if I can get write permissions there to copy these there then.
Steps to perform
Download and install MObile Atlas Creator. You can get it from http://mobac.sourceforge.net/
Create map cache
Create new atlas with OpenStreetMap or Google maps. The working map type is Rmaps SQLite. Choose layers from 10-18. You can choose multiple rectangles for your cache. I did it for the region I normally stay. I used OpenStreetMap MapQuest because I like the way the mark paths and terrain to the map, making cache hunting easier.
Wait until map downloads
Copy the resulting SQLite database into Card\Android\data\com.neongeo.app\files\map
Open Neongeo, SettingsMap ServersEdit mapserverGoogleMaps or OpenStreetMaps (depending your choice). Click on 'Offline Map', and choose your SQLite database. Click 'ok'.
And you are done. Works like a charm.
Reinforcing computer table
I've noticed that my computer table from Ikea had sagged and started to keep disturbing noise. It also wobbled when poked, quite disturbingly. After having watched it for some time, I decided to actually do something about it. I decided to reinforce it.
First, I went to Ikea to see if they still sell it, but unfortunately they no longer did. Oh well, had to do what I had. Next stop was hardware shop where I bought some angle irons and a box of screws, and then to home.
First up, removal of all stuff from the table, sort what goes back.
Quite a lot of stuff, that. The white piece of board with the extension cord screwed on is the white cover plate that goes under the table, but it's been shortened a bit to make room for exhaust air from the PC.
Then, it's time to rebuild the table, and add glue to strategic places. The black thing is a mount rail for my desktop PC, so that I can keep it off the floor, and off the table. The yellow cord is for holding my sub woofer up.
Then it's time to add the white cover plate and screw in the angle irons. They really steady the table a lot. The ones on the cover plate are really quite mandatory, otherwise the cover plate would no longer stay in place as it has been removed a chunk at the end.
Then, the only thing left is to install the "wire rack", and put all the wires in place, along with everything else that goes on the table.
And so, quite pleasing end result is achieved. This last picture was taken later on, as I had to wait for a card reader having lost my connector cable for the camera.
The total cost was about 10 EUR, and I think it was money well spent.