ssl_error_rx_record_too_long and Apache 2.4

This post is mostly to record the problem and cause for the record so that hopefully next time it will show up in my (and others) search engine results, as the underlying cause is a regression (something that worked before no longer works). From memory this is at least the second time I've had to reverse engineer this cause from the symptoms... :-(

If you are setting up a new webserver, using Apache 2.4 on a modern Debian/Ubuntu system (eg, Ubuntu Linux 14.04 LTS), and HTTPS is not working properly -- and apparently your virtual host is not working properly -- there's a change in configuration file arrangement between Debian/Ubuntu's layout for Apache 2.2 (eg, Ubuntu Linux 10.04 LTS) and the one for Apache 2.4 that you need to take into account.

The most obvious symptom is if you try to use HTTPS, and have replicated the setup from an old (Debian/Ubuntu) Apache 2.2 system to a new (Debian/Ubuntu) Apache 2.4 system, and HTTPS no longer works at all. If your web browser (eg, Firefox) displays SSL errors like:

SSL received a record that exceeded the maximum permissible length. 
(Error code: ssl_error_rx_record_too_long)

and trying to connect with openssl s_client results in errors like:

CONNECTED(00000003)
140735307125600:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 348 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---

then this might be your issue.

The next most obvious issue is that the special features of your virtual host do not seem to work -- almost as if your virtual host configuration file is being ignored -- but that can sometimes be harder to recognise, and much harder to find with a search engine, whereas ssl_error_rx_record_too_long is a reasonable search term that gets you close to the cause without hinting what changed between Apache 2.2 and Apache 2.4 configuration on Debian/Ubuntu to cause it.

The cause of the HTTPS error above is that Apache 2.4 is accepting connections on TCP/443 (HTTPS) because the ssl module is loaded, but SSL (TLS) is not active on that port -- so Apache is receiving TLS negotiation and replying with raw HTTP/1.1 that the TLs negotation is not valid HTTP/1.1. Eg, from a packet capture while this issue was happening:

HTTP/1.1 400 Bad Request
Date: Mon, 13 Apr 2015 04:12:35 GMT
Server: Apache/2.4.7 (Ubuntu)
Content-Length: 313
Connection: close
Content-Type: text/html; charset=iso-8859-1

followed by a small HTML document with the default Apache error page. Naturally those bytes make no sense when decoded as TLS, hence the error displayed by Firefox (ssl_error_rx_record_too_long) -- although given how easily this can happen, especially on non-well-known-ports, I'm slightly surprised that Firefox does not have a special case to recognise this has happened and, eg, display a "port speaks HTTP not HTTPS -- configuration error?".

Assuming that you have double checked everything that you copied over from the Apache 2.2 server to the Apache 2.4 server, including file permissions, all the config sections copied over, and that there is actually a working link from sites-enabled to the copy in sites-available, so that the config appears readable in sites-enabled there is one remaining issue to consider which is a change from the Apache 2.2 config layout to the Apache 2.4 config layout: in Apache 2.4 the files in sites-enabled must end in .conf (but in Apache 2.2 they could have any ending).

From the Apache 2.2 /etc/apache2/apache.conf:

# Include the virtual host configurations:
Include /etc/apache2/sites-enabled/

and from the Apache 2.4 /etc/apache2/apache.conf:

# Include the virtual host configurations:
IncludeOptional sites-enabled/*.conf

Which means if you (a) used to use, eg, the domain names as the config file names on Apache 2.2 and (b) just replicated the files and links from the Apache 2.2 server to the Apache 2.4 server, the result will be that Apache 2.4 is ignoring your link /etc/apache2/sites-enabled/example.com -- because it does not end in .conf.

The result of the virtual host being ignored is that by default Apache does not speak TLS on any port, unless you say SSLEngine On, and if you have that only in your virtual host configuration files and those configuration files are being ignored... you will end up with raw HTTP being spoken on TCP/443.

An easy way to test if this is your problem is to try accessing the TCP/443 port with raw HTTP, eg:

telnet example.com 443

GET / HTTP/1.1
Host: example.com

If you get anything that looks even vaguely like a HTTP reply (eg, a HTTP/1.1 200 OK) then for some reason TLS is not active on TCP/443.

The solution is pretty trivial -- rename the links in /etc/apache2/sites-enabled to have extensions .conf, eg, example.com gets renamed to example.com.conf, and reload the Apache 2 configuration. It should then work fine.

An alternative approach is to use a2ensite script (bundled with the Debian/Ubuntu Apache packages) to create the symlink, rather than creating it by hand. But that pushes the problem up one level: a2ensite expects to be run with the basename of the file, and to find a file ending in .conf in /etc/apache2/sites-available. At least you get a more easily diagnosed error message though!

The moral of this story is that if it seems like your configuration is being ignored, investigate further whether it's actually being loaded at all -- and maybe in a later version something will have changed which causes it not to be read at all.

Looking through the Apache 2.4 Ubuntu changelog it appears this change was introduced in the Apache 2.4.1-1 package:

+ Give site configuration a .conf suffix, too. For example the default vhost
  is called default.conf. Moreover, files without .conf suffix are ignored
  upon startup. Please update your site links and confs. Also rename the
  default vhost to 000-default.conf and don't do hacky things in a2enmod

and:

+ Expect a .conf suffix for sites-enabled/sites-available configurations.
+ Remove the default vhost special handling. Instead, we expect the default
  host to be named appropripriately (for example 000-default.conf;
  Closes: #605535).

The main upstream change causing this seems to be that Apache 2.3.6+ supports an Include with a wildcard pattern, which makes it possible to use Include sites-enabled/*.conf in Apache 2.4, but not in Apache 2.2. (It appears that Include of a whole directory does still work, but is not recommended.)