ca md too weak / CA signature digest algorithm too weak

callocorg Uncategorized , ,

Upon upgrading to a new version of the operating system, the clients for the SSL secured Kafka streaming system stopped working. The errors were:

SSL routines:SSL_CTX_use_certificate:ca md too weak

and later on

CA signature digest algorithm too weak

In debugging the issue, I found the following openssl command useful: openssl x509 -in cert-signed.pem -text . The command includes the specific sections about the signature: Signature Algorithm: sha256WithRSAEncryption

After going through all the certificates, it turns out the CA certificate was signed with sha256 already. So while the error message is somewhat right but pointing in the wrong direction of the CA, just the client certificates needed a stronger signature from the CA. To do so, re-perform the signing with explicitly specifying sha256. For example:

openssl x509 -req -CA ca-cert.pem -CAkey ca-key -in client-cert.csr -out client-cert-signed.pem -days 365 -CAcreateserial -passin pass:mysecret -sha256

Replacing the client certificate then pointed to the broker certificates, which needed to undergo the same treatment.

VMware ESXi and Containers

callocorg Uncategorized

One way to conserve resources and work efficiently with containers on ESXi is to use VMware dedicated Linux images for that task: https://vmware.github.io/photon/.

It is compatible with Docker, Kubernetes and other Frameworks. Only the chef-client support seems weak at the moment: Issue 525

Updating VMware ESXi

callocorg Uncategorized

At times it is necessary to update the ESXi hypervisor. I found it relatively easy to work with the command line interface, as it offers opportunities for scripting at the same time. VMwares website can be a bit tricky to navigate, so some links are provided for convenience, I work with version 6.0.0 here. The process I followed below.

Install the VMware vSphere CLI

Download and install the CLI from VMware vSphere CLI v6.0.0

Perform a configuration backup

Ideally you would perform a full backup of all virtual machines before. Then back up the configuration of your hypervisor so you can restore via the original image if needed. On the command line that you open via the start menu on Windows for example, type:

vicfg-cfgbackup.pl --server==10.10.1.1 --username=root -s C:\Backup\esxi_60_cfg.2016-03-09

Download the patch

Download the patch from VMware: https://my.vmware.com/group/vmware/patch#search

In my case it was a major update as a zip file of a little over 300MB. Upload that patch file to a datastore on the server via the GUI for example (right click the datastore line in the summary page).

Check the contents of the patch:

esxcli.exe --server=10.10.1.1 --user=root software sources vib get --depot=[ds]updates/ESXi600-201511001.zip

If the certificate is a unsigned certificate, an error will show and the following parameter will need to be added:

--thumbprint AA:...:AA

Prepare the system

While still in the GUI after the upload, turn all VM’s off and enter the maintenance mode (right click on the server).

Just for inventory purposes, note all the installed packages. As noted earlier, the thumbprint parameter allows to proceed with a certificate that has not been installed on the machine:

esxcli.exe --server=10.10.1.1 --thumbprint AA:...:AA --user=root software vib list

Update and reboot

Perform a dry run of the update. As I use third-party ESXi images, I use the update command so it will not overwrite all packages explicitly as the install command would.

esxcli.exe --server=10.10.1.1 --thumbprint AA:...:AA --user=root software vib update --dry-run --depot=[ds]updates/ESXi600-201511001.zip

After ensuring the dry run competed well, remove the dry run argument and proceed for the whole update. Then reboot the system, exit maintenance mode and start the VM’s as usual.

Alternatively, if you log in via ssh to the ESXi host, you can perform the same from within the system by executing the following from the directory of Patch.zip:

esxcli software vib update -d "`pwd`/Patch.zip"

For completeness, I also verified the system profile:

esxcli.exe --server=10.10.1.1 --thumbprint AA:...:AA --user=root software profile get

Java / Scala Memory Leak Investigation

callocorg Uncategorized

Every now and then a memory leak comes along, in my case typically raised via a java.lang.OutOfMemoryError. For me, the following steps have helped to narrow the issue down.

Step 1 – Consider speed of parallel execution and heap size

Especially in parallel executing programs, the creation of a thread or a Future in Scala is simple and fast. This however can lead to the issue that the Future is allocated but not yet run. If many Futures are created in sequence, their memory allocations will remain in memory and make the program fail.

The solution is to process chunks in parallel that are sure to stay within the memory limits, then wait for this to finish and start the next chunk.

Alternatively, if operating on very large heaps and/or highly concurrent applications, the garbage collection may slow down significantly. Consideration to an alternative garbage collector should be given, such as specifying the concurrent collector.

-XX:+UseConcMarkSweepGC

Step 2 – Investigate the source

To get to the root cause, it helps to investigate the memory on failure in detail. This can be done by adding the following parameters to the execution of Java:

-XX:+HeapDumpOnOutOfMemoryError -verbose:gc -Xmx512m -XX:+PrintGCDetails

Setting the maximum heap to 512MB or a reasonable small size for your particular program helps to hit the limit quicker in a leaking application.

The verbose GC information helps to see the performance of the garbage collector on the fly. The output is similar to the following:

[GC 173630K->74442K(216064K), 0.0066210 secs]
[Full GC 74442K->36371K(208384K), 0.1041700 secs]

The first two numbers indicate the size of objects before and after collection. The number in parentheses is the committed size of the heap, followed by the time it took to perform the small or full garbage collection.

The option “-XX:+PrintGCDetails” adds further detail to this output.

As indicated by the first option, a heap dump will be created on failure.

Step 3 – Analyze

With the heap dump available, various tools can be used to investigate the possible issue in detail. I personally prefer to use Eclipse Memory Analyzer for further investigation: http://www.eclipse.org/mat/

Additional links

Dell Poweredge T430: Silent configuration

callocorg Uncategorized

When unpacking the new Dell T430 server and plugging it in, I was a bit surprised about the noise level, given many people commented on forums about the silent operation of the tower. After installing VMware ESXi I dug through a few documents and online posts and found the tricks:

  • Consider a non-low-power CPU on purchase. As per the Acoustical Performance PDF referenced below, “Configurations with low-power processors (which have lower temperature limits than standard processors), e.g., Intel Xeon E5-2650L v3/E5-2630L v3 processors at 65W/55W, under moderate or heavy utilization, will be about twice as loud as typical configurations.”
  • The CPU is initially, at least in my machine, in the full performance setting without any state change to a lower setting. Fix this via: System BIOS -> System Profile Settings -> System profile set to “Performance per watt”. Alternatively set a custom profile of Performance + C-States/C1E enabled.
  • Adjust the iDRAC fan profile accordingly, in the iDRAC web-interface: Hardware -> Fans -> Setup (tab) -> set the thermal profile to minimum power.
  • Ensure the iDRAC is updated to the latest version, with that comes at times some fine-tuning in terms of fan and thermal control.
  • Consider removing the redundant fan.
  • If all of this does not help, you may want to investigate if you want to replace the loud but also high air volume fans with some quieter ones, e.g. the Noctua industrial range (link). One of the fan connectors on the motherboard is a standard PWM connector and I just used this for the Noctua in the outside optional chassis and used the other connector for the previously external Sunon fan.

The sources:

  1. Dell PDF file “Dell PowerEdge 12th Generation Server System Profiles” by Bruce Wagner, 2014
  2. Dell PDF file “BIOS Performance and Power Tuning Guidelines for Dell PowerEdge 12th Generation Servers” by John Beckett, 2012
  3. Dell PDF file “Dell 13G PowerEdge Acoustical Performance & Dependencies”, Victor Chen et. al., 2015
  4. http://en.community.dell.com/support-forums/servers/f/956/t/19638760
  5. http://serverfault.com/questions/631204/dell-poweredge-r220-fan-speed-high-when-idle-low-under-stress
  6. http://en.community.dell.com/support-forums/servers/f/956/t/19647301

Resolving empty views in Eclipse e4, postConstruct is not called

callocorg Uncategorized

The issue with views turning up empty in e4 might be related to the postConstruct method not being called. First, verify the view part is actually created, for example by printing a statement via System.out in the constructor of the part.

If this does show up, but the postConstruct method is not called, the issue may be the wrong resolution of javax.annotation, which takes care of the dependency injection.

The e4 FAQ addresses this issue as follows:

“Ensure your bundles use Import-Package with a package version to pull in the standard annotations rather than a Require-Bundle on the javax.annotation bundle.” (Eclipse 4 FAQ).

Import-Package: javax.annotation; version=”1.1.0″

In other words, ensure none of the bundles has the javax.annotation bundle under “Required Plug-Ins” but rather imports it with a minimum version of 1.1.0

Working with Scala Futures

callocorg Scala

Working with scala Futures is excellent for performance and non-blocking program flow. However, some conversions are slightly more tedious.

To add a completion action to a future, two non-blocking options exist. First, the Future can utilize a onSuccess (or onFailure / onComplete) call as shown below:


import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global

val f1 = future {
Thread.sleep(10000)
println("Done")
}

f1 onSuccess {
case _ => println("After Done")
}

Second, the result (assumed successful) can be mapped to the next future result:


val f2 = future {
Thread.sleep(10000)
println("Done again, returning 1")
1
}

f2.map(number => println(10+number))

If a Seq of Futures is present, the result of all Futures may be relevant on how to proceed. This is where Future.sequence comes handy:


val futSeq = (1 to 10).map(i =>
future {
Thread.sleep(10000)
println("Done again, returning "+i)
i
}
)

val seqRes = Future.sequence(futSeq)
seqRes.map(s => println(s.sum))

Schema design in Apache Cassandra

callocorg Apache Cassandra

I just stumbled over a good comment from Jonathan Ellis at http://www.datastax.com/dev/blog/cql3-for-cassandra-experts

The rule of thumb is, use one columnfamily for each type of resultset you want to query. Cassandra is not Bigtable or HBase where you have to try to cram everything into very few ColumnFamilies because of implementation limitations.

As a bonus, “one CF per resultset type” promotes good performance, because any time you’re selecting from the beginning of the row, you will get better performance than if you have to scan past other cells.