What I've learned today

A couple of useful Java tricks that I can't find consolidated elsewhere on the web:

Locking down Java: 

I have a Java instance that runs just my one application. (for example, /opt/foo/jre/bin/java) This application talks across systems, and can be set to use SSL. I don't want to pay for a "real" certificate-- I'll use our internal CA, besides, I don't really want to trust Verisign or any of the other 300+ CAs that Oracle/IBM have decided they like. (No offense intended, just paranoia about MITM attacks within my LAN)

Let's assume that I have this part working, and that SSL is being served happily on the first server.
I'll start by extracting the certificate chain from that first server:
:| openssl s_client -host firstinstance -port 8443 -prexit \
           -showcerts >firstinstance.crt
(No, that's not an emotionless command emoticon, it's a pipe)

From there, I can manipulate the Java keystore to trust (or not trust) my targets.
keytool -import -alias firstinstance -file firstinstance.crt \
  -keystore paranoid.jks -storepass changeme -noprompt \
  && cp paranoid.jks /opt/foo/jre/lib/security/cacerts

Did it work?:

Well, let's check.  Using the same Java environment as our app:
$ JRE_HOME=/opt/foo/jre
$ JAVA_HOME=/opt/foo/jre
$ /opt/foo/jre/bin/java -cp . SSLPoke firstinstance 8443
Successfully connected
In theory, a restart of the app should have it pick up the new (single-entry) list of trusted certificates. In practice...

WTF file is it loading from?

Hidden somewhere in the depths of the startup script for the app, something gets redefined so that instead of looking in $JAVA_HOME/lib/security/cacerts (which has been nicely cleaned up), it used /opt/foo/lib/security/cacerts. (But of course, I didn't know that until.. Hammer)
# apt-get install -y sysdig
# sysdig proc.name=java and evt.type=open | grep cacerts & systemctl restart foo
#7554308 14:25:48.107476643 3 java (6613) < open fd=22(/opt/foo/lib/security/cacerts) name=/opt/foo/lib/security/cacerts flags=1(O_RDONLY) mode=0 
Ah, there's the file I need to mangle for my application.

You may have noticed I slipped in the SSLPoke command above. That's a fetch of https://confluence.atlassian.com/download/attachments/117455/SSLPoke.java -- Thanks, #Atlassian!