Odi's astoundingly incomplete notes

I often copy text files via SSH by just opening the target file in vim and then pasting the text into the terminal.

• Ctrl-A Ctrl-C
• ssh targethost
• vim targetfile
• gg  (top of file)
• dG (delete to end of file)
• :set paste (no auto indent)
• i (insert)
• Shfit-Ctrl-V

If the file is quite large, this is very slow and if the target machine is close enough you can hear its hard drive going mad.

So what's going on here? vim is writing to its swap file. Character by character. Dutifully flushing each character out of the buffers. Very good in case you lose power, except that it really slows us down here...

Simple solution: disable vim's swap file with the -n option.

vim -n targetfile


posted on 2013-11-12 21:23 UTC in Code | 0 comments | permalink

The following little unit test measures the memory usage when you convert from char to byte using an OutputStreamWriter. It does that once without buffering and once with a BufferedWriter wrapped around the stream.

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileOutputStream;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;

import junit.framework.TestCase;


public class StreamTest extends TestCase {
    public void test1() throws Exception {
        Runtime rt = Runtime.getRuntime();

        final int max = 10000;
        File f = File.createTempFile("junit", "tmp");

        int gcs = gcs();
        long used = rt.totalMemory() - rt.freeMemory();
        Writer w = new OutputStreamWriter(new FileOutputStream(f), "UTF-8");
        for (int i=0; i<max; i++) {
            w.write("constant");
        }
        w.close();
        assertEquals("increase -Xms -Xmx", gcs, gcs());
        long used2 = rt.totalMemory() - rt.freeMemory();
        System.out.println("unbuffered: "+ (used2 - used) +" bytes");
        
        f.delete();
        gcs = gcs();
        used = rt.totalMemory() - rt.freeMemory();
        w = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(f), "UTF-8"));
        for (int i=0; i<max; i++) {
            w.write("constant");
        }
        w.close();
        assertEquals("increase -Xms -Xmx", gcs, gcs());
        used2 = rt.totalMemory() - rt.freeMemory();
        System.out.println("buffered: "+ (used2 - used) +" bytes");
        
        System.out.println(System.getProperty("java.runtime.version"));
    }
    
    private int gcs() {
        int sum = 0;
        for (GarbageCollectorMXBean mb : ManagementFactory.getGarbageCollectorMXBeans()) {
            sum += mb.getCollectionCount();
        }
        return sum;
    }
}

The result is surprising:

unbuffered: 943896 bytes
buffered: 0 bytes
1.7.0_45-b18

(The actual numbers depend on the heap size). One would assume that unbuffered writes use no memory, and that buffered writes use the buffer size. However this behaviour suggests that OutputStreamWriter's implementation bears a dirty little secret... And where does the buffer memory go? I can just assume that the VM plays dirty optimization tricks here and quickly allocates the transient buffer from thread-local memory without actually resorting to heap.
posted on 2013-11-12 20:40 UTC in Code | 0 comments | permalink

Again I am coming across a customer who has severe misconceptions about security. They are:

There is 100% security
That is so wrong, it's not even a good joke anymore. Security and usability always add up to 100%. Something that is 100% secure is totally unusable and will not fulfil any requirements other than security requirements. Want a secure computer? Turn it off.

Security is cheap
No, security is not cheap. Security always adds inconvenience, it always takes extra work, it always causes extra problems, it always makes things more complex, it is always harder to debug.

Hiding something makes it secure
Also known as Security By Obscurity. Just because something is inconvenient to access, doesn't mean it can not be accessed. This category includes: code obfuscation, jump hosts, locally encrypted data(bases) like DRM (key is accessible by the same user), obfuscated passwords.

Security of a client application is of importance
A client can run any code. Whether that is the original application, a modification thereof, a hacked version, a custom implementation or something compeletely different is irrelevant to security. What matters is what you can do over the network protocol. If your protocol is insecure, then security in your client can not help you.


posted on 2013-08-16 14:34 UTC in Code | 0 comments | permalink

Recently dev-java/sun-jdk-1.6 was masked and removed from the virtual/jdk ebuild. Of course you can still unmask it (in /etc/portage/package.unmask). But Portage still wants to pull in dev-java/icedtea-bin. You really don't need that if you have the sun-jdk. To prevent Portage from pulling it in, simply mark a high version of it as provided.

/etc/portage/profile/package.provided:

dev-java/icedtea-bin-6.999

posted on 2013-07-24 10:46 UTC in Code | 0 comments | permalink

From the J2EE tutorial:

EntityManager em = ...;
CriteriaBuilder cb = em.getCriteriaBuilder();
CriteriaQuery<Pet> cq = cb.createQuery(Pet.class);
Root<Pet> pet = cq.from(Pet.class);
cq.select(pet);
TypedQuery<Pet> q = em.createQuery(cq);
List<Pet> allPets = q.getResultList();

The equivalent JPQL query is:

FROM Pet

The equivalent SQL query is:

SELECT * FROM Pet

Clearly SQL is too simple and boring. You really need a bullshit API to produce a simple SQL query. Of course this will ease maintenance, improve performance, reduce code, prevent bugs, speed up development by a factor of 10 and reduce your project cost by 99% at least. A classic case of overengineering.

posted on 2013-07-23 15:06 UTC in Code | 1 comments | permalink

Sonar Ant task properties use the comma character to separate paths. So a little trick is necessary to integrate that with existing path resources:

<path id="myclasspath">
  <pathelement path="lib/mylib.jar"/>
</path>

<pathconvert property="sonar.libraries" pathsep="," refid="myclasspath"/>

posted on 2013-07-03 11:24 UTC in Code | 0 comments | permalink

Gentoo has updated CUPS to version 1.6. If you run a print server you must configure zeroconf correctly, or your clients won't see any remote printers:

/etc/portage/package.use:

net-print/cups dbus zeroconf
net-dns/avahi dbus mdnsresponder-compat

And allow DNS-SD through iptables:

[0:0] -A INPUT -p udp -m udp --dport 5353 -j ACCEPT

Also configure the CUPS browsing protocol:
/etc/cups/cups-browsed.conf:

BrowseRemoteProtocols none
BrowseLocalProtocols DNSSD,CUPS

I think CUPS tells avahi about registered printers through D-Bus. Then clients discover registered printers via DNS-SD queries.

The clients need working MDNS too:
emerge -av nss-mdns
and change /etc/nsswitch.conf:
hosts:       files mdns4_minimal [NOTFOUND=return] dns mdns4

and should run cups-browsed with this config:
/etc/cups/cups-browsed.conf:

BrowseRemoteProtocols DNSSD,CUPS
BrowseLocalProtocols none

posted on 2013-07-01 18:27 UTC in Code | 0 comments | permalink

When your kernel tells you this: kernel: nf_conntrack: automatic helper assignment is deprecated and it will be removed soon. Use the iptables CT target to attach helpers instead.
It really urges you to include this in your iptables configuration:

*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
[0:0] -A PREROUTING -p tcp --dport 21 -j CT --helper ftp
[0:0] -A PREROUTING -p tcp --dport 6667 -j CT --helper irc
[0:0] -A PREROUTING -p tcp --dport 6566 -j CT --helper sane
COMMIT

Required kernel config:
IP_NF_RAW
IP6_NF_RAW
NETFILTER_XT_TARGET_CT
posted on 2013-01-13 12:08 UTC in Code | 0 comments | permalink

I get this frequently in syslog:

Dec 28 11:44:00 kernel: CPU0: Package power limit notification (total events = 355)
Dec 28 11:44:00 kernel: CPU2: Package power limit notification (total events = 355)
Dec 28 11:44:00 kernel: CPU1: Package power limit notification (total events = 355)
Dec 28 11:44:00 kernel: CPU3: Package power limit notification (total events = 355)
Dec 28 11:44:00 kernel: CPU1: Package power limit normal
Dec 28 11:44:00 kernel: CPU0: Package power limit normal
Dec 28 11:44:00 kernel: CPU2: Package power limit normal
Dec 28 11:44:00 kernel: CPU3: Package power limit normal

Usually when Firefox uses a lot of CPU.  There are many forum posts regarding these messages. But there is nothing to worry about. A bit of background.

These messages are logged by the kernel, in the file: arch/x86/kernel/cpu/mcheck/therm_throt.c
This feature is enabled by the X86_MCE_INTEL config option, which is located under:

Processor type and features
[*] Machine Check / overheating reporting
[*]   Intel MCE features

In dmesg it shows this message, during boot, if the kernel decided to handle thermal interrupts (and not the BIOS):
CPU0: Thermal monitoring enabled (TM1)

So the kernel now gets thermal interrupts from your CPU when certain temperature limits are crossed. The kernel's thermal and power management then decides what to do to decrease the temperature. This is normal operation and nothing to worry about.



posted on 2012-12-28 12:17 UTC in Code | 0 comments | permalink

Gentoo has stabilized iptables 1.4.16. With that some changes to your iptables config are necessary.

Use conntrack instead of state module
before: -m state --state NEW
after: -m conntrack --ctstate NEW

Remove --reap from recent module
before: -m recent --update --seconds 30 --reap -j DROP
after: -m recent --update --seconds 30 -j DROP

Maybe there is more, but I haven't discovered it yet...

posted on 2012-11-24 20:10 UTC in Code | 0 comments | permalink