< min(), max() and the collation dependent ordering of character columns on Greenplum (Postgres 8.2) >

Recently I came across an interesting collation related problem while using a Greenplum (Postgres 8.2) database setup with an UTF-8 collation.

Each character column that undergoes an order by, min() or max() will be sorted according to that collation.

But in some cases, I needed to perform ordering or min(), max() aggregations based on the ASCII content of character columns - luckily, this is quite easily achieved on Greenplum using operator classes and custom aggregates:

create temp table col_test (a text);

insert into col_test (a) values ('"quoted"'),('Abc'),('123'),('Öfföff'),('Zürich'),('Zuhause'),('Zahnpasta');

--
-- order by UTF-8 content
--
select * from col_test order by a asc;

123
Abc
Öfföff
"quoted"
Zahnpasta
Zuhause
Zürich

select * from col_test order by a desc;

Zürich
Zuhause
Zahnpasta
"quoted"
Öfföff
Abc
123

--
-- order by ASCII content
-- 
select * from col_test order by a using ~<~;

"quoted"
123
Abc
Zahnpasta
Zuhause
Zürich
Öfföff

select * from col_test order by a using ~>~;

Öfföff
Zürich
Zuhause
Zahnpasta
Abc
123
"quoted"


The min(), max() aggregation can be taken care of by creating two custom aggregates:

drop aggregate ascii_max(bpchar) cascade;

drop aggregate ascii_min(bpchar) cascade;

create or replace function ascii_bpchar_larger(prev bpchar, curr bpchar) returns bpchar AS $$
begin if (prev ~>~ curr) then return prev; end if; return curr; end; $$ language plpgsql immutable strict;

create or replace function ascii_bpchar_smaller(prev bpchar, curr bpchar) returns bpchar AS $$
begin if (prev ~<~ curr) then return prev; end if; return curr; end; $$ language plpgsql immutable strict;

create aggregate ascii_max(bpchar) ( sfunc = ascii_bpchar_larger, stype = bpchar, sortop = '~>~' );

create aggregate ascii_min(bpchar) ( sfunc = ascii_bpchar_smaller, stype = bpchar, sortop = '~<~' );

The custom aggregates can be used like any builtin aggregate:

--
-- aggregate by UTF-8 content
--
select min(a), max(a) from col_test;
 min |  max 
-----+--------
 123 | Zürich

--
-- aggregate by ASCII content
--
select ascii_min(a), ascii_max(a) from col_test;
 ascii_min | ascii_max
-----------+-----------
 "quoted"  | Öfföff

See http://www.postgresql.org/docs/8.2/static/sql-createaggregate.html for more information regarding custom aggregates and http://www.postgresql.org/docs/8.2/static/indexes-opclass.html regarding operator classes.

< Evolution of a trading system - three years of software development visualized in under eight minutes >

We used gource to create a video from the svn logs of our trading system - the background song is available here for free.

 

< +107% in twelve months of algo papertrading... >

Today, our fully automated algorithmic trading system has been up and running for twelve months.

During this period, the P&L rose from $100k to $207k - an increase of +107.58%

Here is a screenshot of our paper account:

For comparison - during this time, the Dow made +14.31%, S&P 500 +19.88% and Nasdaq +31.28%.

Want to know more? Drop me a line :-)

< Oracle "pipelined" functions and Powerdesigner 16.1 >

During a recent project assignment, I had to generate Oracle 11g "pipelined" functions using Sybase Powerdesigner 16.1.

The stock Powerdesigner does not provide a "pipelined" keyword for Oracle.

But, luckily, Powerdesigner has an excellent metadata system which can be used to add the necessary keyword on demand.

1. from the main menu, choose Database/Edit Current DBMS... 
2. in the General tab, right-click Profile and choose Add Metaclasses... 
3. click on the DatabasePackageProcedure selection box, then OK 
4. right-click DatabasePackageProcedure and choose New->Extended Attribute 
5. name the new attribute IsPipelined, use false as the default value and click Accept 
6. in the tree-selection, go to Script/Objects/DB Package Procedure/Add 
7. in the Value text-field, replace this existing code:
   
   %DBPKPROCTYPE% %DBPKPROC%[ (%DBPKPROCPARAM%)][ return %DBPKPROCRETURN%][ [.O:[as][is]] %DBPKPROCCODE%]
   
   with this one:
   
   %DBPKPROCTYPE% %DBPKPROC%[ (%DBPKPROCPARAM%)][ return %DBPKPROCRETURN%][%IsPipelined%? pipelined][ [.O:[as][is]] %DBPKPROCCODE%]
   
   and choose OK. The DBMS properties window will close and Powerdesigner will ask you to save the changes you just made - click Yes.
8. open the appropriate database function and right-click to open it's properties
9. activate the new IsPipelined attribute in the Extended Attributes tab for "pipelined" functions

< +20.06% in three months of algo papertrading... >

Continuing our fully automated algorithmic trading system test, our paper account P&L rose to +20.06% from February, 11th to May, 17th - this amounts to a gain of around $20k in approximately three months.

Here is a screenshot of our paper account:

For comparison - during this time, the DJI made +9.73%, S&P 500 +9.85% and Nasdaq +9.60%.

< sample backtest equity curve >

Here is a backtested equity curve of our automated trading system since 2000, starting with $100k:

< an outstanding trade >

Normally, our automatic trading system does not do day trading, but today was not normal:

The system bought FSLR at 27.53, then added tranches at 29.17, 31.78, 33.50, 37.00, 40.00 and was finally stopped out at 39.01 during a pull back - net gain for our paper account today with this trade: +$7023.61 - nice ;-)

< +8.6% in four weeks of algo papertrading... >

From February, 11th to March, 8th (four weeks) my partner and I performed a test of our new fully automated algorithmic trading system with a 100k paper account at Interactive Brokers.

The algo was running completely unsupervised - no manual intervention was necessary.

During the four weeks of algorithmic papertrading, the P&L rose to +8.6% or +$8600 - sweet ;-)

P&L 1st week: +3.055%
P&L 2nd week: +5.893%
P&L 3rd week: +7.387%
P&L 4th week: +8.598%

For comparison - in this four weeks, the DJI made +2.89%, S&P 500 +2.19% and Nasdaq +1.62%.

Here is an official statement from the broker:

Here is a TWS screenshot showing the paper account:



We are currently looking for investors - want to know more?

Simply send me an email at agram66 (at) gmx (dot) de - pgp public key.

< The Matrix has got you. >

New Scientist has written an article about a nice study from the ETH Zurich, that shows how interconnected the global capitalistic network we live in has become.

Small teaser from the paper, page 6:


In detail, nearly 4/10 of the control over the economic value of TNCs in the world is held, via a complicated web of ownership relations, by a group of 147 TNCs in the core, which has almost full control over itself. The top holders within the core can thus be thought of as an economic “super-entity” in the global network of corporations. A relevant additional fact at this point is that 3/4 of the core are financial intermediaries.

Here is the article:
http://www.newscientist.com/article/mg21228354.500-revealed--the-capitalist-network-that-runs-the-world.html

and the study itself:
http://arxiv.org/PS_cache/arxiv/pdf/1107/1107.5728v2.pdf

< dlsym(), ISO, C++, gimme a break >

Today, while trying to implement a simple class that can dynamically load new C++ classes into a running program using dlopen() and dlsym() on Ubuntu 11.04 using g++ 4.5.2, I came across this error:

algorithm.cc: In static member function ‘static algo::algorithm& algo::algorithm::load(const std::string&)’:
algorithm.cc:24:67: error: ISO C++ forbids casting between pointer-to-function and pointer-to-object

Couldn't be clearer, could it? According to ISO, there is no way you can convert a pointer-to-object into a pointer-to-function - PERIOD! But wait! We can fix that by adding four simple green lines (comments not counted):

#include "algo/algorithm.h" 

#include <assert.h> 

#define dlsym __gimme_a_break__ 

#include <dlfcn.h> 

#undef dlsym 

#include <string>

// redeclare dlsym to be a function returning a function

// pointer instead of void *
extern "C" void *(*dlsym(void *handle, const char *symbol))();

algo::algorithm &
algo::algorithm::load(const std::string &algo)
{
    // to at least get the illusion of safety...
    assert(sizeof(void *) == sizeof(void (*)()));
    std::string nm = "libalgo-" + algo + ".so";
    void *lib = dlopen(nm.c_str(), RTLD_NOW);
    assert(lib != 0);
    algo::algorithm *(*mk)() =

        reinterpret_cast<algo::algorithm *(*)()>(dlsym(lib, "make"));
    algo::algorithm *a = mk();
    assert(a != 0);
    return *a;
}   // load()

Voilà, everyone is happy again (except the language advocates maybe)...

< Eurozone Banking Stresstest >

Breakingviews has done it's own interactive banking stress test, but with government bonds marked to market:

http://www.breakingviews.com/credible-eu-bank-tests-need-a-higher-pass-mark/1609896.article

In the worstcase scenario (all of the PIIGS countries needing a really short haircut) and under the current Basel II accord, 53 banks would fail the test. I don't know how meaningful the results are, but it is still fun to play around with ;-)

< IWF, EFSF, ESM, WTF?!? >

http://www.openeurope.org.uk/docs/draftesmtreaty.pdf

I won't comment this draft treaty at all - I'll just mention two quotes that seem to fit the bill:

"Insanity is when one is doing the same thing over and over again and expecting different results."
-- Albert Einstein

"It is well enough that people of the nation do not understand our banking and monetary system, for if they did, I believe there would be a revolution before tomorrow morning."
-- Henry Ford

< Kyokushin Summercamp on Sylt >

From 1st through 6th of August I participated as a candidate for shodan in this years German Kyokushin summer camp in List on the island of Sylt. The approximately 40 participants, among them four kids, were housed in the youth hostel "Mövenberg", in the north of Sylt. Here is a picture of some of our group members:

Here is a picture of our accomodation, called "Peter Pan" (you know what Peter used to say: everyone can fly - all one needs is a happy thought ;-) )

The mood was very good and the camp started with an evening training at 8pm on the very first day; during the next days Marek Zmarzly took care of the German competitors as a trainer for the coming world championships in Tokyo: Alireza Shams (1st Dan, Hamburg), Darius Schoeneich (1st Dan, Dortmund) and Milena Kovaleva (4th Kyu, Frankfurt). Other sempais (including me) joined the group to help prepare the guys for the championship.
Our group had to perform a running course each day at 6am and we were drilled by Marek with fighting combinations and sparring.

The other group was trained by Sensais Ulrich Münzer, Darius Czaplarski, Slawomir Urbanczyk and Sempai Peter Gawlik, while the performance of the examinees was always thoroughly monitored by the black belts.
Independently of the groups, there were normally four training sessions per day, and the breaks between the trainings were spiced up by beach-volleyball, Sumo and Rugby.

We visited various nice places on Sylt ;-) Here with my "XXL-Burger" in the restaurant Sansibar:

We all enjoyed the good and mostly sunny weather, the sunbathing and the water training with Gi - the sunsets on Sylt were also very nice:

A spontaneous "after-work" party in front of "Peter Pan":

On the fourth day, the examinees had to show kihon, kata and their stamina under the eyes of the present Sensais and Sempais.
The final test for the shodan candidates, 20 rounds of kumite against brown and black belts, came on the fifth day. I was tired but happy when the examination was over and I was already looking forward to the sayonara beachparty, here Evgeny and I directly after the test:

This is the beautiful landscape where Evgeny and I passed our shodan test:

On the sixth day, we had a final running course at 6:30am (half an hour later as usual due to sayonara partying last night ;-) ), then we all met for breakfast to hear the results of the examinations: Sempai Evgeny and I passed and we are now proud to be Kyokushin Shodans - OSU!

For me personally, going Shodan on Sylt was an experience that I will never forget. I wish to thank all Sensais and Sempais for helping me redefine my own limits ;-)

Some more impressions from northern Sylt:

 

< A true gem... >

I'm always open for new experiences...
http://www.astrogenital.com/
(link is in german)

< Mültikültür ;-) >

<The Joys of C#>

Today I tried to debug my C# code from within Visual Studio 2005 "Professional Edition" - I loaded all the necessary data into the database, massaged it for debugging, and then debugged the program for ca 30 min. While trying to expand a variable in the debug session, I got the following MDA exception:


LoaderLock was detected
Message: Attempting managed execution inside OS Loader lock. Do not attempt to run managed code inside a DllMain or image initialization function since doing so can cause the application to hang.

Yeah, great... just because of this, I had to rollback everything, restart the debugging session, just to find out that the variable expand feature worked this time... something like this never happened to me with neither Java/C/C++/Eclipse nor C/C++/gdb...

<Dilbertism>

What do you say to a manager that wants a guesstimate for porting an RWA calculator from VB to C#, although he demonstrably knows that I am fluent in neither of the two programming languages?

If I were more like my hero, Tom DeMarco, I would probably say that it'll be done when it's done - but since I'm suffering from good manners, I answered that it will take approximately four weeks +/- some uncertainty regarding possible problems that could stem from language specific constructs that might have no simple representation in the target architecture...

I never thought that I would ever find myself in a dilbertesque situation like this again, but here I am... and life goes on ;-)

Perhaps I should suggest the use of the famous Planning Poker game to our development team? There is even an online site for "playing" the planning poker game: http://www.planningpoker.com ;-)

<Mostly Harmless>

As some are aware, there are some "conspirancy" theories floating around regarding December, 21st 2012 - the end of the so called mayan calendar where many associate this particular date with doomsday, the comming of the end of the world.

I used a very scientific method to disproof this assumption - I simply checked if December, 21st, 22nd or 23rd 2012 are Thursdays - they are not. If one of these dates would have been a Thursday, we would be fu*$ed - but since they are not, we're fine ;-)

What has this to do with Thursdays you might ask? This is simply due to the fact, that the earth will come to an end on a Thursday - this is a widely known and accepted fact of life ;-)

If you don't get it, I would recommend reading the hilarious trilogy "The Hitchhiker's Guide to the Galaxy" by Douglas Adams.

In the meantime:

<Beam me up, Scotty...>

...there is no intelligent life down here:

<Bigard is a funny, funny guy>

Oh man, I love this dude: