Author Archives: brillianteyes

Interest rates modeling, algo trading, computer science, machine learning, math and technology.

Using Asio to build Http client (continued)

Well, we need more than send-one-request-receive-one-response.

We need establish a persistent connection and repeatedly send new requests.

Advertisements

Boost Asio as Http client

well, I know poco can do it, but I’d like to kick the tire of Boost library suite. Some people complain that Boost libraries are of varying quality and usefulness.

My goal: I have an existing Excel C add-in which need listen to a HTTP port, receive text stream. Such text is actually serialized (via Boost::Serialized) object, which need to be deserialized and pushed into library’s object cache.

So the key is deep integration with other part of the library. Also, 1) such HTTP get() is data pull model; 2) we may need get from multiple http destinations simultaneously, so true multi-threading is a must (but such MT has to be kept off from main library code).

wrong port: actively refuse (in connect_callback)

The sample is taken from:

http://www.boost.org/doc/libs/1_38_0/doc/html/boost_asio/example/http/client/async_client.cpp

The key is to understand the nested, recursive call structure.

As there is no push in http (i.e. trigger a new get() whenever new msg is ready on server), how to implement the pull?

Now, it is based on the fact Excel function is volatile. So every time Excel re-calculates, it sends a get() request. Note that our get() is asynchronized, so it takes a while before the new content comes in and update object cache. So Excel will well get an existing (older) version of the remote object.

Key fact: the real-time object as of time T won’t be available at least at T+1, i.e. next time Excel re-calculates.

It could be a big drawback, but it is the consequence of pull model.

Review of our object cache: the library is single-thread, so the object cache has no lock in general for best performance. However, asynchorous thread has to update the object cache via a special function where a lock is required. Correspondingly, the Excel volatile function must acquire the same lock before it can read from the cache. In short, the lock is built-in object cache, only applicable for such html thread purpose.

Boost Serialization in polymorphic setup

Boost Serialization is notoriously poorly documented. I am asked to provide serialization capability for a simple class hierachy. To make things a bit more fun, 1) such classes sits in namespace; 2) serialization/deserialization will be in different processes (so we cannot make assumption at runtime which runs first or runs at all).

I use Boost 1.54.0 as of now.

Let us review some basic stuff regarding Boost serialization:

1. Class must have default (param-less) cstor.

Easy to understand: deserializer is like a special cstor(). It must first call the default cstor, then it can populate the member attributes.

2. If we want to use member-function-based serialization, the class can NOT be declared in any namespace.

member-function-based means serialization is implemented as a [private] member function of the interested class.

class FooBar

{

private:

template<class Ar>

void serialize(Ar& ar, const unsigned int version) { … }

}

Why should it be private, see next point.

What if our class is declared within namespace? We have to use non-intrusive serialization (via free functions).

3. Suppose we have a class hierarchy: pure virtual base  Base with two derived classes, named DerivedOne and DerivedTwo.

We should NOT call parent’s serialization code from derived class serialization, instead, using a special base_object template.

class DerivedOne: public Base
{
public:
DerivedOne(double x=0.0, double y=0.0, double z=0.0): Base(x, y), _z(z) {; }
double _z;

template<class Archive>
void serialize(Archive & ar, const unsigned int version)
{
 //Base::serialize(ar, version);  — BIG NO!!!
 ar & boost::serialization::base_object<Base>(*this);
ar &_z;
}
}; //class DerivedOne

*. We have to hint Boost about class hierarchy, or, we get a crash. (true_type == NULL!)

Note: the program walked thru the line of throw, but then crash, kind of weird:

//oserializer.hpp

const boost::serialization::extended_type_info * true_type =
i.get_derived_extended_type_info(t);

// note:if this exception is thrown, be sure that derived pointer
// is either registered or exported.
if(NULL == true_type){  //TRUE
boost::serialization::throw_exception(
archive_exception(
archive_exception::unregistered_class,
“derived class not registered or exported”
)
);
}

We take the preferred approach, explicit registration. i.e. in .cpp file, use macro:

BOOST_CLASS_EXPORT_GUID(DerivedOne, “DOne”)

This macro just expands to other two macros, BOOST_CLASS_EXPORT_KEY2 and BOOST_CLASS_EXPORT_IMPLEMENT.

BOOST_CLASS_EXPORT_KEY2(T, K): build a k/v system for type names/C++ names in cross-platform, cross-compiler fashion. See here for argument C++ built-in typeid() cannot be used.

BOOST_CLASS_EXPORT_IMPLEMENT(T, K): force templates for derived class to be expanded and code is generated. Due to polymorphism, a module can completely manipulate a derived object via a base class pointer. Without such macro, compiler won’t generate code for (serialization) template code.

Canon 6D or 7D, that is a question

My friend D throws me a hard question: buy 6D (1895 usd) or 7D (1489) ? Money is NOT an issue.

 

6D is attractive as it is full-frame, very large sensor size hence super high ISO is possible. 105mm is long enough for any indoor use, even on a full frame and in a big house.

7D makes more sense if you shoot fast motion outdoor a lot (8 fps vs 4) AND you are likely far away from objects (hence cropping better). Indoor fast motion is a different story as lighting is not good. I am not sure very fast shutter will do you any extra good considering you cannot go above ISO 1600 on 7D.

Note: I omit the comparison on shutter life, video capability and other non-essential features. It is a camera, not a wifi hotspot, nor a GPS, period.

 

 

 

Main desktop upgrade

Start with a comparison to see the motivation. Also, my QX56 frequently freezes up!

A (current QX56) B (office) X (new
CPU Q6600@2.4 Xeon W3520 @2.67 I7-3770@ 3.4GHz;
Tech 65nm (Kentsfield) 45nm (Bloomfield) 22 nm
Cache 128+128k/8M/na 128+128k/1M/8M
Mem(sz/freq/latency) 4GB/400/5cyl DDR2 12GB/ 533Mhz/7cyl DDR3
Core/Thread 4/4 4/4 4/8
Score 2973 4943 9426
Power (W) 105 77
Boost 1.42 build (VC10.0) 220 min 85 min
Superpi 1m (sec) 22 14
QuantLib build

The Boost build time is hilarious. QX56 has some other components, I will keep them except to throw in 16 GB DDR3 (latency=9) for $130.

mem: 4 GB DDR2

mobo: MSI-7350 (nForce 650i SLI)

graphics: GTX 650 Ti ( upgraded in Q1 2013, as my old card died shortly I moved in my house)

C drive: Intel 120 GB SSD

D/E drives: WD20 2TB harddrive

 

Based on the chart below, I decide to go with 3770. 4770 is only marginally better, even though it is 4th gen i7 (Haswell) instead of 3th gen (Ivy Bridge). 4770 consumes 84W vs 3770 77W. Well, there is $20 difference. Anyway, I feel good saving $20 even though I shell out $274 for a brand new CPU!

 

Should I go cheap with i5? Well, only i7 has hyper-threading. I will need more threading capability even in VM.

CpuMark

Research on taking offline code back-to-life

I accumulate a lot of legacy code back from college days. They were nicely printed on A4/legal paper. I’d like to run such code. But how? Typing is toooo expensive, as I don’t have the time. This post is to explore some possibilities.

Method 1. Offshore typing. Look to hire a typist in China to do the job. But hard to control the quality.

Method 2. Scanning and OCR!!!

conceptually, this should work, as it is just code (letters only, no graph). But which OCR software should I use for such purpose? Is it practical?

An intuitive discussion on Risk Neutral pricing

back to basic. See Mitbbs: http://www.mitbbs.com/article_t/Quant/31338109.html

 

最简单的binomial例子,假设两个股票,今天都是100块,明天都是升到110或者降到90
,但是一个升的概率大,99%,一个升的概率小,50%,再假设risk-free rate 是0。根
据risk-neutral,call on 这两个股票都应该是5块,但是显然这两个call的收益/风险
不一样,为啥还定价一样?这种情况下理智的人都会去买前一个call吧

或者换个说法,假设有100个独立股票都是99%涨 1%跌,另外100个独立股票50%涨 50%
跌,涨跌幅度一样,那么根据central limit,买100个call on 前100个的收益显然大
于call on 后一百个。但是投资一样,这是为啥?我唯一能想到的区别就是central
limit需要趋于无限才能真正消除risk,在有限情况下始终有非零概率前100个收益不如
后100个,所以风险没有全部去掉,但是为什么风险大小不同但是定价却相同,难道不
应该有risk premium在里面?

你觉得好像option的价格不合情理
其实option完全是redundant的
不合理的源头是stock本身的价格

就好像有个人说,假设一头大象和一只猫一样重,那么把这头大象举起来花的力气
和把这只猫举起来的力气一样大。

然后你觉得,怎么可能呢,现实生活中谁都知道举起大象比举起猫更费力。。

升值概率99%和升值概率1%的俩股票,价格怎么会一样都是100
从离谱的假设出发,即使正确的推导也只会得出离谱的结论

股票本身是strike = 0的call,对吧。

按照你说的这些,你算算股票的价格应该是多少?你那个99%要涨到110的股票根本不可
能卖100.

risk-neutral的唯一原因,是银行每卖一个option,都要买等效的underlying(stock)
来hedge,不论option最后payoff是什么都一样的赚一点spread。搞清楚银行怎么用
option赚钱之后才能理解为什么risk-neutral。

如果你自己投资买option赌股票涨完全不hedge,傻子才会risk-neutral

我来无聊一下。

价格这个东西,不仅反映赢钱亏钱的概率,同时反映投资者的风险偏好。

举例,项目A,50%可能赢1块,50%可能输1块,你愿意玩么?项目B,50%可能赢100万,
50%可能输100万,你愿意玩么?两个项目同样real world期望是0,但愿意付多少钱去
玩,每个人的定价都不一样。你可能愿意花一块钱玩玩,但未必愿意赌一百万进去。

所以real world的概率算出的期望一定要加上risk premium折现回来才能定价。

而Risk-neutral world是指去凑这个概率分布,使得不用加Risk premium就能折合成现
有市场价格。例如,市场观测到项目A的价格是0.1,那么倒推出来项目A的赢钱可能55%
,输钱可能45%,这个概率就是Risk neutral,因为直接期望不用折Risk premium就是价
格。

所以先有价格,而后才凑出来的risk-neutral prob,而且risk-neutural的分布不唯一
的,都有无穷多个解,只是加上log-normal之类的假设以后才唯一的。

“升值概率99%和升值概率1%的俩股票,价格怎么会一样都是100”

这在现实世界里太常见了啊

信息一直在创造也一直在被市场消化,从来都不是一个均衡点
比如垃圾股票都几毛钱,其中就有升值概率差别
差别还可能很大,股票价格就不能反应差别
这个差别当然应该反应到option / future price上面去
(假设也有market maker对这些股票做市)

如果一个理论指导出来说升值概率不同的股票必须价格不同,
如果价格相同则risk neutral option price相同,这样的理论
就不太符合现实。

 

 

 

 

 

What is global markets?

copied from StephenKing in Mitbbs. This article is detailed and fact-based.

 

投资银行一般主要有两条盈利线。Investment Banking和Global Market。前者是由一
群从不睡觉的“banker”组成的黑社会,主要做企业的上市,并购等和资本运作相关的
业务,收取非常可观的费用作为盈利。后者是一帮以“trader”为核心的庄家兼赌徒,
对全球各种金融产品进行定价和交易,赚取利润。

我对Investment Banking的理解仅限于电影“亲密敌人”里的黄立行,所以也无法展开
细说。但是就凭非死不可开盘能卖在42块,我对他们的工作怀有滔滔的敬仰。

来详细说说Global Market全球市场这一块吧。全球市场大致可分为Fixed Income固定
收益类产品(外汇,大宗商品一般归入Fixed Income)和Equity股票类产品。由于固定
收益产品的交易一般只针对全球各类基金,银行和央行(个别外汇交易除外),所以大家
对它的了解也许不如股票交易。但是其实无论从资金量,交易量,利润等等来看,固定
收益的规模都是股票的几十倍甚至上百倍。

两者除了规模上的不同外,固定收益类交易主要是基于世界经济的实际表现和预期,以
及在各种经济指标(例如利率,信用等级)之上,对产品价值进行严格的计算,或是用
数学模型对一些衍生产品进行缜密的理论推算。大部分产品并不是在exchange上交易,
而是通过broker/sales,通过价格竞争/磋商来完成交易(OTC)。所以作为交易员所需的
素质,除了超强的世界经济金融信息的把握和反映能力外,还需要有很强的数学,统计
学功底和英语交流能力。对于某些更加倚重数学模型的产品,例如利率/外汇期权,
Exotic Structures等,还需要一定的计算机编程动手能力。固定收益类的交易账本比
较复杂,交易员的流动也相对小些。

而说起股票交易,大家应该是很了解的。股票交易主要是基于公司发布的经营信息,以
及宏观经济状况,来对公司价值进行正确估算和交易。换句大家熟悉的说法,股票交易
就是在一片嘈杂和混沌中,被不断f**k和看别人被f**k的SM游戏。Ooops。好吧,说的
文雅一点,我觉得股票的走势,从短期来看就是random walk。我不知道股票交易员需
要什么素质。我80岁的奶奶炒股翻了三番,且全部Cash out。我自己个人的股票交易账
户至今仍然是海底两万里。我说的并没有全错,因为投行的股票交易员普遍是毕业不久
的来自各个名校稀奇古怪专业的年轻本科生,精力旺盛,下班后一个比一个疯狂。我如
果不是在自己家里醒来,最后记得的一定是东倒西歪的股票交易员们和酒瓶。

金融危机后,投行不能再用自己的资金像基金账户那样交易(prop trading),而
broker性质的market making以及大宗股票交易,也越来越多的被“机器人”和自动
matching系统(algo trading,dark pool)取代,所以大量股票交易员失去工作,且
这个趋势不可逆转。其实股市上真正的大赢家是那些有资金/信息/技术资源优势的基金
。投行的主要盈利来自于交易佣金,尤其是通过提供自动化交易平台,和research部门
联合pitch idea的sales trading,以及大宗股票交易的佣金等。这样的盈利模式和普
通散户并没有什么根本性的对立。所以大家不必对投行MM们有太深的阶级仇恨。

在投行的交易大厅,头上戴着免提耳机坐在中央位置,面对一堆电脑屏幕,吼来吼去的
是交易员。投行交易员一般是market making性质,被称为“卖方(并不表示他们不买
)”,这不同于基金的交易员(“买方”)。像纽约,伦敦这样的一线金融城市,一个
投行往往需要50名左右的固定收益类交易员,东京,悉尼,新加坡这样的金融城市,规
模往往只有三分之一左右。固定收益类的所有盈利都由这几十名交易员创造。交易员周
围围坐着Sales队伍,交易员们通过他们把触角伸向全世界各种基金,商业银行等其他
玩家。交易员和Sales是合作伙伴,同时也有着与生俱来的利益冲突。因为Sales只在乎
能否成交,他们需要向客户pitch idea。而交易员需要盈利。有时候双方会爆发矛盾。
交易员同时也通过面前嘈杂复杂的电话系统,和电话另一头的broker们连接在一起,通
过broker们和其他投行连接,一同形成一个interbank市场。交易员可以在这个市场里
获得信息,hedge自己的risk。圈子再往外扩展,坐的是research,矿工和码工们(大
部队集中在投行其他楼层里)。这里往往是中国人,印度人最集中的地方。他们为交易
员提供各方面的支持。圈子的最外围是风险管理,Finance,后台文案等后勤支持。以
上也就是所谓的前台(front desk),中台(middle office)和后台(back office)。

由于Global Market所有人员的生存靠交易员在前线拼杀,所以每个交易员都有需要完
成的盈利目标。如果完成的好,除了丰厚的底薪外,一般能拿到3~10%的盈利作为奖金
,如果没有完成,将随时面临下岗的危险。这和买方交易员略有不同。一般来说,基金
交易员能拿到盈利的30%甚至更多,但是基金对失败亏钱的交易员的容忍度几乎为零。
一个优秀的交易员可以每年创造一亿美元以上的利润。一般交易员也应该能创造几百万
美元的利润。交易员的圈子其实很小,互相之间都很熟悉,每个人的表现大家也都大概
清楚。所以一个表现糟糕的交易员也很难再在其他地方找到交易员的工作。

每个交易员会根据能力,被分配一定的风险额度。类似于一名股票交易员能动用的资金
额度。例如,一个债券交易员的风险额度可能包括,利率每移动一个基本点,你的亏损
不能超过十万美金。对于一个衍生产品交易员来说,风险额度的描述会复杂的多。这些
都有风险管理部门随时监控。

现在来说说具体操作,也许有人会认为,交易员的工作,尤其是Market making,应该
很简单。不过是随时hedge你的风险,然后赚取bid-offer的差价。举个简单的股票例子
。假设某股票股价600美元。交易员会定599买,601卖,买卖之间的差价就是交易员的
利润。但是实际操作完全不是这么简单。随着市场的波动,交易员将不可避免的建立仓
位。市场的瞬息万变使hedging非常困难。很多市场,比如exotic,几乎永远是one-way
market。要涨大家买,要跌大家卖。这使得hedging变得不可能。就算是一个普通的利
率期货市场,找到一个完美的hedge也是几乎不可能的。另外投行之间的激烈竞争,也
使得market making变得非常困难。比如市场上这个股票中间价是600元,但是我觉得这
是一个垃圾股票,我只想卖不想买。这时Sales对你吼,“xx基金对这只股票进行询价
!Where is your two-way market??(买方交易员一般不会告诉你他是要买还是卖)”
。作为一名market maker,哪怕你不想买,你也得报一个买入价和一个卖出价。也许你
会说,那简单,就报一个很低的买价好了。这是很愚蠢很粗鲁的。因为如果其他投行都
报598,599买入,你哪怕报595买入,客户以后再也不会把你作为他下次询价时先打的3
个电话之一。这基本上就意味着你将永远的失去这个客户。所以你的正确做法是,对市
场必须非常熟悉,报出598.95的价格,使你的价格看上去极具竞争力,但是却恰好不会
赢。而有的时候,你明明不想交易,但为了观察市场的动向,弄清楚基金到底是想买还
是卖,不得不报出非常tight的买卖价格,比如599.95买,600.05卖。甚至600 choice
。这样保证你的价格无论是在买价中还是卖价中都是接近最好的。这样做的后果是没钱
可赚,甚至亏钱,即所谓的winner’s curse。但是你通过交易知道了客户的意图,也
掌握了市场信息。这对某些重要客户来说尤其重要。遇到比较难对付的客户,他们往往
会将大单拆成小单,同时和多家投行成交。当你和客户成交后转向interbank市场试图
去hedge时,发现所有人都持有相同的仓位都在寻找的hedge,这时价格立刻会向你所期
望的相反方向移动。以上所说的只是日常交易中需要随时注意的几个例子。同时不要忘
了,即使再复杂的组合期权报价(exotic除外),都必须在10几秒或几十秒之内完成。
一旦你说出价格,Sales/Broker喊出“done”,就没有后悔的余地(所有交易员、
sales电话都有录音)。任何mis-pricing都会被市场上的其他玩家hunt down,并导致
可观的损失。所以说market making并不是一项吃差价的简单的工作。

Phew~~ 写累了。草草收尾吧。

Bought a new toy Perplexus Twist

The image is from Amazon.

 

It is the 4th of the series. Marked difficulty level 7/10. I spent 1 hr and only went to marker 10 (I admit I restarted from marker 5 where the ball dropped off)

 

 

Annotation on sklearn.gmm sample for iris data

Data set and manipulation

Iris data is the popular ML dataset with 120 samples. See UCI’s introduction:

Iris Data Set
Download: Data Folder, Data Set DescriptionAbstract: Famous database; from Fisher, 1936
Data Set Characteristics:   Multivariate Number of Instances: 150 Area: Life
Attribute Characteristics: Real Number of Attributes: 4 Date Donated 1988-07-01
Associated Tasks: Classification Missing Values? No Number of Web Hits: 374769

Use the following stmt to check the data details

?sklearn.datasets.load_iris()

The GMM classification is as below.

the data loading code

iris = datasets.load_iris()

# Break up the dataset into non-overlapping training (75%) and testing
# (25%) sets.
skf = StratifiedKFold(iris.target, k=4)
# Only take the first fold.
train_index, test_index = next(iter(skf))

X_train = iris.data[train_index]
y_train = iris.target[train_index]
X_test = iris.data[test_index]
y_test = iris.target[test_index]

n_classes = len(np.unique(y_train))

Now let us recall that the data types in Python:

dict: {a:b, x:y}

list: [x,obj,b]   — list can hold heterogeneous elems. len(), l.append(x), l= range(n)

array:

Note iris.data is of type numpy.array, a popular alias of type numpy.narray, which represents multi-dimensional numerical data.