12 May
2010
12 May
'10
3:16 a.m.
Hi Fons
On Tue, May 11, 2010 at 4:19 PM, Fons Adriaensen <fons(a)kokkinizita.net> wrote:
On Tue, May 11, 2010 at 03:36:08PM -0700, Mark Knecht
wrote:
Yes, a specific (small) range of frequencies and some ability to see
if the energy meets some minimum level of interest.
Actually though, as a test of an idea I had and
just because I've got
the data here, I'm using stock market data, decomposing the data into
bands of energy if you will, and then summing the bands back together
to see how close I come to the original data. It works fairly well
most of the time, but stock data can have big directional moves at
times which appears as a big transient event. When those events
complete then the slower filter bands ring and cause large
displacements in the output.
So you want a set of filters with the following properties:
1. each filter has a frequency response with a shape that allows
an intuitive interpretation as a 'bandpass', i.e. a range of
frequencies, and
2. the sum of the filter outputs equals the original
input.
This is not really a requirement, but if they did then I'd have a more
visual demonstration that the process was working.
This is a *hard* problem. Basically none of the
classical filter
types have these properties. There are filter sets that can do
this but they are quite esoteric.
'Ringing' is not really a problem here, each of the individual
filters may ring, as long as this is cancelled by the others
when you add the outputs.
Not sure I totally agree with this as it assumes that in the end I
listen to the reconstituted sum of all the filters. On the other hand,
if I listen to only a few then without the other 'negative'
contributions I end up hearing the ringing in the few I'm listening
to.
An FFT will provide a set of filters having property (2), but
each filter has a sin(x)/x shape, which is not really a bandpass.
You can 'improve' the shape by windowing, but that destroys (2).
And anyway using an FFT like this is a 'block' operation - how
a sample is treated depends on its position within the block.
Windowing requires overlapping blocks, and they won't add up
to the original input.
I think also that an FFT assumes some sort of longer-term steady state
repetitive nature which isn't in the stock data. I could take a
representative sample - the last 1000 bars, and then treat that as if
it was a repeating cycle and extract frequencies. I'm not sure what
that will do for me and the sample points are likely to introduce lots
of new artifacts. Still, worth thinking about.
<SNIP>
Now if the purpose of the filtering is to extract 'features' of
the data, you really don't need property (2). And given the
nature of stock market data, I suspect that classic audio filters
are not the right way to extract such features. I'd have a look
at wavelet filters for example.
Ciao,
Will do. Thanks.
Cheers,
Mark