How to calculate running Standard Deviation

[Steve E.]

I have a file where the field in each record holds a sample value, in this case the number of stems in vegetation plots. For example:

Record 1 - #stems = 0

Record 2 - # stems = 1

Record 3 - # stems = 0

Record 4 - # stems = 2

etc.

As this data accumulates I would like to calculate the sample standard deviation as a running function. For example:

Record 1 - #stems = 0

Record 2 - # stems = 1 - s[records 1 to 2] =1.4142

Record 3 - # stems = 1 - s[records 1 to 3] =1.0

Record 4 - # stems = 2 - s[records 1 to 4] =0.8164

etc.

Unfortunately, the default standard deviation function in FM is only for all records. It only calculates the function for the entire population of records and will not produce a running amount.

I know I could do repeated finds for a subset of the records, use the summary standard deviation function, and then transfer that number into another field for the particular record. But it seems so crude. I'd like to simply have a calculation field that would calculate and retain the running standard deviation. Does anyone know of a simple way to do this?

[Genx]

How many records are there likely to be in total?

[Steve E.]

For each set, no more than 500.

[Genx]

Lol, I remember very little about stDev since last year... I'm never doing statistics again!...

But see if the attached looks right.

test.zip

[mr_vodka]

Wouldn't the standard deviation between

Record 2 - # stems = 1 - s[records 1 to 2] =.5

Record 3 - # stems = 1 - s[records 1 to 3] =.4714045207910317

Record 4 - # stems = 2 - s[records 1 to 4] =.7071067811865475

[Genx]

That's what i get if i use the StDevP() function and use 0,1,1,2 for the stem values.

[mr_vodka]

http://www.csgnetwork.com/stddeviationcalc.html

http://invsee.asu.edu/srinivas/stdev.html

was telling me something different. Dont know which is supposed to be used in this case but the one thing that that is certain is that I have no idea where he was getting those other figures at.

Edited March 2, 2007 by Guest
reworded

[Genx]

I'm not arguing, i think those figures were just made up . Steve'll probably know which one he needs in the end, i forget what they are for anyway lol, all i remember is linear regression, variance, median, mean and that's about it.

[Steve E.]

Beautiful! Just what I needed.

And I understand your feeling about statistics. I used to feel that way, but I've had way too many experiences of driving several hundred miles to (for example) census a rare plant population and then found several months later when I analyzed the data that there weren't enough samples to have reasonable confidence in the results. And its a lot worse when you try to compare different years and find that you can't really say if the population is the same, declining or increasing. Now I probably spend as much or more time planning than actually collecting data.

Thanks.

[Steve E.]

I need the sample standard deviation, not the population standard deviation. Population standard deviation is the square root of the population variance; sample standard deviation is the square root of the sample variance. For my particular use right now, I need the sample standard deviation. Of course, that's easy enough to change if I need to.

[Genx]

If you ever need more than 500, just increase the number of repetitions.

[comment]

I think I would do it this way.

RunningStDev.fp7.zip

[Genx]

I was thinking of doing that, but what if the records are entered out of order and then sorted for some reason?

... Or doesn't it matter in this case?

[comment]

There are pros and cons to each method. I should have mentioned that mine ignores the found set, too.

[comment]

OK, here's a solution done the Filemaker way. It considers only the found set, and records can be sorted in any order.

RunningStDev2.fp7.zip

[Genx]

I so have no clue what you did there, but i'm sure it was brilliant so I'll pull it apart later. I always learn things from your files comment so thanks for always posting so many alternative ways of doing things Your my role model.

[comment]

Yeah, right... I wonder how long it will be before I beg to work for you.

[Genx]

Lol, I think you give me too much credit, but still it's nice to hear that from someone like you (even if it is jokingly )