RFC - Repeat Function Idea

[The Shadow]

Range ( ) and ForEach ( ) therefore seems the natural way to go, and I'm now just waiting to see what FMI engineers would answer.

Thanks Ugo, I'm sorry to disappoint you but the only answer the engineers can publicly provide is via a new release of the product - no one I know has enough power to be allowed to promise new features. :)

[comment]

It's a tough one. What you suggest is mighty convenient, but slightly inconsistent. Logically, I would expect all ¶ to simply pass through, with all the processing done in-between them. It's too bad there's no way of knowing where in the list you are, because then I would say leave the separator entirely to the user. As it is, your way is probably the best compromise - and you could say it follows the logic of List(), which also skips empty values.

[comment]

Oops. Another problem with the idea of automatic separators: what if you wanted to combine values, e.g. return each pair of consecutive values as a single value?

[The Shadow]

Yeah, maybe its not worth it to add this - we could just stick with letting the user do whatever they want with the newlines themselves.

It's a bit of a hassle that way, but its the most flexible, I suppose.

[comment]

I was just thinking that if you have the other function (I think Counter() would be the best name for it), there's very little you couldn't do - even with a single iterator. So to solve Genx' problem:

Let ( [

v = ;

end = ValueCount ( v )

] ;

ForEach (

i = Counter ( 1 ; end ) ;

GetValue ( v ; i ) & Case ( i < end ; ¶ )

)

)

The counter is really the key here, because for anything a bit more clever you'd want i to be an index, rather than an actual value. Checking the previous/next value, for example.

[fabriceN]

What's wrong with Self? I hadn't noticed any complaints on the forums.

Oh... well... a function that only works in auto-enter calcs and conditional formatting, dispalying an error message in every other case... that's a premiere !

Why can't I use it in a script, a custom function, or a Replace fied content... ? these are where it would be really useful.

Why can't I set a variable to Self + 1 ?

For the rest, you perfectly understood my meanning. So yes, ForEach() would be great, but needs a Repeat() little sister.

Before custom list, I used a recursive function that might be comparable to ForEach, which I put a link to here only to help others understand what we're talking about, if needed... There is also a Matt Petrowsky video article .

http://www.bh-a.com/downloads_1.1.html#DL04

Of course it is recursive, slow and limited to 10,000 values, but it saved me very often.

As ValuePosition is concerned, I use a simple calculation (but true, it's part of my "template", so I wouldn't consider working without it

http://www.briandunning.com/cf/62

[comment]

Why can't I set a variable to Self + 1 ?

Yeah, that too.

As ValuePosition is concerned, I use a simple calculation (but true, it's part of my "template", so I wouldn't consider working without it

http://www.briandunning.com/cf/62

That function you pointed to has some problems:

1. The start parameter operates in a text context - which is rather useless in a value list context. You could simply hardcode it to 1.

2. It's not thought out or written very well. If you're OK with giving up on the start parameter, you could write simply:

ValueCount ( Left ( listOfValues ; Position ( ¶ & listOfValues & ¶ ; ¶ & item & ¶ ; 1 ; occurrence ) ) )

But if you want to include a functioning startValue parameter, you need to do some extra work:

Let ( [

checkedList = RightValues ( listOfValues ; ValueCount (  listOfValues ) - startValue + 1 ) ;

pos = Position ( ¶ & checkedList ; ¶ & item & ¶ ; 1 ; occurrence ) ;

shortList = Left ( listOfValues ; pos ) 

] ;

Case ( pos ; ValueCount ( shortList ) + startValue - 1 ; 0 )

)

And that still isn't right, because it won't take negative occurrence. I believe you're better off going recursive for that. So it's a far cry from being as simple as Shawn would let us believe.

Now, how about a native SortValues() function?

Edited November 25, 2007 by Guest

[The Shadow]

True, I was just thinking of the basic Position functionality, if you wanted the start / occurrence thing also, it would be rather a bit more work. Still, I'd already capitulated that ValuePosition() was a good candidate for a builtin.

Now, how about a native SortValues() function?

Yes, that would be another nice one, the issue here I think is what datatype do you want to sort with, since a list of values no longer has type information for its component values.

Symbolic constants could be added, with a default of Text to allow this to be specified.

    SortValues( "12¶1¶2¶27¶23¶", Ascending, Number ) -> "1¶2¶12¶23¶27¶"

    SortValues( "12¶1¶2¶27¶23¶", Ascending, Text ) -> "1¶12¶2¶23¶27"

That would make a nice addition to the value-processing functions, anything else come to mind?

[fabriceN]

Well, CustomList allows sorting, de-duping, filtering a list with another one with several criteria (contains, starts with, ends with, does not contain... with a case sensitive option), give all values of current foundset...

I also use it verymuch as an audit tool all field values of current record (including repetitions), list of fields used as keys...

[comment]

the issue here I think is what datatype do you want to sort with

Exactly. The data type parameter is a must.

anything else come to mind?

LOL, a lot of things, but not necessarily related to this...

I just keep thinking about another version that somehow combines the two functions together.

Let's say it has the syntax of:

For ( var ; counterStart ; counterEnd ; { increment ; } expression )

As you see, it doesn't take any "real" input. All it does is evaluate the expression, where you can use var as the current counter value. Any actual values to be processed would need to be referenced in the expression, e.g. GetValue ( yourList ; var ) or Left ( text ; i ).

Now, you may ask why bother with such a simple thing, when you can easily construct this with a custom function. But the thing is that nesting won't work with such a custom function, where the expression is passed as a parameter (or at least I haven't been able to make it work). But with this function, the var would be carried forward, as we discussed above, so that:

For ( i ; 1 ; 3 ; For ( j ; 1 ; 2 ; i & j & ¶ ) ) = "11¶12¶21¶22¶31¶32¶"

Granted, it's a little more work for the developer this way. For example, when processing a list, you'd have to pre-calculate the value count, rather than just let the list run out by itself. But at the price it's also more flexible (not limited to value lists), and I can't find anything that you cannot do with this one function that would be possible using ForEach() and Counter() together.

[Agnès]

Hello !

Well, CustomList allows sorting, de-duping, filtering a list with another one with several criteria (contains, starts with, ends with, does not contain... with a case sensitive option), give all values of current foundset...

Fabrice is a Fan : ( me too )

But, no, CustomList doesn't do all that, these are calculations that we put inside who do it.

I try to sort data with this but I am to block because of "max", calculation is not yet ok.

For filter values, it's ok with FilterList ( ListA ; Attribut ; ListB ; CaseSensitive) (use CustomList )

http://www.briandunning.com/cf/771

you can write for exemple

FilterList ( List ( Table::MyField ) ; "Contains" ; Left ( FieldX ; 1 ) & ¶ & Middle ( FieldY ; 3 ; 3 ) ; "" )

it isn't too distant from Range() or ForEach().

we can indeed put 2 lists together or more, to make the processing which one wants

but :

both lists were iterated over at once, so for example:

ForEach( [ i = "a¶b¶"; j = "2¶3¶" ]; i & j & ¶ )

would result in: "a2¶b3¶"

If you put List ( MyField1 ) for i and List ( MyField2 ) for j and if MyField2 contains values with ¶, the result will be false

With CustomList, I must write calculation like this to concatenate my 2 lists (or more) :

Let (

$Rc = ¶ ;

CustomList ( Start ; End ;

"Substitute ( GetNthRecord ( MyField1 ; [n] ) & "" "" & GetNthRecord ( MyField2 ; [n] ) ; $Rc ; "" "" ) " ))

CustomList has the advantage of treating the values or words or part of text one by one and not by group like list.

But is limited beacause I don't want to push the numerical list beyond 20000 nor lines of packages of evaluate beyond 18700, the result takes too a long time then to wait

please, I do not have the claim to have made the things which it is necessary, just to have tried to find answers to my problems of processing of the lists and if that can advance a little bit, I take part with pleasure.

my unhappy English does not allow me all to understand, sorry if I am out of the subject.

Thanks

Agnès

[fabriceN]

Two ideas here :

Shawn, am I wrong or once the calculation engine would be able to do such a thing, it wouldn't be so hard to upgrade the List() function to a version 2, that would allow to nest an expression

List ( this & that )

for this= "a¶b¶¶d" and that = "1¶¶3¶4", it would return "a1¶b¶3¶d4".

The reason why I'm thinking of this is that it would seem consistant that empty values are omitted as we are used to, by List function... so you could return them in ForEach.

I don't understand why List function behaves this way : empty values are interesting, and so easy to get rid of, but that's the way it works now. So obviously a new version of List should remain compatible...

Second idea : well, we have now FilterValues, that returns values found in two lists, but what about the opposit : find values that are not in the second list ?

I do this with CustomList, as I said before, but it could be a native function per se.

[The Shadow]

I don't understand why List function behaves this way : empty values are interesting, and so easy to get rid of, but that's the way it works now.

List() was written to be an aggregator, the same as Avg(), Sum(), Count(), etc. Thus, it pretty much works like they do, which is to drop empties. It wouldn't have had to, but that seemed like the right thing to do at the time.

Second idea : well, we have now FilterValues, that returns values found in two lists, but what about the opposite : find values that are not in the second list ?

Yes, sort of a list subtraction kinda thing, I've needed that a couple times myself.

[The Shadow]

Second idea well, we have now FilterValues, that returns values found in two lists, but what about the opposit find values that are not in the second list ?

I think with the additon of ValuePosition(), this becomes easy:

SubtractValues( values, minusValues ) =

ForEach( v = values; If( ValuePosition(minusValues; v) <> 0; v & ¶))

So, I think ValuePosition() is the thing to discuss next:

  ValuePosition( text; searchValue [; startIndex [; occurrence]] )

this would be like Position, but with optional arugments that default to one.

Also, I think it would be nice for the searchValue argument to acutally be multiple values, and ValuePosition() would only return the index if all of the values were present and in the same order:

  ValuePosition( "a¶b¶c"; "b¶c¶" ) returns 2

and:

  ValuePosition( "a¶c¶b¶d"; "b¶c" ) returns 0 (not found)

Any comments are appreciated.

[comment]

Hi Shawn,

If you had ValuePosition(), you ALMOST wouldn't need the Counter() function, because - to generalize your example - you could use the processed values to count themselves:

ForEach ( v = listOfValues; Let ( i = ValuePosition ( listOfValues ; v ) ; ) )

I am saying 'almost', because with a real counter, the ForEach() function could process anything, not just a list of values - a text string, for example. Also, at the risk of repeating what I said earlier, I really think it would be more powerful to have the INDEX (i.e. the output of the counter) passed forward to a nested function, rather than an actual value.

Regarding ValuePosition()

Allowing searchValue to have multiple values (actually a block of values) would be a nice feature - although I don't see much use of it. That is, I'd rather have one without it sooner than with it later.

[The Shadow]

Sure, but ForEach with Range/Counter gives you the choice to have either, or both:

Let( myList = "apple¶bear¶cat";

  ForEach( [idx = Range(1,ValueCount(myList)); v = myList]; idx & ". " & v & ¶) )

resulting in:

  1. apple

  2. bear

  3. cat

Passing *only* the index is inefficient, as a good percentage of the time that index would be passed back to a GetValue() call, which is just waste, since we could have had the value itself for cheap/free rather than having to look it up again.

ForEach can *also* process a string:

  ForEach( v = Range(1,Length(myString)); Middle(myString,v,1) )

So I'm not seeing the extra value of a counter-only ForEach.

---

An issue I see with your example to avoid Range is that it would be giving back the wrong index if the listOfValues had duplicates.

[The Shadow]

Regarding ValuePosition()

Allowing searchValue to have multiple values (actually a block of values) would be a nice feature - although I don't see much use of it. That is, I'd rather have one without it sooner than with it later.

Well, if the function exists, it has to do *something*. The choices I see are:

   1) Return an error.

   2) Grab the first value from the list, and only use it.

   3) Search for all the values supplied.

If you don't care, then that's fine, but a complete design for a function needs to at least consider these things.

[comment]

Passing *only* the index is inefficient, as a good percentage of the time that index would be passed back to a GetValue() call

That is true, if you're mostly going to process value lists. I realize that's where your idea originally started, but I am thinking of something more general, and much more versatile. The way you have it, it's designed specifically to process value lists, but you can also make it general by adding a counter. To me, it seems more elegant having a general function as the basis, and adapting it to process value lists by using GetValue() - just as you could use Middle() to process a string character-by-character, or GetNthRecord() to process a found/related set. This way, the average amount of added processing is evenly distributed among the types of usage.

[comment]

a complete design for a function needs to at least consider these things

Hm. It's not that I don't care, but it's hard to come up with a practical example where I would need this. It also depends on the other issue we're discussing. If ForEach() is going to be designed to deal first and foremost with value lists, then value lists are the thing. In such case, I think I would want ValuePosition() to return a separate index value for each searched value - also as a list. Otherwise I would go with #3 - search for the entire block.

[bruceR]

FWIW I talked with Agnes early on during the development of her CF. It's an interesting concept but eliminating recursion doesn't really accomplish anything. I have a much simpler version, that does use recursion. It has recursion limits but the limit is higher and performance is essentially identical. See http://www.briandunning.com/cf/751

[The Gardener]

I have been following this discussion with interest. Much is over my head, but am learning. Checked out your custom function. Also, went to your website, your homepage but cannot enter the site, ie. go to any pages. ?

[The Shadow]

I do think value processing is much more common than word / letter processing. Maybe its just the way I think about things but it seems like most of the questions that need a CF (for looping) on this forum are about value processing.

I was going to mention, part of the reason I think "Range" is a better name that "Counter" is that the function is useful in other contexts besides the loop. It can preserve the type of the argument up until the list needs to be constructed, so:

  Let( d = Date(12,7,2007); Range( d; d+(4*7)-1; 7 ))

results in: "12/7/2007¶12/14/2007¶12/21/2007¶12/28/2007¶"

which could be used to break a time range into 15 minute increments also, and that kind of thing. I think the name "Counter" implies its only useful in conjunction with ForEach, which I don't see as being the case at all.

[fabriceN]

Dear Bruce,

Agnès sent me this recursive version ways before you published yours and advertised it as being a creation of yours, without even mentionning her name in the comments...

Unfortunately, it is still much slower than CustomList, and limited to 10,000 items. More, I'm not sure to understand what's so simple about it all 3 parameters are exactly the same as CustomList's.

Give back to Ceasar...

Edited December 3, 2007 by Guest

[fabriceN]

I absolutely agree with you Shawn. Range() is much more powerful than Counter(), and much more explicit.

As far as word by word processing is concerned, I have to say that I developed FilterWordsByTest() before FilterValuesByTest() because I needed it on some project, but I haven't used it much since that, unlike FilterValuesByTest, which I have used at least on a daily basis until CustomList was released.

So again, you're right B)

[Agnès]

[sorry to take it so personnally, but I cannot muzzle myself with what Bruce said. It's surely off-topic, but FWIW]

FWIW I talked with Agnes early on during the development of her CF. It's an interesting concept but eliminating recursion doesn't really accomplish anything. I have a much simpler version, that does use recursion. It has recursion limits but the limit is higher and performance is essentially identical.

We don't "talked with Agnes early on during the development of her CF" !! isn't not the true...

I started to write CustomList() at the beginning of April, with the assistance of Ugo and Fabrice,

[color:blue]I published CustomList in [color:red]April http://www.fmsource.com/forum/index.php?s=&showtopic=33993&view=findpost&p=139817

And I show CustomListRec, Recursive version of CustomList In [color:red]June http://www.fmsource.com/forum/index.php?s=&showtopic=33993&view=findpost&p=139872 I published the recursive version to show that the NotRecursive version was much faster

You send me a mail in [color:red]August, when I published CustomList In the BrianDunning's site, to say to me that one needed a getAsNumber for start and End

and we discussed the recursivity and I sent to you a mail with the Recursive version to show you, by you specifying that it was slower and that I did not want especially to publish it on the BrianDunnig's Site !

I believe the NotRecursive version is much more interessante and more rapid than recursive version

I am can be impulsive but I do not understand that anyone take the work of the others without at least specifying it, and while letting to believe that it is its work, its idea.

and even if you changed my recursive version a little, I do not understand that you published it without to at least to have asked it me.

IMHO , I think is not a copy but a "plagiat" to my cf and my work,

and I doubt :

and performance is essentially identical

.... then, Just for fun....

Test on MacOS_10.4 | TowerG5 | PowerPC G4 | Intel with FM 8.5 - The same calcul for the recursive and the NotRecursive, 2000 records. simple calcul, SetField, not with loop

CustomList ( 1; 2000 ; "[n] & "." & GetNthRecord (TableA::Name ; [n] )" )

G5 -> 1 sec | PC : 3 sec | Intel Immediate Result

ListFunction ( 1; 2000 ; "[n] & "." & GetNthRecord (TableA::Name ; [n] )" )

G5 -> 9 sec | PC : 21 sec | Intel 3 sec

4000 values just with Intel CL 2 seconds, LF 8 seconds

it's a difference no ?

May I add that the Non recursive version can be used by FM8+ users without any need for an Advanced version ?

Thanks,

Agnès

[off Topic : sorry]

Edited December 4, 2007 by Guest
........

[comment]

The thing that bothers me about the name "Range()" is that range is the input to the function. A function, I think, should be named by what it does or by what it outputs. This function takes a range and expands it into a list by enumerating - roughly the opposite of this: http://www.briandunning.com/cf/734

I do think value processing is much more common than word / letter processing. Maybe its just the way I think about things but it seems like most of the questions that need a CF (for looping) on this forum are about value processing.

Well, everybody has their own perspective. A very rough count of my CF folder shows list processing accounts for less than 15%. Text is about 40%. I have about 120 files there, and I'd say at least half of those were produced following a question on the forums.

[Agnès]

[Edit First Post]

Edited December 3, 2007 by Guest
..... to muzzle myself........ ;)

[fabriceN]

Sorry... I unleashed the beast B)

[Ugo DI LUCA]

Hi,

Staying on the subject, I'd agree with Mike the name "Range" may be too tied to the output, but then "Count" is too close to a counter/loop thing as well.

What Shawn decribes with

"Let( d = Date(12,7,2007); Range( d; d+(4*7)-1; 7 ))"

may be achieved without any extra new function with a simple repeating field. This would be sufficient to rename my calcField, eventually the Custom Function I'd be using, but it still is a ******* repeating field calculation, whatever the function I used. Should I be naming it "Repeat ()" ?

This is the basic problem when functions may be used for so many things at one time. Thought, thinking it loud, I would think that this kind of function may be one of the first "Set" prefixed function.

Edited December 3, 2007 by Guest
******** = ****

[Lee Smith]

Swear in French or Italian, and see if you get the ********

[AudioFreak]

Swear in French or Italian, and see if you get the ********

rofl B)

[comment]

Perhaps it's worth noting that a list of values is ALWAYS going to be Text. So to me, it doesn't matter so much if the output is "12/7/2007¶12/14/2007¶12/21/2007¶12/28/2007¶" or "733017¶733024¶733031¶733038¶". Actually, I can think of situations where I'd prefer the latter.

[Ugo DI LUCA]

Actually Numbers() could then do the trick as well

And Lee, I'm trying not to be rude...that's all B)

[comment]

No, "Numbers" is Apple's spreadsheet. But Numerator() would work for me.

[Lee Smith]

Lee, I'm trying not to be rude...that's all

And I was hoping to learn a few new words.

Dam

Lee