The PubMed Horcrux: Nine XSL to import one XML. Can the import be recursively scripted or somehow automated?

[Quito]

Hi FMForums,

I import one PubMed XML nine times, each time with a different XSL file, in order to capture all of the available content into it's corresponding fields and tables. Yet, it's very tedious to run nine imports. I'm thinking that perhaps there's a way of telling FileMaker which XML is going to be imported and then loop the import with each XSL. Can this be handled with a script? What would that script be?

Very kind regards,

Daniel

[ggt667]

I assume these 9 XSLTs are due to you wanting these pieces of data in each their table in FileMaker?

Personally I prefer to do `xsltproc XML2FMLXMLRESULT.XSLT XML > XML.fmpxmlresult.xml` scripts in linux / unix shell scripts, crontab or otherwise. Sometimes I write XSLTs in which I can use to send to FileMaker server as XML/RPC or JSON payload, utilizing curl's `-d`-option

As pr example for a project I did last year:

$ exa -lT xslt
drwx------    - user 28 Feb 17:48 xslt
.rwx------@ 64k user 27 May  2021 ├── Furniture2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 Mar  2022 ├── Window2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 May  2021 ├── Door2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 May  2021 ├── Electric2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 May  2021 ├── Panel2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 May  2021 ├── Bathroom2filemakerCWPpayload.xslt
.rwx------@ 64k user 13 May  2021 └── Livingroom2filemakerCWPpayload.xslt

Typically run as:

curl -XPOST -H "Content-Type: application/x-www-form-urlencoded" -d "$line" "http://user:[email protected]/fmi/xml/FMPXMLRESULT.xml"

Wheres $line is ecah of the lines outputted by:

xsltproc anything2cwppayload.xslt originalpayload.xml

I run such scripts either crontab, svcadm, or initiated by email, depending on the desired trigger point[s].

Edited March 29, 2022 by ggt667

[comment]

4 hours ago, Quito said:

perhaps there's a way of telling FileMaker which XML is going to be imported and then loop the import with each XSL. Can this be handled with a script? 

Yes. Start the script with Insert File, as reference only, into a variable. After the user has selected a file, the variable will hold the path to the selected file. Use the variable as the filepath in the subsequent Import Records steps. I don't think there's a way to "loop" among those - you will have to perform each one  explicitly. But that's not really an issue.

 

[Quito]

10 hours ago, ggt667 said:

I assume these 9 XSLTs are due to you wanting these pieces of data in each their table in FileMaker?

Personally I prefer to do `xsltproc XML2FMLXMLRESULT.XSLT XML > XML.fmpxmlresult.xml` scripts in linux / unix shell scripts, crontab or otherwise. Sometimes I write XSLTs in which I can use to send to FileMaker server as XML/RPC or JSON payload, utilizing curl's `-d`-option

As pr example for a project I did last year:

$ exa -lT xslt
drwx------    - user 28 Feb 17:48 xslt
.rwx------@ 64k user 27 May  2021 ├── Furniture2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 Mar  2022 ├── Window2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 May  2021 ├── Door2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 May  2021 ├── Electric2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 May  2021 ├── Panel2filemakerCWPpayload.xslt
.rwx------@ 64k user 27 May  2021 ├── Bathroom2filemakerCWPpayload.xslt
.rwx------@ 64k user 13 May  2021 └── Livingroom2filemakerCWPpayload.xslt

Typically run as:

curl -XPOST -H "Content-Type: application/x-www-form-urlencoded" -d "$line" "http://user:[email protected]/fmi/xml/FMPXMLRESULT.xml"

Wheres $line is ecah of the lines outputted by:

xsltproc anything2cwppayload.xslt originalpayload.xml

I run such scripts either crontab, svcadm, or initiated by email, depending on the desired trigger point[s].

Thank you, ggt667,

From what I can understand, this solution can be run:

a.) From Terminal (and probably also as an Automator app). 

b.) From within FileMaker itself (where a script using curl performs the workflow)

I'll give it a hard look in the next couple of days as this seems to be exactly what I want to accomplish. Will this skip over the (10 second) pause FileMaker has when using the usual Import Records > , XML Data Source, Specify XML Data Source File Specify & Use XSL style sheet File Specify GUI?

Best,

Daniel

[ggt667]

Do you run this in FileMaker client or server?

For client you can have any Mac, Linux, or Unix system prepare the data for you automated. As for the client if you run a client on an unattended Mac, you can also fully automate this, however FileMaker Pro a k a client requires GUI access and automation. Yet as long as no user uses this mac you can have a shell script that pulls down the data, and then open a fmnet link that triggers a script from a schedule.

For server you can have this done fully automagitically.

Edited March 29, 2022 by ggt667

[Quito]

6 hours ago, comment said:

Yes. Start the script with Insert File, as reference only, into a variable. After the user has selected a file, the variable will hold the path to the selected file. Use the variable as the filepath in the subsequent Import Records steps. I don't think there's a way to "loop" among those - you will have to perform each one  explicitly. But that's not really an issue.

 

Thank you, Comment,

OK, it seems I should combine both ggt667's solution with yours as a script that inserts the XML (as reference) not into a container but into a variable (presently don't know how but I'll get there). Then add the Import Record steps referring to the XML filepath. As I cannot loop it in a script, should I create nine Import Record buttons? One for each XSL I want to import? That would reduce the workload, but the amount of buttons seems problematic for future users.

Best regards,

Daniel

1 minute ago, ggt667 said:

Do you run this in FileMaker client or server?

For client you can have any Mac, Linux, or Unix system prepare the data for you automated.

For server you can have this done fully automagitically.

It's on my Mac client.

[ggt667]

Does users really have to to this by a click of a button?

29 minutes ago, Quito said:

It's on my Mac client.

Then I would write a shell script more or less like this; in order to prepare the FMPXMLRESULT.xml files for FileMaker, that way FileMaker will have its own XML format predigested, and the import itself will use less resources as this script will already have front loaded the desired format.

curl -kL -o ~/data/newdata.xml https://api.domain.tld/linkToSource.xml
xsltproc ~/xslt/somesource2fmpxmlresult.xslt ~/data/newdata.xml > ~/data/newdata.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult2.xslt ~/data/newdata.xml > ~/data/newdata2.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult3.xslt ~/data/newdata.xml > ~/data/newdata3.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult4.xslt ~/data/newdata.xml > ~/data/newdata4.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult5.xslt ~/data/newdata.xml > ~/data/newdata5.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult6.xslt ~/data/newdata.xml > ~/data/newdata6.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult7.xslt ~/data/newdata.xml > ~/data/newdata7.fmpxmlresult.xml
mv ~/data/newdata.xml "newdata-$(date).xml"
curl -kL fmp://127.0.0.1/yourfile.fmp12?script=scriptnameImportScript

Not sure at which intervals you would run this? You can use this application to make a Mac schedule to trigger the script at desired intervals: https://apps.apple.com/us/app/lingon-3/id450201424?mt=12 If you are to use crontab you will have to use full paths for everything in the script.

For reference

https://fmhelp.filemaker.com/help/16/fmp/en/index.html#page/FMP_Help%2Fopening-files-url.html%23

While the above probably works as a static; below will leave a dynamic trail; and would probably be better for production?

ti="$(date +%F_%H%M%S)"
curl -kL -o "~/data/newdata$ti.xml" https://api.domain.tld/linkToSource.xml
xsltproc ~/xslt/somesource2fmpxmlresult.xslt "~/data/newdata$ti.xml" > "/tmp/newdata$ti.fmpxmlresult.xml"
xsltproc ~/xslt/somesource2fmpxmlresult2.xslt "~/data/newdata$ti.xml" > "/tmp/newdata$ti2.fmpxmlresult.xml"
xsltproc ~/xslt/somesource2fmpxmlresult3.xslt "~/data/newdata$ti.xml" > "/tmp/newdata$ti3.fmpxmlresult.xml"
xsltproc ~/xslt/somesource2fmpxmlresult4.xslt "~/data/newdata$ti.xml" > "/tmp/newdata$ti4.fmpxmlresult.xml"
xsltproc ~/xslt/somesource2fmpxmlresult5.xslt "~/data/newdata$ti.xml" > "/tmp/newdata$ti5.fmpxmlresult.xml"
xsltproc ~/xslt/somesource2fmpxmlresult6.xslt "~/data/newdata$ti.xml" > "/tmp/newdata$ti6.fmpxmlresult.xml"
xsltproc ~/xslt/somesource2fmpxmlresult7.xslt "~/data/newdata$ti.xml" > "/tmp/newdata$ti7.fmpxmlresult.xml"
curl -kL "fmp://127.0.0.1/yourfile.fmp12?script=scriptnameImportScript&param=$ti"

You could probably use $ in place of IP, if you know that the file is always open.

Edited March 29, 2022 by ggt667

[comment]

22 minutes ago, Quito said:

a script that inserts the XML (as reference) not into a container but into a variable (presently don't know how but I'll get there).

It's an option you will see in the Insert File script step, under Target.

 

22 minutes ago, Quito said:

As I cannot loop it in a script, should I create nine Import Record buttons?

No, of course not. You add 9 Import Record steps to the same script. User interaction ends with the selection of the file to import. From that point it's all done by the script.

 

22 minutes ago, Quito said:

it seems I should combine both ggt667's solution with yours

I don't think so, but perhaps I am missing something.... What exactly is the advantage you see in going at it this way?

 

Edited March 29, 2022 by comment

[Quito]

2 hours ago, comment said:

It's an option you will see in the Insert File script step, under Target.

 

No, of course not. You add 9 Import Record steps to the same script. User interaction ends with the selection of the file to import. From that point it's all done by the  script.

 

I don't think so, but perhaps I am missing something.... What exactly is the advantage you see in going at it this way?

 

It worked! The whole import process (Dialogs off) took two minutes.

Thank you sooo much, Comment.

[Quito]

2 hours ago, ggt667 said:

Does users really have to to this by a click of a button?

Then I would write a shell script more or less like this; in order to prepare the FMPXMLRESULT.xml files for FileMaker, that way FileMaker will have its own XML format predigested, and the import itself will use less resources as this script will already have front loaded the desired format.

curl -kL -o ~/data/newdata.xml https://api.domain.tld/linkToSource.xml
xsltproc ~/xslt/somesource2fmpxmlresult.xslt ~/data/newdata.xml > ~/data/newdata.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult2.xslt ~/data/newdata.xml > ~/data/newdata2.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult3.xslt ~/data/newdata.xml > ~/data/newdata3.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult4.xslt ~/data/newdata.xml > ~/data/newdata4.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult5.xslt ~/data/newdata.xml > ~/data/newdata5.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult6.xslt ~/data/newdata.xml > ~/data/newdata6.fmpxmlresult.xml
xsltproc ~/xslt/somesource2fmpxmlresult7.xslt ~/data/newdata.xml > ~/data/newdata7.fmpxmlresult.xml
mv ~/data/newdata.xml "newdata-$(date).xml"
curl -kL fmp://127.0.0.1/yourfile.fmp12?script=scriptnameImportScript

Not sure at which intervals you would run this? You can use this application to make a Mac schedule to trigger the script at desired intervals: https://apps.apple.com/us/app/lingon-3/id450201424?mt=12 If you are to use crontab you will have to use full paths for everything in the script.

For reference

https://fmhelp.filemaker.com/help/16/fmp/en/index.html#page/FMP_Help%2Fopening-files-url.html%23

Thank you, ggt667,

For the moment, Comments' solution was easier for me to implement. Yet, I do see this shell script useful for my future versions. I just have to look at it harder.

Again, thank you very much!

Daniel